Preheat: Preheat your oven to 350°F. Grease a 9x13 inch baking dish.
Prepare the Pecan Base: In a medium saucepan, melt the butter over medium heat. Stir in the brown sugar, maple syrup, and heavy cream. Continue to cook, stirring constantly until the sugar is completely dissolved. Add the chopped pecans and salt to the saucepan. Stir until the pecans are well coated with the syrup mixture. Pour the pecan mixture into the prepared baking dish, spreading it evenly across the bottom. Set Aside.
Prepare the French Toast Mixture: In a large mixing bowl, whisk together the heavy cream, eggs, vanilla extract, cinnamon, nutmeg, and orange zest (if using) until well combined.
Assemble the Casserole: Dip both sides of each slice of brioche bread in the egg mixture and place on top of the pecan mixture in the casserole dish without overlapping. Let the casserole sit for about 10 minutes to allow the bread to soak up more of the egg mixture. This step ensures a custardy texture in the final product.
Bake the Casserole: Bake in the preheated oven for 35-40 minutes, or until the top is golden brown and the casserole is set. Once baked, remove from the oven and let the casserole cool slightly before serving. Use a spatula to take the slices out of the baking dish and flip them over onto the plate so the pecan side is up. Baking the bread right side up allows the bread to get nice and crispy.
Serve: Top with more maple syrup,a dusting of powdered sugar and whipped cream if you’d like!