In a mixing bowl, whisk eggs, milk, cinnamon and nutmeg.
Peel and dice the apples. In a smaller mixing bowl, combine melted butter, brown sugar and cinnamon. Toss with diced apples.
In a greased 8×8-inch glass baking dish, tear 3 slices of bread and add to the bottom. Then add a layer of the diced caramel apples.
Tear the other 3 slices of bread and spread on top of the apples.
Pour egg/milk mixture into the baking dish. Cover with foil and let stand in the refrigerator for at least an hour (overnight, ideally!). Preheat oven to 350.
Bake covered for 30 minutes, then uncover and bake another 15-20 minutes, or until eggs have cooked through.
Serve Caramel Apple Breakfast Casserole with fresh fruit and sausage or bacon.