For Charlotte: Remove crusts from bread and allow slices to dry out. Tear bread into pieces and place in food processor. Pulse until crumbs are pea-size.
Melt butter in a skillet over medium heat and toast crumbs until golden brown.
In a bowl, combine brown sugar, lemon zest, cinnamon, nutmeg, and ginger.
In another bowl, combine apples and lemon juice.
Warm jam in microwave (25 to 30 seconds, medium power) to make easier to spread.
To assemble: Preheat oven to 350°F. Generously butter sides and bottom of 8-inch casserole or springform baking pan.
Spread 1-½ cups bread crumbs on bottom of prepared pan. Cover with half of apple slices. Sprinkle with one-third of brown sugar mixture. Spread half of apricot jam on top. (Pastry brush or soft spatula works well.) Repeat with 1-½ cups bread crumbs, remaining apples, one-third of brown sugar mixture, and remaining apricot jam. Combine remaining bread crumbs and remaining one-third of brown sugar mixture. Sprinkle on top. Bake for 35 to 40 minutes, or until lightly browned.
For topping: In a bowl, whip cream with confectioners’ sugar and brandy (if using), until stiff peaks form.
To serve: Cut Charlotte into slices and top each slice with whipped cream, caramel sauce (warmed, if desired), and a pinch of sea salt.