Preheat the oven to 450°F with a rack in the center position. Grease an 8x8-inch baking dish with 1 tablespoon olive oil.

Add the potatoes and toss with 1 tablespoon of the oil, 1 teaspoon of the salt, and the black pepper. Roast until tender, 20-25 minutes.

Meanwhile, heat the butter and remaining 1 tablespoon oil in a medium saucepan over medium heat. Once melted, add the shallot and cook, stirring, until tender, about 5 minutes. Add the garlic and cook until fragrant, about 1 minute more.

Add the cream, stock, rosemary sprigs, lemon zest, lemon juice, mustard, cayenne, and capers. Increase the heat to medium-high and bring to a simmer. Reduce the heat to medium-low and cook until it begins to thicken, about 5 minutes.

Sprinkle the cod all over with the remaining 1 teaspoon salt.

Remove the potatoes from the oven, nestle in the fish, and pour the sauce over the top. Bake uncovered until the fish is flaky, another 10-15 minutes.

Sprinkle with parsley, if using, before serving.