1 cup soft unsalted butter (2 sticks) (divided)
1 1/3 cups Sugar (divided)
2 large eggs (at room temperature)
1/2 teaspoon pure almond extract
1 cup almond flour
9 firm plums or pluots (about 2 pounds)
3/4 teaspoon ground star anise
1/4 teaspoon ground nutmeg
  Whipped cream or vanilla ice cream for serving (optional)