Whence the sexes?

There are many explanations of the evolutionary value of sex in terms of gene exchange (I particularly like this one). But these don’t explain the evolutionary value of having sexes: of the differentiation between males and females. A species of hermaphrodites would get all the genetic benefits of sex, but without the massive cost of half its population being unable to bear offspring. On average, each individual could have twice as many offspring, unless other problems arose. And indeed, most plants are hermaphroditic - but only a few animals. So why aren’t most animals hermaphrodites? A quick search doesn’t turn up any widely accepted answer, so I’ve brainstormed a few possibilities. I may well be missing something obvious; if so, let me know.

1. Specialisation of labour. If there’s a strong division of labour between males and females, then maybe it’s harder for hermaphrodites to gather enough resources to support offspring. But in many species the males contribute little in terms of resources - e.g. in orang-utans, who are very solitary.
2. Developmental or metabolic costs of being hermaphroditic. Maybe it’s just a very expensive adaptation. But this seems unlikely to be the main factor - the relevant baseline is the existence of males at all, which is a huge energy cost.
3. Difficulty of evolving hermaphroditism. Maybe this is just hard for evolution to find? But non-reproductive hermaphroditism seems like it arises via mutations pretty frequently, so I’d be surprised if reproductive hermaphroditism were unachievable by evolution. And in fact there are a few hermaphroditic species - so why haven’t they spread much more widely?
4. Difficulty of fixating hermaphroditism. A hermaphrodite in a species without many hermaphrodites is likely not as attractive to females as most males are, nor as fertile as most females are. So maybe, even after arising, the trait will be selected against. But on timeframes where sexual desires can themselves evolve, all else equal we should expect stabilising sexual selection towards the best combination of fertility and attractiveness. E.g. it would be undesirable to be impregnated by overly masculine conspecifics, because the resulting offspring would be less fertile themselves. So this adds a bit more difficulty to reaching the hermaphroditic equilibrium, but doesn’t answer the core question of why that’s a less fit equilibrium.
5. Self-fertilisation. I remember reading a while back that self-fertilisation has a strong short-term advantage, despite losing out on the long-term benefits of sex (this old paper has a section on “selection in favour of self-fertilisation”, although I haven’t read it in detail). So maybe the answer is that hermaphrodites end up evolving ways to self-fertilise, which is harmful in the long run, and so group selection prevents the trait from becoming too widespread. This effect plausibly occurs in plants, but not strongly enough to prevent most of them from being hermaphroditic. And presumably it's significantly harder for plants to prevent self-fertilisation (since their pollen spreads widely) than it is for animals. I’m open-minded to the possibility that a combination of these explanations is responsible. But none of them seems particularly strong to me; so I'm guessing that the biggest effect comes from:
6. Physical dominance. Maybe animal competition to impregnate fertile conspecifics is grounded in physical power, so that dominant males could just prevent hermaphrodites (who invest less in muscle and brawn) from having sex. In some sense this is a variant of the first possibility: the comparative advantage of muscle is just so strong that the best solution is to have a division of labour. But it focuses not on problems posed by the environment, but rather on problems posed by one’s own species. If true, it feels a bit sad: that there could be a much better solution if it weren’t for the threat of physical force. But it does seem pretty plausible to me - especially because hermaphroditism is much more common in plants, which can’t use the strategy of physical dominance.
1. The main argument against it is that males in some species don’t compete via shows of force - e.g. birds which sing to attract mates. But birds are unusual in other ways too - e.g. over 90% of bird species are monogamous (as compared with less than 5% of mammals), which makes it more plausible that the "specialisation of labour" hypothesis explains their sexual differentiation. So I'd be interested in pointers to any literature on how many non-monogamous species lack physical male competition.
2. Also, in some species several different mating strategies remain in equilibrium (including strategies which involve surreptitious mating unnoticed by a dominant male). So even if physical dominance is the best strategy, is it really so much better that it can crowd out all the others?

Getting more clarity on this topic isn’t a priority for me, but I do think of the question as one small data point that might help ground big-picture abstractions about competition and cooperation (in a comparable way to how knowledge of how insect colonies work provides an interesting metaphor for thinking about society).