i don’t understand how “legitimacy” factors in here?  unless you’re suggesting that people assume only men can be legitimate, but that seems at odds with the rest of this question

i’m not suggesting it’s quite so overt.  suits tend to imply some form of authority or management, and there exists an association between authority/management and men.  (otherwise the “glass ceiling” wouldn’t be a thing.)  men are thus vastly overrepresented among CEOs and politicians and other suit-wearers you’re likely to see or encounter if you’re not in business yourself.  combine the two, and you’re likely to assume someone in a suite is male.

this isn’t actively antagonistic sexism; it’s the usual low-level background influence on how we perceive gender.  power, confidence, authority, even height — we interpret these all as masculine traits, strongly enough to override wearing a skirt.

i wish well-meaning people were not so eager to write off the little things as “well that’s not really sexism, because _____.”  no one wants to think of themselves as a misogynist.  but getting caught up in potential labels and trying to weasel out of them, instead of actually examining our thought processes, won’t improve anything.

incidentally, among orchestra conductors, men outnumber women at least 4 to 1.