I’m not quite certain the point you are making here. Is the implication that because humans typically have two hands, those that do not are not a group that can be described? Or that they can be, but only should be as the product of developmental errors?
We don’t generally, where we know exceptions exist, refuse to acknowledge their existence. Saying sex is a binary is saying there are only males and only females. That’s literally what binary means. Like binary notation either uses 0 or 1. If it was possible for sometimes to have a 2, it wouldn’t be binary anymore. That’s a different thing.
This is especially true for something like sex that is based on a grouping of traits, genes, expressions, etc. which are not universally 0 or 1. Sure, we generally agree on a category when some are different, but there’s some points where it’s not so stark. Hence, the binary fails because there can be overlap and grey.
Nobody is saying we have to stop using male and female to describe sex in most cases, especially in a medical setting. But if you had a child born intersex, and the doctor turned to you and said, “Nah, my gut says male. Nothing will be different,” you’d probably ask for a second opinion.
Then it’s not a binary system. It’s a system with two extremely dominant members. Those are different things. You can be more binary in specific contexts e.g., gametes and egg vs sperm.
I’d be very cautious about the healthy description in reference to intersex people. I don’t believe you are trying to say anything nefarious, but there’s a reason it shows up in eugenics arguments.
I didn’t say sex was a spectrum, though perhaps someone else you were speaking with did. I wouldn’t use spectrum for sex, since there are multiple differentiating factors with differing measures.