Not sure if you're joking or being sarcastic, but here are a few examples where the mere absence / presence of a Y can't determine sex & gender.

• Sometimes a person has XY chromosomes, but the body developed in a female manner because the fetal cells were resistant to testosterone. Such a person has good claim to being a woman (she developed that way) or a man (he has a Y chromosome; his brain could have been sensitive still to the testosterone to still develop male-coded)
• Sometimes a person has XX chromosomes, but the body developed in a male manner. Usually (though not always) this is caused by part of a Y chromosome ending up on an X. Such a person has good claim to being a man (he developed that way) or a woman (maybe she lacked enough testosterone to male-encode the brain).
• Other conditions such as XXY combinations, or chimeras (some cells XX, some cells XY), or other intersex conditions where some body parts develop male, some female.

To me personally, I view trans people as a type of intersex person. It seems entirely possible that you might have a person whose brain cells were more or less resistant to testosterone and/or exposed to testosterone and truly is a man/woman in a woman's/man's body. You don't need to bring choice or culture into it - I think biology alone provides good evidence to believe trans people about what gender they are.