Is it okay to tell someone that’s mixed race that they aren’t really that race (read context below)?

I’m mixed...I’m often told I’m not really this or that and not gonna lie...it bothers me (doesn’t define or inhibit me, but yet- it sticks in the back of my mind). It makes me feel like I’m too light for one side and too dark for the other that neither are willing to embrace me as their own. Ppl don’t dare have the audacity to tell someone that’s mulatto that they’re not black so why is it okay for someone to say this to me? What do I say back?