Don't listen to them. We're all allowed to have whatever interests we like as you said, since after all we're all individuals 😇
On a side note, I absolutely love anime, and even though I'm a boy, I actually know a number of different girls at my school who are really into anime too 😉