How do I stop being so sensitive to this?

I have this guy friend who is like a brother to me. Thing is, one of my close friends doesn't like him, and another girl I hang out with but don't really consider her a friend anymore doesn't like him either. Now, I know that if someone says something bad about him, he says he doesn't care. However, it annoys me a lot whenever they say something mean/rude about him. Especially because he's been there for me when they haven't been.

The girl who I'm not really friends with brought up how I don't like one of her friends for no reason, but there is a reason. Anyway, she made it seem like I just dislike people without having an explanation for it. She told two other people this as well...which I don't understand because I never bring up her not liking my guy friend or make her feel crappy for it? And I knew I couldn't say anything back because I'd just be digging myself into a deeper hole.

Anyway, I guess it's just hard hearing people say mean things about him because I care about him a lot. How do I avoid this though?