Why am I such a rug to be walked all over, the black sheep?

It's like everyone! Everyone! My family, my friends, my neighbors all seem to take out their frustrations on me even if they are talking about someone else they say YOUUUU! Me? Not me them! They lash out on me and it's deminishing my self esteem, my view of myself, I'm starting to think maybe I am all of these things they accuse me of, or they just know that I'm too nice to lash back out on them, thus making me an easy target!