Why do people give us autistic people such a hard time?

We can’t help the way we are OK? So sick and tired of people yelling at us! Stop yelling at us it’s not gonna help us improve! One of my neighbors yelled at me today because I didn’t hear him right away. I told him not to yell at me and he said, “Well if you would have actually have been LISTENING and not wondering around, I wouldn’t have yelled at you!!! I hate repeating myself and you just frustrate me so much! LISTEN TO WHAT IM SAYING DAMN IT!!” I just walked away, then ran into my room crying.
He is my elderly neighbor’s son and he’s not very nice to me. He’s never been nice to me at all and whenever we come over he always gives me this nasty look when I try to be nice.