Coping mechanisms?

So I'm 19 and a 60-odd year old man thinks it's acceptable to mutter abuse at me when I walk put of my house. He won't do it if my father or brother are there, it's like bullying but what am I meant to do about it? I don't throw mental health around loosely, but it's really getting to me, as I've had nothing to do with him. The reason he dislikes me is because he upset my dad or my dad upset him (a little of both maybe) and because he can't get at my dad he's doing it to me. I've said nothing and done nothing, even his partner is fine with me, but it's bordering on harassment because it's every time I leave my house and this person is there, he starts spouting off. It's getting harder and harder to ignore because of the things being said, until I'm able to leave there's nothing I can do to stop him, I just need to be able to deal with it.