Is there any place which is bully free? Even my village has bullies

I used to think my village people were nice and caring. I live in a city now. But last year when I visited my village Falakata in april 2017 I was 16 yr old boy and went with my grandpa to see his carpenter shop. Its near a high road so big trucks and buses move there. Suddenly I saw some 3-4 college boys eating food at the roadside stall and were staring at me badly and discussing something about me. They all were like 18-20 yr olds. I was scared and suddenly they were calling me too. I got scared and went to the shop and said to my grandpa. When my grabdpa came out they took their bicycle and when they went far they saw me and showed middle finger and saying some abusive words and went away. I am still thinking why they did that. Maybe they r bullies in college too. But I was taller than them I am 5'7(170-171 cm) and they were 5'1-5'4 range. I still am shocked that how come they know the middle finger and why did they do it. Is it because they never see me the full year or because I looked funny? One of them looked like an Assamese too while rest were Bengalis. I asked my grandpa about it and he said maybe they were jealous because I am tall and goodlooking. But god only knows the reason.