Could removing a fence on my own property lead to disputes with neighbors, and how should I handle it?

Why isn’t it anybody else’s fault for what happened to someone? My big brothers would steal food from me, and wouldn’t share an equal amount of food given by adults with me when I was a kid which caused me to be weak, and bullied by minors.