if someone adds me without asking why do they count against my limit?

I didn't ask them, most of the time never spoken to them so why if someone adds me as a friend does it count against my total number of blocks to remove them? I want to save my blocks for the truly annoying, 500 may not be enough.