People that have treated me better when I haven't worn a hijab think it's a compliment

but I respect them a whole lot less because of it.

You suck ass. Your value of someone based on how they dress is quite frankly pathetic.