Why does it bother me so much when people aren't smart enough to recognize when I'm using metaphors, and they decide I'm just stupid?

Title says it all. It just drives me nuts when that happens. And I don't know why. I don't get why it bothers me that stupid people think I'm stupid.