As a white male you DO have white male privilege. Just because you don't recognize it doesn't mean it doesn't exist.
And how could you? It's all you've ever known? Understand, this doesn't mean that no white male ever experiences a lack of privilege. But you do have it. I don't know why this notion upsets you so much.