Possibly a dumb question…

Ok, so I know this is likely a stupid question and yes I need therapy but, I am curious to pose it to the masses and hear the response.

In a relationship, friendship, co-worker or any situation that it applies, if you tell them that something that is a hot button issue and upsets you (not a hot button issue for them) and their response is you’ve done it to me a hundred times so get over it.

If I have done it to them even though it doesn’t bother them am I in the wrong?