I Have a Question

When did the word ninja become a racial slur? I'm so out of the loop.
I asked a guy at work for his help reaching something high on a shelf. And he jokingly asked me why I didn't climb up to get it.

And I said,"I'm not a ninja."
And he told me it was racist. He was joking still. But he asked me if I ever used the n word before.

I couldn't figure it out.