How should I react to this?

You said co-worker not supervisor. You have no need to justify yourself to anyone else that isn't your supervisor.
That comes from a retired Supervisor of many, many years.

I used to have a very difficult co- worker. I had to see a psychiatrist for job stress. The shrink asked me, "Can you tell me in just a few words what is stressful about your job? "

I answered, "I can tell you in TWO words: Frank H_____."
The shrink signed and said, "You know, I've been asking patients that question for many years now and it's amazing how often they answer with a name."

My problem solved itself when the guy got fired for writing a very nasty sexually harassing letter to the director of the institution in which I worked.

In retrospect, I probably should've taped him and sent it to someone in authority but at the time I feared that I might be fired if I used surveillance equipment which was against the rules where I worked.

Do you have a supervisor to discus this with...the coworker seems irrational

Why explain yourself to him unless he's your boss?

Do you report to him? If not, tell him to mind his own business. And give him a look that says I mean it, creep.