It’s always amused me a bit when people reply

It's a common misunderstanding.

When someone asks me a "do you mind?" question, I'll usually reply, "no, I don't mind" (if I really don't mind). Answering it that way helps to avoid any misinterpretation.

I was speaking to a group of peers (i.e. I wasn't in 'teacher' mode) in a seminar on one occasion when someone spoke over me. 'Can I interrupt?'

'You just did,' I replied, dead pan.

Laughter followed because we knew each other well and offence was understood not be given or taken.

Your response sound perfectly clear to me.

I don't think I have experienced this.

Or maybe I am guilty of it and have not realized it! 😂

All they see is "No"
I've had that happen too.

It's weird to realize they're looking for a 'yes' or 'no' as if they directly asked for permission, instead of what they actually asked. 😂

English is difficult, in some aspects more so for native speakers than for those who learned it as a second language.

Maybe their English is their 3rd or 4th language ....;)