Are we more lonely when we are alone, or when we are surrounded by people who don't understand us?

Definitely the latter.

Latter

I rarely ever feel lonely with the voices in my head

Definitely the latter and not the former.
The latter - like at many workplaces.

Had a panic attack, long ago, being in a crowded situation. In a place that I am totally familiar with. I practically grew up there.

Long story to that.

Short reason how I overcame that.

I got away from the crowd. Of course I knew how to, being so familiar with the place.

It's why I avoid crowds though. Especially in unfamiliar places. Which is too numerous to say.

Never had such a situation alone. And why removing myself from the crowd worked.

I was a latchkey kid growing up. I spent a lot of time alone.

I don't feel lonely either way.

But being surrounded by those people would be more irritating or annoying than lonely.

I've never felt lonely, just alone because of the reason the question stated.

Alone. People who complain about being lonely in company don’t realize the importance of potential connections. Having an opportunity can sometimes be worth more than the opportunity itself. Think of Cast Away and Wilson. He had nothing and was forced to make something from that. That version of loneliness is much more profound than the kind you’ll find in people whose standards for connecting are too high.

"Nobody likes me, everybody hates me, I guess I'll go eat worms."

It's worse to be with people who make us feel alone, than to be alone.