I don't know what to do with my eyes

In a typical social interaction. When I am talking directly to someone, it's easy, you make eye contact. What do you do if no one is talking or no one is talking directly to you?? WHAT ARE YOU SUPPOSED TO DO WITH YOUR EYES?? lol. I usually pick a random point in space and stare at it. But this makes some people be like "what are you looking at?" It has made therapists think I am hallucinating before, even.

Seriously what do y'all do with your eyes if someone is not speaking directly to you?