I hate when people tell me to stop looking down when they are talking to me

I don't know why people tell me that. I don't look startled when someone asks me a question and I don't look down. If I do it's not like it's on purpose or intentional. Besides, if I do and maybe I do, it's just for literally a split second when I wasn't expecting to have a conversation with them in advance. Besides, it's not that big of a deal. I mean I have to look somewhere and it's not all the time. I mean if I do it at all.