Is it wrong to call women "Sweetheart" on the phone?

I work as a telephone agent. My last caller insisted on calling me "sweetheart" during the call. He was an older gentleman. I get this from time to time, especially from older men in the American South. When guys talk like that, I feel a mix of feelings as a woman, both valued but also patronized.

Do you think it's good or bad when men talk to women this way? Why do you think so?