The reason you feel conflicted is because it’s a term of endearment (which could be endearing) that assumes a level of intimacy that doesn’t exist. If my wife calls me baby… I don’t blink an eye. That intimacy exists for us and doesn’t feel out of place. It would if some person (man or woman) called me baby where there is no intimacy to be assumed with a term of endearment. It is the same with sweetheart. @
sarabee1995 and I have talked about this many times over the years as some men on here will also use terms of endearment misplaced, since no level of intimacy like that exists between them and I. In the south, I think women were more so viewed as property for a long time where any man could assume a level of intimacy with any women with or without her consent and it’s looked at differently. Same as their opinion of not saying sir is considered being impolite, even though historically those were terms used for certain levels of nobility and not just for regular people. I don’t take issue with sir and ma’am at all, even though it technically doesn’t belong but assumed intimacy I can. It feels awkward at best and actually very creepy and sexualizing without my consent at worst.
“Sir is used to address a man who has the rank of baronet or knight; the higher nobles are referred to as Lord. Lady is used when referring to women who hold certain titles: marchioness, countess, viscountess, or baroness. It can also be used of the wife of a lower-ranking noble, such as a baron, baronet, or knight.”
https://www.merriam-webster.com/wordplay/a-field-guide-to-british-noble-titles-whats-a-duchess#:~:text=Sir%20is%20used%20to%20address,baron%2C%20baronet%2C%20or%20knight.