Disappointed with myself still thinking I need to be polite

Went to the Goodwill outlet with my daughter yesterday. There was a man with a cart standing against the wall next to the last bin. I assumed he was probably waiting for his wife with their cart. As I looked through the bin he said something about one of the items i was considering. I told him what is was but language gap got in the way and the coversation felt awkward so I moved on. Few minutes later I was at a different bin on the other side and he was next to me again. He slowly started looking through things too while asking me something about if I had a boyfriend, if I lived alone, etc. I told him I'm married and hoped that would end his interest in me. He came around me a few more times after that and tried talking with me more. I ignored him and would move away.
I stayed alert and watched for him as i left. He obviously wasnt there with anyone and he wasnt seriously shopping for anything, but only made it look like he was.
I was being polite and that was wrong in this situation. I should have looked him in the eye and told him I dont like him talking to me and he will leave me alone or I will get the manager. I missed a good chance for my 13 year old daughter to see a valuable lesson on how to protect her personal safety. It would have stayed with her and helped her in the future.
Am I ever going to get away from this belief of the need to be polite in all situations? Its more ingrained in my brain than I thought.