Can someone track your phone location without your permission?

I think someone is tracking my phone. He sent me a google map location, and I clicked on it. He commented on what part of town I must live in. I’ve since turned location off.