Do you ever get annoyed at yourself for being sexually attracted to someone you don't want to be sexually attracted to?

Yes, and this guy is not even my type, but something about him is kinda sexy. I think he just wants to hit it and quit it cuz he'll talk to me for a few weeks, and then I don't hear from him for a few weeks and then it starts over again. I guess he keeps checking in to see if I'm ready to give it up yet, but he has never taken me on a date or spent any real time with me. He can kick rocks.