What makes a man want a relationship not just sex?

All the guys I meet have more fun with others than they do with me...
I go with the flow but the moment we have plans especially booked and I want to be there on time.
Somehow its always a problem.
They might be late once and I let it slide but the 2nd I get upset.
I hate tardiness and I don't think I should stop being myself.
I have to be honest.
This is when they start getting all distant and only seeing me on their terms