Am I being unfair

I’ve been seeing this guy from work for a few months now he isn’t out yet he only came out to me because we were on a night out and he got drunk He tells me he loves me and wants to be with me but he always ghosts me when he’s with his mum or his friends I called him out on it and he denied it it really hurts my feelings