I need advice

Ok so first things first I am asking for no judgement please. Any hateful judgy comments will be removed..... I am really struggling with a guy at work. We’ve been back and forth back and forth for almost a year. We will talk and then decide to cut it off then we keep goin back to each other.... see here is the kicker I’m married and he’s in a relationship with a big wigs daughter. I know it’s wrong and I’m a horrible person . if I’m unhappy I should just leave but I’ve been with the same person for seven years and I’m scared ....it’s all I know. He is lazy and doesn’t work and let’s me be the provider ( not an excuse) but still. What I’m confused on tho is this. Me and the other guy actually met up in person after work last week, up until then it hasn’t been physical. We made out it didn’t go any farther and to be honest it wasn’t the magic I thought it would be. He’s talked to me very little since then and I’m just trying to understand from a guys perspective what could be going on..... is it guilt? Is it that it just was THAT BAD? Is he trying to pretend it didn’t happen because he feels bad? Is he afraid to feel something more deeply? I don’t understand. I know it’s not because he simply got what he wanted because I know him well enough by now that I know that’s not it. We’ve both been very conflicted for awhile and discussed it with each other.... so is that what this is is he confused? Also this isn’t the first time he’s gotten weird on me and has acted like he hasn’t wanted to talk to me much.... it’s like a cycle.... but if he really wanted to he’d make time for me right ?