Is it me or should trying count

My husband and I have been together a total of two years, married one, and when we met he had been in the lifestyle for around 25 yrs and I had only heard of it through movies and book (I was very vanilla). He was honest about everything, he wanted to have other men fuck me while he watches and have sex with other women. I was concerned at first, but thought he was worth it and I would try. There were problems, for me, at the beginning seeing him with other women men, but I didn’t let him know. Then the next few women that he had sex with I was vocal about not wanted by them to come back. These woman were crazy! Homeless, asking for things, difficulty getting them out of the house, just people I didn’t want in our lives. And he thought I was “cock blocking” him. I tried to explain, but he proceeds to tell me that’s not what why, I just didn’t want him with other women. Months go by and we play with other couples and if anytime something happened, aka he didn’t fuck the woman, it was my fault. I did something, acted a certain way, when I don’t remember doing any of that. We argue, he says I’m being selfish and just getting what I want, more dick, and I don’t care about him, giving him what he gives me. I ask him to help me then, if I’m doing something and not seeing it how can I change? He never says anything helpful. He says things like “just be nice”, “let him express himself like I’ve been able to”. That’s not helpful for me. So I spend the next couple months getting info on swinging and jealousy and how to over come. The next several sessions with woman are good, no problems. But then because I’m getting more D than he’s getting P I’m the reason for it not happening. He’s said to me before that I should let him take control of talking to the woman, he has the experience and knowledge where I don’t. Ok! But then yells at me for not doing anything. I wait till people txt us to talk about it, that’s true, I’m not use to it being the front most thing on my mind. He says that all I have are excuses, but then doesn’t give any advise on how to improve things. I ask him to help me so I can help him. He throws that back at me saying why does he have to do everything. Then this last weekend, a couple comes over, we’ve played with before, but he hadn’t slept with her yet. Everything was going well, but then the other husband says that his wife isn’t ready yet, she had a hysterectomy 6 weeks ago. He was ok with it, her and I bonded more, which he liked. He even tells me later “wow, you really have changed, I didn’t see anything like before”. The next day an incident happened and I’m stressing about work and a few “real life” issues. We get a txt from a guy friend says he had a girl over. I ask him if he wants to go, he says no, I’ve had a long few days and work in the morning. And yep, the next day he throws it back in my face saying I did things to make it not happen again! WTF!!!! Am I crazy?? When he tells me something I believe he’s telling me the truth, but now he says he said he didn’t want to go to see if I would encourage it, like he does for me. That’s BS!! All the arguing, he says I’ve had two years and nothings changed, I’m getting D and he’s denied P. I was so proud of myself for things to start going in the right direction. But apparently, I’ve done nothing and I’m so selfish. Did I mention that I said I didn’t need more D and we can just work on him? Yeah he says then I would be denying him another thing he loves. I DONT KNOW WHAT TO DO!!! I love him, I want to swing with him, but no matter what I do it’s not good enough. He keeps bringing up the bad experiences that had happened as if they were all that way and those were all over a year ago!!! He tells me to do one thing and then wants me to do the other. I feel like I’m m being torn in two and everything I’m trying isn’t working, but he won’t do anything to help me. I’m not great with women, always had male friends, abusive childhood and in an emotionally abusive relationship for almost 15 yrs. I just started to have my independence and work on my self esteem for less than a he before we met. There’s a lot about a lot that I don’t know. He says I know what I’m doing and just don’t care about him. Please help!!! I’m thinking a swingers couples therapy? We have a rule, we play under the same roof, but I told him if I’m the problem that he can go somewhere without me. He says I’m trapping him and I don’t mean it. I’m so frustrated, and so is he, but I can’t better our marriage without his help. What should I do????