Does everyone need to be comfortable

Am i wrong to believe that all 4 people in a couple swap need to be havinf fun and enjoying themselves or is it ol for one person to not be ok or gettinf along with the other couple in order to have fun? Recenrly ran into an issue with this where my spouse and i were talking to a couple. My spouse really liked them, and i did not so i wanted to move on but everyone pointed me out as the bad person. I would not have had fun. I would not have enjoyed myself. And i would not have eevne been able to perform even though the oerson i was performing for was extremly attractive. Am i wronf or are rhey wrong? I want serious answers we want to fix this so we can continue enjoyinf the lifestile together