So tired of being the bad guy for protecting me and my own peace.

Do what you need to do to take care of yourself. You are your priority & nobody else's opinion matters. If it were me, I'd probably tell them to go f themselves.