I want to be a good person and have a peaceful life. And if my life isn't peaceful, I want that conflict and struggle to be about a greater good.

So if you're creating a significant amount of conflict and stress , and it is over something trivial, I will not allow you in my life. I am finite and I have finite patience and emotional resources. I don't believe in unconditional love being synonymous with unconditional suffering. I also don't believe in either as stand-alone parameters which are either present or absent.

Is it selfish to let people go? Yes, selfish and right.