If something costs me my peace, it's too much.

If I don't like how I feel around someone after having given them a chance, I don't associate with someone. Doesn't matter if it seems petty, it's my life.