Why do people say “to be fair” when something is mentioned?

That pretty annoying. When one of my friends called my mother a selfish bitch for abandoning my dad and I when I was seven and I said, “How dare you insult my mother?! You keep cutting me off when I mention her.” My best friend said, “Well to be fair, you mom DID leave you guys for her own selfish needs and she doesn’t really call you so she was telling the truth and you know it. I mean why do you wanna stay with a woman that doesn’t want you in your life.”