If someone's own family has issues with and turned away from someone, would that be a big red flag?

I would say so most likely. Your family is your first support and who typically love and accept you over all others. If they have those type of issues with you, it can point towards you being mostly the problem. They wouldn't do it for NO reason. Some people are good at playing the victim.