Would you give someone a chance even if you don't vibe but stays with you?

No probably not. I have spent far too much time with people who were wrong for me. It's not fair on anyone involved. Let them go and find someone they can connect with.