I'm a Loner Most of the Time

I often feel like being an INFJ is what makes me a loner. I can only deeply connect with certain types of people. I think only other INFJ's can understand this frustration. Its difficult enough to know that this is the rarest personality type in the world. I don't know if any one else here can relate or not, but I just wanted to get it off my chest.