Does a lack of self confidence really make someone unattractive?

I'm not going to lie, I struggle with my self confidence and have since I was a teenager. Though I've gotten much better, it's still an uphill battle and I definitely have my highs and lows (and the lows can get really low).

Anyway, I've always thought that my family has lied to me about how attractive I am, because I have definitely never been treated like I'm attractive and in fact, usually get treated like the opposite. I've brought this up and the excuse is that I have no self confidence and it shows, and this is why I don't get seen or treated as attractive. This just doesn't pass the sniff test to me. Is there really any truth to this, or is it likely my family really is lying to me?

Apparently another thing is that I'm "elusive". It's true that I don't really like the idea of getting close with anyone and like to keep my distance. I still don't see how this would affect how many people notice me or how they treat me upon meeting me for the first time (as friends or in a professional setting; I don't date).

It doesn't really matter anyway since I'm not actually trying to attract anyone, but I don't want to get buffaloed by well-intentioned people who love me into accidentally thinking I'm better in any way than I really am. I feel like that would be a bigger blow than having low self confidence.