Everything has to do with how you see yourself. If your self-image is low enough, even positive and accepting attitudes will be seen as negative.
No one can make you happy or unhappy. Everything is choices. Until you see yourself as being whole and complete you will attract those that feed off your low self-esteem. Until you start believing in yourself, no one else will.
Don't feel confident? Always know more than they think you do.