Would you say its almost dangerous to associate with someone with a really low self esteem?

My best friend is mad at me because I brought up that he could be autistic. I'm almost positive he is. He is very sensitive. It bothered him for days one time when I told him he had expressive eyebrows. One day I said I wanted to draw a picture of him and he flipped out. So now that I have brought up this autistic thing (in a gentle and nice way), he ran and cried to his new girlfriend and now she had to gripe me out for being such an "ass". People baby him all the time, his girlfriend even rocks him to sleep! I'm the only one that's real with him. But it has made me realize, no matter how nice a person is, if they have low self confidence, it could backfire and end up making you look like the bad guy.