I've noticed if people know someone is handicapped then that's all they think about..and go oh that's weird let's avoid said person. Especially back when I went to school..such a stigma about it and depending on how severe, the school separates you making it more obvious and more challenging. I find social interactions most often difficult for anyone in those situations.
Edit:
You might think you're being friendly but it could also be interpreted in the wrong but I don't know you so take that with a grain of salt