How do you tell someone with a disability that you don't want to be friends with them?

It's not because he has a disability. I never even got to know him. All of a sudden he was all buddy buddy with me and I was majorly creeped. Then someone told me he had a disability. I tried to be friends with him but we just don't click and I never have any fun with him, which is sad because I need more friends. He even tried to make me jealous about his girlfriend once.