I'm a terrible person nobody wants to be near with

Ever since I can remember I've been an outcast... I've always thought it's because I was shy, so I've built up my confidence, but still poeple, in various groups that don't know each other, start to avoid me after some time. The more the poeple know me, the less they want to spend time with me. I had several poeple telling me how hard it is to be with me and talking to me. Sometimes when I tried to aproach poeple, they ignored me or even told me, they don't specifically me to contact them any more. I asked a trust person for help at my university, only to have him 2 hours telling me, how indeed he as well doesn't think I'm a capable person and that I have to change. I don't know how to change, and it's me, not the others, cause it's the same since primary school, through high school, till university...