Let's be honest here. Lot of anime fans, just like gamers, are stuck in child like mental state by choice and don't want to explore other aspects of life.
They kind of creep me out, but in Japan (they source), they have been around for decades. It used to freak me out, watching all these men, on public transport, with their anime magazines, and that was 25 years ago.
Anime, like many other art forms is a way of expressing oneself. It isn't for me, but I'm not everyone. As for the pervy anime, perv is everywhere. You choose to watch it or you don't.