Please define "dirty old man"

I never understood this expression.
Just because I'm older and enjoy everything about sex , does that make me "dirty"?

Aren't older men allowed to be sexual?