I found the key!

If you want a guy to stop talking to you, just tell them you work in a factory with the maintenance team!

And just like that: *poof* gone