What should you do if you're worried you've ruined a potential relationship with a thoughtless joke?

What are effective ways to handle neighborhood bullies when they seem to have no fear of consequences?