I Wasted My Time On Someone

So the person I spoke of when I first posted in this category showed up with his entire family to the place I work and sat down in the dining room to eat 2 nights ago. Of course, it's a public place and they can come and go as they please, but knowing I was there they still stayed. It bothers my boyfriend to no end that they did that. I told him last night that there is nothing I can do about it, but he tells me I should tell him that I'd appreciate if they don't come in there anymore. I think it would be rude to say that. And they probably won't care what I think. Honestly, I don't want to tell them not to come in there because they weren't trying to bother me. What are everyone's thoughts on this?