It depends, but if you read the wider room...
I see that you're an immigrant, but you don't say to where...
If you're an immigrant to SPAIN, Bolivia, Colombia, El Salvador, then maybe you should be working on learning Spanish.
If you're an immigrant to Botswana, Germany, Greece, Japan then maybe the question is a bit more out of place, but for an exception that I'll mention below.
If you're an immigrant to the US, especially certain parts, then it's a reasonable question, again given the thought next:
You say they're customers... do you want their money??? If so and they speak Spanish then maybe it's a reasonable investment in your career to learn the languages of the people that want to give you money, but want to be able to communicate their needs to get the exact service or goods they need. So if you're in a high Spanish-speaking are then isn't it reasonable to be asked, "Hey, I have these dollars, are you able to offer me the information I need to gain these dollars?" ??
If you are in the United States then it is less likely that you'll find someone who speaks Polish. Not saying that is a good or bad thing, but looking at the marketability of your linguistic offerings you might be not-what-the-boss needs. English would be most needed, Spanish 2nd, 3rd might be a tie between German, French and Portuguese. (again depending on region). I think you'd find some urban areas might still have a need for clerks or customer support that speak Polish, but overall (like across the country) even some domestically native languages (Cherokee, Navajo, Creole) would out-demand Polish.
Just some perspective.
Also you mention their angry looks -- what was the presentation of YOUR attitude going into the conversation. Maybe given one affect they and you could have walked away with a hearty laugh. Given some other way of expressing it everyone walks away with a bad taste in their mouths. I know in your narrative you'll always be the good guy, but if I was your boss I'd have noted the experience and looked for other signs of whether this employee (you) will work out or if you're scaring away customers.
