So here's a really odd question for the Brits on SW

The problem is that the English St George's flag has become very much associated with the anti immigrant groups and that's increasingly happening with the Union Jack too. I'll be honest if I saw someone walking along the street today waving either I wouldn't immediately consider them showing patriotic ferver but probably an anti immigrant sentiment.