As abused, mistaken, and misapplied as labels (and not just racial ones) may be every single day, it is hard to see what else we could do. If "race" was invented by Western minds in the 15th/16th centuries, "all of humanity together" was invented by them in the 19th/20th centuries, and is at least as divorced from reality. The human brain evolved into and under circumstances in which people seldom or never dealt with anyone whom they did not know personally. When members of tribal societies do encounter strangers unexpectedly, the way in which this is usually handled (violently and often summarily) cannot be seen as encouraging to anyone wishing to suppose humans would all just get along in the absence of social engineering.
On the contrary, it is all the social engineering and labeling that actually allows us to live immersed in a sea of strangers. For as many incidents in which labeling is used as an excuse for "meanness" at any degree of severity, there are about a hundred more in which labeling is used to make us feel we can safely engage in peaceful, day-to-day interactions with others.
Without this basic sense that certain strangers can be trusted, it goes without saying that a society of millions would collapse overnight. But we know that not all strangers can be trusted, and we do not have time to sit down and get to know everyone we encounter (often enough, people end up having been "wrong" to trust someone they thought they knew well for months or years, anyway). Again, it is hard to see what else we could do aside from look for some clues to and try to correlate traits in whatever way we can to arrive at conclusions like: "Oh, look, this guy is like me or someone I already know and trust in these ways. Maybe that means he will be okay, I guess I can try to deal with him."; or: "Uh oh, this guy reminds me in these ways of someone who did something bad to me or someone I know. I'd better avoid him just to be safe." The most that is sensible is to debate over what kinds of labels are and aren't fair, rational, or (most importantly) pragmatic to apply.
Even in the most warlike of state-level societies, death-by-violence-per-capita averaged over any length of time which might meaningfully indicate the norm is lower by far than it is in most tribal societies. And this is so largely because of social engineering, which conditions people to believe:
(1) In spite of all feelings to the contrary, you are actually in the same "group" as thousands or millions of people whom you will never even meet.
(2) You can, within some tolerable limits, expect to safely deal with even people you don't know in this group without having to worry about being attacked or killed.
(3) Here are some good indicators that a stranger you encounter is in the same group as us.
(4) If someone in our group breaks one of the sacred rules not to commit certain kinds of harm against other group-members, you can count on us, the social engineers, to send some strong people to kill or otherwise remove this covenant-breaker from the group. So if you could refrain from seeking justice on your own and collapsing our precariously-maintained complex society into a chaotic cycle of revenge-killings, that would be lovely.
Tellingly, a common religion has been the longest and most widely-used by far out of of all of the available excuses to trust strangers. Religion may be on the decline in the West these days, but the mere belief in supernatural elements is far from the crucial factor. It is "believing the same things"/"using the same form-language to interface with the world" that most inclines people to trust strangers. While the distinction between the exotic, colorful appearance of white people and the other "races" they encountered in their spread across the globe is striking and something one could hardly fail to notice, the ultimate roots of using "race" in the modern sense as an "us"/"them" indicator lies in the fact that in the 15th and 16th centuries, a white stranger was likely to be a fellow Catholic, and a non-white stranger was likely to have never yet heard of Christ.
In fact, nobody believes race alone is a sufficient group-indicator, whatever they may say. A white supremacist today may well snub blacks and consider them "others" on principle, but he certainly does not feel group-solidarity with all white people on principle, and in fact most whites are "others" from his perspective as well. Only white people who believe what he does — the beliefs he chooses to exalt into his group-identity happen to be those that consist of commentary about "race" — are apart of his group. Blacks may be treated exceptionally poorly by him on principle, but most whites are not treated exceptionally well relative to how most civilized people treat the average stranger. If he goes out of his way to be exceptionally kind to a random white person, it is not because he is white, but because his whiteness signals him as a potential convert to the white supremacist's belief-system.
Well, there is much, much more that could and probably ought to be said on this, but it’s probably ran on longer than most will read as it is (and also, it took me longer to write this than I expected and I now have to head out for a bit), so I’ll just leave it there, as far as this post goes, anyway.