Here is the problem..
Trump did not simply tell them to go back.
Here is the tweet in full..
"and viciously telling the people of the United States, the greatest and most powerful Nation on earth, how our government is to be run. Why don’t they go back and help fix the totally broken and crime infested places from which they came. Then come back and show us how you did it."
The media always leaves off the last sentence. He isn't telling "brown people" to go back because he doesn't want them here. He is specifically referring in this tweet to Rep. Ilhan Omar (D-MN) who came to the United States as a refugee from the violence in Somalia. Rep. Omar has criticized not only Trump, but America and the American people in general. You can agree or disagree with Trump or Omar, but the media is portraying this as Trump simply telling people to go back where they came from, and that's disingenuous.