Why are people of color so against being called colored people? It means the exact same thing.

Not really. Colored people was a Jim Crow term during the Civil Rights era.
Over time people evolved and started using people of color. And that just means a person who is non White.