Why are people of color so against being called colored people? It means the exact same thing.

Same reason why black people generally dislike being called 'negro' but are fine with being called 'black' even though the former is literally just french for black. It's weird and has a racist history behind it. Language is often arbitrary and vibes based like that, best not to get hung up on it.