Why are people of color so against being called colored people? It means the exact same thing.

So-called 'white' people are not 'white'. So called 'black people' are not 'black. Every human alive is 'coloured'. Remember that true black and white are not actually 'colours' as such.