Why are people of color so against being called colored people? It means the exact same thing.

That is odd, isn't it? But these expressions have their histories and they do have different connotations. In general it makes sense to pay attention to what people prefer to be called.