Why do people say they go/went to college when it's a university?

Huge difference between a community college and a university. Yet the degrees are applicable to both. Not all universities will accept a college degree though while a college usually will accept a university degree. Note many private colleges are not called universities as well. So the university will not accept the degree. They may accept certain course units. Depends on the university.

Basically it's prestige. It's supposedly harder to get into a university.

All my math and science courses are transferable to most universities. While most of the other courses are not transferable. Like my logic and philosophy courses are not transferable.

And so my degree is not transferable.