How do people just not feel concern for others?

How do people figure out where they draw the lines of who they want to care about and who can get bent? How can they tolerate that some people are treated differently based on which side of an invisible line they were born on while still believing all men are created equal? Is it really that rare to just care about everyone?