What do you say to people who act surprised when you give up on something or someone you loved?

I told someone I might be moving away. His response was "I guess you really did give up". No idea what he meant...