Why do I get irritated when people don't respond to me?

Please just at least say something. When my dad calls me out on my crap, I say sorry. I would say sorry until he responds. I hate it when he doesn't respond when I said sorry. He would said, "I HEARD YOU! I don't HAVE to respond to that. I heard you once and that's enough."
Why don't people respond to sorry like others? Is sorry a hateful word?