Do you ever feel guilty for stuff you've never done?

Sometimes I think about a situation I found myself in sometime in the past and I realize I could have done something really rotten. And somehow I feel guilty about it even though I didn't do it.