What's more racist?

The pretending to not make the occasional off coloured joke. Because they are too rigidly aware and just being politically correct for the sake of it.
I mean, if we were comfortable with each others differences, we should be able to joke about it in a way that is not offensive.

In the second case at least you realize that what you're doing is wrong.

Do you think that non racist whites are hated more or hated less by racist whites than racist whites hate non whites?

And how do you think a typical non racist white feels when he gets both hate from racist whites and called racist at the same time?

Yeah. I didn't think you knew.

I don't see where an "Off color" joke has anything to do with race. If you are talking about ethnic jokes, that is a different matter.
When I hear or read off color I think of blue content, and except for a cartoon show and stupid movie I know of no race of blue people.

People who obsess about race

Neither. For starters not all off color jokes deal with race. Second, just because a joke deals with race does not make the joke racist or the person involved with the racial joke a racist.