Is it racist to not want to date someone of a specific race?

Perhaps. But arguably the most benign form, unless you take it to the point, like some do, where you don't think anyone should date "whomever".