Is it just me or is coloring with yellow kinda frustrating?

I try not to use yellow unless I specifically see it in my vision of a drawing. It seems like it takes longer to shade with yellow, especially if I need to add much darker colors around it later. By the way, I just draw as a hobby, nothing professional.