I think sometimes there are people who need a little more attention than others. My children talk about when I paid more attention to one or the other. And they are right. There were periods in their childhood when one needed more attention that the other and then they switched. And today when they talk about it explain why this one or that one needed more help, more attention. I think if the pastor felt like he wasn't doing what he is supposed to do, he would cut it short. Remember when the children came to talk to Jesus and the disciples chastised them, Jesus said "Let them come". They weren't a bother.

Bonus- encourage your pastor. It is always well recieved.