Should I give up child support card because my child isn't with me right now?

My sons father knows that I'm homeless and trying to save money to get back on my feet so every dollar counts. He just called me yelling at me because he can't pick my baby up at 2:30pm because he gets off work at 5. I'm still waiting on my second pay check from my job. I also live 4 hours away from him in another state. I'm trying to save as much money as I can get on my feet so I can take care of my baby. But now he wants me to send him the child support card that he pays me for 43.00 a week. He drives a challenger and has a place but I'm homeless driving a 2007 hatch back