Have you ever refused a gift?

My crush gave me a monetary gift for my birthday. I told him that he did not have to give me anything, but he said he wanted to. Would I be wrong to sneakily give the money back to him, like slip it in his pocket or something?