Is it ok to let your baby stay with your parents for a few day?

I just needed to get a little break so I let my baby stay with my mom and my friend, my sister and others said they could not do that. They said I should wait until the baby is 3 years of age before I allow him to stay over.