I Have a Question

Please give me ur opinions....... So my boyfriend has a best friend who takes advantage of him but can't see it. So a few months ago my boyfriend moved in with me. We came to an agreement that his friend can come over any night during the week for a few hours and he could spend one night and one day on the weekend with us. We have gotten into spats about him thinking I wasn't being fair. I should let his friend come and go as he pleases. U should know my boyfriend is/was only 22. Am I in the wrong?