Should I be worried that he left his boxers and socks for me to wash for him to collect next weekend?

I wouldn't say it's something that should "worry" you, but I'd definitely consider this a red flag. Especially if there was no prior understanding/agreement between the two of you about who does laundry for whom, and when. Most definitely something you should talk to him about.

No matter what kind of relationship you have, YOU have to define your own boundaries, and demand that people respect those. You may like/love him, but that's no reason to let anyone walk all over you ... EVER.