My effing next door neighbor keeps walking into our house through the backdoor

Like, I get it, we are buying his mom's house and are now selling all her belongings.....but just walking in is weird af. Sometimes he knocks but not always...