Do people hate on you because you work remotely?

A former coworker that I still talk to regularly always has something snarky to say on the days that I work from home. It's annoying and immature and I called him on it today. As long as my work gets done and my boss has no complaints, I don't see what there is to be snarky about. He admitted that he's a bit jealous. 🤨