Yes, you can be mad at him if that suits you.
And ultimatums are always hard to hear...
But honestly, he may have just done you a favor...or several.
1) now you know what he is 'deep down' like...
2) carrying weight leads to all kinds of health problems... cardiovascular, gastric, pulmonary, fatty liver disease are just a few.
3) you are unhappy that your clothes don't fit, So change that.
4) and you yourself said :

losing weight is something I want

Wanna know an easy way to drop 100+ lbs?

Well...yes.

But lose it young!

If you are going to lose weight,lose weight for you and to better your own health. Don't do it for him. Once you lose weight, ditch him. If someone puts conditions on their love for you, that's not love. The one who loves you will still love you for better or for worse. Love yourself first.

Obesity is fatal. You put yourself in this position. At least he was honest enough to tell you. He fell in love with a different “you”, perhaps one that cared how she looked.

There’s the sympathy response to this post and the reality response. Which are you honestly looking for?

You already know the truth…

Who cares? he isn't worth you time anyways.

Is that all he sees in you? Leave him. He is full of lust and that's all he cares about because I guarantee you, even if you lost 150 pounds, he'd lust after somebody else. There's no quenching that because it's an uncontrollable passion that drives him, in which he seems so preoccupied with in his heart and mind, to being wholely sexually driven to infidelity and never finding satisfaction or even a sense of gratitude for a decent woman like you being in his life.

At least he’s honest

Dump him. If he is this way now, when middle age hits both of you and looks and weight start to go he will leave you or cheat. He was an ex for a reason.