How to convince an older guy ...

There's an attraction/chemistry between me and this older guy I used to work with. We've known each other for around 5 or 6 years and probably see each other a handful of times every year. Over the last year, we've talked about our attraction to each other, on and off. The most physical it's got has been him caressing my thigh. He doesn't think that I should 'waste' my time with him, and that I'm better off with someone closer to my own age. I don't disagree. But, we have great chemistry. I'm not expecting a long-term relationship with him - he isn't expecting that with me either - so I don't see what the big deal is if we perhaps explore the chemistry a bit more. He thinks that he feels like he's taking advantage of me, even though that's not the case.

How can I convince him that I can make these decisions for myself?

(I'm late 20s, he's around 50 but looks a lot younger.)