Is it too late to learn how to swim when you're in your 20s?

No. People have learned much later than that. There's a class at the "Y" here for seniors.