In most cases, that "slower worker" would be let go if they couldn't keep up with the rest.
If someone can't help their shortcomings they might want to talk to their employer about what's holding them back.
If the boss doesn't know that there's an issue, the "slower worker" just looks like they're slacking.

Everyone should do the best they can. You and your colleagues don't have to worry about who does how much. That only matters to the employer when deciding if everyone is worth their salary. And if he doesn't complain, you have nothing to worry about.

If you are trying your best, then they should try to understand. What more can you do? Don't feel bad.