Can an employment agency hold it against you if you took a while to get back to them?

Why didn't you just return it to the office? If you do something like that yes it does affect things, I'm sorry.

I mean, it depends. Both on the personality of the person working at the agency and how long “a while” is.

If it’s just a couple days then you’re likely fine, if it’s a few weeks then maybe not