Should I have defended myself?

This happened a few months back when I worked at the movie theater. I used to work at Harkins. At Harkins, we often have more than 1 team lead on the same shift, sometimes there's up to 5 on any given shift.

Well one day I was told by one team lead to clock out and go home. On my way out, I get confronted by my other team lead and he questions where I'm going. I tell him I'm clocked out and that I'm going home. This upsets him and he gets in my face like he's trying to intimidate me and asks me who told me I could leave. I tell him the other team lead told me to and then he gets more visibly mad. He then proceeds to tell me that I'm only supposed to listen to him.

I don't remember there being some kind of ranking system on which team lead has more power, if there is, they certainly didn't train me on it and they definitely didn't tell me who was higher up on the ranks but either way if that was the case, why did this supposedly lower ranking Team Lead tell me to clock out if it wasn't okay?

I'll admit, the other team lead dismissed me 7 minutes early but I mean it's not like I was gonna knock out some big task in that amount of time. It's not my fault the guy hates being a Team Lead. He gets paid $1 more per hour than the regulars to deal with way more BS and stays later. Not my fault he accepted the promotion.

Anyway, I kept my mouth zipped during the whole interaction and he couldn't make me clock back in so I left. The next day I greeted him like nothing happened.