Why didn't I tell the guy sitting in my window seat to move?

I am on a plane. I buy my tickets early in order to secure a window seat. I don't like the aisle.
I have priority so I board pretty early. I get to my seat and some guy is strapped in tight.

Im not afraid of anyone but sometimes I just choose the path of least resistance. So I said hello and he seemed friendly enough so I just sat down in the aisle seat. At least its not a middle seat.
But then Im wondering did he really not realize that he didn't purchase a window seat?