Pressing the crosswalk button when people are already waiting is like announcing, in public, that you trust none of them at all.

1st cousin to hitting the elevator button when there is a group of people waiting for the doors to open

Not pressing the crosswalk button yourself is a sign of arrogance on your part in assuming that everyone ahead of you is going where you want to go.