Funny you should mention "manly hands." I think part of it is the hand should be at least normal sized. I think when a big man has hands the same size as a small woman, it's not very manly. I have this photo of two people with roughly equal sized hands to illustrate my point:
