If you work in an office environment where heels and hose are the norm, then I could see "comfy" socks being viewed as being a bit casual, since they would likely be worn with a less casual shoe. If that's not the case, then I don't know why they would even care.
I wear outrageous patterned and colored socks most days, ranging from cartoon characters to works of art. However, most times men don't have their choice of socks on display, so it's not evident to others.
