What’s wrong with wearing a v-neck t-shirt and jumper/sweater

I prefer the look of v-neck to crew neck t shirts, but whenever I wear one people comment as if it’s odd, especially if it’s with a v-neck sweater as well