Is it bad that I don't wanna wear a bra when I'm around other people?

Nope. I hate them, so I can certainly understand your feelings. But if you can find a really comfortable sports bra that doesn't bind or pinch, try to wear it as much as possible. It will keep gravity at bay a bit longer.