My friends having a birthday thing at a LGBTQ drag karaoke bar. Am I gonna feel uncomfortable there as a straight guy?

The more seriously you take it the more you'll struggle. The more light hearted you take it the smaller the divide is likely to be. They're only people. One or two may try to hit on you or tease you but there's arseholes in every group of people, I imagine the majority will be fine.