Why do passive aggressive people annoy you?

I’ve been called out for being passive aggressive and I don’t voice my opinion or stand up for myself. I apologize for a lot of things because I don’t wanna upset someone.
My two friends tell me to stop agreeing with everything and speak my opinion or choose something. When I keep saying whatever they want, they get frustrated. Really?