Why does having some sort of a fetish makes a person so weak?

It depends on the person. If someone's a slave to their fetish it can be used to manipulate them in which case yes, that'd be weakness. Most just know what they like but it's not all they are.