Why do I feel guilty about setting limits on who can attend family events, and how can I overcome this feeling?

If you see through manipulative people so easily, what makes intelligent people still get caught in their traps?