Can I ignore my jury duty letter? Or is that too risky? I really don’t feel comfortable attending.

Just go.
They'll pay a rate for the time you attend and there's every chance you won't be needed anyway