Do you struggle with telling people "No" or when you feel you might let someone down?

Not once. If the answer is no, then it's no.
Someone, somewhere, will always be let down (or think thy have been).