I feel it totally depends on what you are talking about. There are so many people who take great delight in telling you negative things about which you can do nothing, probably didn't need to know, and wish you could unhear. Just because it's true, doesn't mean it has to be told.
One example. You are in a relationship with someone you hope to go the distance with. You've had better lovers. Do you really think he/she needs to hear those words? Or could you maybe find a better way to encourage him/her to be more of what you want? Is that considered withholding?