Is fantasizing about you wife having pleasure with other men wrong ?

No, it is quite common. Actually experiencing it is not wrong either, as long as she consents to doing it. Most women find it hard to understand why we would like that if we truly loved them, but it is impossible for any one person to satisfy all their partner's fantasies and desires, and if we truly love them we want to be fully pleasured. Also it is a turn on for a man to know that he has a very sexy and willing wife who is still going home with him afterwards because he is more than just a sex object to her.