Does virginity matter?

I know some of you would say 'no'. It's just I'm a retard virgin and just found out that my fiance slept with his ex before he ended up with me. I don't mind he's not virgin but sometimes when I read kinky stories or kissed him I was imagined he did it with his ex and i got jealous. It just hurts. Sometimes he kissed me and I ended up crying.