That used to happen to me all the time with the 3rd ex. Little did I know, at the time, that the doubting and the accusations were giant red flags...that there was something very wrong with HIM. It took forever for me to figure it out. Those were just the beginning signs of domestic abuse, which, typically, escalates.
If I might make a suggestion....RUN!!!!