Bathrooms two toilets

Ok so me and my partner had an argument last night
We have two toliets
about using bathroom he doesn’t like using down stairs for going number two as it too small which i agree. However I needed the loo yesterday I really wanted to go upstairs one. But then he wanted to go upstairs. but I said I needed to use the upstairs because it was quiet tv was on. he said he would wait for me to finish . And go loo. I asked him why he couldn’t go down stairs while I was up stairs he said he didn’t want to leave me waiting upstairs he normally longer then me due to his IBS x but I felt the pressure of having to hurry up because he wanted upstairs toilet even though he wasn’t as in pain as me .
Yes I changed his routine With his Aspergers Austism for going upstairs I normally take down stairs for him I couldn’t actually go in the end. He was angry and upset that I was suggesting to him go down stairs. Was I wrong should I be just easy going ? He started blaming me for him not being able to go as well