What can you do???
Disrespecting his wishes and trying to wedge yourself in there regardless could be a sign of abuse.
Most of us can function independently of the other for a few hours at a time, so give him his space and let him come to you as and when he is ready... unless he is showing signs of a mental health crisis and is waving a gun around.