If I tell someone "go back to your quiet place" - would it be considered rude?

I casually asked a guy whether he is a quiet person. He said "I will be like this only. I think you like an aggressive man". To which I responded "It's not about "aggressive". 2 people can have a conversation without being aggressive. Anyway go back to your quiet place". Didn't hear from him since.
I don't feel like apologizing either.