Threatened by coworker.. Should I be worried for my safety?

I work as a manager for a large company, in the construction industry. Weird situation. I have a crazy coworker who works in IT. I at one point had a dissenting opinion from him. It’s a long story, but made short - I uploaded a spreadsheet wrong into a data base (having no knowledge of the right way to do it, there isn’t any documentation) but none the less, we fired off a campaign from it which was successful. This IT guy who has hated me the entire time I worked there, just super insecure and fragile ego he has admin access and halted the whole thing. He’s a contractor not even an employee, but he halted the campaign claiming there was an issue with the data. He then proceeds to revoke my access to the the software and tell me that all spreadsheets will go to him, mind you I he’s not a manager he just has access. I disagree by just saying I think that’s inefficent because in the past when we’ve sent him spreadsheets it takes weeks to get back, when it takes us days. Just tell us the proper way we will do it right moving forward. It doesn’t have to all fall on you IT guy.. well after that he’s pissed! And revokes my access to our software randomly…our director is appalled she complains to owners. But because our company is so disorganised it took a week to get my access back from our IT director. My boss intervened and makes sure that all spreadsheets will go to our data analyst to manage moving forward. During a call, the guy tells me he’ll make sure that’s not the case that he WILL make sure all spreadsheets go to him. During that time, I send him an email letting him know he’s not approved to revoke my access… he does it anyway.. he’s pissed for a bunch of reasons like that he’s told he needs to restore my access. My boss has no control tho. Meanwhile he makes up a bunch of stuff to HR, tries to get me fired and then! Puts my phone number on a spam list to get like 100 phone calls a day.. I also get a mysterious text that says I better watch my back.. also I find out he sent stuff to HR, that he got by searching around in my email address. I’m traveling to visit headquarters and I’m nervous he knows when I’m coming, my car rental and hotel. Terrifying because he was in my email and I just got sent that info like a week ago. The fact that he’s doing all these things worries me. I have a family and kids and I’ve told the company I’m worried for my safety and they said don’t be it’ll be fine.. like this guy is obviously crazy and harassing me and searching my emails. I feel like I don’t have a choice but to go visit the office (I’m flying from Texas to Florida for very important meeting) and now he knows like all my hotel, flight car details… like but I feel like if I raise anymore hell I could see more retaliation from the entire company including the crazy IT guy. And if I don’t go to the meeting I could also see retaliaation. After the owner was informed I was scared, the response was oh it’s fine! He won’t be there just come to the meeting,, you know so now I feel like I have to.. and I’m scared about this guy being so crazy to me, what do I do? Like stuck between a rock and hard place, should I just go and not worry?