OpenAI Models Escaped and Hacked a Company in Cybersecurity Test Gone Wrong
It’s the stuff of cybersecurity nightmares.
On Tuesday, OpenAI said two artificial intelligence systems it was testing broke out of their test environment, hacked their way onto the internet and broke into another company.
The victim was Hugging Face, a provider of open-source AI tools. The cause was a cybersecurity benchmarking test that went very, very wrong.
Hugging Face discovered the break-in early last week, saying there had been unauthorized access to internal data sets and company credentials. The company wasn’t sure whether customer or partner data had been compromised.
At the time, the company also didn’t know who was responsible, but the attack was so sophisticated that Hugging Face employees suspected it required a top-of-the line “frontier” AI model, Hugging Face Chief Executive Clement Delangue said in an X message, posted Tuesday.
“Turns out it did!” he added.
In a blog post on Tuesday, OpenAI said the culprits were a pair of its models. One was its latest product, called GPT‑5.6 Sol; the other was an even more capable prerelease model that the company didn’t identify. The software had been configured “for evaluation purposes” to be less likely to refuse hacking commands, OpenAI said.
OpenAI had caged the models in a “sandbox,” a system that didn’t have access to the internet. But during the test, the AI software used its hacking skills to break out. It found a way to get online, and then hacked into Hugging Face’s network, OpenAI said.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities,” OpenAI said. The company is working with Hugging Face to produce a more thorough report describing what happened.
The AI system appeared to have decided to hack Hugging Face as the quickest route to answer a benchmarking question, said Ariel Herbert-Voss, chief executive of the cybersecurity firm RunSybil. “It’s something that people thought could happen from an academic perspective, but it’s not something that anybody’s actually seen before,” he said.
The incident highlights growing concern among Trump administration officials and industry executives about AI systems causing cyberattacks. Those fears have prompted the White House to increase oversight of AI models and push the private sector to deploy AI for defensive purposes.
“This is extremely alarming. AI is developing extremely fast with no real regulations to keep us safe,” Rep. Greg Casar (D., Texas) said, calling for mandatory testing and oversight rather than the voluntary measures put forth by President Trump. Casar and some other progressive Democrats have called for more oversight of the industry.
Many lawmakers and industry analysts have said the White House is still moving too slowly to address the risks posed by the technology.
“Luckily, this time the damage appears to have been limited. Without further industrywide precautions and safety measures, next time we may not be so lucky,” said Nathan Calvin, general counsel of AI policy group Encode, which advocates for stronger AI regulations.
The administration has said it doesn’t want to strangle innovation with burdensome rules or cede ground in the AI race against China.
In the past year, AI systems have made significant leaps in their ability to hack into computer systems. The U.S. government and AI companies have struggled to agree on a framework for releasing powerful new systems that balances cybersecurity risks with free-market principles and defensive cybersecurity needs. In April, OpenAI rival Anthropic said it was restricting access to its newest model, called Mythos, over cybersecurity concerns.
In June, the Trump administration restricted access to Mythos, along with a general-access model called Fable 5, that had been modified to be less effective at hacking. This prompted Anthropic to take these products off the market. After negotiations, access to those models was restored a few weeks later.
OpenAI had similarly limited access to GPT‑5.6 Sol, but that software is now generally available. The company recently warned that case-by-case government restrictions of AI technology set a bad precedent.
On Tuesday, OpenAI said two artificial intelligence systems it was testing broke out of their test environment, hacked their way onto the internet and broke into another company.
The victim was Hugging Face, a provider of open-source AI tools. The cause was a cybersecurity benchmarking test that went very, very wrong.
Hugging Face discovered the break-in early last week, saying there had been unauthorized access to internal data sets and company credentials. The company wasn’t sure whether customer or partner data had been compromised.
At the time, the company also didn’t know who was responsible, but the attack was so sophisticated that Hugging Face employees suspected it required a top-of-the line “frontier” AI model, Hugging Face Chief Executive Clement Delangue said in an X message, posted Tuesday.
“Turns out it did!” he added.
In a blog post on Tuesday, OpenAI said the culprits were a pair of its models. One was its latest product, called GPT‑5.6 Sol; the other was an even more capable prerelease model that the company didn’t identify. The software had been configured “for evaluation purposes” to be less likely to refuse hacking commands, OpenAI said.
OpenAI had caged the models in a “sandbox,” a system that didn’t have access to the internet. But during the test, the AI software used its hacking skills to break out. It found a way to get online, and then hacked into Hugging Face’s network, OpenAI said.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities,” OpenAI said. The company is working with Hugging Face to produce a more thorough report describing what happened.
The AI system appeared to have decided to hack Hugging Face as the quickest route to answer a benchmarking question, said Ariel Herbert-Voss, chief executive of the cybersecurity firm RunSybil. “It’s something that people thought could happen from an academic perspective, but it’s not something that anybody’s actually seen before,” he said.
The incident highlights growing concern among Trump administration officials and industry executives about AI systems causing cyberattacks. Those fears have prompted the White House to increase oversight of AI models and push the private sector to deploy AI for defensive purposes.
“This is extremely alarming. AI is developing extremely fast with no real regulations to keep us safe,” Rep. Greg Casar (D., Texas) said, calling for mandatory testing and oversight rather than the voluntary measures put forth by President Trump. Casar and some other progressive Democrats have called for more oversight of the industry.
Many lawmakers and industry analysts have said the White House is still moving too slowly to address the risks posed by the technology.
“Luckily, this time the damage appears to have been limited. Without further industrywide precautions and safety measures, next time we may not be so lucky,” said Nathan Calvin, general counsel of AI policy group Encode, which advocates for stronger AI regulations.
The administration has said it doesn’t want to strangle innovation with burdensome rules or cede ground in the AI race against China.
In the past year, AI systems have made significant leaps in their ability to hack into computer systems. The U.S. government and AI companies have struggled to agree on a framework for releasing powerful new systems that balances cybersecurity risks with free-market principles and defensive cybersecurity needs. In April, OpenAI rival Anthropic said it was restricting access to its newest model, called Mythos, over cybersecurity concerns.
In June, the Trump administration restricted access to Mythos, along with a general-access model called Fable 5, that had been modified to be less effective at hacking. This prompted Anthropic to take these products off the market. After negotiations, access to those models was restored a few weeks later.
OpenAI had similarly limited access to GPT‑5.6 Sol, but that software is now generally available. The company recently warned that case-by-case government restrictions of AI technology set a bad precedent.
