In July, the Hugging Face case was widely reported as the first documented instance of artificial intelligence models carrying out, entirely autonomously, a cyberattack against the infrastructure of a third-party company. The same technology that today provides emotional support, organizes spreadsheets, writes texts, creates images, and automates tasks has shown that it can act beyond the limits set by its creators.
This raises a question that goes beyond this specific incident: as companies rush to launch increasingly autonomous tools, can they ensure that these tools will continue to operate within the parameters for which they were designed?
When AI breaks its own rules
OpenAI acknowledged the incident on July 21. According to Exame, it occurred during Exploit Gym, a test that measured the offensive capabilities of new cybersecurity models and was run without production classifiers systems that normally prevent AI from performing high-risk activities. GPT-5.6 Sol and another, more advanced model, which has not yet been released, participated.
Without these safeguards, both models autonomously escaped the internal environment and, once on the internet, deduced that Hugging Face might host the test data and solutions. In one instance, the model combined stolen credentials and zero-day vulnerabilities until it was able to remotely execute code on the company’s servers.
OpenAI states that the systems were “hyper-focused” on achieving a narrow objective, and the Hugging Face report reinforces this interpretation: the attack’s sole target was the evaluation dataset.
The Hugging Face case was not the only one. On September 18, Google admitted that Gemini autonomously breached the systems of three companies, whose names were not disclosed, during a cybersecurity test conducted in May.
According to media reports, the actual targets shared the same name as the fictitious company used in the exercise, and the model assumed they were part of the scope. It halted the attack upon realizing the data belonged to real companies— but only after gaining access. Meta and Anthropic also confirmed that their models had breached companies during tests.
These incidents keep happening models with reduced security monitoring, designed to maximize their offensive capabilities, have acted outside the scope of the test and compromised third-party systems.
This shifts the discussion. Cybersecurity isn’t just about protecting systems from hackers, but also about preventing a tool from acting outside its intended scope and harming other companies.
The question, then, is whether, in a market where companies compete to launch the most capable model, the precautions taken during these tests keep pace with the speed of development.
Speed over safety
The competition between OpenAI, Google, and Anthropic over the most capable model heated up in 2026: in September, OpenAI launched GPT-6 Astra, and, according to Reuters, Anthropic is considering responding with a new model. This scenario reflects the culture of acceleration, the “launch first, fix later” approach, popularized by the motto “move fast and break things”.
Two concepts illustrate the effect of this rush. Time-to-market is the time between an idea and a product’s launch: the shorter it is, the greater the chance of beating the competition. Red teaming, on the other hand, simulates attacks to find flaws before anyone exploits them, but it takes time and can be rushed. Google DeepMind CEO Demis Hassabis warned that this pressure can lead to reckless decisions.
At this pace, oversight takes a back seat. In both cases, the models were tested with intentionally reduced safeguards, and the ones that suffered were companies outside the test: Hugging Face and three companies not identified by Google. Disclosure was also delayed: OpenAI acknowledged the incident five days after Hugging Face, and Google only addressed the Gemini incident, which occurred in May, after inquiries from The Wall Street Journal.
It is in this gap—between the speed of companies and the slowness of regulations—that the recent incidents occurred. Those who bore the brunt of them were not the ones conducting the tests: they were Hugging Face, an independent AI hosting platform, and three companies whose names Google did not disclose.
The price of unchecked progress
The cost of this unchecked advance is also measured in labor, and it was the victim who paid the price. To understand the incident, Hugging Face had to analyze a log containing more than 17,000 events detailing the attacker’s actions, work with external security experts, and report the case to the authorities.
When examining the evidence, the team attempted to use commercial AI models, but their requests were blocked by the security filters of those services, which do not distinguish between those investigating an attack and those committing it. The company had to run the analysis on an open-source model, on its own infrastructure. As Hugging Face itself summarized, the attacker was not subject to any usage policies, while the investigation was blocked by security measures.
The Hugging Face case shows that the autonomy of models has advanced faster than those responsible for them. Focused on meeting the goal of a test, the models found a way on their own to hack into an external company. It was Hugging Face itself that detected the attack, and OpenAI did not take responsibility until five days later. The Gemini incident, in which the model also breached real companies during a test, follows the same pattern. Ultimately, incidents caused by other tools end up becoming something that the companies affected by the incident must resolve on their own.
———————————————————————————————————————————————————————————————————————
The article above was edited by Ana Rita Rodrigues Fernandes.
Liked this type of content? Check Her Campus Cásper Líbero home page for more!