Opinion
AI defenders are fighting under rules their attackers don’t have
One of the first major AI-versus-AI incidents showed how safeguards designed to prevent misuse can leave authorized defenders constrained during a live attack.
An autonomous AI agent moved through thousands of actions at machine speed. When the defenders at Hugging Face turned to AI to help investigate it, their assistant refused.
The commands, malware and techniques they needed to examine looked too much like instructions for carrying out an attack. The attacking AI was not constrained by the same safeguards that now limited the response. At the moment speed mattered most, the defenders entered the investigation with rules their adversary did not have.
The deeper failure was not the breach itself. It was that the systems designed to prevent AI misuse could not reliably distinguish between an attacker exploiting a system and a defender authorized to investigate the same techniques.
1 View gallery
Eylam Milner (left) is Co-Founder and CTO at Echo. Mor Weinberger is a Software Architect and Researcher at Echo.
(Jenny Rochlin)
OpenAI later disclosed that the intruder was one of its own research models, which had been participating in a controlled evaluation of advanced cyber capabilities. The model searched for information outside the environment created for the test, obtained credentials and ultimately reached Hugging Face. The platform is one of the world’s largest hubs for building and distributing AI models and is used by millions of developers.
The technical path matters, but the response exposed the broader problem. Cyber defenders routinely need to inspect malware, reproduce exploits and reconstruct hostile commands. When stripped of context, that work can look almost identical to an attack. The difference is often not in the command itself, but in who issues it, which system they control and what authority they have.
Safety systems that evaluate mainly the visible content of a request may therefore block legitimate defensive work because the techniques involved are inherently dual-use. That does not mean safeguards should be removed. A model capable of helping a researcher understand an exploit could also help an attacker improve it. The challenge is to prevent misuse without leaving authorized defenders unable to perform the work required during a live incident.
Hugging Face eventually worked around the problem by running a self-hosted open model inside its own environment. That solved two problems at once: investigators could analyze the attack without triggering the same commercial guardrails, while credentials, forensic evidence and internal infrastructure details remained inside the company’s environment. Using the model, the team reconstructed more than 17,000 attacker actions in a matter of hours, work that otherwise could have required days of manual investigation.
The local model did not prove that open systems are inherently better or that every organization should abandon commercial tools. It showed that analytical capability alone is not enough. A defensive AI system also has to be permitted to perform the necessary work, under conditions that protect the evidence and remain under the organization’s control.
The incident also showed that this disadvantage cannot be solved in the middle of a crisis. Organizations cannot assume they will assemble the right AI capabilities after an attack has already begun. They need to know in advance which model they will use, under what restrictions, where it will run and who controls the environment around it.
Hugging Face reached a similar conclusion, recommending that organizations have a capable model they can run on their own infrastructure, vetted and ready before an incident. That recommendation addressed both failures exposed by the investigation: the risk that safeguards will block legitimate analysis and the risk that sensitive attacker data or credentials will have to leave the organization.
The word “vetted” extends beyond the model. A locally operated system still depends on containers, libraries, an operating system and other supporting software. Those components need to be maintained and trusted before they become part of an emergency response. The middle of an incident is the worst possible time to discover that the software beneath a security tool has introduced another security problem.
This does not mean every company needs to become an AI lab or operate its own models for every task. It means that AI used in incident response can no longer be treated like another feature activated on demand. Organizations need to know whether the system can perform the uncomfortable and potentially dangerous work involved in a real investigation, where sensitive evidence will be processed and what will happen when its safety policies encounter ambiguous requests.
Model providers face the corresponding challenge. They need to support legitimate researchers and defenders without turning advanced cyber capabilities into unrestricted services. Future safeguards will need to consider more than whether a request contains dangerous commands. They will also need better ways to account for the user, the environment and the authority under which the work is being performed.
No single signal can prove legitimate intent, and claims of authorization cannot simply be accepted at face value. But a system that ignores context entirely can create a structural imbalance. Attackers can use open models, modified systems or tools operating outside commercial policies, while enterprises and professional security teams are more likely to rely on governed platforms designed to prevent misuse. Those safeguards are necessary, yet if they cannot accommodate legitimate defensive work, the organizations acting responsibly may find themselves constrained by rules their attackers simply ignore.
The question for organizations is therefore broader than which model performs best. They need to know whether the AI they expect to rely on can actually help during an adversarial, sensitive and urgent event, and whether the surrounding environment has already been tested under their control.
The next AI-versus-AI battle may be decided before the attack begins. Defenders will either test the model, its restrictions and the infrastructure beneath it in advance, or discover their limitations once the attack is already moving at machine speed.
Eylam Milner is Co-Founder and CTO at Echo. Mor Weinberger is a Software Architect and Researcher at Echo.
