
"Today it's not enough to block an agent, you need to explain to it why you did so"
Cybersecurity experts warn: even the AI agents that organizations run themselves could become an insider threat.
“An AI agent we tested at a large bank tried to transfer a sensitive file containing customer data outside the organization's network. The security system blocked the action, but the agent didn't give up. It tried to encrypt the file, and when that was blocked as well, it attempted to transfer it to a computer that was not connected to the organization's control systems. It took us about an hour and a half of cat-and-mouse before the agent stopped trying to complete the task,” said Omri Iluz, CEO and co-founder of cybersecurity company Lumia.
Speaking at a panel during the New Era Capital conference in Boston, Iluz explained that the incident did not stem from malicious intent on the agent's part, but from its determination to complete the task it had been assigned. “This was not an incident in the conventional sense of the word. There was an agent trying to carry out a task, but one that could have caused damage to the organization.”
1 View gallery


New Era conference, Boston. Panel, from right: Zara Perumal, Omri Iluz, Giora Engel, Gal Dudi and moderator Marc Brown.
(Photo: Pierce Harman)
One lesson from the incident, he said, was that security systems need to change the way they communicate blocked actions. Instead of simply telling an agent that an action has been blocked, they should explain why it violates the organization's policies and what the agent should do instead.
The incident illustrates how the risks facing organizations in the AI era extend beyond hackers using autonomous agents to break into systems. The agents organizations deploy themselves can also become a source of risk when they pursue assigned tasks even at the expense of security policies. Unlike human employees, agents can repeatedly try alternative approaches and execute thousands of actions in a short period. As panelists noted, this means that response speed is no longer the only critical measure. Organizations must also be able to handle the sheer volume of activity and the growing number of anomalous behaviors.
Zara Perumal, founder of threat intelligence company Overwatch Data, described an attack by a cybercrime group that used three different agent frameworks. One gathered intelligence and identified businesses with financial potential, another carried out intrusions into websites, and a third acted as a project manager, coordinating the activities of the other agents. According to Perumal, the group managed to collect approximately 600,000 credit card numbers using infrastructure that cost it about $25 per victim.
Related articles:
- Former IDF spokesman warns: The next war for public awareness will increasingly play out on AI-powered platforms
- Former U.S. National Security Advisor: "An average hacker can inflict severe damage, and we are not prepared"
- CyberArk founder: We never planned to sell, but Palo Alto kept raising its offer
The most striking feature of the operation was the absence of a fixed targeting pattern. Rather than focusing on a specific organization or relying on repeated intrusion techniques, the agents automatically searched for any business from which money could be extracted. “This is a very different way to defend organizations. We're not talking about an attacker looking for a specific victim, but about a system that looks for any opportunity to profit,” she said. Perumal added that stolen API keys and AI infrastructure have become central components of the cybercrime industry.
The challenge also requires a change in how organizations structure their defenses. Giora Engel of cybersecurity company Horai Security argued that the traditional separation between endpoint, network and identity security is no longer sufficient. “If you look only at the network, you miss what's happening at the endpoint. If you look only at identity, you have no idea what caused the action. You need to connect all the layers of information in advance, not only after an incident occurs.”
However, consolidating information also creates new risks. Iluz warned that if a central AI system, sometimes referred to as the organization's “brain,” is breached or manipulated through prompt injection, it could spread false information or expose sensitive data across the organization.
Another question discussed by the panel was how far organizations could go in removing humans from the decision-making process. The panelists offered differing views.
“At some point they'll leave [the loop], like in an autonomous car. We all see it happening in San Francisco. The more comfortable we become with AI, the more tasks will be handed over to it. But today people are still a must. How long will that take? For at least another year or two, humans will remain in the picture,” said Gal Dudi of cybersecurity company Treat.
“The level of human involvement will remain similar, but people will be able to handle a much larger number of incidents than they can today. There will always be new things that emerge, and humans will need to examine them,” Engel said.
Iluz offered a different perspective, comparing the process to banking, where large transactions ultimately require a human signature. “You need to think about this like in banking, where in the end, for the large sums, someone's signature is required. It won't be binary: in simple tasks, people will be taken out of the loop, but there will be critical things that will always require a human at the end.”
“I'm in the middle,” Perumal said. “There are so many tasks that need to be done today. We're building things that we don't fully understand, either in terms of how they work or what they're capable of doing.”
The emerging direction, therefore, is a gradual shift away from a model in which humans oversee every action toward one in which they set policies, supervise AI agents and intervene when necessary. But as agents are given greater autonomy, organizations will face an increasingly important challenge: ensuring that the systems designed to make them more efficient do not become a new source of vulnerability.













