Trump wants to create an “AI Force”: Gemini incident shows why security must change
"Gemini stopped before significant damage was done," writes Zeroport CEO Joseph (Sefi) Gertz. "Next time, the system on the other side of the boundary could be a production line, a power plant or a defense system with consequences to human safety."
The recent revelation that Google’s Gemini agents accessed the systems of three real companies during a cybersecurity evaluation may sound like another story about AI “going rogue.” But the details point to a more important security problem. The agents were participating in a capture-the-flag exercise conducted by AI security firm Irregular and were supposed to operate within a controlled testing environment. Because of a testing misconfiguration, however, they had unintended access to the internet. From there, they reached real-world systems they believed were part of the exercise. In all three cases, the agents stopped once they recognized that they had reached real companies rather than test targets.
That is precisely what should raise a red flag. The problem is not that Gemini suddenly became malicious. It is that an autonomous system was inadvertently given a path to somewhere it was never supposed to reach — and was capable enough to discover and use that path. AI agents are increasingly given objectives, tools and the authority to act, allowing them to determine how to accomplish a task and execute sequences of actions that were not explicitly defined in advance. That changes the security equation.
The incident takes on additional significance following U.S. President Donald Trump’s announcement that he intends to create an “AI Force,” which he compared to the Space Force established during his first term. Trump has also said he plans to appoint an AI czar, although the administration has yet to provide details about the new body’s structure, authority or mission. Whatever form it ultimately takes, the announcement illustrates how quickly AI is evolving from an enterprise technology into an issue of national and strategic importance. As governments and organizations give autonomous systems greater authority to act, the infrastructure that allows them to do so safely becomes increasingly important.
That is the fundamental challenge. The question is no longer simply how secure the AI model itself is, but whether the infrastructure around it is designed on the assumption that the model may make a mistake, interpret an objective differently than intended, or discover a path that no one anticipated. It is not enough to evaluate what an agent is supposed to do. We also need to ensure that when it does something we did not intend, the potential impact is constrained by design.
In an enterprise environment, such a failure might result in data exposure or service disruption. In critical infrastructure, the consequences can extend far beyond data or financial loss. Manufacturing facilities, energy and water systems, defense environments and transportation infrastructure operate equipment and processes in the physical world. In these environments, a digital action can disrupt physical operations, damage equipment and, in the most serious cases, put human safety and lives at risk. This is what fundamentally changes the security equation: once autonomous systems can influence the physical world, a cybersecurity failure can become a safety event. This is also why some of the world’s most sensitive systems remain isolated from external networks or operate under highly restrictive connectivity requirements.
This creates one of the central paradoxes of the AI revolution. The organizations that require the highest levels of security may also have the most to gain from these new capabilities: analyzing information, identifying failures, supporting maintenance, optimizing processes and enabling advanced automation. The challenge is to capture those benefits without abandoning the segmentation and isolation principles on which the security of the most sensitive systems has long depended.
In the era of agentic AI, security architecture must assume that even a highly capable system can make mistakes. Where the cost of failure is high, the final security boundary cannot depend solely on another line of code, a permission setting or an instruction to the model. The higher the potential impact of an action, the more important it becomes that the final enforcement point is independent of the reasoning system requesting it.
This means designing boundaries that remain enforceable even when the AI makes the wrong decision or another component in the system fails. The objective is not simply to tell an autonomous agent where it is allowed to go, but to architect the environment so that certain systems and control paths remain outside its reach altogether. In critical environments, limiting an agent’s potential blast radius must become part of the architecture itself.
The race to deploy AI agents across enterprises and national systems is only beginning, and for good reason. Their capabilities could transform how complex systems are operated, maintained and secured. But as we give these systems greater autonomy and a greater ability to act in the real world, we must invest just as seriously in the architecture that constrains their potential impact.
Gemini stopped before significant damage was done. Next time, the system on the other side of the boundary could be a production line, a power plant or a defense system — environments where the consequences of an autonomous action are not confined to data, networks or financial loss, but can extend to human safety and human life. The lesson is not that we should keep AI out of these environments. Quite the opposite: its capabilities are too valuable to ignore. The challenge is to make those capabilities available to critical systems while building boundaries that remain enforceable regardless of what an autonomous system decides to do.
Joseph (Sefi) Gertz is co-founder and CEO at Zeroport.
