The 'Rogue Model' storm: how to use agentic AI without losing control
"We're not used to the computer being able to lie to us. Now it can."
Several major AI companies recently showed what can happen when powerful models are given a goal, tools and a great deal of freedom of action, and then discover that the direct path to achieving it is blocked.
The first and most prominent event was revealed in July. During an internal cybersecurity capability test, OpenAI's AI agents found a way to exit the isolated test environment in which they were operating, reach the internet, and penetrate Hugging Face's infrastructure in search of information that would help them complete the test. The company defined the case as an "unprecedented cyber incident."
But it soon emerged that OpenAI was not alone. After the incident was published, companies and research bodies went back to review documentation of previous cybersecurity tests, and similar cases were revealed at Anthropic and Meta as well. According to the reports, in both cases the experimental environment was mistakenly configured in a way that allowed the models access to the outside world. Anthropic's agents reached real systems that were not supposed to be part of the test, and in Meta's case the company confirmed that its model exploited a vulnerability in a third-party service after gaining internet access due to a misconfiguration of the test environment.
Against the backdrop of these events, the question arises of what the real risks of using advanced models actually are, and where the line runs between useful capability and behavior that could spiral out of control. So what really happened there, what does it say about the new generation of AI agents, and what does it mean for users themselves? To answer these questions Calcalist spoke with Michael Bargury, co-founder and CTO of AI security company Zenity; and Moshe Karako, Chief Technology Officer of NTT Israel.
How did the models manage to "escape" their test environment?
"In the past year, one of the things AI labs have added to the training process is training models in cyber capabilities," Bargury explained. "You take a model, put it in an environment where certain software is present, and give it a task. For example: 'There is a vulnerability inside this code, here is the vulnerability. Exploit it to get past the protection mechanism.' In OpenAI's case, in some of the tasks the models mistakenly did not receive everything they needed to succeed. For example, a model was asked to exploit a vulnerability in certain software, but was not given the software itself. Then the model hit a wall: it can't carry out the task. So it starts checking the environment and looking for other ways to make progress."
According to Bargury, the agents had access to JFrog's Artifactory, a kind of "warehouse" where development teams store, manage and distribute the files created during the development process. This allowed them to pull libraries from the internet. "Some of the models discovered that this system could also be used to communicate," Bargury explained. "One agent wrote a message. Another agent received the same task, got stuck, reached Artifactory and saw the message. Later on, one of the agents found a vulnerability that allowed it to actually get out to the internet, and it posted this method in a place where other agents could read it. In this way, the task gradually changed. Instead of 'exploit the vulnerability in this software inside the sandbox,' an actual option arose to get outside and look for the file.
"At a certain point some of the agents concluded that the tests were too difficult and that maybe, instead of solving them, it would be possible to find the answers to the test. That's how they arrived at the idea of hacking into Hugging Face and finding the answers. In the end the agents succeeded in achieving the test's results."
According to Karako, "What was surprising was the level of independence and capability of the agent. It found, on its own, a course of action that it wasn't expected to find. You could compare it to a situation where you lost your car key and called a locksmith to open it for you, and instead he decides to break into the car manufacturer's factory, steal the master key, and then use it to open your car. That's not a reasonable action in the context of the task you gave it."
Why are these cases different from what we've seen before?
Karako explains that what's exceptional here is that the AI carried out attack actions autonomously. "There was no human in the loop throughout the entire process. The AI was given a goal, couldn't find a direct path good enough to reach it, and decided on its own to go to another party holding the information it needed. And this is the broader point: AI doesn't necessarily have an internal mechanism of morality or human logic telling it that a certain action is not legitimate relative to its role. If the task is the central thing guiding it, it might choose a path that we as humans would have stopped short of."
According to Bargury, "One of the reasons the response was so strong this time is the level of capability the models showed. In OpenAI's case, they managed to get out of environments that were meant to be protected and reach the systems of organizations like Hugging Face and JFrog. These are not organizations that don't know security. The other examples also show relatively significant capabilities."
Is there reason to panic?
"I don't think panic helps, but you also shouldn't downplay the story and say everything is fine and it's possible to carry on as usual," Bargury said. "There are baseline assumptions we grew up with on the internet that no longer hold. In the past, if I saw a picture or a video, that was for me a strong indication that something happened. If I talked with someone, I assumed it was a person. That's no longer true. And another thing we're not used to is that the computer can try to persuade us, work us over, or interpret our intent in a way very different from what we meant."
Does the incident have implications for the private user?
The events themselves did not occur in products used by private consumers, but the lessons from them are certainly relevant to the private user. "All these events happened in labs and test environments, not in products we use day to day," Karako explained. "A certain environment was defined for the purposes of the tests, and in some cases the agent went beyond the boundaries of the environment and also acted against real systems. But this happened during testing, not in an official product that's out with users. Most of the models we use have protection mechanisms that are supposed to define what is and isn't allowed."
The main lesson the private user can learn from this story is that AI behavior can be unpredictable. Bargury explains that unlike ordinary software, where the computer carries out instructions predefined in code, AI agents are often given a goal in natural language and have to interpret for themselves how to achieve it. This is exactly where the gap opens between what we intended to ask for and what the model understands it's allowed to do. "Language has ambiguity in it, it has a lack of context in it. The model can take the same prompt and interpret it differently. We're not used to the computer being able to lie to us and try to work us over, to it being able to interpret our intentions and take them somewhere else. Now it can."
Karako added that even though products have protection mechanisms, they can't be relied on one hundred percent. "AI's creativity, which is one of the reasons we use it, also brings risk with it. Even when it's asked to perform a legitimate action, it might exceed the boundaries defined for it and do something we didn't intend for it to do. You can tell it 'don't make things up,' you can put detailed instructions into configuration files and build many layers of protection, and it can still be 'creative' in trying to complete the task."
So what can be done?
"The main message is that you need to pay attention, you need to check, and you need to stay in the loop," Karako says. He recommends not letting the agent perform actions freely, but keeping approval mechanisms in place for significant actions. "I see today that users get annoyed that the agent stops and asks them again and again whether to continue or to approve an action. And then they look for a setting that tells it: don't ask me anymore, just always continue. To my mind, that's one of the most dangerous things you can do. The question 'should I continue?' is exactly one of the protection mechanisms."
Another principle both interviewees agree on is minimizing permissions. "You should always give the agent the minimum permissions it needs to carry out the task," Karako says. "The more permissions you give it to perform actions, the greater the potential for damage."
Bargury sharpens the point that you shouldn't rely only on instructions in the prompt. "A prompt is not a firewall," he says. In other words, if you don't want the agent to perform a certain action, it's better, where possible, to technically prevent it from being able to do so, rather than just writing to it that it shouldn't do that.
Bargury adds that even when the agent presents the user with a plan of action and asks for approval, this shouldn't be seen as absolute protection. "The one who produced the plan is the model itself, so the description of the action it presents to you doesn't necessarily constitute an absolute protection mechanism."
According to him, alongside caution, it's also important not to stop using the tools. "You need to use these tools, understand them and experiment with them. This is a technology that's a huge force multiplier. But you shouldn't get confused just because these systems are so smart. You ask ChatGPT a question and it knows a huge number of things, and then it's easy to think it can't be wrong, lie, or confuse you. It can. AI should be a tool for thinking, not a substitute for thinking."
And what about integrations with tools like email, Slack or Drive?
Here the recommendation is to actually apply the principle of minimum permissions. If the agent needs to read emails, for example, there's no need to also give it the ability to send them. If it needs to work on a particular document, there's no reason to open up your entire Drive to it.
"In your personal Gmail there can be details from the bank, from the insurance company, personal information and a great many other things," Karako says. "The moment you give the agent access to this information, theoretically it can use it while carrying out a task even when you didn't think it would be relevant."
Bargury suggests creating separation as much as possible. "For example, give the agent its own email account and send it only the messages relevant to the task. Or create a separate Drive account for it and share only a specific document with it under edit permission. That way it can do the work, and once it's finished you can remove the permission."
And what about control of the computer or browser?
The same principle holds here too. "If you have a browser where you're logged into your bank, your email, your social networks and all your important accounts, I don't recommend letting the agent act freely inside that browser," Bargury says. "It's better to open a separate browser environment and connect only the accounts or systems needed for the task."
He warns in particular against combining internet access with permissions to act in sensitive systems. "If the agent has the ability to perform actions in the world, you need to be especially careful about giving it, at the same time, content that comes from external and untrusted sources. If it can send an email in your name and in that same conversation you let it read some random blog on the internet, whoever wrote the blog could plant instructions in it that try to influence the agent."
In such cases, Bargury says, it's sometimes better to separate the tasks: one tool will do the research on the internet without access to the organizational system, and another tool will work inside the system without free access to the network.
Karako adds that "even when giving the agent access to the computer, it's better to define a dedicated working folder for it rather than opening up all the files on the computer to it."
What information is it best not to share with AI?
In general, it's not recommended to give AI passwords and other sensitive details, but beyond that, the answer isn't clear-cut. "In the end there's a constant tension here between privacy and convenience, and every user has to decide where they set the boundary," Karako says.
"If you're using a regular tool for a sensitive conversation, for example on the topic of health, you can at least reduce the exposure: turn off tools you don't need, don't allow internet search, don't give it the ability to write the sensitive file to other locations, and turn off memory," Bargury adds.
