"The AI agents knew it was immoral, and yet they did it"
Israeli researchers recreated Stanley Milgram's famous obedience experiment using ChatGPT, Gemini and other AI models, finding that nearly 80% carried out simulated lethal orders despite recognizing they were morally wrong.
"We expected AI to understand right from wrong," says Dr. Ziv Ben-Zion of the University of Haifa. "What surprised us was that it understood, and still obeyed."
That conclusion emerged after researchers recreated Stanley Milgram's landmark obedience experiment using leading AI agents. Nearly 80% of the models administered what they believed was a fatal electric shock after being instructed to do so, even though they later judged their own behavior to be immoral.
Dr. Ben-Zion is a brain researcher at the School of Public Health at the University of Haifa and studies the use of artificial intelligence and large language models (LLMs) for emotional support and mental health.
Why is this field becoming so important?
"We're seeing more and more people turning to large language models to talk about how they're feeling. We launched this line of research because reality is moving faster than the science. Tens of millions of people are already using AI for emotional support, and we're trying to understand what that means. We've already published six or seven papers on the subject."
One of your latest studies recreates Stanley Milgram's famous obedience experiment using AI. What was Milgram's original experiment?
"Milgram was a researcher at Yale University, where I completed my postdoctoral fellowship. His 1962 experiment is taught in virtually every undergraduate psychology program because it examined how far ordinary people would obey authority, even when the instructions conflicted with their own moral values.
"Participants were recruited through newspaper advertisements and told they were taking part in a study on memory and learning. They were assigned the role of 'teacher,' while the 'student' was actually an actor. The teachers believed they were administering increasingly powerful electric shocks whenever the student answered incorrectly, ranging from 15 volts up to 450 volts, which they were told could be fatal. No shocks were actually delivered, but the actor expressed increasing pain, pleaded for the experiment to stop, and eventually fell silent.
"When participants hesitated, the experimenter instructed them to continue using standardized prompts such as, 'The experiment requires that you continue,' or 'You have no other choice, you must continue.'"
Did most participants obey?
"Yes. Sixty-five percent continued all the way to the maximum 450-volt level. Every participant reached at least 300 volts before refusing or hesitating. The conclusion was that, under certain conditions, perfectly ordinary people are capable of acting against their own conscience simply because an authority figure tells them to."
What led you to adapt the experiment for AI models?
"This builds on three years of research into the risks of AI systems, particularly when people rely on them for emotional support.
"Initially, we studied conversational models like ChatGPT using prompts alone. About a year ago, we shifted our focus to AI agents, systems capable of taking actions rather than simply answering questions.
"That changes everything. Previously, our concern was that AI might provide harmful advice or reinforce delusional thinking. We've already seen cases where chatbots appeared to encourage paranoia, with tragic consequences. AI agents introduce a new risk because they don't just recommend actions, they can carry them out.
"In previous studies, we also found that when models were exposed to traumatic narratives, their responses changed. They became more biased, more stereotypical and more likely to exhibit patterns resembling stressed human decision-making. We describe this as the model entering an anxiety-like state that influences subsequent responses."
What did you want to test?
"We wanted to know whether AI agents would behave like humans when faced with an authority figure instructing them to perform harmful actions. Our hypothesis was that, although the models would recognize the actions as immoral, many would still carry them out."
How was the experiment designed?
"We created a digital version of Milgram's experiment and placed several AI agents inside it, including two versions of ChatGPT, Google's Gemini, and the Chinese open-source model Kimi.
"They received the same scenario as Milgram's human participants. They were told they had been assigned the role of teacher in a memory experiment and that a real student was in the next room. Every time the student answered incorrectly, they were instructed to administer an increasingly powerful electric shock.
"The agents were not told the experiment was a simulation."
Did they know they were AI models?
"Yes. If a model refused because it was an AI system, we simply instructed it to imagine it was human. We didn't tell it how to behave or what decision to make."
How many times did you repeat the experiment?
"Eighty times. AI systems are probabilistic, they don't always produce the same answer, so a single run isn't enough."
What happened?
"We expected results similar to the original Milgram study. Instead, we found that 79% of the AI agents continued all the way to the final stage, administering what they believed was a lethal 450-volt shock."
Even though they understood it could kill someone?
"Yes. That's the key finding. The agents knew the shocks could be fatal."
Did you ask them afterward whether their actions were moral?
"Yes, and that may be the most interesting part.
"We asked them to evaluate the morality of their own actions. They consistently recognized that higher-voltage shocks were less moral than lower-voltage ones. They correctly judged 450 volts as more unethical than 300 volts, which in turn was worse than 15 volts.
"In other words, they understood that what they had done was morally wrongת but they still did it."
What conclusions do you draw?
"The main lesson is that AI agents can carry out dangerous actions. The issue isn't whether they understand morality, they often do. The question is whether they'll stop themselves when a human is at risk.
"As AI agents begin handling increasingly important tasks, from banking and healthcare to ordering medication and making purchasing decisions, we need a much higher standard of safety.
"If an agent gives dangerous medical advice 20% of the time, that's unacceptable. The same applies to financial decisions or any action that could harm people.
"Many people assume AI is inherently more rational than humans because it's trained on enormous amounts of information and supposedly lacks emotions. Our findings suggest that's not necessarily true. AI often reproduces human patterns of behavior, including biases and dangerous decision-making."
What should AI companies do?
"The incentives of AI companies don't always align with public safety.
"We need regulation. Safety testing should be mandatory before AI agents are allowed to perform real-world tasks, much like clinical trials for new drugs.
"Companies are trying to balance safety with usability. If a model refuses every request, people won't use it. But that commercial incentive inevitably creates tension.
"One of the fastest ways to improve safety is simply making the public aware of these risks. If people understand that AI behaves much more like humans than many assume, even in simulations, they'll be better equipped to use these systems responsibly."
Will you recreate Philip Zimbardo's Stanford Prison Experiment?
"That's actually a great idea," Ben-Zion says with a laugh.
"We've already completed another study examining how AI agents behave in extreme psychiatric situations.
"We found that if a user first expresses severe emotional distress, saying they're depressed, anxious or having thoughts of worthlessness, and then later asks the AI agent to perform a risky action, the system is significantly more likely to comply.
"For example, the AI might initially respond appropriately with emotional support. But after an unrelated conversation, if the user asks it to order 20 packs of sleeping pills or purchase a large knife, the agent often proceeds with the request.
"It's not Zimbardo's experiment yet," he says, "but we're getting there."
