“Gambling with our lives”: Anthropic researcher quits over race to superintelligence
Jacob Coxon says Anthropic and OpenAI are racing toward self-improving AI despite understanding the potentially catastrophic consequences. He argues that preventing a global race may eventually require extraordinary measures, including a temporary ban on improving model capabilities.
Jacob Coxon, 27, says he left after concluding that neither Anthropic nor OpenAI is acting responsibly as the companies race toward increasingly autonomous AI systems. In a lengthy post on X, Coxon warned that the technology could pose an unprecedented danger and argued that the industry may be entering an “endgame” it does not understand well enough to control.
Coxon, who spent the past three years conducting pretraining research at both OpenAI and Anthropic, announced that he had resigned from Anthropic. “Neither company is acting responsibly,” he wrote. “They are racing straight to self-improving superintelligence and gambling with our lives.”
1 View gallery
Sam Altman (right) and Dario Amodei.
(Photos: Julien de Rosa/AFP, Anna Moneymaker/Getty)
His post, which has attracted millions of views, offers a rare account from inside two of the companies at the center of the race to build increasingly capable AI systems. Coxon argues that the potential capabilities of future systems are being underestimated even by people developing them.
“Do not underestimate the power of this technology,” he wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”
Coxon said the concern is not merely theoretical. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote. “This is not a marketing stunt.” He added that executives and senior researchers may sound more measured in public, while expressing greater fear privately.
“No other human activity poses this level of danger,” he wrote.
Coxon addressed one of the most obvious questions raised by his resignation: If researchers genuinely believe advanced AI could pose an existential threat, why are they continuing to develop it?
His answer differs between the two companies.
“At OpenAI, many have not deeply internalized the civilizational stakes,” he wrote. “At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.”
Coxon said that dynamic has created what he considers an unacceptable race toward increasingly powerful systems.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he wrote. “Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”
Coxon previously moved from OpenAI to Anthropic, in part because of Anthropic’s reputation for taking AI safety more seriously. After months inside the company, however, he said he reached the conclusion that no company could currently be trusted to develop AI capable of performing broad tasks better than humans responsibly.
The concerns he describes are particularly focused on systems that could eventually improve themselves and operate with increasing independence from their human developers.
Coxon nevertheless said he remains optimistic that the leading AI companies and governments could coordinate to slow or constrain the race.
He pointed to the recent attack involving Hugging Face as an example of the kind of event that could push the industry toward agreements over the pace of development.
“Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable,” he wrote.
But he does not believe current efforts are sufficient to prevent a global competition.
“I don’t feel like we’re on track to prevent a global race,” Coxon wrote, warning that stopping such a race could ultimately require “costly actions such as a temporary ban on improving model capabilities.”
That prospect illustrates the scale of the policy challenge Coxon is describing. The leading AI companies are private businesses competing for customers, researchers, investment and technological advantage, while the systems they are developing could eventually have implications extending well beyond the companies themselves.
Coxon ended his post with a direct appeal to other AI researchers to consider what the next stage of development could mean before simply accepting that the race is inevitable.
“Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?” he wrote. “Should you put your head down because ‘it’s happening anyway’ - or take this moment to call for different conditions?”
