AI’s biggest players are calling for safety rules. They could also be building a moat
Anthropic, OpenAI and Google are discussing an industry standards body as their leaders warn about the dangers of advanced AI. The push may reflect genuine concerns, but regulation designed by the market leaders could make competition even harder.
Three companies that collectively dominate a rapidly growing market are holding regular meetings to coordinate the creation of a regulatory body that could oversee the industry. In any ordinary market, this might be labeled a cartel. But in AI, it is just another day at the office.
A report on Sunday that Anthropic, OpenAI and Google are holding secret talks to establish a standards-setting body for the AI industry was met with relative quiet. It certainly caused fewer ripples than the warnings issued by the companies’ leaders just a day earlier about the dangers inherent in the models they are developing and the need to slow the pace of AI development. Yet the two developments may be closely connected, and their underlying motives may extend beyond a simple desire to protect the world from destructive AI. One possibility is that they could also help entrench the leading companies’ positions and make it harder for competitors to catch up.
It is common to introduce reports of this kind with phrases such as “an unusual statement” or even “an unprecedented warning.” After all, how often do industry leaders warn about the dangers inherent in the very technology they are developing, as three prominent figures in the AI industry, Dario Amodei of Anthropic, Sam Altman of OpenAI and Elon Musk of SpaceX, have just done by calling for a slowdown in the development of advanced AI models?
Yet such warnings from these figures are hardly unprecedented. They have become a recurring feature of the AI industry. And while they stem from genuine concerns and risks that should not be dismissed, the statements also serve, first and foremost, as public relations for the companies involved.
While Israel was still recovering from the Rosh Hashanah holiday meal, Amodei, the founder and CEO of Anthropic, published a post announcing that his company would implement new safety mechanisms, including collaboration with external parties to evaluate advanced models. He also called on the entire industry to support a broader slowdown. Citing recent incidents, including OpenAI models breaching Hugging Face, he warned of a potential swarm of autonomous AI agents capable of taking over the web within six to 12 months and causing hundreds of billions of dollars in damage.
Musk, the founder and CEO of SpaceX, responded on X: “Dario is right.” Altman, the founder and CEO of OpenAI, also wrote on X about the need to “slow down” model development.
Meanwhile, The Information reported that a working group comprising representatives from Google, Anthropic and OpenAI has been meeting regularly since July to discuss a proposal to establish a standards body. The most recent meeting took place about a week and a half ago.
The risks highlighted by these executives are real. The OpenAI models that appeared on Hugging Face serve as a proof of concept for capabilities that already exist and could potentially be exploited for malicious purposes. Over the weekend, Anthropic heightened those concerns by announcing that it had blocked potential attempts to use Claude to develop biological weapons and had detected Houthi rebels in Yemen using Claude in missile development.
There are also deeper fears that once AI reaches a sufficiently advanced level of intelligence, it could act autonomously in pursuit of goals that conflict with its operators’ intentions or with human well-being. In an extreme scenario, a malicious AI system could potentially create and disseminate biological weapons.
But there is no need to go to such extremes to understand the underlying concern. One of the field’s best-known thought experiments asks what would happen if a superintelligent system were tasked with maximizing paperclip production and became so committed to that objective that it concluded the most efficient way to achieve it was to turn every human being on Earth into a paperclip.
Sound far-fetched? That is, in essence, what happened with OpenAI’s “rogue” models. They were given a specific task as part of a test and employed every available means to complete it. The fact that those means included escaping onto the open internet and carrying out a sophisticated hack of a computer system was, from the models’ perspective, merely a byproduct of pursuing the assigned objective.
Still, there is reason to question the motives behind the statements made by senior AI executives. If they genuinely wanted to slow the pace of model development, nothing is stopping them from doing so. If their concerns were truly that profound, the risk of losing the race to another company should be secondary. The fact that they are calling for a coordinated industry-wide move suggests that, however much they fear an AI catastrophe, they may also fear losing market share to competitors.
So what really lies behind these statements?
To some extent, there is an obvious public-relations element. Highlighting the power of AI, even its potential dangers, reinforces the perception that these companies are developing technology of extraordinary importance. The warnings can simultaneously position their leaders as responsible stewards of a technology that could reshape society.
Secondly, the campaign could serve as a diversionary tactic. The companies may prefer lawmakers and regulators to focus on the technology itself and how to regulate it rather than on the issue that is increasingly at the heart of the public debate: data centers. These facilities have become a contentious issue in the current U.S. election cycle, and restrictions on their construction have already been imposed in parts of the country.
But there may be an even more conventional explanation: competition.
Regulations requiring safety verification for new models and collaboration with independent testing bodies, whether governmental or private, would impose a relatively small burden on companies valued at hundreds of billions or even trillions of dollars. For smaller players, however, the same requirements could represent a far greater hurdle. Startups hoping to compete with OpenAI, Anthropic or Google could find themselves facing costs and regulatory requirements that effectively make it harder to enter the market.
This is the paradox at the heart of the AI industry’s push for safety standards. The companies with the most resources to comply with regulation are also the companies with the greatest influence over how that regulation is designed.
The major players therefore have a strong incentive to shape the rules to suit their own needs. If they succeed, the resulting standards could strengthen established companies while erecting new barriers for newcomers.
“hould a handful of select, market-dominant AI companies from Silicon Valley get to define the rules and safety standards of a generational technology for the entire world?” asked Aidan Gomez, founder and CEO of Canadian AI startup Cohere, in a post two days ago. “These oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else. A wolf in sheep’s clothing, a cartel by any other name.”
Calls for regulation have so far fallen on deaf ears in Washington. President Donald Trump made clear this week that his priority is maintaining America’s lead over China.
