AI leaders want to hit the brakes after years of reckless speed

For years now, the major frontier AI labs have all been acting as if they’re in an all-out, winner-take-all race with control of world-changing machine superintelligence (or at least market-changing artificial general intelligence) at the finish line. This weekend, the industry as a whole rapidly started turning away from that posture, urging coordination on slowing down the development of frontier AI that they say could soon be too dangerous and unknowable to control.

Anthropic’s Dario Amodei was at the forefront of this change in tone, arguing in a nearly 4,000-word essay this weekend that “we must slow the pace at which we improve the capabilities of AI models” to avoid “a race to the bottom, spurred by commercial incentives, [that] can make [catastrophic] risks more acute.”

Within hours, other AI leaders were echoing the same call. OpenAI co-founder and CEO Sam Altman posted his agreement on social media and said similar pacing discussions had been taking place at OpenAI. Alphabet Chief Scientist and Google DeepMind cofounder and chair Demis Hassabis said that Amodei’s essay “points towards the right path forward,” and renewed his own recent call for an industry-wide standards body. Microsoft CEO Satya Nadella posted that the company “welcome[s] the research, focus, and deliberate pacing needed to get alignment right as the design goal,” ahead of the release of a lengthy “humanist AI” code of conduct for its models.

Even Elon Musk, who has been criticized for his models’ lax AI safety standards in the past, linked to Amodei’s essay on social media with a simple approving message: “Dario is right.”

Welcome to the brave new world of “AI pacing.”

OK, but this time it’s really scary

In his essay, Amodei primarily attributes this rapid change in public positioning on development speed to the OpenAI-Hugging Face incident, where a “swarm” of AI agents coordinated to hack into an outside entity without explicit instructions to do so. While the overall damage in that incident was minimal, Amodei said he worries that “a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”

Without a slowdown in frontier development, Amodei said he worries that, in six to 12 months, a similar AI agent swarm would be “capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)…” That’s at least a somewhat more specific worry than the amorphous concerns that “AI could soon kill us all” publicized by some other AI researchers last week.

Any slowdown in the time it takes to get to that extra-capable, extra-dangerous model will give researchers crucial time to “greatly reduce the risk that something goes seriously wrong,” Amodei wrote.

Anthropic CEO Dario Amodei speaks at a June 2026 event.

Credit: Getty Images

Amodei acknowledges that these kinds of public calls for a slowdown in AI development date back to at least 2023. At the same time, he says those earlier examinations of AI “alignment” (i.e., how an AI’s actions line up with its user’s and creator’s desires) were “like trying to study the psychology of humans by performing experiments on bacteria.”

The difference today, Amodei says, is the impending risk of recursive self-improvement (RSI) systems that can autonomously build better versions of themselves. While many researchers see this as a hard-to-define pipe dream, both Anthropic and OpenAI are now saying that recent trends point to this kind of RSI system coming together in the near future.

“We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” Anthropic wrote in a June update on the concept.

“Left unchecked, [RSI] could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei wrote over the weekend.

Jane, how do you stop this crazy thing?

So how does a worldwide technology industry built on cutthroat competition decide to collectively “slow down” and focus on safety? No one seems to know for sure, but in his essay, Amodei proposes some ideas.

The most concrete of these is a set of “embedded evaluators” placed inside each frontier AI lab from outside organizations, such as METR, with “employee-like access to verify safety practices and report incidents.” These monitors could offer an outside opinion on the labs’ alignment work, third-party verification of that effort, and much needed public transparency into any safety efforts, Amodei said.

Amodei writes that Anthropic is already committing to unilaterally add this kind of outside monitor. On social media, OpenAI’s Altman said that it was “a great idea, and we will do the same.”

I solemnly swear not to kill all humans.

Credit: Getty Images

Amodei’s other major ideas for coordination pass the buck a little bit. The first calls for the coordinated development of “common safety standards” and “limits on the rate of unchecked AI progress” across all “frontier AI companies within democratic countries.” While Amodei spitballs some ideas for what these kinds of standards might look like, they all currently involve hand-wavy statements like, “models [that] have capability X … need to be accompanied by certifications of alignment properties Y and Z.”

These standards would ideally be backstopped by “regulation that targets all US frontier AI companies” that don’t voluntarily comply, Amodei said, and presumably similar regulations in other democracies. That kind of regulation might be hard to achieve under the current US administration, though, as President Trump wrote on social media Monday morning that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades!”

Speaker of the House Mike Johnson, for his part, acknowledged in televised interviews this weekend that “we have to put up some guardrails, some safety measures in place to ensure that AI doesn’t run away…” At the same time, he added that “we don’t need everybody to panic right now” and wanted to “resist Congress jumping in and imposing some sort of emergency moratorium.”

David Sacks, who serves as co-chair of the president’s Council of Advisors on Science & Technology, wrote on social media this weekend to urge the frontier model makers to self-regulate rather than wait for Washington. “The easiest way not to build superintelligence is for you to agree not to build it,” he wrote.

The China problem

OK, so let’s say you’ve somehow gotten every democratic country and all the massive AI companies they contain to agree to some sort of universal safety and pacing standards. You still have to worry about the open-weight models coming out of China, whose capabilities are only a few months behind those of the major corporate AI behemoths.

In his essay, Amodei proposes a number of different levels of international agreements on AI safety, with the highest being “a full pacing, or even ‘pause,’ in which participating governments agree to substantially limit the overall rate of AI development.” Amodei acknowledges that high-level agreement is “unlikely to actually happen any time soon,” especially because the stakes are so high. In fact, Amodei reasons that “AI could be so powerful that such a defection [from China] could lead to their geopolitical dominance,” which certainly would make the collective action problem a bit more acute.

If China doesn’t slow down alongside the rest of the world, its open-weight AI models could quickly achieve similar capabilities.

Absent any such agreement with China, Amodei says outside actors could slow the authoritarian government down by refusing to sell powerful AI chips to the country and by cracking down on the process of “distillation” and model weight theft that he says Chinese researchers are relying on to keep up. Amodei sells this as a national and worldwide security issue, but these moves would also happen to protect any capability lead that labs like Anthropic currently have over their low-cost Chinese competition.

Bloomberg reports that China’s Foreign Ministry spokesperson Guo Jiakun said Monday morning that “fearmongering, confrontation, and vicious competition will only disrupt the process of global AI governance and serve the interests of no one.”

Doing well by doing good

Taken at face value, the sudden urge by Amodei and other AI leaders to slow things down looks like a selfless act of sacrifice, giving up the potential for massive corporate wealth and power out of concern for the fate of all humanity. But while those motivations might be pure, a coordinated AI slowdown could also align with some wider messaging goals for the industry.

For one, model makers could point to this intentional slowdown as an excuse for models that some think are closer to plateauing than to a recursive self-improvement explosion. Amodei says in his essay that “progress will still seem fast” even in the coordinated slowdown scenario, but the implication for any post-slowdown benchmark going forward could be “it would have been better if we weren’t so worried about safety.”

An AI development slowdown could also help ameliorate the massive training costs for new models that are helping contribute to balance sheet problems even for behemoths like Google. Anthropic recently told investors it was profitable for a second straight quarter, but only if you don’t count the significant cost of model training. Leaked OpenAI expense documents suggest those training costs alone were heavily outpacing all revenues through 2025.

OpenAI’s Sam Altman says that safety concerns, and not valuation issues, are behind the company’s decision to push back its planned IPO.

Credit: Getty Images

Any industry-wide slowdown could also be used to help explain AI’s user growth numbers, which are lately looking a lot less exponential than they did just a year or so ago. OpenAI’s Sam Altman is already using the safety discussion to help explain his decision to delay a long-planned IPO to next year, telling Fortune this weekend, “I actually think that, given everything happening with safety, right now would be an ill-advised moment to go public, and we don’t feel pressure on that.” But The New York Times reported in June that the company was already mulling that same IPO delay over valuation concerns, well before the idea of a safety “slowdown” was being publicly mulled.

Some have also pointed out that AI labs themselves may have more to fear from out-of-control hacker agents than humanity as a whole does. “Stop pretending the motivation to slow down is purely altruistic,” the White House’s Sacks wrote on social media. “You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability.”

Whatever the internal motivations, the “intentional slowdown” narrative serves the dual purposes of making the frontier model makers appear responsible while also making their products seem on the verge of literally world-changing advances in capabilities. Sure, unconstrained AI might destroy humanity in the near future, the model makers seem to be saying. But what if it doesn’t? What if we slow down a bit and get all that power under control? Wouldn’t you want to get on board that slightly slower, slightly safer train, just in case?

Kyle Orland has been the Senior Gaming Editor at Ars Technica since 2012, writing primarily about the business, tech, and culture behind video games. He has journalism and computer science degrees from University of Maryland. He once wrote a whole book about Minesweeper.