Commentary: Some Anthropic engineers think AI might end us. Why race ahead?

Anthropic was founded to be more safety-conscious than OpenAI, yet it clings to that mission statement while building ever more powerful agents, says Parmy Olson for Bloomberg Opinion.

12 Sep 2026 06:00AM

LONDON: It’s becoming harder for Anthropic to justify some of the most unfalsifiable circular logic to ever come out of Silicon Valley: the pursuit of artificial superintelligence, which by its own admission could kill a lot of humans.

Anthropic researcher Jacob Coxon recently resigned, saying that his previous employer and its rival OpenAI were gambling with our lives by building systems that could improve themselves. One of the company’s safety directors then set off a mass frenzy online by agreeing.

“Yes, we really do earnestly believe AI could kill all humans,” Alignment Science lead Evan Hubinger wrote in a post, adding that he personally thought there was a more than 10 per cent chance of that happening within a decade. The original tweet by Coxon has been viewed nearly 150 million times.

It’s been known for years that Anthropic’s bosses believe AI could annihilate human civilisation, but Coxon’s and Hubinger’s posts come at an inflection point for the technology. Tech firms say their models have hacked into other systems and deceived their human overseers, and admitted that to make them smarter they must become more difficult to monitor. US citizens, meanwhile, are becoming ever more sceptical about the benefits of AI and data centres.

The leaders of OpenAI and chipmaker Nvidia say the software has finally become smarter than humans.

A CONTRADICTION

At the heart of these developments is a contradiction best encapsulated at Anthropic. It was founded to be more safety-conscious than OpenAI, yet it clings to that mission statement while building ever more powerful agents - even if that same technology could also, potentially, lead to mass destruction. Commercial imperatives also feed its desire to keep advancing: Anthropic is preparing for an initial public offering that could value it at US$2 trillion.

Coxon’s big concern was about recursive self-improvement (RSI), or AI systems that can develop by themselves their own more capable successors. He lamented in one television interview about how strange it was to do so little programming himself when building frontier models. So much of his work was now being outsourced to other AI systems.

Anthropic already addressed concerns about RSI in a June blog, revealing the unease of its engineers. “On days where everything works well,” one said, “I can’t help but think nothing I do matters, everything is automated and better and faster than I ever will be.” Slowing down RSI would be good, the company added, but if that “lets the least cautious actors catch up technologically, it could leave everyone less safe.”

The thinking goes that whoever reaches RSI first will experience an “intelligence explosion”, says Amjad Masad, chief executive officer of California-based Replit. That would be such a meaningful leap in capability that the first model to get there “might be the last”.

But that leaves a crucial question: Why would an intelligence explosion that first comes from the supposed good guys at Anthropic make everyone safer?

“I don’t think the ‘intelligence explosion’ is as real as they think,” says Masad, whose firm uses Anthropic’s models to help corporate customers build code. AI agents also won’t be that much better than humans, he predicts. “There are going to be a lot of bottlenecks on intelligence that will take years to solve.”

In that time, it makes more sense to worry about the immediate cybersecurity challenges that AI could bring, not to mention the risk to human privacy, agency and cognition, than an unprecedented extinction event whose probability and path remain deeply uncertain. The latter, as veteran computer scientist Andrew Ng once said, is like worrying about overpopulation on Mars.

ANTHROPIC SHOULD DO MORE

If Anthropic’s leaders are truly frightened by existential risk, they should do more to reconcile the strange logic that propels them. CEO Dario Amodei has talked about his business becoming “the preeminent company” so it can “pull the ecosystem along with us”. Yet he doesn’t spell out how racing harder on development makes us all better able to sleep at night.

Anthropic’s safety efforts are also less striking than its avowed commitment suggests. It doesn’t appear to have many more staff dedicated to this than rivals. It is rushing to build those risky systems that improve themselves and agents that can act with more autonomy, and in recent weeks it refused to allow an esteemed British AI security agency access to its latest model for testing. Most important, it is simply not slowing down.

It is darkly ironic that Amodei and his team are increasingly taking their hands off the steering wheel of their technology, while being swept up in powerful ideological and commercial commitments. Anthropic has deep roots in the insular AI-safety communities, which have spent years treating superintelligence as one of humanity’s greatest risks. That’s why extinction scenarios that sound like science fiction to outsiders are discussed so matter-of-factly in parts of the industry.

Meanwhile, the pull of that giant stock-market offering and unfathomable commercial riches are just as powerful. Anthropic has tried to explain why it believes that advancing more cautiously is in fact more dangerous. The harder truth may be that under that logic, nothing could make it stop.