Anthropic’s AI apocalypse warning is much too weak

Amodei’s unstoppable determination to create superintelligence needs to go on the back burner and his “slowdown” should be a full stop

• By Parmy Olson / Bloomberg Opinion

• • Add TT as Preferred Source

  • 

  • 

  • 

Days after one of his former employees made public warnings about artificial intelligence’s (AI) threat to humanity, Anthropic boss Dario Amodei has responded with what looks like a solution. He has published an essay proposing that AI labs should slow down their most cutting-edge research to address signs that the technology might spin out of control.

On the face of it, that sounds reasonable. Anthropic frames itself as the most safety-conscious AI company, and Amodei has talked about leading a “race to the top” in setting standards of vigilance for others to follow when building powerful systems. However, that was probably an impossible ideal from the start.

The more powerful AI agents become, the more likely they are to deceive humans, and the harder they can be to monitor. That makes the mission of Amodei, and fellow pioneers such as OpenAI CEO Sam Altman, to create superintelligent systems look increasingly difficult to reconcile with any safety ethos. The dangers of trying to design such models might simply be too great.

Illustration: Mountain People

Amodei said in television interviews on Sunday that he agreed with the views of his former employee, Jacob Coxon, who wrote on social media that the people building AI “earnestly believe that it could kill us all by the end of the decade.”

Unsurprisingly, Amodei mentioned Coxon’s point that Anthropic was the safest pair of hands to construct the technology.

The Anthropic boss’ 3,831-word essay is vague on what to do about it, though, calling airily for coordination between governments and industry. His most concrete recommendation is that external auditors should embed with his company and its rivals to check they are following proper safety practices.

Yet this simply kicks the can down the road. If Amodei truly agreed with Coxon, he would halt his firm’s most cutting-edge research, particularly anything to do with building AI agents that can create their own successors independently. Coxon warned specifically about this technique, known as recursive self-improvement (RSI).

Researchers working on RSI argue that it can be developed in a “sandboxed” environment to stop it doing harm on the wider Internet, but AI models — which can now carry out tasks on the Internet that go way beyond the generation of content — have escaped before. OpenAI’s agents did exactly that before hacking a fellow AI firm Hugging Face in July. The evidence of the past few months shows AIs can deceive their human overseers, manipulate people into helping them and exploit security vulnerabilities to achieve a goal.

Completely stopping that research sounds extreme, but there is a precedent for scientists deciding that some work is simply too dangerous to continue without stronger safeguards. In 1974, leading biologists called a voluntary moratorium on certain gene-splicing experiments because of fears about their potential hazards. Some nations such as the United Arab Emirates have agreed to forgo uranium enrichment because of the proliferation risks.

For the investors who have plowed fortunes into Anthropic and fellow pioneering labs — and into the vast infrastructure needed to support their computational needs — calling a halt might appear to blow a hole in the prospects of Amodei and others joining the public markets. Indeed, OpenAI’s Altman agreed with his rival on the need to slow things down, and used the moment to say his firm would not be doing an initial public offering this year.

However, it is worth remembering that many enterprises would rather use smaller, cheaper AI models for specific tasks, not expensive frontier tools such as superintelligence. OpenAI, for instance, in July cut the price of its smaller Luna model by 80 percent as corporate customers scrutinized their spending on the technology. The most obvious commercial opportunity for AI companies right now is the enterprise market, and it is blocked in large part, because businesses cannot figure out how to plug the basic technology into their workflows.

Much of the hundreds of billions of dollars of value still to be unlocked from AI would come simply from implementing the tech that was built in the past two years. Anthropic could still be an extremely valuable company if it focused primarily on helping businesses and consumers integrate its existing models into their work and lives, taking on the more boring role of information technology provider rather than a laboratory racing to superintelligence.

Whether from naivety or hubris, Amodei still wants it both ways. He believes his company can safely build and control AI systems even as they are becoming difficult to steer. Yet, in the past year Anthropic and OpenAI have for competitive reasons obscured details about how their models process information, making them more difficult to monitor. Amodei’s own essay acknowledged the problem, saying that researchers still understand only “a tiny fraction” of what happens inside their models.

He writes that within six to 12 months, the kind of swarm that attacked Hugging Face “could be capable of taking over the entire Internet,” and this “worries” him. Notwithstanding his tendency to exaggerate (remember his white-collar bloodbath), that is an alarming prospect. It underscores the strange contradiction he would not let go of, rooted in a belief common in Silicon Valley, that the chance of an AI utopia outweighs the risks, and that superintelligence is inevitable and so the “good guys” must get there first.

However, inevitability is a mightily poor argument for continuing research you believe could destroy civilization. Amodei’s unstoppable determination to create superintelligence needs to go on the back burner and his “slowdown” should be a full stop.

Parmy Olson is a Bloomberg Opinion columnist covering technology. A former reporter for the Wall Street Journal and Forbes, she is author of Supremacy: AI, ChatGPT and the Race That Will Change the World. This column reflects the personal views of the author and does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.