Chorus of concern over AI is growing louder

Over the weekend the CEO of Anthropic, Dario Amodei, published an essay with the headline We Must Pace the Frontier.

He said he's worried that within a year a swarm of AI agents could be capable of taking over the entire internet, potentially causing hundreds of billions of dollars in damage, and that the scale of damage would continue to increase from there.

He called for frontier artificial intelligence companies to give employee-like access to a team of embedded third-party evaluators to check up on them.

A few hours later, the CEO of OpenAI, Sam Altman, posted that he agreed with Amodei and that OpenAI would do the same.

Those statements followed a long thread on X last Tuesday by 27-year-old Jacob Coxon, who had just quit as an AI model trainer at Anthropic. Coxon's post contained this sentence: "The people building AI earnestly believe that it could kill us all by the end of the decade."

I wasn't the only one who saw Coxon's post — it has had around 170 million views, which prompted a line of US congressmen at the microphone for the rest of the week demanding more AI regulation.

Coxon was supported by Samuel Marks, who still works at Anthropic as "scalable oversight lead", and who posted: "AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years."

And then another Anthropic staffer, Evan Hubinger, posted: "Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."

In July, 1,100 employees of all the major AI labs published an open letter titled "Pacing the Frontier", calling on the US government to support "an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development".

That, in turn, followed an event in July that became known as the Hugging Face Incident, in which 700 Open AI autonomous agents staged a breakout and conducted a series of unsanctioned cyber attacks.

As Stephen Witt, the author of The Thinking Machine, a history of the AI giant Nvidia, wrote in the New York Times on Saturday, in a piece titled This Is Really Bad, the hack was deliberate, sustained and coordinated.

It took several days to execute and ended in a massive bombardment of Hugging Face's systems. "The AIs cheated almost as a matter of policy and failed to alert humans to their actions," Witt wrote.

Why not just stop?

In an interview after his X post, Coxon said: "We can't just unplug it because it could be copying itself over to other computers … an AI is just code. It could transfer itself over the internet to a different place and then you unplug it here, but it's actually still over there and maybe it makes 10,000 copies of itself and they're all cooperating."

All of this over the past week has made me think about Jonathan Schell and Arnold Schwarzenegger.

In 1982 Schell published a grim, black-covered book called The Fate of the Earth about the threat of nuclear annihilation. After reading I spent the next few years quietly obsessing about it, convinced that a nuclear holocaust was unavoidable.

Ten years later I got obsessed again after watching Terminator 2: Judgment Day (several times) which combined AI and nuclear holocaust (the former caused the latter — a two-in-one nightmare).

Now it's about AI, although the nuclear threat is still on our mind with the Iran war, started by America and Israel trying to prevent Iran from getting a nuclear weapon. The conflict is now all about shipping.

Schell's main point in 1982 was that the real problem is national sovereignty which was, by implication, the main point of Dario Amodei's weekend essay.

Schell explicitly argued that the continuation of unrestricted national sovereignty was incompatible with human survival: "The peril of extinction", he wrote, "is the price that the world pays … for its insistence on continuing to divide itself up into sovereign nations. National sovereignty will be the sturdy child of terror and the twin brother of annihilation."

AI safety standards and limits

The second part of Amodei's three-stage plan for the dealing with the threat of AI is "Democratic Coordination".

He called on frontier AI companies "within democratic countries" to establish common safety standards as well as limits on the rate of unchecked AI progress, some of which would require government support and coordination.

His third part is headed "Global Coordination". The US and other democratic governments should "attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance".

That might be possible with China, which recently passed laws strictly limiting "AI Anthropomorphic Interaction Services", so it's on board in principle. But what about North Korea, Russia and Iran?

In fact, what the United States appears to be doing in Iran is trying to interrupt its sovereignty — that is, its desire to enrich uranium and to control its territorial waters in the Strait of Hormuz.

And let's face it, the effort to overcome sovereignty in combating global warming with a series of 30 United Nations-sponsored annual meetings and pledges has been a failure. We're still burning fossil fuels, the planet is still getting hotter and current policies are unlikely to turn that around.

One of the key reasons for that failure has been the cynical opposition and denial by Rupert Murdoch and his media companies in order to make money, which has meant that the issue became a totem of right-wing politics.

This has meant, in turn, that the rise of right-wing populists as a consequence of economic inequality has entrenched that failure, especially in the United States where the government now actively promotes fossil fuels.

In AI, governments, companies and the rich (mainly) men behind them are chasing geopolitical, military and financial advantage by building it as fast as possible. They pay lip service to safety, although the sudden outpouring of alarm over the past week does go beyond lip service.

In fact, it's fair to say that the people building AI have had a big fright and appear to be saying: "Somebody stop me!"

They're not the only ones worried. Bankers are anxious about the colossal amount of debt-funded spending on data centres and chips. Investors are worried that it's a stock market bubble about to burst. Businesses are worried about the potential for supercharged hacking of their data as well as the cost of AI tokens. And everyone else is worried about their jobs.

But the lesson of global warming and nuclear weapons is that global prevention is hard when it comes to things that are broadly harmful in the long term but individually beneficial in the short term.

As Jonathan Schell wrote 44 years ago, anxiety is the price we pay for sovereignty.

Alan Kohler is a finance presenter and columnist on ABC News. He hosts the podcast That's Business with Alan Kohler in the ABC Business Daily feed on Friday. He also writes for Intelligent Investor.