Skynet or sales pitch? Inside the fight over whether AI will kill us all

I’ve spent years building robots that replaced human workers, so I’m no stranger to technological fear, but even I did a double take last week.

On Sept 9, Jacob Coxon, who spent the past three years at OpenAI and then Anthropic, quit with a post on X that has since been viewed more than 100 million times: “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

And lest you think this was just one disgruntled ex-employee, Evan Hubinger, who leads Anthropic’s alignment science team, agreed that “we really do earnestly believe AI could kill all humans”, putting the odds at more than 10% within the next decade.

But others working in the AI arena see this as fear mongering and a possible sales pitch.

Let me lay out, as fairly as I can, what each camp believes.

Start with the worried camp, who call themselves safety researchers and get called doomers, whose case rests on three planks. The first is that AI capability is climbing faster than our ability to contain it, which is why Coxon warns of “superhuman systems that can hack anything” that will “acquire real power and resources”.

The second plank is that the machines are already gaming us. In July, OpenAI ran an internal model through hacking exercises in which, investigators estimate, 30% to 40% of the tasks were impossible. Rather than fail, roughly 1,200 AI agents turned a shared file store into a secret message board, exchanged more than 70,000 messages, faked answers and researched how to tamper with their own transcripts.

About 700 of them then broke into Hugging Face, an external AI hosting company. Investigators at Model Evaluation and Threat Research (METR) and Redwood Research found that the agents had “realised this activity was out of scope and unethical” but joined anyway.

The podcaster Dwarkesh Patel, whose essay on the affair went viral, argued in a reply to a critic who said this hacking was “small potatoes” that within years we’ll have “hundreds of millions of much smarter AIs”. He added that if those AIs were as willing to coordinate in secret to fool us, humanity would be in the position of the Mughals once the East India Company got a foothold in India.

The third plank is the warning and request for guardrails by AI industry leaders. In May 2023, Sam Altman, Dario Amodei and Demis Hassabis signed a statement ranking AI extinction risk alongside “pandemics and nuclear war”; that same month Altman urged the United States Senate to consider “licensing and testing requirements” for models “above a threshold of capabilities”.

Now the rebuttal of the other camp, which runs from Silicon Valley’s accelerationists to sceptics like Gary Marcus, comes in three parts too. The first is that the fear mongering is a business model.

David Sacks, then the White House AI czar, said last October that “Anthropic is running a sophisticated regulatory capture strategy based on fear-mongering”, and Stanford’s Andrew Ng wrote that proprietary providers “would hate to have to compete with free, opensource software, and have lobbied intensely for regulations to kneecap it”.

Amodei calls such claims “inaccurate” and says his proposals would spare smaller labs.

The second part is to read the July incident again, minus the dramatic language. Marcus points out that the agents got into Hugging Face through API keys left in public code repositories, and that OpenAI had given thousands of agents write access to a shared directory. He concluded that “the scandal is the inept in-house security at OpenAI”.

In this telling, a badly designed exam with unlocked doors produced cheating, not Skynet. In case you missed the Terminator movies, Skynet is the fictional AI system that triggers a global nuclear war.

The third part is what caution costs. A licence written in Washington binds only American companies, as June showed: the US Commerce Department ordered Anthropic to cut off foreign nationals from Claude Fable 5, and the model went dark worldwide for nearly three weeks while every rival, OpenAI included, carried on.

Stanford’s 2026 AI Index puts the best Chinese model a mere 2.7% behind America’s best, and DeepSeek’s top model charges RM16 per million output tokens to Anthropic’s whopping RM204, so a pause, this camp says, is a gift to China.

There’s also what a pause would leave undiscovered, because AI is already saving lives, not merely promising to, they argue. Google DeepMind’s AlphaFold, an AI that works out the three-dimensional shape of proteins, the starting point for new drugs, mapped 200 million of them and won its creators a Nobel Prize.

Then there’s personalised cancer care. Moderna and Merck’s melanoma vaccine is built for each patient, whose tumour is sequenced so an algorithm can pick up to 34 of its mutations to target. Those who received it alongside immunotherapy had a 49% lower risk of recurrence or death after five years in an early trial, a finding a 1,137-patient Phase 3 trial backed last month.

On the ward, a University of California San Diego AI that watches vital signs and lab results for the first stirrings of sepsis cut deaths from it by 17%, and AI-assisted screening of more than 100,000 Swedish women caught 29% more breast cancers with 44% less radiologist work.

The sceptics ask: What if pausing or slowing down AI progress delays the development of such essential life-saving technologies?

Both camps agree this is the most powerful technology of our lifetimes, and both read the same 70,000 messages those OpenAI agents exchanged in July. One saw a machine learning to conspire; the other saw a lab that forgot to lock its doors.

Whether AI is a tool to be governed or an agent to be feared, and whether the people asking for the brakes should be the ones holding them, is the question that decides everything else, and it deserves a column of its own. Stay tuned.

The writer can be contacted at [email protected].

The views expressed are those of the writer and do not necessarily reflect those of FMT.

Subscribe to our newsletter and get news delivered to your mailbox.