We Are Summoning An Alien Species: Ex-OpenAI & Anthropic Researcher

Ex-OpenAI & Anthropic Researcher Jacob Coxon who recently quit the Dario Amodei-led company behind Claude sparked global shockwaves by warning that the people building AI themselves "earnestly" believe that the technology could wipe off entire humanity by the end of the decade. Coxon's latest warning states that trying to build super-intelligent AI is like summoning an alien species-a mind we don't fully understand and cannot control.

"The concept of superintelligence, it's been likened to summoning an alien species. So thinking about it in terms of an alien mind we don't fully understand is pretty terrifying even without any concrete scenario," he said during a TV show.

Coxon is basically sounding an alarm on the fact that creating an uncontrollable alien intelligence is inherently terrifying, even before you consider any doomsday scenario.

And he's not the first to speak of the doomsday scenario - AI eventually killing us all if the current pace of development continues. But he's definitely the first whose warning has resonated far and wide including with prominent AI leaders building the technology. In fact, Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman and Grok boss Elon Musk have all come together now and spoken about the possibility of orchestrating a slowdown in development. In fact, Musk himself has time again pointed to a 10 percent possibility of AI wiping off humanity in 10 years. Geoffrey Hinton, a Nobel Prize-winning computer scientist widely revered as one of the 'Godfathers of AI' has repeatedly warned of a 10-20 percent chance of human extinction due to AI. In fact, Hinton along with other tech and AI pioneers such as Yoshua Bengio, Yann LeCun, LinkedIn co-founder Reid Hoffman, former Google CEO Eric Schmidt, Google DeepMind co-founder Demis Hassabis, among several others have made global appeals for increased guardrails around unchecked AI.

Coming back to Coxon who seemed to have triggered off a domino effect of sorts said that for him it was a "continuous process of realisation that I was genuinely not happy with the way things were going and I needed to make a significant change." 
The Insiders Who Became Outsiders

There have been several insiders before Coxon who have quit to warn of the perils of AI. Earlier this year, in February, Coxon's colleague Mrinank Sharma, the former head of Anthropic's safeguards research team resigned after expressing deep concern that the world is in "peril" and citing internal friction and pressures to set aside critical safety work for rapid deployment
In 2024 there was a serious exodus at OpenAI.

Jan Leike, co-leader of OpenAI's Superalignment team resigned explicitly stating that safety culture and processes had "taken a backseat to shiny products." 

Then there was Ilya Sutskever, OpenAI co-founder and Chief Scientist who co-led the Superalignment team alongside Leike. He left over fundamental disagreements regarding the trajectory of the company's safety research.

And there was Daniel Kokotajlo, a governance and alignment researcher who quit after losing confidence in the OpenAI leadership's capability to handle future advanced AGI systems responsibly and refusing restrictive non-disparagement exit terms. AGI or artificial general intelligence is loosely defined as a point where AI is able to match or surpass human intelligence.

Other AI safety researchers at OpenAI who were part of the 2024 exodus included William Saunders, Leopold Aschenbrenner and Pavel Izmailov. Aschenbrenner was in fact let go over an alleged improper disclosure of internal company information. He disputed the reasoning, claiming he was targeted after writing an internal memo warning the board that security was insufficient. Aschenbrenner also turned down an equity payout worth nearly $1 million because he refused to sign a non-disparagement/nondisclosure agreement tied to his departure.