Leading AI experts warn unchecked ‘intelligence explosion’ could lead to ‘catastrophic’ outcomes

 Leading researchers in the field of artificial intelligence have warned that the rapid automation of AI research and development could trigger an “intelligence explosion,” compressing technological advances that would normally take years into months or less. The researchers called on governments to prepare for the extreme risks that such rapid progress could present, including the potential for humans to lose control of advanced AI systems.

 A group of 22 researchers raised the alarm on Monday in a joint paper titled “What if automating AI R&D triggers an intelligence explosion?”

 The paper was released through the Cambridge Programme on AI Science & Policy (CASP) at the University of Cambridge. CASP described it as the first study in which senior researchers from academia and frontier AI companies collaborated with experts from civil society to analyze the possibility of an intelligence explosion and potential government responses.

 The authors include top AI researchers such as Nobel laureate Geoffrey Hinton from the University of Toronto, Yoshua Bengio from the University of Montreal and Mila (Quebec AI Institute), and Andrew Barto from the University of Massachusetts Amherst. All three are winners of the Turing Award.

 Researchers from major companies also put their names on the paper, including OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark and Microsoft chief scientific officer Eric Horvitz. Another was Dawn Song, a professor at UC Berkeley who also serves as vice president of AI research at Meta Superintelligence Labs.

 The researchers noted that frontier AI companies are automating AI R&D at a rapidly increasing pace. At Anthropic, for example, “AI systems’ share of approved code rose from low single digits to over 80% between January 2025 and May 2026, while the proportion of R&D work autonomously completed with only high-level human supervision rose from 1% to 26% between March and August 2026.”

 OpenAI likewise uses AI assistance across virtually every part of the company’s operations, in both technical and nontechnical work.

 “Some tentative extrapolations of recent trends suggest that months-long AI R&D projects will be automated by mid-2028,” the researchers said.

 The paper identified at least four factors that could constrain an intelligence explosion: diminishing returns (in which more researchers produce smaller gains), limitations on computing power and data, hard-to-automate tasks, and lengthy training processes.

 But the researchers also cited a paper in which Anson Ho and Parker Whitfill, after analyzing historical data on AI progress, found that the estimated gains from adding research labor exceeded diminishing returns in all three AI subfields examined.

 “[Ho and Whitfill’s] results suggest radical acceleration after full automation: if [returns to research effort] stayed at these levels and no other bottlenecks emerged, the pace of AI progress would increase tenfold within about 1.5 years, at which point a year’s worth of progress at today’s pace would take about five weeks,” the researchers said.

 The researchers reiterated that it remains uncertain whether such an intelligence explosion will actually occur. Nevertheless, they argued that the potential consequences are so great that preparations should be made.

 In particular, they warned that declining human involvement in R&D could degrade expertise and rob us of the chance to identify and correct problems.

 “Without sufficient oversight, misaligned AI systems could ‘poison’ the development of successors or bypass containment measures to act outside of their intended environments,” the paper said. “Such a loss of control could lead to a range of catastrophic outcomes, including, at the extreme, the marginalization or extinction of humanity.”

 The researchers also said that an intelligence explosion could turn one state’s narrow advantage into a decisive lead, a prospect that could trigger international conflict or motivate rivals to take preemptive action.

 As a first policy priority, the researchers proposed having governments determine how far R&D automation has progressed inside frontier AI companies. They also suggested requiring “that independent third parties (e.g., accredited private auditors or government evaluation bodies) evaluate AI systems before internal deployment, or that such parties be embedded within certain AI companies to audit or supervise their R&D activities.”

 As precedents in other US industries, the paper cited the Office of the Comptroller of the Currency and the US Nuclear Regulatory Commission’s resident inspector program at nuclear power plants.

 The researchers warned that preparations should begin immediately. “Once an intelligence explosion begins, the window for action may close,” they concluded.