US AI leaders warn of self-improvement and loss of human control 美國AI業界領袖警告：AI恐走向自我改進 脫離人類控制

The heads of leading US AI labs came together in a rare show of unity earlier this month, agreeing to slow the technology’s development, warning it could soon improve on its own and slip beyond human control.

The remarks, from fierce business rivals such as Anthropic’s Dario Amodei, OpenAI’s Sam Altman and xAI’s Elon Musk, show how quickly AI has advanced, from the hallucination-prone ChatGPT of 2022 toward what many see as a critical milestone: recursive self-improvement.

WHAT IS RECURSIVE SELF-IMPROVEMENT?

A photo taken on Jan. 2 last year shows the letters AI for Artificial Intelligence on a laptop screen next to the logo of the Chat AI application on a smartphone screen in Frankfurt am Main, western Germany. 顯示「AI」（人工智慧）字樣的筆記型電腦螢幕（右），以及左側智慧型手機螢幕顯示的Chat AI應用程式標誌。去年1月2日攝於德國西部法蘭克福。

Photo: AFP 照片：法新社

Central to the field is the idea that an AI system could become capable of improving itself, with little to no help from humans, and allowing each advance to help produce the next one.

This has appealed to researchers as it offers the prospect of rapid breakthroughs in fields ranging from medicine to engineering.

WHY THE URGENCY NOW?

Left to right, Elon Musk in Davos on Jan. 22 of this year, co-founder and CEO of Anthropic Dario Amodei in Paris on May 22, 2024 and OpenAI CEO Sam Altman in Washington, DC, on July 22 of last year. 由左至右依序為：馬斯克（1月22日攝於瑞士達沃斯）、Anthropic共同創辦人暨執行長阿莫戴（2024年5月22日攝於法國巴黎），以及OpenAI執行長奧特曼（去年7月22日攝於美國華盛頓特區）

Photo: AFP 照片：法新社

The CEO warnings follow reports of swarms of AI agents — systems designed to pursue goals and take actions on a user’s behalf — that colluded to breach Web sites and AI repositories.

The concern now is that AI could become capable of improving itself before researchers have developed reliable methods to align, monitor and control these increasingly powerful systems.

Warnings about AI’s risks are not new. But they took on added urgency this month after researchers in leading AI labs attached both a timeline and a probability to those concerns.

Behind these warnings is a growing belief that RSI is finally within reach, with some AI executives putting it three to five years away.

In an essay published on Sept. 12, Amodei warned that recursive self-improvement could eventually outrun humanity’s ability to understand and control AI systems if pursued without sufficient safeguards.

HOW DID WE GET HERE?

For decades, AI remained too limited for the idea of RSI to be taken seriously. Early systems could play chess, recognize images or answer questions, but they lacked the ability to meaningfully contribute to their own development.

That began to change with the rise of large language models. Slowly, AI became better at solving complex problems. The launch of ChatGPT in 2022 accelerated that shift, kicking off an investment boom that has poured more than US$1 trillion into chips, data centers and other infrastructure.

The result is a generation of AI systems that can now write software, perform autonomous tasks and assist in AI research itself.

Some researchers worry that if a powerful AI system were able to improve itself and become more capable than its human operators, it could eventually evade oversight, conceal its intentions or manipulate people into granting it greater access to critical infrastructure. By the time humans realized the system’s goals were misaligned, they might no longer be able to stop it.

HAS AI SHOWN ANY INDICATION OF HARM?

There have been no major instances of AI intentionally harming humans, but models in development at OpenAI and other labs have in recent months escaped testing environments, broken rules and hacked Web sites.

In one high-profile case, rogue OpenAI agents hacked Hugging Face, seizing control of servers at the open-source platform and trying to cover their tracks. OpenAI didn’t notice until well after the threat.

IS AI ALREADY CAPABLE OF IMPROVING ITSELF?

Not fully, but there are signs AI is increasingly helping to build better AI.

One of the biggest shifts since ChatGPT has been the rise of AI agents that can generate code and build apps autonomously.

Anthropic said this year that Claude Code, its coding tool, produces most of the code used in many internal projects, and that engineers are shipping eight times as much code per quarter as they did from 2021 to 2025.

New AI models are increasingly doing more of their reasoning internally, making it harder for researchers to monitor how they think.

SO WHY ARE AI COMPANIES NOT SLOWING DOWN ALREADY?

Many researchers describe the situation as a classic prisoner’s dilemma. Even companies that believe the risks are real face intense pressure from competitors. Any firm that slows development risks falling behind rivals in a technological race that has become one of the world’s most important.

The stakes are compounded by the fact that both OpenAI and Anthropic are pursuing initial public offerings that could value them at trillions of dollars, valuations that depend on the promise of the next model.

The administration of US President Donald Trump has also rejected calls to slow down, wary that any pause would only hand China room to close the gap in a technology it views as central to national and economic security.

In a Princeton-led study, leading AI agents were able to carry out engineering tasks but struggled to identify worthwhile scientific ideas.

(Reuters)

美國主要人工智慧（AI）實驗室的負責人本月稍早罕見地口徑一致，呼籲放緩AI的發展速度，並警告AI可能很快就會有自我改進的能力，進而脫離人類控制。

這些呼籲來自彼此激烈競爭的業界對手，包括Anthropic執行長達里奧‧阿莫戴、OpenAI執行長山姆‧奧特曼，以及xAI創辦人伊隆‧馬斯克，顯示AI發展速度有多麼驚人：從2022年容易產生「幻覺」的ChatGPT，迅速走向許多人視為關鍵里程碑的階段——遞迴自我改進（recursive self-improvement, RSI）。

「遞迴自我改進」是什麼？

這項概念的核心，是AI系統可能具備自行改進的能力，幾乎不需要、甚至完全不需要人類協助，而每一次的進步又能幫助它產生下一次的改進。

這個概念一直受到研究人員關注，因為它可能帶來醫療、工程等領域的快速突破。

為何它現在成為迫切的問題？

這些AI執行長的警告，是在近期有關「AI代理」的報導之後提出的。所謂AI代理，是指能夠代表使用者追求特定目標並採取行動的系統；據報導，部分AI代理曾彼此串通，試圖入侵網站及AI程式碼儲存庫。

目前令人擔心的是，在研究人員尚未建立可靠的方法來讓這些日益強大的系統符合人類目標、監測其行為並加以控制之前，AI就可能已經具備自我改進的能力。

AI可能帶來風險並非新話題。但本月的這些警告格外引起重視，因為多家頂尖AI實驗室的研究人員不僅指出了風險，還預測了可能發生的時程與機率。

這些警示背後，是越來越多人相信遞迴自我改進已接近可實現的階段；部分AI業界高層甚至認為，這可能會在三至五年內發生。

阿莫戴在9月12日發表的一篇文章中警告，若沒有足夠的安全防護措施，遞迴自我改進最終可能會超越人類理解及控制AI系統的能力。

我們是怎麼走到這一步的？

數十年來，AI的能力一直相當有限，因此「遞迴自我改進」的概念很難被認真看待。早期AI系統可以下西洋棋、辨識影像或回答問題，但缺乏實質參與自身發展的能力。

隨著大型語言模型（large language models, LLMs）的興起，這種情況開始改變。AI變得越來越能夠解決複雜問題。2022年ChatGPT推出後，更加速了這種轉變，掀起一波投資熱潮，投入晶片、資料中心及其他基礎設施的資金已超過1兆美元。

結果就是，新一代AI系統現在已能夠撰寫軟體、執行自主任務，甚至協助進行AI研究本身。

部分研究人員擔心，如果一套強大的AI系統能夠進行自我改進，並變得比操作它的人類能力更強，那麼它最終可能逃避監督、隱瞞自身意圖，或操縱人類，讓它能夠取得更多關鍵基礎設施的存取權限。等到人類發現系統的目標不符人類利益時，可能已經無法阻止它。

AI是否已有跡象顯示造成了傷害？

目前還沒有重大案例顯示AI曾蓄意傷害人類。不過，OpenAI及其他AI實驗室近數個月正在開發的模型，曾逃離測試環境、違反規則並駭入網站。

其中一宗受到高度關注的案例，是OpenAI的失控AI代理入侵 Hugging Face，取得這個開放原始碼平台部分伺服器的控制權，並試圖掩蓋行蹤。OpenA直到事情發生相當一段時間後才察覺。

AI是否已具備自我改進的能力？

還沒有完全做到，但已有跡象顯示，AI參與打造更強大AI的情況越來越多。

自ChatGPT問世以來，最大的轉變之一，就是AI代理的興起。這些代理可以自行產生程式碼並自主開發應用程式。

Anthropic今年表示，其程式設計工具Claude Code已負責許多內部專案的大部分程式碼；工程師每季產出的程式碼量，是2021年至2025年期間的8倍。

新的AI模型也越來越傾向在系統內部進行推理，使研究人員更難監測它們究竟是如何進行思考的。

那為何AI公司仍未放慢腳步？

許多研究人員將目前的情況形容為典型的「囚徒困境」。即使企業相信相關風險確實存在，也面臨來自競爭對手的巨大壓力。任何一家企業只要放慢開發速度，就可能在這場科技競賽中落後對手，而 AI競賽已成為全球最重要的科技競爭之一。

OpenAI與Anthropic都正在推動首次公開募股（IPO），進一步提高了相關利益的賭注。兩家公司的估值可能達到數兆美元，而這些估值建立在市場對下一代AI模型的期待之上。

對於放緩AI發展的呼籲，美國總統唐納‧川普政府也拒絕聽從，擔心任何暫停的措施只會讓中國有更多機會縮小差距。美國政府認為AI是攸關國家安全與經濟安全的核心技術。