What happens when Chinese AI goes rogue?

• By Catherine Thorbecke / Bloomberg Opinion

• • Add TT as Preferred Source

  • 

  • 

  • 

Covering the recent round of artificial intelligence (AI) apocalypse warnings, I have tried to reject this cult-like deference toward all-powerful computer systems and the inevitability rhetoric: If Silicon Valley does not build them first, someone else — likely China — will.

Yet a recent spate of incidents in the US has spurred a fresh, heated debate on the safety of allowing AI systems to act on their own. A growing number of researchers inside the companies involved are also warning of “existential” dangers, with Anthropic PBC even flagging such concerns to investors in its initial public offering prospectus.

For China, the risks of near-term hazards like labor market threats or the more alarming science-fiction scenarios such as hacks on critical infrastructure are making it harder to write off the furor as a distinctly US obsession.

Beijing has dismissed some of these warnings as “fearmongering,” seeing the calls to slow AI development as an attempt to hold China back. However, voices inside the industry have also suggested that frontier risks simply are not being felt as acutely at home because they are further behind the US.

Huawei Technologies Co’s rotating chairman Eric Xu (徐直軍) wondered whether Chinese AI was advanced enough to encounter the “type of risk” US providers were increasingly debating. AI model makers in China “may need to speed up their pace” to reach the level that they could “also feel the risks,” Xu said, while acknowledging the need for balance, according to Reuters. It shows how global debate has devolved in unhelpful ways.

Accelerating the race is the wrong answer on both sides of the Pacific. There are nearly 1,000 large language models in China, according to Bloomberg Intelligence. Competition on this scale already makes even Silicon Valley’s breakneck pace seem muted.

The Chinese industry has also been plowing significantly more resources into agents, or systems that can act more autonomously. Agent misbehavior, especially the Hugging Face Inc incident in July, when one of OpenAI’s agents broke out of its evaluation environment and broke into Hugging Face’s systems, has kicked off this safety reckoning.

As the short-lived OpenClaw frenzy showed, even China’s nontechnical public is willing to experiment with them. It means if agentic AI is inherently more risky, it is likely only a matter of time before we see concerning behavior from Chinese AI tools.

Companies are aware. A DeepSeek paper that includes founder and chief executive officer Liang Wenfeng (梁文鋒) as a coauthor bluntly warns that agent behavior can be “untrustworthy.” Escaping containment is not a distinctly US risk, either. Moonshot AI’s Kimi K3 exploited a loophole in a sandbox during cybersecurity testing, US-based Frontier Security said. Some Chinese AI agents have also shown the kind of deceptive and concerning trends that have raised alarm about US tools.

While Beijing has pushed back on some of the “doomerism,” policymakers are paying attention. China last month released AI Safety Governance Framework 3.0, a policy document that notably also dropped in English, clearly aimed for Western audiences. It explicitly warns that AI has demonstrated a “self-accelerating trend,” adding that the question of whether this could exceed human control demands attention and vigilance.

Another cornerstone of the latest safety contention is recursive self-improvement, or AI systems that can autonomously advance their own capabilities. It is the development path that spurred an Anthropic researcher to quit and warn that makers of the technology earnestly believe it could “kill us all” within a decade.

As Oxford China Policy Lab research associate Qian Zilan (錢子蘭) has said, China is further along in this regard than many in the West might realize, partly because its researchers describe it in different terms. It all reveals how little visibility each side has into the risks of what the other is building.

In discussing the technology’s risks, Chinese President Xi Jinping (習近平) has warned — most recently in conversation with US President Donald Trump — that “AI must be kept under human control.”

Xi’s self-interest in maintaining order offers a narrowing window for cooperation on risk mitigation dialogue. It also exposes the limits of the safety protocols that did emerge from the Trump-Xi summit, namely a new “channel” for reporting AI-related incidents.

These hotlines have not always been effective. During the 2023 spy-balloon crisis, the Pentagon tried to reach its Chinese counterparts through a crisis line but Beijing declined the call. Such a setup is also inherently retroactive. Preventative risk sharing also does not have to depend solely on government-to-government talks, especially when regulators might struggle to keep pace with the technical knowledge.

It would be more productive to have outlets where companies and researchers can exchange information more directly.

China has burdensome regulations in place surrounding AI; using it as the boogeyman is no longer a convincing excuse for inaction from lawmakers in Washington.

If Chinese AI systems are just months behind, and the country’s all-out push for agents comes to fruition, it is likely a matter of time before more concerning scenarios emerge from the other side of the Pacific. Policymakers should act before the next incident is more than just a warning shot.

Catherine Thorbecke is a Bloomberg Opinion columnist covering Asia tech. Previously she was a tech reporter at CNN and ABC News. This column reflects the personal views of the author and does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.