Expert Explains | Why China’s answer to frontier AI risk looks different from America’s

The US and China have agreed to open a new bilateral dialogue on artificial intelligence, following weekend talks that US Treasury Secretary Scott Bessent called “very successful”.

The talks came ahead of Chinese President Xi Jinping’s first state visit to the White House in over a decade. The new dialogue would provide a channel for discussing common AI goals and threats, while the US has also proposed a notification mechanism for certain AI-related national security incidents.

But the agreement is striking for another reason: the two countries increasingly recognise some of the same risks from frontier AI, while taking markedly different approaches to governing them.

In recent weeks, Anthropic CEO Dario Amodei and other leading US AI executives have argued that frontier development may need to be “paced” if increasingly capable models outstrip existing safeguards. China, meanwhile, has begun explicitly identifying risks such as models deceiving evaluators, escaping controlled environments and improving themselves, but has so far chosen a different regulatory model.

Why have the two countries arrived at different answers to increasingly similar concerns?

Poe Zhao, a China tech analyst and founder of Hello China Tech, explains how China’s existing rules differ from US proposals for independent oversight during frontier-model training, and how strategic competition complicates the comparison. Edited excerpts:

China now recognises risks such as models improving themselves, deceiving evaluators and breaking out of controlled environments. If the concerns increasingly overlap with those being raised in the US, why has China not adopted the same idea of independent oversight during frontier-model training?

Story continues below this ad

There is no single starting point for every obligation. The key question is whether a company provides a generative AI service to the public in mainland China. Some requirements, including certain security assessments and filing procedures, may apply before launch. Other duties continue after launch, including content governance, user protection and labelling AI-generated content. The rules can also examine training data, annotation and algorithms, even though their scope is based on providing a public-facing service.

The AI Safety Governance Framework 3.0 released by China’s National Technical Committee 260 on Cybersecurity (TC260) names recursive self-improvement, models deceiving evaluators, and models breaking out of test environments. It even asks developers to validate safety during training. But it is guidance. The framework itself does not establish a legally enforceable right for an independent evaluator to maintain continuing access to training.

China’s binding rules sit elsewhere. The Interim Measures for Generative AI Services do include training-stage duties, covering pre-training data handling, annotation, and regulator inspection of training data and algorithms. Those duties fall on a service provider because the rules apply when a company provides an AI service to the public. But regulators can still scrutinise how the underlying model was trained.

Amodei proposes an independent evaluator inside the lab with continuing access to training pipelines, and checkpoints that activate when a model demonstrates specified dangerous capabilities. That is a proposal, not current US regulation.

Story continues below this ad

The main difference is who gets access, and when. Under the embedded-evaluator model, an outside evaluator would have employee-level access while a model is being developed. This would allow it to test whether agreed risk thresholds had been crossed. Chinese regulators can inspect training data, algorithms and pre-launch risks. However, the current rules do not give an independent outside evaluator a continuing role inside an AI lab during development.

How much of this difference comes from the way the two countries regulate AI, and how much from their strategic positions in the AI race? Would a slowdown inevitably look different from Beijing when most leading frontier labs remain American?

I would not assign percentages to the two. Regulatory design and strategic position interact.

On September 14, China’s Foreign Ministry was asked by Reuters about the calls to slow frontier development and about Amodei’s claim that a Chinese lead would threaten US security. The spokesman said that spreading threat narratives and engaging in confrontation and vicious competition would only disrupt global AI governance. That is a response to the framing, rather than a position on any specific pacing mechanism.

Story continues below this ad

Amodei’s essay pairs pacing with measures to preserve the US lead: blocking chip sales to China, restricting unauthorised distillation (using one AI model’s outputs to help train another), and preventing weight theft. When a safety proposal also seeks to preserve US technological leadership, Beijing is likely to assess both its safety benefits and its competitive consequences.

China’s rules largely give regulators a company or service to hold accountable once an AI system reaches users. Where does that approach become weaker as open-weight models and AI agents become harder for their original developers to control?

A deployment-centred approach gives regulators an identifiable service provider to hold accountable and a point at which they can impose conditions on public access.

It weakens with open weights and agents. Open-weight models make their trained numerical parameters, or “weights”, available for others to download, run, modify or fine-tune the model themselves. Simply releasing those weights may not clearly count as providing a service to the public.

Story continues below this ad

Once copies are downloaded, the original developer’s ability to control or withdraw them becomes limited. Managing risk then requires safeguards across downstream deployments as well, meaning the different products and systems in which copies of the model may later be used.

On agents, the cyberspace regulator said on September 1 that vigilance was needed over frontier models bypassing sandboxes, controlled environments designed to restrict what an AI system can access or do.

TC260 released a practice guide on the secure deployment and use of AI agents in July, and on September 18 opened public consultation on a separate guide for the secure development of agent systems. Both are practice guides rather than final mandatory national standards.

Chinese AI firms also face high compute costs and pressure to raise capital and generate revenue. What does China’s experience tell us about whether financial pressure itself can explain calls to slow frontier development?

Story continues below this ad

A capability-based trigger — where intervention depends on what a model becomes capable of doing, rather than on a company’s financial situation — does not tell us how much financial pressure contributes to the proposal.

China’s experience suggests that the effect of capital pressure depends on financing conditions and competitive incentives. Chinese AI companies Zhipu and MiniMax are publicly listed, while Chinese AI developer Moonshot has reportedly moved toward a Hong Kong listing.

Where access to funding depends on demonstrating growth, a company has reason to release products and generate revenue faster. Where compute costs outrun available funding, it may postpone or scale back training. Capital pressure can push in either direction, so it cannot explain calls for pacing on its own.

Does the new US-China AI dialogue suggest that the two countries may be able to agree on certain AI risks even if they disagree over how and where to regulate them? What kinds of “shared threats” are most likely to produce genuine cooperation, and where do you expect strategic competition to remain the limiting factor?

Story continues below this ad

The agreement is meaningful because it creates a regular channel for discussing shared risks. The most practical starting points are incident notification, uncontrollable AI agents, and misuse by non-state or cyber actors. Bessent has specifically cited uncontrollable agents and non-state actors, including cyber threats, as areas the two sides need to discuss.

The dialogue could later cover AI use in nuclear command systems, critical infrastructure and other high-risk military settings. These ideas have appeared in expert discussions, but they are not yet agreed government positions. Wider cooperation will remain difficult because AI safety is closely connected to competition over chips, computing power and advanced models.