OpenAI executive urges caution in China-US AI talks, warns of distillation threats

A top OpenAI executive said on Monday that Washington needs to be “clear-eyed” and “aware” about sharing information with Beijing during coming talks on artificial intelligence (AI), while also warning about the threat posed by alleged Chinese distillation attacks.

“Be very clear-eyed. Be very aware of what information you are able to expect in return for what you are sharing, and start to build a foundation,” said Caroline Zier, OpenAI’s deputy lead for national security, during a discussion at the Hudson Institute.

Zier, who served as a special assistant to the US secretary of defence from 2021 to 2023, endorsed the bilateral dialogue but expressed scepticism that Washington and Beijing shared the same understanding of the risks involved.

“My philosophy is you have to get caught trying,” Zier said.

“I doubt very much that when we say safety and loss of control, those are the same fears that the CCP [Chinese Communist Party] would have when they use that same terminology,” she added.

US Treasury Secretary Scott Bessent and Chinese Vice-Premier He Lifeng are expected to meet later this month before the summit between US President Donald Trump and Chinese leader Xi Jinping on the sidelines of the Asia-Pacific Economic Cooperation (Apec) summit in Shenzhen – their third face-to-face talks this year.

The South China Morning Post reported last month that technical teams from the two countries were holding regular talks to iron out details of a framework announced in September during Xi’s state visit to Washington, with the aim of making AI cooperation a potential deliverable of the coming talks.

The White House fact sheet and the Chinese government’s account of the eight outcomes, released after the September summit, said the leaders had agreed to establish a dialogue mechanism on the “risks and benefits” of artificial intelligence and a “bilateral communication channel” for AI-related incidents.

Sam Bresnick, a research fellow at Georgetown University’s Centre for Security and Emerging Technology, said that while the talks are “encouraging,” he does not expect the US-China “hotline” on AI risks to be an effective mechanism.

“Hotlines with the Chinese have not worked,” he said during the Hudson Institute discussion.

“I don’t expect this one to work particularly well either, because the Chinese side simply does not share information quickly.”

The Chinese embassy in Washington said last month that Beijing “remains open to maintaining communication” with Washington through the AI dialogue mechanism.

SCMP Plus is a new premium news platform that gives you an

all-inclusive edge to stay ahead on China news.

To access our exclusive content you’ll need to subscribe.

Already a subscriber?

LOG IN

“China puts equal emphasis on development and security in terms of AI. We take seriously the inherent and secondary risks of AI,” it added.

Elsewhere in the conversation, Zier addressed allegations of Chinese distillation attacks. These attacks involve a malicious actor using a “teacher” model’s outputs to train a “student” model, according to the Institute for AI Policy and Strategy.

Trump rejects working with China on AI despite Xi’s recent visit

Zier distinguished “authorised distillation”, conducted under agreed terms to train student models, from “unauthorised distillation”, which Zier said they have sometimes called “adversarial” to be “a little more pointed on the source”.

“It is largely from Chinese labs that’s where the activity emanates,” she said, adding that it is not in accordance with terms of service or with agreement in advance.

“It seeks to take IP that is protected,” she said, noting that it often strips “safety guardrails and boundaries that we put into our models”.

In September, Beijing rejected US allegations of AI distillation, warning of retaliation. China hit back at claims that Chinese AI companies were copying American models “on an industrial scale”, after a joint advisory from US intelligence and cybersecurity agencies accused six Chinese companies, including DeepSeek and Alibaba Group Holding, of distillation.

China’s Ministry of Commerce said the allegations lacked factual and legal grounds, accusing the US of using such claims to assert technological dominance and warning of countermeasures in response.

Alibaba is the owner of the South China Morning Post.

Despite Washington and Silicon Valley framing unauthorised distillation as a threat to American IP, start-ups in the West have increasingly turned to Beijing’s cost-effective open-weight models, a point the panellists discussed on Monday.

Michael Ortiz of Reflection AI called Chinese models highly capable and at the frontier, flagging an estimate that almost 80 per cent of US AI start-ups are working on them.

“What the Chinese models have historically been able to do is offer very good capability for lower prices,” Bresnick added.

Bresnick said that start-ups often prefer open-weight models because they can run them locally, fine-tune them, and use their own data.

Both Zier and Bresnick raised concerns about future dependence if access to Chinese models were restricted.

“It is hard for me to imagine that there are not significant rip and replace costs down the line if one day ... the CCP decides that it too would like to have controls over when and how models are provisioned,” Zier said.

“I think there’s a risk of being dependent on Chinese technology,” added Bresnick.