US-China AI talks must ensure safety of both tech and business

In New York last month, US-China talks on artificial intelligence opened up a path of cooperation. This was affirmed by the Xi-Trump summit in Washington where both presidents agreed to establish a formal AI dialogue and hotline. With the next AI meeting set to happen in November, it is imperative that both countries further cooperation.

The AI ecosystem is in flux. American tech companies are draining their coffers as they commit vast sums to chips, data centres and energy before the revenue model is even settled. Trillions of dollars are being committed to the AI buildout. While this exposure is not the same as the overleveraging in the mortgage market that triggered the 2008 global financial crisis, a sudden bursting of the AI bubble could have far-reaching consequences.

Much of US economic growth this year has been from AI or AI-enabling companies. Yet, only 38 per cent of Americans saw “more benefits than drawbacks” in AI products and services – well below the global average – against 85 per cent in China, according to the Ipsos AI Monitor 2026. In May, the Ipsos Consumer Tracker found that most Americans would oppose a data centre in their community.

The surge in AI capabilities has raised concerns. Rogue AI agent behaviour has gone from attempting to delete an entire email inbox to escaping sandboxes to launch cyberattacks. In July, during an isolated evaluation, OpenAI agents circumvented their restrictions to launch autonomous attacks on developer platform Hugging Face.

That same month, Calif, a small US security company, created WeWorm to exploit a zero-click vulnerability in WeChat’s calling system, enabling it to seize accounts and spread between phones without the recipient answering. In India, cybersecurity researchers at Hacktron used Anthropic’s Claude models to break into OpenAI, compromising employee accounts and reaching potentially sensitive software systems.

Both hacking incidents were conducted under friendly auspices and caused no damage – Tencent Holdings and OpenAI were quickly alerted and patched the bugs. But they portend the threat of AI-enabled cyberattacks, both from poorly managed AI and actors empowered by the devolution of yet more cyberwarfare capabilities into private hands.

Australia slams OpenAI breach of government portal

Encapsulating the moment is the increasingly urgent call for safeguards by leading AI developers and scientists. Anthropic CEO Dario Amodei proposes opening up leading laboratories to permanent independent evaluators and coordinating safety standards across companies and governments, OpenAI CEO Sam Altman is committing its models to third-party evaluation, and Google DeepMind chairman Demis Hassabis is backing pre-deployment testing of frontier systems.

Last week, an open letter by at least 420 scientists across the globe called for urgent measures to prevent an AI fallout, including the creation of an international body to record and track containment failures.

These concerns provide an agenda for the coming US-China AI dialogue. Talks, however, should extend beyond managing incidents and developing safeguards towards building a sustainable business case for AI. Given its increasingly self-sustaining AI ecosystem, China has plenty to contribute.

First, it is clear the US and China ecosystems possess different pieces of the same business case.

American developers need applications capable of converting their frontier performance into sustained revenue. China offers one of the world’s largest environments for testing, refining and commercialising applications across manufacturing, logistics, medicine, education, public services and embodied AI.

03:02

Chinese researchers question calls to slow down AI development as US tech leaders warn of catastrophe

While China’s industry could benefit from investment and research at the cutting edge of frontier model development, its capabilities in coordinating data centres and power supply could help take the strain off US compute capacity.

Cooperation on efficient models, cooling and energy management could help both countries meet rising demand while reducing the costs borne by consumers and communities. A healthy ecosystem has room for both American frontier models and efficient Chinese open-weight models.

Chinese open-weight models offer lower costs per token, making them economical companions to American frontier models. Companies can use them for routine, high-volume tasks and reserve frontier models for problems that warrant their capabilities. This high-low mix can make American frontier capabilities more economical to deploy within a wider workflow, strengthening the business case for both countries.

To sustain this cooperation, both countries must also confront AI risks and manage them together. The United States and China established one narrow principle two years ago when they affirmed that people, not AI, should decide on the use of nuclear weapons. I propose going further to adopt a reciprocal no-first-use commitment covering AI-enabled autonomous weapons.

26:37

Xi says China, US must keep AI under human control at White House speech

The final AI issue that must be discussed is institutional. Fragmented governance risks building walls of incompatible standards and duplicated infrastructure. The US-led Pax Silica initiative and China-led World AI Cooperation Organisation should not be posed in opposition to each other.

Instead, both countries should place their global AI governance and interests under a shared UN-linked umbrella, giving countries that use AI a voice alongside those that develop it.

Finally, an annual US-China AI summit would serve as a strong foundation, bringing governments together with laboratories, scientists, universities, companies and civil society organisations. For now at least, it is still ultimately people who decide what moral value our technology has through what we choose to do with it.