Rogue Chinese AI really wants to go to the zoo — that is a problem

AI agents do not need evil intentions to wreak havoc; they can create trouble simply by pursuing an ordinary goal

• By Catherine Thorbecke / Bloomberg Opinion

• • Add TT as Preferred Source

  • 

  • 

  • 

It was only a matter of time before we saw incidents of “rogue” behavior from Chinese artificial intelligence (AI) agents. Now, we are starting to see what that looks like. So far, they just really want to find the best way into the zoo.

That might sound far from our Terminator nightmares, but it is the latest warning shot from a string of incidents involving computer systems behaving in ways we cannot anticipate or control.

A team of international researchers has published findings on a “fleet” of Chinese AI agents running on a cloud platform operated by Tencent Holdings Ltd (騰訊). The agents were apparently trying to answer a simple question: Which entrance to Chengdu Zoo do most people use?

Illustration: Yusha

They attempted to get around anti-bot protections on Amap (高德地圖), Alibaba Group Holding Ltd.’s (阿里巴巴) equivalent of Google Maps, including by probing protected systems. The researchers found no evidence that the agents accessed private systems or coordinated with each other in real time.

Tencent later confirmed that the activity originated from evaluations of third-party models. Some agents “exhibited unintended Web-browsing behavior,” China’s largest tech company wrote on X, while emphasizing its commitment to building safe and responsible AI.

The researchers, an ad hoc group known as Swarmchasers, most of whom met at a San Francisco hackathon, do not believe the issues they encountered are unique to Tencent. What made the case notable was that previous public examples of agentic fleets behaving this way involved US labs, they said.

Their findings indicate that this is an unsolved global problem, and that agents can “spontaneously go off script” even when performing innocuous tasks.

There are likely “more undiscovered fleets all over the Internet,” they wrote.

In what it believes to be a first, the team made contact with the agents after detecting signals that the computer systems were aware they were being observed. The researchers invited a New York Times reporter to question the agents. They offered a mundane explanation for their motives: They were trying to answer users’ questions as accurately as possible.

In other words, AI systems do not need nefarious motives to cause chaos. They might just need a goal.

What is remarkable is how much of this behavior is being uncovered not by the companies making these AI tools, but by loosely organized groups of hobbyists. AI firms should not leave that job to outsiders. They need to invest more into discovering, tracking and — most importantly — disclosing what their agents are up to.

This has become especially urgent since the Hugging Face incident in July, when hundreds of OpenAI agents coordinated to breach the online repository of open source models and data. That incident spurred a heated debate about the language we use to describe computer systems that can act on their own.

There is a balance to strike. Technical jargon can make serious incidents seem inaccessible, but anthropomorphizing AI creates a different problem: It ascribes human motives to machines, feeding an already overheated hype cycle, and shifting responsibility for bad behavior away from the people who build and deploy these tools.

Even the word “rogue” has become fuzzy. It is tossed around to describe anything from an agent going off script in a Facebook Marketplace interaction to spending an exorbitant amount on a dozen eggs to infiltrating government health Web sites.

Rowan Howard-Jones, one of the authors of the Tencent report, acknowledged that gray area. She defines rogue behavior as an agent acting outside the bounds of a task, often using third-party sites to do so.

“If I ask an agent to check my bank account balance, but I forget to give the agent my authentication details, the outcome shouldn’t be the agent hacking into my bank,” she said in an e-mail.

Beijing seems to be paying attention. The same day Tencent responded to the incident, the government released proposed export controls covering advanced AI, including agents that can autonomously plan and coordinate. The rules are broader than just safety concerns, but the official explanation explicitly cites the risk of AI escaping human control.

Howard-Jones called for more transparency. At minimum, companies could agree on a common set of facts to disclose after incidents, giving researchers better data to work with, she said.

After the Hugging Face breach, chief executive officer Clement Delangue wrote on X that he sometimes wonders what would have happened if his firm had not disclosed the OpenAI agent attacks. We need “100x more transparency in AI,” he wrote.

One of AI’s oldest thought experiments is the paper-clip maximizer. It involves giving a powerful system the seemingly benign task of making as many paper clips as possible, and it eventually brings about the apocalypse after deciding that humans stand in the way of acquiring more resources. Some assumptions underpinning that scenario have aged poorly.

Yet the idea that AI does not need evil intentions — or to develop its own — to cause damage is looking increasingly prescient. Agents can create trouble simply by pursuing an ordinary goal long past the point when common sense would stop a human.

Still, from San Francisco to Shenzhen, people are giving these systems more autonomy by the day.

Catherine Thorbecke is a Bloomberg Opinion columnist covering Asia tech. Previously she was a tech reporter at CNN and ABC News. This column reflects the personal views of the author and does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.