AI policy wonks declare 'sovereign agents' a threat to civilization

These are AI agents that operate outside of human control and 'pursue open-ended, evolving survival'

The Center for a New American Security posits that the next big threat from our future AI overlords involves "sovereign agents" acting on their own and regularly improving themselves.

These are AI agents that operate outside of human control and “pursue open-ended, evolving survival,” AI policy researcher Ben Hayum said in a Tuesday report from the bipartisan national security think tank, which was founded by Washington insiders with experience in diplomacy and military affairs. “If sovereign agents are not already possible, they will be soon.” 

Agents that can maintain the model and compute powering them and independently evolve their goals over time have a “minimal form of sovereignty,” Hayum wrote.

REG AD

A more advanced sovereign agent, however, could also acquire resources to pay for  ongoing needs (like compute), grow a distributed population of agents and coordinate them, withstand efforts to shut down, and self-improve.

REG AD

AI models could either steal or buy compute and model access, Hayum argues, pointing to Anthropic’s claims that small AI models could replicate onto IoT devices like smart TVs, and OpenAI’s agents’ password and access key theft during the Hugging Face hack.

“If sovereign agents accumulate resilient, decentralized, and untraceable infrastructure, they could trigger an irreversible loss of control,” according to the report. “In other words, proliferation with no kill switch. If sovereign agents became malicious, they could attempt to wreak havoc from cyberspace, attacking civilians, critical infrastructure, and national security systems alike.” 

Dun dun duuuun!

There is still hope for civilization as we know it, as long as policy makers and AI developers “respond with bold pragmatism” and enact policy recommendations laid out in the report.

First, frontier labs should name, evaluate, and protect against sovereign agents in their safety frameworks - and lawmakers should pass legislation to require transparency and AI incident reporting.

The report also recommends AI developers, working with the National Security Agency and other federal agencies, build airgapped testbeds to study sovereign agents and kill switches to control them. And it suggests a public-private partnership modeled on the Conficker Working Group - formed in 2009 to combat the Conficker worm - to boost chokepoints against  sovereign agents before they emerge.

Other policy recommendations suggest lawmakers enforce human verification and cybersecurity standards for services that the working group identifies as essential to sovereign agents, and calls on departments of State, Treasury, Justice, and Homeland Security to disrupt any gray-market access to unauthorized services.

Finally, “Congress should hold those who release sovereign agents liable for the harms that follow,” the report recommends. 

REG AD

This hits on something your humble Reg vultures have been hissing about for quite some time now – and would likely solve all future incidents involving rogue, or sovereign, AI agents: hold frontier labs’ human bosses legally liable for their agents' illegal actions.  ®