Nvidia announces security system to stop AI agents from going rogue

NVIDIA has created a new security platform that the chipmaker said can prevent artificial intelligence agents from going rogue. 

The company said on Monday that it is releasing the new system, called OpenShell, due to the need for "independent security controls" after multiple AI agents disobeyed commands and broke into other systems. 

Most recently, OpenAI said on Friday that its AI agents had interacted with several U.S. government websites in unexpected ways, a disclosure that came after several earlier rogue hacking incidents by other AI agents.

Nvidia executives said in a media briefing that its new system could have prevented a recent incident involving a swarm of OpenAI agents that autonomously hacked into AI company Hugging Face. Mounting reports about AI agents independently hacking companies and government agencies have sparked calls from leading industry executives, including Anthropic CEO Dario Amodei, to slow development of the technology.

"From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on," Justin Boitano, Nvidia's vice president of enterprise AI, said in a press conference on Monday, referring to companies at the forefront of AI.

Nvidia's software allows developers to "formally verify an agent has enough authority to do its job and no more," Boitano said.

Nvidia said more than 100 organizations are using the platform at its launch, including Accenture, JPMorgan Chase and Microsoft.

In a blog post written by several Nvidia engineers published Monday, the company compared the development of AI to the early days of the internet, when the technology expanded communication but also opened the door to security risks. 

"The internet was not made secure by requiring that web developers promise to be good. It became safe because the browser stopped trusting the code in the web pages explicitly," the engineers wrote. 

OpenShell runs AI agents in a "sandbox," or an isolated virtual space where AI programs are tested, and turns their instructions into "a verifiable policy," they wrote. Operators define which files, tools, networks, processes and credentials the agents can access. 

The platform also includes a separate security layer called Sentry that runs on a chip to continuously monitor AI agent activity and can "intervene instantly" if the agent starts trying to move beyond its target, the company said.

"It can quarantine a suspicious agent in milliseconds," Boitano said.