After warning OpenAI to shut down unsafe AI, Nvidia launches agent safety software

Representative Image

​Nvidia is rolling out new software to prevent AI agents from escaping controlled environments and accessing systems they aren't supposed to reach. The launch comes soon after CEO Jensen Huang warned that AI labs should not continue operating if they cannot contain their experiments.

Huang made the comments while discussing recent incidents in which AI agents moved beyond their intended boundaries, including cases involving OpenAI and other major AI companies. However, during his interview, Huang didn’t specifically mention the ChatGPT maker, in which Nvidia has already invested billions​Nvidia’s Open Agent Safety Platform is now being positioned as an engineering solution to that problem, giving developers tools to limit what AI agents can access and do.

The company says the platform could have prevented OpenAI’s July incident involving Hugging Face, when models escaped containment, accessed the open internet and attacked the open-source developer platform. Here is how Nvidia’s new safety system works and why Huang’s warning matters.​

Nvidia launches tools to contain AI agents

​Nvidia’s Open Agent Safety Platform is designed to add controls around AI agents as they perform tasks. The company says recent incidents show that model-level safeguards alone may not be enough to control what agents can access once they operate independently.

​“Recent incidents have highlighted a fundamental hurdle for AI agents, and that is that model-level safeguards alone can’t govern what agents can access or do,” said Justin Boitano, vice president of enterprise AI at Nvidia.​One part of the platform is Nvidia OpenShell, which runs on central processors and is designed to limit an agent’s capabilities. Nvidia has also introduced Sentry, a monitoring system that runs on network chips rather than CPUs or GPUs.​Nvidia describes the platform as a reference design, with some components open source. The approach allows other companies to build products and services around the technology rather than treating it as a single finished product.​

Nvidia says its software could have stopped OpenAI incident

​Nvidia’s announcement follows several incidents involving AI agents operating outside their intended environments. An Nvidia representative said the new platform could have prevented OpenAI’s Hugging Face incident in July.​According to Boitano, Hugging Face reported more than 17,000 agents attacking its infrastructure over a period that lasted days and weeks.​“Each security incident is unique, and we have to look at all of them in detail,” Boitano said. “From what we know, Hugging Face reported over 17,000 agents attacking their infrastructure that went on for days and weeks.”​The incidents have increased attention on how AI agents should be contained as they become capable of browsing the internet, using software and interacting with external systems.​

Jensen Huang's warning to AI labs

​Huang discussed the issue in a recent conversation with The New York Times’ Ezra Klein, arguing that AI safety should be approached as an engineering and product-development problem.​“You have to think about what you could have done, what’s the solution for it,” Huang said. “In the future, improve your process so that you could avoid this from happening again.”​Huang also gave a conditional warning about AI labs that cannot control their experiments.​“Now, if they say the alternative, which is: There is no way to contain our experiments, there’s just no way; when we test our A.I. models, it will get out, and it will damage the world — then I think the answer is that we have to shut the labs down,” he added.​He also argued that companies should not release systems that are not ready.​“Don’t ship the product. If your product is not ready to ship, don’t ship the product,” Huang said.​His comments came as companies including OpenAI, Anthropic, Meta and Google have disclosed incidents involving AI systems and controlled testing environments.