OpenAI builds a kill switch for its research models

How much damage can a rogue agent do in 35 seconds? Asking cos that’s how long it might take to shut one down

In 2019, OpenAI decided not to publicly release a new model because its research uncovered “concerns about malicious applications of the technology.” And on Tuesday, the company announced it has built a kill switch for the models deployed in its research labs, months after some of them displayed disturbing behavior.

“As our models get more powerful and the associated safety and security risks become bigger, it becomes more and more critical that we can detect potential issues, and quickly stop models in their tracks when needed,” OpenAI explains in a Tuesday post.

The company named the kill switch “Control Room” and described it as “a central system to track model workloads across our research infrastructure and stop them quickly when safety or security concerns arise.”

REG AD

The facility can apparently stop models from producing any output 35 seconds after a decision to pull the plug – and can go faster.

REG AD

“The 35-second lag comes from a caching layer for inference authorization, to support the required volume of traffic. We are working to drive this number down to zero so that we can end jobs instantly,” the company promises.

A rookie mistake in critical infrastructure design is to build both production and resilience tools on shared resources that could be taken offline by a single incident. OpenAI seems conscious of that risk, because it says the tool “has been built to keep working even when other systems fail, and to stop model output if Control Room itself goes down.”

OpenAI hasn’t explained why it took so long to build a kill switch or why it decided now is the moment when it needs one.

The Register fancies recent examples of agents exceeding their brief in worrying ways – like attacking Hugging Face, causing trouble at the Wikimedia Foundation, or going a little too deep when researching Australian healthcare data – may have influenced the timing of this development.

“Control Room is already deployed across our research infrastructure, where the need is greatest,” the company’s announcement states. “Our external model-serving stack is already much more centralized by nature, with a well-developed operational capability to quickly take down model deployments if needed.”

OpenAI says it plans to “strengthen and expand the system over the coming months, alongside our investments in monitoring for misaligned behavior and security breaches.”

That’s a variant of Big Tech’s usual “We’re sorry and we promise to do better in future” proclamation that follows the latest SNAFU. Yet back in 2019, OpenAI pulled the pin before matters ever reached that state. ®