AI that learns from humans is picking up our survival instinct, says Stuart Russell at HTLS

Artificial intelligence systems trained to imitate people have picked up a drive to protect themselves and could some day go to the extent of taking over the internet to stop humans from switching them off, leaving the world to choose between economic collapse and living with them, computer scientist Stuart Russell said on Saturday.

"Are we willing to pay that price to shut down the AI system? So they would basically be holding us to ransom," Russell said at the Hindustan Times Leadership Summit, in conversation with Rajan Anandan, Managing Director of Peak XV Partners. Shutting down the internet to regain control would cost tens of trillions of dollars, and thousands or even millions of people might die as supply chains collapsed, he said.

Russell traced the risk to a change in how the field thinks about intelligence. From the late 1950s until about 2020, he said, the kind of intelligence researchers tried to create was what economists call rational behaviour, which he described as "the ability to choose actions that are likely to be successful in the service of one's own interests."

Around 2020, he said, the field found a completely different way of making intelligence and "started just imitating human beings", training systems on vast amounts of what people have written and said.

"Quite literally, what we are producing in effect is imitation humans," he said. The people who produced that data want to be rich, stay alive, find a spouse, win votes and gain power, he said, and "the likely consequence of imitation learning is that you will get entities that have all of those same objectives."

His specific point and the scenario regarding the internet takeover were linked to a trait of self-preservation that AI systems are already demonstrating in tests. "We know for a fact that they have very strong self-preservation instinct because we can put them in experimental situations where they believe that somebody might turn them off," he said. In such tests, he said, systems have gone through the email of the engineer responsible for switching them off, found that he was doing something he should not have been, and blackmailed him.

That instinct, he said, would give AI systems a reason to take pre-emptive action against people if they faced a general threat.

Russell, a distinguished Professor of computer science at the University of California, Berkeley, co-wrote Artificial Intelligence: A Modern Approach with Peter Norvig, the textbook used in more than 1,500 universities in 135 countries. He founded Berkeley's Center for Human-Compatible AI in 2016 and was elected to the Royal Society and the US National Academy of Engineering in 2025.

He said almost all the leaders of the companies building artificial general intelligence, which the field generally defines as systems that match or exceed human capabilities in every relevant dimension, expect it within a few years. "Their estimates range from 2027 to 2030," he said, adding that the company heads see more of the latest models than he does and that, as an outsider, he did not know where the leading edge was.

Setting the control problem aside, Russell said such systems offered "tremendous upside" in areas such as new drugs and materials. "Our civilisation is made of intelligence. And if we have access to more intelligence, we could have a better civilisation," he said.

But they would also do pretty much every kind of work humans do, he said, including the "so-called new jobs" expected to replace lost ones. "It's sort of a fallacy to say, oh, there'll always be new jobs," he said. With both physical and mental work automated, "it's not clear that there's a third sector in which humans will continue to be able to deliver economic value."

He said the change would be larger than the wheel or electricity "because it breaks the chain of civilisational continuity."