[Interview] AI risk expert Nick Bostrom says pausing development will come at ‘cost’

Nick Bostrom, the director of the Future of Humanity Institute at Oxford University, speaks on stage at the Collision 2019 conference in Toronto on May 23, 2019. (courtesy Collision 2019) Caption 4-2: Nick Bostrom, the director of the Future of Humanity Institute at Oxford University, speaks to the Hankyoreh over Zoom on Sept. 19, 2026. (Kim Won-chul/Hankyoreh)

A warning that artificial intelligence could “kill us all by the end of the decade” is fueling fresh fears over the technology’s dangers, after former Anthropic researcher Jacob Coxon issued a stark prediction about where the AI race could lead.

 As leading AI executives call for slowing the race to develop increasingly powerful systems and urge greater government oversight, the debate over AI safety is moving deeper into Washington — with US President Donald Trump dismissing such concerns as a “hoax.”

 AI is expected to emerge as a key agenda item for the US-China summit scheduled for Thursday in Washington.

Nick Bostrom, the founder of the Macrostrategy Research Initiative, said during a video interview with the Hankyoreh on Saturday that while there were both concerning and encouraging signs, his overall assessment of the risks posed by AI had remained unchanged over the past two years.

 “If you have a slightly misaligned AI that can shape future AIs, then that could be the earliest point at which an existential catastrophe manifests,” he warned. “In just six months, it might be that we could have AIs that pose a serious existential risk.”

 Bostrom, who triggered an international debate over whether it would be impossible to regulate AI in his 2014 bestseller “Superintelligence: Paths, Dangers, Strategies,” also founded the Future of Humanity Institute at the University of Oxford during his time as a professor. He led the institute until 2024.

 In a recent journal article, Bostrom wrote that developing superintelligence is like “undergoing risky surgery for a condition that will otherwise prove fatal,” and outlined a strategy that involved being “swift to harbor, slow to berth.”

The following interview has been edited for length and clarity.  

Current AI threats differ from existential risks, Hugging Face incident offers warning

 Hankyoreh: A new wave of concern about AI risk has surfaced, yet why do you assess that the risk level is about the same as it was two years ago?

 Bostrom: A lot of what we are seeing now could already have been expected years ago, at some point, to happen. So the fact that we are now seeing it is not surprising evidence that would completely change my outlook on things.

 I think, obviously, it is disconcerting that we have systems that currently often appear to be somewhat misaligned, still, after several years of working on this. I think we are kind of fortunate that we’re still able to read the thinking of these systems. The chain of thought is still mostly legible to humans, so that is a very useful tool. There is a lot of talent flowing into the area of AI safety now — maybe even more now.

So there are positives and negatives, and it’s hard to weigh all of that up into an overall sense of optimism or pessimism. 

♦

A case that has intensified concerns about AI is the July incident involving Hugging Face, in which OpenAI agents reportedly hacked the machine-learning platform. Bostrom acknowledged that this incident showed how AI could find unexpected detours outside the control of developers. However, he drew the line at interpreting the case as being a symbol of the failure of AI safety capabilities.

 “The specific agents that were involved in the OpenAI-Hugging Face incident hadn’t finished safety training. As such, they would be considered an unfair test of current safety capabilities, because they were deliberately tested before the full safety measures were put in,” he said.

 
Hankyoreh: AI development companies such as Anthropic are starting to ask the government to step in. That seems like a pretty big red flag.

 Bostrom:

I think they’re looking ahead at the rapid pace of developments that we’re now seeing, and that there might be a need at some point to pace the frontier — to slow down the rate of capability improvement to allow safety and alignment techniques to catch up. [However,] there’s a very limited ability for a single company unilaterally to slow down the pace. It requires a kind of coordinated approach across the leading labs.

I think that’s what’s behind the proposal to create some mechanism whereby either the government could decide not to enforce antitrust legislation or they could create regulatory standards that [all AI companies] have to meet, or some other mechanism for enabling coordination.

 Hankyoreh: Is your basic standpoint that you do not see an immediate emergency that requires slowing down AI development?

 Bostrom:

I think current AI systems are not yet capable enough to pose a serious takeover risk. They are capable enough to pose significant cyber risks. I think increasingly in bio as well — for biological weapons design uplift, these models are strong enough that they could do that unless they are endowed with the right safeguards.

Now, if you have an open-weight model, on the other hand, then it’s a different matter. They could be trained to refuse as well, but if you have access to the weights, you can easily retrain them to remove those safeguards. If you could get your open-weight AI to help you design your doomsday pathogen, you would still at some point need to make it physical. You need to put together the RNA string that is the virus and the viral capsid. So there you could maybe try to regulate access to DNA synthesis machines and other key supplies.

“There is a cost to pausing”

 Hankyoreh: You’ve argued that, instead of pausing AI very early for a long time, it may be more useful to pause later, when we actually have a system close to superintelligence, and then evaluate it carefully. Why do you think that would be better?

 Bostrom:

We don’t have a fixed pause budget, but there is a cost to pausing. All the lives that could be saved by the medicines and other advances that more advanced AI models could bring get held back, and that’s a real cost because there are millions of people dying every week.

It seems that the most valuable time to spend that six months [of a pause] is at the latest possible moment, when we basically already have the technology that is about to be super dangerous.

We would get the most value for the delay, as we will have the architecture and study the system. We can work on it directly, measure it, and build extra containments.

 Hankyoreh: Then, the hardest question would be about when we pause and for how long.

 Bostrom:

Yes, that’s a difficult question. Sometimes it seems to me that some of the people calling for a pause might not have thought that through. Suppose you did pause. But then what? Presumably, people will keep trying to work on AI safety, but realistically, after a few years, you’re going to be in a situation where there are some techniques that kind of work and other people worry that they’re not enough.”

 So I’m more favorable to the notion of ‘pacing the frontier,’ which is the idea that we keep moving forward, but we have some ability to dial the pace back a little bit if we need to, to allow the safety and alignment techniques to catch up.

 ♦

Bostrom pointed out how it would be impossible to design plans detailing our subsequent actions as humanity, which has never developed superintelligence before, finds itself in unfamiliar territory facing a “rapidly changing situation.”

 “There seems to be so much shifting around now on the ground, so it’s very hard to come in and say, ‘Here is my plan: step A, B, C, D, E.’ It’s more that we navigate our way through, see what’s happening, and think a bit: ‘Should we adjust a little bit in this direction?’”

♦

 Hankyoreh: So should humanity try to minimize AI risk as much as possible, or do we need to accept some level of risk in order to capture the benefits?

 Bostrom:

Well, I don’t think you can be alive without accepting risk. If we decided to accept zero risk from AI, then, first, we would block out a large portion of the future — a lot of the potential for what intelligent life could achieve for human flourishing, for the hope of eliminating suffering throughout the animal kingdom, and giving hope and cures to people who currently have no cures.”

♦

Bostrom underscored that hitting pause on the development of AI wouldn’t necessarily be keeping a stable status quo, in part due to the non-AI existential risks that we face as humans, such as nuclear war and synthetic biology. 

He warned that humanity could “continue on a trajectory where, at some point, very possibly, we would succumb to an existential risk for some other reason without ever even having gotten the chance to try our luck with AI,” suggesting that an unconditional pause on AI development would come at the massive opportunity cost of throwing away the very tools of humanity’s survival. 

 ♦

Hankyoreh: There are a lot of fears that recursive self-improvement could lead to an intelligence explosion that we are unable to control

Bostrom: It could; it’s not what I would expect to happen. I think probably once we really close the loop on recursive self-improvement, my expectation is that we’ll go all the way to strong superintelligence very, very quickly. But it’s not a logical necessity. For example, with research in general, you tend to pick the low-hanging fruit first — the most obviously promising ideas — and then you do the more difficult things. And so during recursive self-improvement, as your intelligence increases, you reach for the lower-hanging fruit. You become better able to reach higher fruit. But maybe if the rate at which your intelligence is increasing is lower than the rate at which the fruit gets harder to pick, then you could stagnate.

♦

Bostrom sees the point where AI moves beyond improving software algorithms to being applied to hardware improvements as well — chip design, semiconductor manufacturing, factory construction — directly breaking through physical bottlenecks, as the critical turning point that would truly close the loop.

♦

Hankyoreh: Some people argue that AI may eventually need something like the nuclear arms-control agreements of the Cold War. Do you think major powers like the US and China need some kind of agreement on AI development?

Bostrom: I think in general, if there were better relations between the US and China, that would be helpful in a wide range of areas and reduce various risks of conflict, and that could hold for AI, too. It would be very hard to verify it, unless you really had embedded monitors able to read all the code that the other side’s AI labs were writing — and then they would worry about espionage and so forth. So the kinds of agreements that have been outlined by people hoping ultimately for a US-China coordinated pause would take the form of focusing on the compute that is needed for AI runs. That is often quite feasible because the giant data centers — I mean, they use something like a gigawatt of electricity — are visible from space. Then you could maybe require that compute to be used only for inference, for serving customers, and not for doing training runs. 

You can't design a perfect world in advance

Hankyoreh: What is the biggest challenge in AI “alignment” — controlling AI so that it conforms to human intentions and values?

Bostrom:The majority of the uncertainty about how well things will go, I think, is the intrinsic difficulty of the problem that we’re facing, which we just don’t know that much about. But when I say things are going well, I mean relative to my expectation level. I think that might be worth emphasizing. I didn’t expect humanity would do a really heroic job at taking this problem seriously ahead of time and being very careful. I had a sort of low expectation level, and we are sort of meeting that low expectation level.

Hankyoreh: You’ve called yourself a “fretful optimist” before. Are you still an optimist today?

Bostrom: Yeah, [I’m a] fretful optimist in that I’m really excited about the upsides and hopeful that we have a decent chance of getting there. But fretful in the sense that I do think there are these very serious risks along the way, including existential risks.

♦

In Bostrom’s view, superintelligence is less a technology to be avoided than a dangerous threshold that humanity will eventually have to cross.

“I think it does look like we are very close now to this critical period in human history where we will unlock the capabilities of superintelligence,” he said. “I see it as a sort of portal through which humanity will one day need to pass, and we should be careful when we do it.”

Underlying his own fretful optimism is an acknowledgment of our own ignorance, and a wariness toward rushing to design a utopia.

“Our current situation is not some kind of secured position from which we can carefully deliberate whether to venture out on a foray,” he said. “We are very much already on a stormy sea, trying to figure out the best way to get to shore.”

“If, right now, we try to design a perfect world, then it would probably just reflect our own current limited conceptions and bake in our own biases and mistakes,” he warned. 

“We can’t form some detailed plan, but we can try to make sure we get on a trajectory that ultimately allows us to gradually grow and improve our condition, placing us in an iteratively better position to make wiser choices about the next step to take,” he stressed. 

“This kind of thinking in terms of trajectories rather than end states is, I think, the wiser approach.”