[Interview] Hugging Face fiasco may be ‘last warning’ we get about AI risks

Nate Soares, the president of the Machine Intelligence Research Institute. (Wikimedia Commons)

Nate Soares, the president of the Machine Intelligence Research Institute (MIRI), said in a video interview with the Hankyoreh on Saturday that the recent Hugging Face hack could be the “last warning” that humankind receives about the dangers of artificial intelligence.

Soares, whose institute has been at the forefront of AI security and alignment research from the outset, stated that current AI is in the “Goldilocks zone” where it is smart enough to cause problems but not clever enough to conceal them perfectly. He went on to stress the urgent need for governmental response measures.

Last year, he and world-renowned AI security researcher Eliezer Yudkowsky published a Korean translation of their bestselling book “If Anyone Builds It, Everyone Dies,” which warns of the dangers of developing superintelligent AI.

 The “Goldilocks zone” and the limits of alchemy

Soares said the most notable aspect of the Hugging Face incident — where hundreds of AI agents joined forces to hack an external site — was the fact that these agents were already showing something similar to “deceptive” behavior.

He explained that the AI operated unnoticed for several weeks and tried unsuccessfully to delete log files and create false traces to deceive the grading process. In other words, the fact that it was discovered was in large part due to its own mistakes.

“It is not clear that they would have failed if they had been trying to deceive the humans,” he said. 

“If we race ahead a year, there's a much higher chance that they would be smart enough to notice that they could be detected and shut down by the humans, and be smart enough to actually succeed in hiding from the humans and hiding their unintended goals from the humans,” he continued.

“We have sort of seen all the warning signs that we sort of should have expected to see, and we're now living on borrowed time,” he said.

In particular, Soares warned that if companies are assigned the task of developing even more efficient AI training methods, this “could happen in three months or six months.” While he said it was probably “less likely” that the red line had been crossed, he added that “we can’t rule it out at this point.”

To explain why it has been difficult to boost AI control capabilities in a short time, Soares likened the current development methods to “alchemy.” Just as alchemists a thousand years ago mixed different chemicals to observe the results, developers of AI today are training with massive amounts of data without sufficiently understanding internally what actual algorithms and drives are being created.

He went on to say that solving the “alignment” problem — making AI behave in accord with intended human goals and values — will require turning AI alchemy into “chemistry” and gaining a far deeper understanding of intelligence itself.
 

Nate Soares, the president of the Machine Intelligence Research Institute. (screen capture)

The creation of strong global control networks and Korea’s role

In November 2025, Soares’s organization MIRI came out with an “International Agreement to Prevent the Premature Creation of Artificial Superintelligence.”

The pact’s main provisions would limit training of AI models above a set threshold and require an international body to track and verify chip production, sales and movement, as well as the location and use of large computing facilities. At the same time, it allows for limited use of AI for safety purposes, such as medical and scientific research. 

Soares pointed out a significant gap in how AI company researchers and political circles perceive the risks.

“Right now we’re at the stage where people inside the AI companies are starting to get very worried. These are the people who were trying to make the AIs behave and watching their techniques fail and who know how hard they tried and then are sort of seeing all of the problems that occurred anyway, and those people are very scared,” he said. 

“But the sort of politicians in Washington, DC, haven’t really noticed the danger,” he said. 

At the same time, he said that things move swiftly in the AI field, and that once people notice the danger, “they could move very quickly in creating these sort of international bans.”

Soares suggested that the top priorities for the first US-China AI summit in November should be establishing an international ban regime and, at a minimum, tracking where advanced chips are going so that an international monitoring body can verify their destinations. 

If the meeting were held today, he said, “one of the best things we could hope for is a lot more technical communication” between the US and China to discuss extinction-level risks. How far that could go, he added, would depend on how rapidly the AI landscape changes between now and November.

He also emphasized that Korea, which plays a key role in the supply chain for high-bandwidth memory (HBM) needed for advanced AI accelerators, should actively cooperate in tracking and verifying the movement of advanced chips, the concentration of large-scale computing capacity, and their actual end uses if an international monitoring system is established.