AI’s greatest threat is biological, not robotic

Anthropic researcher Jacob Coxon recently quit and posted on social media that it was because “the people building AI earnestly believe that it could kill us all by the end of the decade.”

Evan Hubinger, who leads the artificial intelligence company’s alignment science team, put the odds above 10% in the next 10 years. Anthropic Chief Executive Officer Dario Amodei called for a slowdown in the AI frontier to help mitigate the risks, an idea echoed by OpenAI CEO Sam Altman and xAI CEO Elon Musk.

You might be imagining doom from killer robots with thick Austrian accents. But the greatest risk comes from cells, not steel. An Institute for Security and Technology survey of 111 national security practitioners found that 70% believe AI meaningfully increases — or soon will — bioweapon risk. Imagine an AI-aided researcher inserting an immune-signaling gene into a poxvirus that suppresses the immune response so thoroughly it kills every unvaccinated animal it infects and most who are immunized.

That’s a nightmare. But it’s not science fiction. It happened more than 25 years ago without any help from AI. Mousepox with the gene for interleukin-4 did exactly that to mice, and it was described in a paper published in the Journal of Virology in February 2001. The Australian researchers who built it weren’t even trying to kill mice; they were just trying to limit mouse reproduction.

The natural world provides aspiring terrorists with plenty of pathogens. Making them worse is something that we can already do, albeit with some difficulty. AI might make the process easier, but it’s never been the primary roadblock. The hard part was taking a virus’s genome (natural or engineered) and turning it into something dangerous. That still needs a lab and trained scientists. And the skills one would need aren’t easily learned from textbooks. They’re what’s called “tacit knowledge,” which has long been thought to require experience and apprenticeship.

In 1917, France asked America to build copies of its 75mm field gun. Even with complete plans, the United States was unable to produce a single copy in the 18 months before the war ended. That’s because the drawings never described the screw threads on the gun carriage, and when Americans measured an actual gun they found six types, none of them used in this instance. As military historian H.A. De Weerd wrote, “the only secret about the French recoil system was the difficulty of its construction.” That’s tacit knowledge.

Over time, tacit knowledge often becomes less important. A factory today could make those guns from emailed files. Although tacit knowledge remains an important safeguard, that process is just beginning in biology. I’ve been concerned about its effects on biological risk since long before AI was a thing. In 2009 I wrote a paper with MIT’s Kenneth Oye and Boston University’s Scott Mohr arguing that synthetic biology was an attempt to minimize the importance of tacit knowledge in bioengineering, and that it could increase biological threats.

AI may have far more impact on the importance of tacit knowledge than synthetic biology ever did. The biologist Roger Brent and RAND’s Greg McKelvey, Jr. argue that we should retire the assumption that tacit knowledge is even required. They showed that foundation models available in 2024 provided “accurate instructions and guidance for recovering a live poliovirus from a construct built from commercially obtained synthetic DNA.”

Venture capital firms are also investing in “automated labs” which could, at least in theory, eliminate much of the human skill element in experimental biology and rapidly increase the rate at which experiments can be conducted. That’s great news for defenders, and for anyone who wants medical science to progress more quickly. But without guardrails, it could also be great news for attackers.

One key safeguard is to mandate that any sequence of DNA which is synthesized must be screened for possible use in a pathogen. Today most synthesized DNA is bought from companies, but desktop DNA synthesizers are on the market. Many, but not all, DNA synthesis companies already screen, but a Microsoft-led team showed that AI protein design tools can already produce sequences the screening software misses, though whether those sequences would still work is unproven.

The federal government currently requests, but does not mandate, screening, and a revised framework has been in limbo for the 16 months since an executive order called for it. It should mandate screening, and DNA screening companies and the frontier AI labs should pool their resources to create bounties to reward anyone who beats AI-based DNA screening. And as automated labs proliferate, they should only work with verified researchers from accredited research institutions and be subject to the same screening requirements.

These are no-lose policies. They’d be a good idea even without AI, and they’re a better one with it. Implementing them would be much easier with federal help. Unfortunately, instead of chipping in, the White House has opposed AI regulation, with U.S. President Donald Trump saying that “AI taking over the World, destroying Humanity, and all other things bad, is a HOAX” and that “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!”

Just in case that High-IQ President needs some expert advice, it might have been helpful to have it institutionalized within the government instead of dissolving the National Counterproliferation and Biosecurity Center and Homeland Security’s counter-WMD office. The administration even suspended updated screening frameworks that would have closed the loophole letting novel agents through.

Until the government decides to step in, leaders in the private sector will need to fill the gap. Anthropic has already reported blocking five attempts to use Claude for research that could support biological weapons development, and says it can no longer assume its newest models fall below the threshold for meaningful help in building them. Even a minor attack using a biological agent that AI helped develop will surely result in industry-stifling regulation. We should make sure we don’t end up like those Australian mice.

[bio]Gautam Mukunda writes about corporate management and innovation. He teaches leadership at the Yale School of Management and is the author of “Indispensable: When Leaders Really Matter.“[bio]