‘Entirely false’ AI-generated intelligence report ‘almost started a war’ with China - report

As the US military ramps up plans to implement artificial intelligence, officials warn that mishaps could lead to “catastrophic miscalculations.”

An intelligence report disclosing that a Chinese ship in the Middle East was transporting components of a nuclear weapons program immediately set off alarm bells across the US military before it was determined that the intelligence was false, a Friday CNN report revealed. 

The US military immediately made plans to intercept the vessel, CNN said, citing four sources familiar with the matter. Two of the sources said that armed members of the US military were planning to board the ship, and military planes were in the air as well, according to the two other sources.

Before the planned operation, however, a special operations command analyst found that the report had been generated with artificial intelligence. A chatbot used by the analyst had inaccurately identified the material in the ship, according to CNN. 

CNN said the news outlet was unable to learn what the cargo actually was.

The report was “entirely false,” CNN cited one of the sources as saying, but “almost started a war,” as a US operation against a Chinese ship would have potentially sparked an armed conflict between the two countries.

The incident occurred as US President Donald Trump awaits Chinese President Xi Jinping’s visit to Washington on September 24. 

The leaders are expected to discuss the use of artificial intelligence as experts urge both governments to consider rising concerns tied to its increased usage.

Officials in the US military and intelligence community are pushing to integrate AI into “nearly every facet of their work,” said CNN’s report, “from analyzing the huge volumes of raw intelligence the US collects and selecting targets for strikes, to more mundane applications like managing budgeting, logistics and supply chains.”

Hegseth pushes for increased AI usage

The rationale for AI’s rapid adoption is that it can facilitate and quicken battlefield decisions, like which targets to strike or which military assets to move, according to CNN. The US can’t afford to fall behind in the technology if it must one day face China or another adversary that could potentially be one step ahead, according to officials.

US Defense Secretary Pete Hegseth released an “Artificial Intelligence Acceleration Strategy” in January to speed up the military’s use of AI, CNN reported.

“We will unleash experimentation, eliminate bureaucratic barriers, focus our investments, and demonstrate the execution approach needed to ensure we lead in military AI,” he said during a speech.

The strategy also aims to democratize “AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels,” CNN cited a memo announcing the strategy as saying.

The efforts, however, are decentralized, with multiple US officials claiming that different parts of the government are using different tools under different orders and safety standards, according to CNN’s report.

The episode with the Chinese ship underscores the profound risks of using AI. Analysts have long feared that the use of this new and relatively poorly understood technology could lead to a “catastrophic miscalculation” if countries are relying on poor or corrupted data, said CNN.

In this case, the analyst asked an AI chatbot about an intelligence report that originated from the Hawaii-based US Special Operations Command Pacific, according to CNN’s report. It wasn’t clear whether the chatbot was commercially available or a US government product.

The bot, according to CNN, fused together open-source intelligence with secret signals intelligence in government holdings to reach the conclusion. The analyst then used AI to create an intelligence report for military officials.

US military’s AI is ‘commercial stuff wearing lipstick,’ official says

A former senior US official said that the US military and intelligence analysts’ internal tools are “mostly just copies of the commercial stuff wearing lipstick,” according to CNN.

Another source said that “AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide, the report added.

The chatbot’s “hallucination” was not an isolated incident across the intelligence community since these tools began proliferating across government, according to one of CNN’s sources.

The report added that the rapid implementation of AI has increased pressure on analysts to “produce and disseminate intelligence faster, opening the door for mistakes.” And young analysts are more likely to trust chatbots “uncritically.”

“AI allows you to get to a bad idea faster,” one of the sources said.

US Special Operations Command Pacific and the Pentagon did not respond to CNN’s request for comment.