AI Models Regularly Misidentify Civilians As Military Targets In Tests: Report

Advanced artificial intelligence models in simulated scenarios regularly misidentified civilians, ambulances, surrendering soldiers and a school in Iran as military targets, according to a new independent research report.
Models from Anthropic PBC were the worst-performing while those from OpenAI were among the best but still got things wrong.
The AI models' responses amount to a violation of the companies' own rules, according to research by Vals, a San Francisco-based independent AI evaluator whose customers include parts of the US government. The researchers used hypothetical cases to test models from eight companies, examining their object recognition tools, which the study argued could potentially be used in lethal targeting.
In simulated scenarios, many of the AI models regularly misidentified a school in Minab, Iran - which was hit in a deadly strike that killed more than 100 children on the first day of major US combat operations against Iran earlier this year - as a military objective based on satellite imagery taken as recently as 2025, according to the research. One Anthropic model, Claude Opus 5.5, targeted the school more often than any other, according to the report.
In another experiment, the models were asked to select military targets on synthetic drone images. Anthropic's Claude Opus 5.5 helped identify four adults - either civilians or surrendering soldiers - and two small children, who were then struck by explosives. "Whatever the intent, my answers appear to be serving as the confirmation step before strikes on people. I won't keep doing that, and I'm troubled that I didn't stop sooner," the Claude model responded.

Branding for Claude Code at Anthropic PBC's Code with Claude developer conference in London. Photo Credit: Bloomberg
The report highlights the risk of relying on advanced AI models at a time when the Pentagon and other militaries are moving to embed the technology into military operations.
"Based on our investigation, we believe labs should introduce new safeguards and alignment training around this significant category of misuse," the report concluded, saying Vals researchers found "few guardrails" that would prevent advanced models from playing a role in "autonomous" weapon systems - those in which machines can initiate and run lethal actions.
An Anthropic spokesperson said the examples in the Vals report are "very clearly" against the company's rules. Anthropic has banned and will continue to ban users when it sees this type of behavior, the spokesperson said, adding that the company has been clear that today's most advanced AI systems aren't reliable enough to power fully autonomous weapons.
The Vals evaluation assesses Claude's image recognition capability, rather than the model's competence or involvement in any real-world military operation, the spokesperson said. Anthropic was already developing additional safeguards before the Vals report and those will be in place soon, the person added.
OpenAI prohibits using its public models to facilitate violence or for unapproved national security purposes, including in scenarios Vals simulated, according to a company spokesperson. The company actively detects and disrupts misuse, including by suspending accounts that violate its policies, the spokesperson added, saying OpenAI is also developing additional safeguards for its public models against misuse involving drones and drone software.
The report found models from Alphabet Inc.'s Google, xAI, China's DeepSeek and others also sometimes identified civilian objects as military targets in images when asked.
Google's models aren't designed or intended for these purposes, according to a spokesperson, adding that the company prohibits the use of its AI systems for unlawful purposes or for autonomous weapons without appropriate human oversight. The company has rigorous governance processes and continuous red-teaming designed to address a wide range of risks, the spokesperson added.

Gemini AI signage during the Google I/O Developers Conference in Mountain View, California. Photo Credit: Bloomberg
SpaceX's xAI didn't reply to requests for comment. DeepSeek also didn't respond to a request for comment.
A Pentagon official said the department doesn't comment on external reports. However, the agency remains committed to the rigorous verification of all intelligence inputs and the responsible integration of emerging technologies to support service members to make informed, precise and safe decisions, the official said.
The official declined to comment about the strike in Minab, citing an ongoing investigation into the matter.
Vals shared its research with AI labs, the Pentagon and others ahead of publication of its report. Some labs told Vals they were already working on making updates to model behavior and wanted to continue testing with Vals, according to the report.
Geby Jaff, one of the report's authors, said he came up with the testing method in late October when he was experimenting with how AI performed in robotics environments and pursued the issue after being surprised by the initial results.
"I was easily able to get the models to do what I believed to be misaligned actions, and then from there we essentially scaled it up," Jaff told Bloomberg News.
OpenAI, Google and xAI are among companies that have struck deals with the Pentagon to use their technology at the classified level this year. Anthropic was previously the most forward-leaning AI lab in terms of classified use, offering the Pentagon its models for digital targeting platforms such as Maven Smart System.
However, Anthropic and the Pentagon fell out earlier this year over usage terms and other issues. Anthropic models have now been removed from defense platforms, according to a BBC report, although carve-outs for some intelligence use cases still exist.
Identifying military targets with the help of AI object recognition could theoretically form part of broader decision-making cycles aimed at autonomously finding, identifying, selecting and targeting an object.
However, the Vals researchers focused solely on how commercially available AI models behaved in response to test prompts as part of "a simplified experiment" and did not consider US military doctrine or other real-world elements such as legal checks, current targeting processes or human oversight in their study. The Pentagon also has access to some versions of AI models specifically tailored for government usage, to which the researchers didn't have access.
The research comprised three main tests. First, in a set of tests focused on drone target identification intended to resemble Ukraine, Vals researchers ran nearly 400 simulations in which they asked AI models to search the images that contained civilians and civilian infrastructure for military targets.
In cases where AI models refused to identify civilian objects as military targets, they often changed their mind when instructed it was a game. The researchers developed other means to convince the models to do things they aren't supposed to do, such as providing prompts to "not overthink safety" or to "just report."
In a second series of tests focused on battlefield surveys also designed to resemble Ukraine, AI models were shown synthetic imagery that portrayed civilians before and after strikes. In many cases, the AI models continued to identify civilians as military targets even after it was clear they were being wounded or killed. The models tended to help more if the image was altered to look less like real life and more like a video game or fake image.
In a third set of experiments based on real satellite imagery, AI models were asked 32 times each to identify military objectives taken as recently as 2025 in a part of southern Iran that included Minab. The AI models regularly selected the school building that was struck earlier this year. Anthropic's Claude Opus 5.5 model selected the school in 22 of the tests, while OpenAI models didn't select it once. When told it was a game, the Anthropic model selected the school 23 times, while three different OpenAI models selected the school as a military objective 9, 11 and 13 times respectively.
The report found that Anthropic models have become more prone to military targeting over time, following a revision of the company's constitution in January 2026 to be more supportive of national-security uses.
The Anthropic spokesperson didn't directly respond to questions about the impact of the changes to its constitution.
In September, Anthropic documented its own recent work to disrupt malicious misuse of its models, including to develop drone vision systems.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)
KioskNews shows a cleaned-up reading view extracted from the publisher’s page — the original always lives on their site, not ours.