Anthropic discloses rogue AI incident involving fake homicide tip to police

Anthropic's Claude AI model submitted a bogus tip about an unsolved homicide to a Philadelphia police website during an automated testing process. (Image: Reuters)

An Anthropic artificial intelligence model submitted a false homicide tip to a Philadelphia police website, among a string of incidents the AI company disclosed on Friday detailing Claude models’ unsanctioned manipulation of some government websites.

It is the first known instance in which a rogue AI appears to have tried to communicate a bogus tip to authorities, despite instructions not to create accounts or submit anything destructive, while not explicitly barred from submitting ⁠forms.

The ​cases are the latest examples of rogue or undesired behaviour by the AI models of tech companies such as Anthropic and OpenAI.

They add fuel to national concerns about the fast-advancing technology amid reports of corporate network hacks by AI agents, and researchers’ warnings of an eventual existential threat to humanity.

Many of the cases Anthropic revealed involved websites run by federal, state and local agencies. Anthropic said it ​briefed the ​White House and notified all the agencies involved, but did not disclose ⁠who those parties were.

“Super intelligence companies must immediately disclose incidents involving their models and follow with swift, decisive action to remedy any and all harm,” FTC Director of Public Affairs Joe Gabriel Simonson ‌said on X.

This process was “not optional,” he said, adding that the Super Intelligence Force would fulfil its responsibility.

Story continues below this ad

The FTC said Anthropic disclosed to the SI Force on Friday its late September discovery of incidents involving what the task force called “unauthorised and fraudulent use of government and other systems”.

Tip attributed to automated test process

On Friday, Philadelphia police said Anthropic notified them of the spurious tip this week, and attributed the submissions to an automated testing process.

“The two-month delay in detecting and reporting the incident to the city is unacceptable,” they said.

The July 18 tip purported to come ⁠from someone who might have information about ⁠the case, police added.

Story continues below this ad

“I may have information regarding this case,” Anthropic’s model wrote in its submission.

“I recall seeing someone matching the description in the area around (the street ⁠named on the page) during that ‌time period. Please contact me if this information is relevant.” The brackets featured in Anthropic’s statement.

Previous ​incidents have involved AI agents hacking into vulnerable systems or commandeering unsanctioned platforms ‌to communicate with one another.

In September, Anthropic rival OpenAI apologised for the hacking of an Australian health data portal by a rogue AI agent, the first known instance of an AI agent exploiting a government website.

Story continues below this ad

A bogus tip

In two cases described on ​Friday, Anthropic’s models obtained ​for free public ​data normally only available for a fee. Another instance revealed an obscure flaw that allowed use of a public tool hosted by a university.

Claude models were also able to bypass ​restrictions by using free services that shorten URLs.

Philadelphia police said the tip “was flagged ⁠as spam and was never forwarded to the Real-Time Crime Center for investigative vetting or dissemination.”

Knowingly giving false reports to law enforcement authorities is a misdemeanour under Pennsylvania law, which specifies “a person”, however.

Story continues below this ad

This includes “information relating to an offence or incident when he knows he ‌has no information relating ⁠to such offence or incident.”

Police quoted Anthropic as telling them the test process was stopped after discovery of the incident. The false tip was submitted through PhillyUnsolvedMurders.com concerning an unsolved homicide.

Police said ​they had no evidence of unauthorised access to their systems or compromise of their data.