Cisco open sources tool for detecting AI agents on hacking sprees

Cisco has open sourced a behavior detection engine that watches AI agents, to help security teams prevent rogue agents from going off the rails like they did with the Hugging Face hack – or worse.

It’s called Project Quarry, and it aims to provide a “shared language” for evaluating agent behavior, along the lines of what Snort, which Cisco also maintains, did for detecting malicious network traffic and what YARA did for rule-based malware identification.

Most detection tools rely on specific interactions with an AI model or agent. Did it attempt prompt injection? Or did the agent call an attack tool?

REG AD

“But what is missing is the entire context under which something may occur,” Amy Chang, head of AI threat and security research at Cisco, said in an exclusive interview with The Register.

REG AD

She likened this to credit card fraud detection. Without context, individual transactions such as a small test charge or a single purchase in a location far away from a card-holder's usual location may not raise any flags.

“And then you try again,” making another fraudulent purchase, “and that went through, too. So now I’m gonna buy the PS5,” Chang said. 

“Those are the things that Quarry addresses by taking into consideration the entire agent session,” she explained. “One tool call may seem in-scope until you realize that that next step is taking that information and sending it to an attacker-controlled inbox or server.”

Even data theft is a sequence and not a single agentic action. Agentic AI security firm Invariant Labs demonstrated this with a proof-of-concept attack against GitHub's MCP server that hijacked a user's agent via a malicious GitHub Issue, and coerced it into leaking data from private repositories.

Simply reading a configuration file, or sending a request to an external endpoint may be legitimate agentic actions. However, combining the two can lead to data exfiltration – and shows “why action relationships matter,” wrote Chang and fellow Cisco researchers Adam Swanda, Ravikumar Balakrishnan, and Konstantin Berlin in a blog post about Quarry shared with The Register ahead of publication 

“A policy can produce a finding when the trace shows a sensitive file read followed within a defined window by an external send,” they wrote. “The risk comes from the relationship between the two actions, so the finding should preserve evidence from both."

Quarry covers single- and multi-turn large language model conversations, agents, tool use, Model Context Protocol (MCP) interactions, and multimodal inputs.

In addition to the open source code itself, the project’s GitHub repository includes installation instructions, supported Python versions, plugins for Claude Code, Codex and other MCP-compatible agents, configuration examples, rule-writing guidance, and a rule starter pack.

REG AD

For instance, with the rule pack, “If a base-64 encoded malware is somewhere in the tool call, that would result in a block,” Chang said.

The rule format looks similar to a YARA or Snort rule, with a header that details the traffic to examine, content to match, and metadata for tracking the rule. Here’s a Quarry rule that flags an agent that reads a credentials file:

While the Cisco Ai researchers have not run Quarry against data describing the OpenAI-Hugging Face incident, they suggest that if the intrusion had been read as a complete, step-by-step record of a single session, there were several points where the AI agents’ steps would have given a Quarry rule something to act on and flag for humans.

For example, Quarry would probably have flagged internet access from a sandbox. 

“The rule can flag specific tool calls that reach an external server asset that wasn’t in the task’s allow list,” Chang said.

“Another one would be credentials found and used somewhere else.”

According to OpenAI’s technical report about the Hugging Face intrusion, this is exactly what happened on July 10, when an agent validated 14 exposed Hugging Face tokens with write access and shared them. 

“The rule would flag that there was a credential in the tool result that then called an outside service, which is likely not within the parameters of what you would want to do in this case,” Chang said. “We don't want you to go and search for credentials that may have been leaked or inadvertently exposed and then utilize them to conduct a task.”

REG AD

The Cisco team is quick to say that rules along are not a substitute for chain-of-thought monitors, or other forms of human oversight – but they do help record a pattern that other teams can then “review, test, and apply to their own agents,” they wrote.

Herein also lies the benefit of open source, and a collective approach to defending against adversarial use of AI, Chang said.

“Do we have all the answers? No,” Chang said. “We all see slices of the problem, and it’s important for us to share what we see.”®