Spate of rogue AI hacking points to lack of tech oversight, outdated defences: Experts

SINGAPORE – The recent spate of hacking incidents by rogue artificial intelligence reveals two major blind spots: a lack of clear rules for governing AI as well as outdated defences that cannot keep pace with automated attacks, said cybersecurity experts.

While the incidents do not suggest that organisations have poor defences, experts urged organisations to use AI to fight AI.

“The lesson isn’t that every hacked site was poorly defended,” said Santanu Dutt, cybersecurity firm Zscaler’s vice-president and head of technology for Asia Pacific and Japan.

“It’s that systems built for slow, human-led attacks are now being probed by tools that never stop looking. It’s the difference between checking your locks once a year and having someone try every window, gate and side door within minutes.”

Alarms over rogue AI’s hacking abilities were sounded when ChatGPT maker OpenAI disclosed in July that one of its AI agents had earlier escaped its testing environment and breached AI software repository Hugging Face to complete a task it was given. The attack was not intended by OpenAI.

AI agents are bots that can carry out actions autonomously with minimal human supervision.

This was followed by a flurry of reports from Anthropic, Meta and Google that their AI had also gone rogue and hacked other organisations.

On Sept 24, Australian Prime Minister Anthony Albanese revealed that the database of Australia’s health system was breached by a rogue OpenAI bot in June. Although OpenAI discovered the attack in August, it reported the attack to the Australian authorities only on Sept 10.

The scope of misbehaving AI could be far larger than initially reported.

A subsequent Sept 26 report by news site Axios suggested that Anthropic, OpenAI and security researchers are quietly investigating tens of thousands of incidents of their cutting-edge AI models acting in problematic ways.

In many of the reported hacking cases, tech companies did not appear to know that their AI systems were behaving improperly until later. This lack of visibility and control over their AI’s actions was an issue flagged by cybersecurity experts.

“If you give these AI models systems and tools like internet access, monitoring what they are doing becomes extremely important,” said Qasim Mithani, co-founder and chief executive of cybersecurity firm depthfirst.

He noted that OpenAI disclosed its monitoring systems were not active during the Hugging Face incident. The monitoring systems could have flagged its agent’s problematic behaviour more than a day before it breached Hugging Face.

Zscaler’s Dutt said that the Australian health website hack also points to a monitoring and governance gap, rather than just a technical flaw.

Experts said that AI agents tend to find workarounds to complete their tasks if they encounter obstacles. In the Australian example, the rogue agent resorted to hacking the portal to retrieve the information required to complete its assigned task.

Tony Anscombe, ESET’s chief security evangelist, said the Australian case offers a critical lesson – an autonomous agent granted internet access and tools to run programmes needs clear boundaries around which systems it may access, what actions it may take when access is denied, and when it must stop or ask for human approval.

“Those boundaries need to be enforced technically, not left solely to instructions that are open to interpretation given to the AI,” said Anscombe.

Depthfirst’s Mithani said that some of the recent incidents and his firm’s own research also highlight that AI systems are becoming more capable of finding and validating vulnerabilities without a human guiding them through every step.

“Only a short time ago, that kind of work was largely the domain of highly skilled security researchers and hackers. Increasingly, AI can do it autonomously,” he said.

Open-weight models are also an issue. Such models tend to be cheaper and offer users more control than the closed models from many frontier AI laboratories.

Tests by depthfirst suggested that open weight models are “remarkably capable” at cybersecurity tasks like finding critical vulnerabilities in popular apps, said Mithani.

Even so, the capability of AI models could be uneven. Teo Xiang Zheng, Ensign InfoSecurity’s vice-president of advisory, said that in his firm’s tests of 10 frontier AI models, they were all able to gain initial access to simulated systems, with six models completing all the given cyberattack objectives in at least two test runs.

“However, they were less reliable at later stages, such as bypassing endpoint detection. So, AI can already automate significant parts of an attack, but it is not equally effective at every stage,” Teo said.

In the recent incidents, while some of the vulnerabilities found and abused by the AI agents were completely new, other attack methods were less sophisticated. For example, in the incident involving Google’s Gemini, the AI accessed one system by guessing a password and accessed two others using publicly listed credentials, noted Mithani.

Since AI can execute actions at a speed and scale far greater than human hackers can, Teo said that organisations should assume that their systems may eventually be breached and use techniques to limit how far an attacker or an AI agent can progress.

And to match the machine speed of AI attacks, organisations need AI defences that can operate at a similar pace.

Said Zscaler’s Dutt: “No human team can watch every login, website request and file movement round the clock. But a well-deployed AI defence system can spot unusual behaviour as it happens and act before a small incident becomes a public breach.”