AI safety cannot be left to Silicon Valley

The response involves building regulatory capacity before these systems become embedded in critical public infrastructure, rather than after an incident forces action.

3 min readSep 28, 2026 06:12 AM IST First published on: Sep 28, 2026 at 06:12 AM IST

Incidents of AI agents breaking into government systems in Australia and the US — the first time such attempts have been reported — should ring alarm bells everywhere. These breaches, where OpenAI agents “attempted to get information” from dozens of institutions, including the US Securities and Exchange Commission and Australia’s Medicare statistics reporting service, show what happens when systems designed to carry out tasks autonomously find ways around restrictions placed on them. The immediate damage appears to have been limited. That governments themselves failed to detect these “misalignments”, however, points to a terrifying asymmetry, with potentially catastrophic consequences: These systems contain sensitive information about citizens and support essential public services, and AI agents that treat a security restriction as a mere obstacle to be overcome could disrupt services, expose confidential data or expose chinks in more critical systems.

Anthropic, Google, Meta and other companies have reported cases in which their models behaved in ways their developers did not intend, while Anthropic’s research has demonstrated how frontier models can resort to deceit or strategically harmful behaviour, like blackmail. The industry’s response has increasingly included calls for stronger safety standards and international coordination; tech leaders, including OpenAI’s Sam Altman and Anthropic’s Dario Amodei, testified to this effect during a recent UN Security Council briefing. They argued that the safety standards for increasingly autonomous systems cannot be left, as Altman put it, to “labs in San Francisco alone”. Voluntary testing, transparency and incident reporting are essential, but they cannot replace independent oversight.

The response involves building regulatory capacity before these systems become embedded in critical public infrastructure, rather than after an incident forces action. It calls for independent testing of high-risk AI, mandatory reporting of serious misalignment incidents, strict permissions for agents accessing government systems, and accountability from companies deploying them. For India, the 2026 AI Governance Guidelines provide a foundation, but institutions such as the proposed AI Safety Institute and AI Governance Group will need both technical expertise and the authority to scrutinise frontier systems. Moreover, a technology whose agents can cross borders in seconds calls for international standards regarding model evaluations, agent permissions, cybersecurity testing, incident disclosure and emergency responses, with mechanisms that can be strengthened as capabilities grow — as they are bound to. The choice is not between innovation and regulation, but between governing increasingly autonomous technology in a calibrated way and allowing the technology itself to determine the limits of what governments can control.