'Heed the warning': NSW premier urges caution with AI after data breach

The NSW premier has urged the world to heed the warnings of artificial intelligence bosses about the technology's potential dangers, after a state dataset found it could be vulnerable to exploitation.

The NSW Bureau of Crime Statistics and Research (BOCSAR) was this week notified by the Australian Signals Directorate that a dataset underpinning a publicly-accessible crime mapping tool had been identified by OpenAI as potentially vulnerable to a data breach.

BOCSAR, a crime and criminal justice research agency, said there was no evidence that the vulnerability had actually been exploited or that a data breach had occurred.

A statement from the agency said: "While the dataset does not contain personal information and any risk associated with exposure is considered low, BOCSAR is reviewing the concerns raised and working to strengthen its systems and safeguards".

Earlier on Thursday, Prime Minister Anthony Albanese announced an OpenAI agent had hacked into a Medicare data portal in June, but that the company had not notified the federal government until September.

Mr Albanese said the AI agent had accessed public and non-public files but that it did not appear anyone's personal Medicare details had been accessed.

BOCSAR, the Australian Institute of Health and Welfare and the Victorian Department of Health "may (also) be impacted", Mr Albanese told a press conference in New York.

'A big concern'

NSW Premier Chris Minns said he had spoken with Mr Albanese about the issue.

"As I understand it the AI agent was told not to access these parts of the website, this information, and it did so anyway," Mr Minns said.

"Now that's a big concern for us in New South Wales. We need to and will examine its impact both on technology and the vulnerabilities in our confidential information or information that is not public-facing."

Mr Minns said Cyber Security NSW would work with Commonwealth intelligence agencies to "get to the bottom" of the AI involvement with BOCSAR.

NSW Greens industry spokeswoman Abigail Boyd called for an "urgent, forensic and wide-ranging" audit of all NSW government systems to check for breaches.

"BOCSAR is not the only vulnerable source of potentially sensitive information. The NSW government hosts highly sensitive information through public-facing websites like the HealthStatsNSW website, as well as the Data.NSW website which is an open data portal hosting 17,164 NSW public sector datasets," Ms Boyd said.

Premier urges world to 'heed the warning'

Anthropic chief executive Dario Amodei and OpenAI chief executive Sam Altman this week addressed the UN Security Council on the need for global cooperation on the greater regulation of AI development and use.

"In our history, there have been times where countries who compete and don't always like each other very much still come together for shared interests and the collective good in the face of a powerful new technology," Mr Altman said.

"We believe this must be one of those times."

Mr Minns said the world should heed those calls, which were made less than two weeks after Mr Amodei, Mr Altman and Tesla founder Elon Musk urged the world to "slow the pace" on AI development.

"These are the people who will most specifically financially benefit from the expansion of AI," Mr Minns said.

"If they're saying it should slow down, I think that everybody else should heed the warning and look very closely at what this very powerful, potentially damaging technology can do with public information."