Missed security update led to 9½ hours public healthcare service outage on Oct 13

SINGAPORE - A “preventable lapse” involving a missed security update to a shared domain name system led to a 9 1⁄2 hours public healthcare service outage on Oct 13, Parliament heard.

Affected services included appointment bookings, visitor registration, medication dispensing and billing, said Senior Minister of State for Health Tan Kiat How on Nov 3.

He added that about 4,000 outpatient appointments were rescheduled and some patients experienced delays of up to two hours. There were however no reported adverse incidents affecting patient safety as a result of the disruption.

The outage affected all three of Singapore’s public healthcare clusters – NHG Health, National University Health System (NUHS) and SingHealth.

“This was a known change and a preventable lapse,” he said, replying questions from MPs about the cause and extent of the disruption, which is the longest in recent years.

MPs Joan Pereira (Tanjong Pagar GRC) and Poh Li San (Sembawang West) asked about the reason for the disruption and whether backup systems were in place, while Dennis Tan (Hougang) asked if measures had been implemented to reduce the risk of such systemwide disruptions.

Specifically, the required security update to a shared domain name system (DNS) service was not completed before a scheduled global change, Tan said. DNS works like an address book, transforming website addresses into numerical IP addresses which systems use to locate and connect to applications.

Tan said that while national health-tech agency Synapxe had contracted a service provider to manage the necessary update, it was not carried out. Synapxe is reviewing the provider’s contractual obligations and will take appropriate action for any established lapses, he added.

“Regardless of the outsourcing arrangement, Synapxe remains accountable for the resilience of its critical systems,” he said.

Tan reiterated that the incident was not the result of a cyberattack, and that there was no unauthorised access to patient data, which was not compromised or lost.

While abnormal activity was detected, time was required to identify the underlying problem, said Tan, who is also Senior Minister of State for Digital Development and Information.

Synapxe is strengthening monitoring and fault diagnosis to enable faster recovery, he said, apologising for the inconvenience caused to patients and their families, as well as the resulting additional work for healthcare workers.

Singapore’s public healthcare systems have backup infrastructure, including two data centres. “However, both depended on the same underlying DNS service and security configuration. Switching between them would therefore not have resolved this disruption,” he said.

Synapxe will review such shared dependencies and strengthen alternative recovery arrangements, he added.

“Healthcare institutions activated downtime procedures, with additional staff supporting operations and clinicians using selected read-only patient records and manual workflows to continue essential care,” said Tan.

To strengthen resilience against such incidents, the authorities are improving the tracking and verification of critical technical changes, said Tan, adding that shared dependencies are also being addressed and recovery arrangements improved.

He added that continuity of care during such incidents is also being addressed by improving downtime systems, staff readiness and regular exercises, including for partial outages.

The Health Ministry will oversee the corrective actions and verify that they are effective, said Tan.

“Synapxe has appointed an expert panel comprising representatives from other organisations that have similar engineering profiles to review the investigation findings and proposed remediation measures and identify further improvements,” he said.

While digitalisation brings important benefits to healthcare, resilience must improve as systems become more interconnected, he said.

“We must make our technology more reliable, while ensuring healthcare workers can continue caring for patients safely even when technology fails.”

In a supplementary question, Yip Hon Weng (Yio Chu Kang) asked if having Synapxe oversee the tech functions of the three healthcare clusters posed a concentration risk.

Tan said the global DNS service shared by the clusters and their systems was a “trust anchor” – a term used for authoritative entities in computer security.

Decentralising the healthcare IT systems would not necessarily have prevented the disruption, but could instead have made diagnosis and recovery much more difficult,” he said.

Yip had also asked whether the Epic system used by private clinics here would be affected by such a concentration risk.

Tan said the authorities would look at such risks and dependencies as it sought to implement a shared electronic medical records system across providers.