'Preventable lapse' by service provider caused 9.5-hour public healthcare IT outage on Oct 13: Tan Kiat How

SINGAPORE:  A "preventable" lapse by a contracted service provider to complete a required security update led to the nine-and-a-half hour disruption to services across Singapore's public healthcare clusters on Oct 13, Senior Minister of State for Health Tan Kiat How told parliament on Tuesday (Nov 3). 

About 4,000 outpatient appointments were rescheduled, and some patients experienced delays of up to two hours, Mr Tan said. No adverse patient-safety incidents attributable to the disruption were reported.

The disruption impacted services across all three public healthcare clusters - NHG Health, the National University Health System (NUHS) and SingHealth.

Services affected included appointment bookings, visitor registration, medication dispensing and billing.

Healthcare institutions activated downtime procedures, with additional staff supporting operations and clinicians using selected read-only patient records and manual workflows to continue essential care, Mr Tan said.

National health technology agency Synapxe had contracted a service provider to manage the necessary update, but it was not carried out, Mr Tan said.

"Synapxe is reviewing the provider's contractual obligations and will take appropriate action for any established lapses.

"Regardless of the outsourcing arrangement, Synapxe remains accountable for the resilience of its critical systems," Mr Tan said.

The public healthcare systems have backup infrastructure, including two data centres. But both depended on the same underlying DNS service and security configuration, Mr Tan said.

"Switching between them would therefore not have resolved this disruption. Synapxe will review such shared dependencies and strengthen alternative recovery arrangements."

The required security update to a shared Domain Name System, or DNS was not completed before a scheduled global change, Mr Tan said, likening the DNS to an internet address book that helps systems locate and connect to applications.

"This was a known change and a preventable lapse," Mr Tan said.

The disruption on Oct 13 caused inconvenience and concern to patients and their families, and created additional work for healthcare workers, said Mr Tan.

"We apologise. It should not have happened." 

The investigation found that this was not a cyberattack, and there was no unauthorised access to, compromise or loss of patient data, he added.

Noting the duration of the disruption, Mr Tan added that monitoring detected abnormal activity, but it took time to identify the underlying problem and restore services safely.

Synapxe is strengthening monitoring and fault diagnosis to enable faster recovery, Mr Tan said. 

EXPERT PANEL

Mr Tan said the government is strengthening resilience in three areas: prevention, by improving the tracking and verification of critical technical changes; and containment, by addressing shared dependencies and improving recovery arrangements. 

The third area involves continuity of care, by improving downtime systems, staff readiness and regular exercises, including for partial outages

"MOH will oversee the corrective actions and verify that they are effective," he said. 

Synapxe has appointed an expert panel comprising representatives from other organisations that have similar engineering profiles to review the investigation findings and proposed remediation measures and identify further improvements.

"Digitalisation brings important benefits to healthcare. But as our systems become more interconnected, resilience must improve too. 

"We must make our technology more reliable, while ensuring healthcare workers can continue caring for patients safely even when technology fails," said Mr Tan.