‘A crazy perfect storm’: Satya Nadella explains what caused Microsoft Outlook outage

At a recent interview, Satya Nadella discussed Microsoft's outage, the software bug behind it and the lessons for the company's cloud infrastructure. (Express Image/Agencies)
AI Narration
Listen
00:00
Days after a widespread Microsoft outage disrupted services for users worldwide, CEO Satya Nadella has opened up about the incident and revealed what went wrong. The CEO attributed the incident to a latent bug that was discovered only during a particular traffic pattern. Nadella spoke about the cause behind the global outage of Microsoft’s suite of services on CNBC’s show Mad Money, hosted by Jim Cramer.
The bug had reportedly existed for several months but had not been detected during testing. It surfaced only when Outlook experienced a specific combination or pattern of user traffic. Simply put, the system worked normally under ordinary conditions, but failed under an unusual workload. “It was one of those crazy perfect storms where there was a latent bug that got sort of found only during a particular traffic pattern,” Nadella said.
“Of course, we should have done the testing before. We should have figured out how to have the bulkheads such that you didn’t have a global outage,” he added.
Bulkheads is a software engineering concept in which different parts of a system are isolated to prevent a failure in one component from impacting the entire network of systems. According to Nadella, a failure in one server or network gateway should remain isolated; other servers and regions could continue handling Outlook traffic. This also means automatic failover mechanisms would not accelerate the spread of the failure to otherwise robust systems. ‘Failover’ here means when a system automatically switches to a backup when the main system stops working, helping keep the services running.
Nadella’s acknowledgement is important as the outage exposed weaknesses in Microsoft’s software testing, infrastructure resilience, and failure-containment measures. According to Microsoft’s preliminary investigation, three factors contributed to the disruption: a software bug introduced in July, increased traffic following a Teams failover, and reduced server capacity, with nearly 14 per cent of the affected gateway’s capacity already unavailable.
The outage that was reported on November 12 led to a series of major disruptions impacting some of the biggest corporations in the US, airline operations, and banking services. Further, the incident also highlighted how technology is deeply interwoven into the everyday lives of millions and, most importantly, how businesses that are reliant on single technology providers are often at risk.
According to Nadella, with AI helping companies to detect software bugs faster, the sheer number of updates needed to keep these systems running could likely cause instability in older systems. “The thing that AI has done is it allowed us to find these bugs and fix these bugs. But that means you’ve got to then patch them, and that creates instability,” he was quoted as saying by the publication.
KioskNews shows a cleaned-up reading view extracted from the publisher’s page — the original always lives on their site, not ours.