Single defect in software update likely was cause of global computer outage

In the end, a single file of faulty software code buried deep in an automatic update for security software may have been all it took to bring global commerce to its knees.

The widespread computer outages to Windows-based systems that began at 12:45 a.m. Friday and continued through the day were tied to an automatic software update by security firm CrowdStrike that conflicted with the most widely used computer desktop platform by Microsoft, CrowdStrike confirmed Friday.

The defect, which impacted only Windows users, was part of an automated update delivered via CrowdStrike’s Falcon security suite, which includes antivirus and desktop protection, "cyberthreat intelligence, managed threat hunting and security hygiene," the Austin, Texas security company said.

"Many systems received this update that literally shut down following this downloading due to a problem with the software," Gov. Kathy Hochul explained Friday morning. It may take days or more to get them all back up and running, experts say. 

CrowdStrike, a well-regarded cybersecurity firm that worked with Hochul’s administration in 2022 to make its security system free to counties across the state, confirmed it identified the problem as part of a recent "content deployment," in which updates are pushed out to clients’ computer systems.

Security firms such as CrowdStrike, in an effort to stay ahead of fast-moving cyberattackers, are constantly upgrading and sending out threat detection software and fixes, with internet-based delivery systems that automate the process. Typically, such fixes are widely tested before they are sent out. It's still unclear how the fault slipped through the cracks, but Nick Nikiforakis, an associate professor of Stony Brook University's computer science department, said he couldn't recall a similar faulty update having such a widespread impact. 

The CrowdStrike software sits in a core part of the Windows operating system with "very high-level  privileges on the machines it runs," said Nikiforakis. "When it crashes it can take the whole operating system with it." 

To respond to Friday's outages, CrowdStrike developed an automated fix that uses the same deployment system to fix the defect, but not all systems use the Microsoft cloud-based delivery system to receive it. 

In the interim, those with systems still facing the so-called blue-screen error notices are being directed to a manual fix that involves rebooting computers in safe mode, and manually deleting the offending file.

But that fix won't be a fast one for the potentially hundreds of thousands of impacted systems worldwide, and the price tag ultimately could be high.

"The cost of touching every single computer to fix this is going to be excruciating," said Michael Nizich, an adjunct professor of computer science at the New York Institute of Technology and director of its Entrepreneurship and Technology Innovation Center.

While CrowdStrike has asserted that the software bug is “not a security incident or cyberattack,” some appear worried that threat actors could use the incident to perpetrate an attack. Suffolk County in a note to employees noted the bug is “creating a scenario where bad actors are attempting to take advantage of the situation by playing off employee concerns.” CrowdStrike itself warned customers to “ensure they’re communicating with CrowdStrike representatives through official channels" when contacting the company.

Still to be seen is how long it will take to restore all the impacted computers to normal working order, and who will pay for the damage, according to Nizich. 

Nizich said bugs like the one impacting CrowdStrike’s customers are likely to continue and expand as more systems turn to so-called cloud-based computing that keeps data and applications stored remotely, but still relies on computer desktops with operating systems to access and run them. “With cloud-based systems, we’re seeing global outages like never before,” he said.

By Mark Harrington

mark.harrington@newsday.comMHarringtonNews

Mark Harrington, a Newsday reporter since 1999, covers energy, wineries, Indian affairs and fisheries.