Many businesses will probably take days or even weeks to recover fully from Friday’s unprecedented computing outage, IT experts have warned, after a faulty software update from the company they trusted to secure their systems caused massive global disruption.
CrowdStrike, one of the world’s largest security vendors, blamed an update to its Falcon software for a bug that broke countless Windows PCs and servers, grounding planes, postponing hospital appointments and taking broadcasters off air around the world.
Cirium, an aviation analytics company, said on Saturday that airlines had cancelled a further 1,848 flights, mostly in the US, though Australia, India and Canada were also affected.
The outages were all the more shocking given CrowdStrike’s strong reputation as many companies’ first line of defence against cyber attacks, analysts said.
“This is the first time that a widely deployed security agent, that is designed to protect machines, is actually causing them to break,” said Neil MacDonald, analyst at IT consultancy Gartner.
The only remedy for Windows users affected by the “blue screen of death” error involves rebooting the computer and manually deleting CrowdStrike’s botched file update, requiring hands-on access to each device.
That means it could take days or weeks to apply in businesses with thousands of Windows machines or a shortage of IT workers to administer the change, experts say.
“It seems that millions of computers are going to have to be fixed by hand,” said Mikko Hyppönen, chief research officer at WithSecure, a cyber security company.
“The most critical machines like the CEO’s laptop are already fixed — but for the average Joe in finance it’s going to take a while until someone comes over to fix your laptop.”
Exacerbating the impact of its error is the large scale and the high-profile nature of many of CrowdStrike’s users.
The Austin, Texas-based company said it had more than 29,000 business customers at the end of 2023, and has claimed in marketing material that its software is used by more than half of the Fortune 500.
“Despite [CrowdStrike] being actually a fairly large company, the idea that it would shut down the world is extraordinary,” said Marshall Lux, visiting fellow at Georgetown University’s McDonough School of Business.
The global ripple effect illustrates “the interconnectivity of all these things” and “concentration risk in this market”, Lux added.
Software vendors “have clearly become so large and so interconnected” that their failures can damage the global economic system, wrote Citi analyst Fatima Boolani in a note to clients. This could invite greater political and regulatory scrutiny.
Gartner estimates that CrowdStrike’s share of revenues in the global enterprise endpoint security market — which involves scanning PCs, phones and other devices for cyber attacks — is more than double that of its three closest rivals: Trellix, Trend Micro and Sophos. Only Microsoft is larger.
In CrowdStrike’s latest earnings call in June, chief executive George Kurtz said there was “a widespread crisis of confidence amongst security and IT teams within the Microsoft security customer base” following a series of high profile cyber incidents affecting the Big Tech giant.
CrowdStrike, which was founded in 2011, said it saw a surge in demand after Microsoft said earlier this year that its systems had been breached by state sponsored hackers.
In May it launched a product designed to work alongside Microsoft’s own Defender antivirus protection tool.
On Friday, as Kurtz apologised to CrowdStrike’s customers, he emphasised that the incident was “not a cyber attack” and insisted that CrowdStrike’s customers “remain fully protected”.
But security researchers warned that fraudsters could take advantage of the chaos to impersonate Microsoft or CrowdStrike agents for phishing scams.
“We see this happening with every major cyber incident that is in the news,” said Vasileios Karagiannopoulos, an associate professor of cyber crime and cyber security at the University of Portsmouth.
Cybersecurity firm Secureworks said its researchers had observed several new CrowdStrike-themed domain registrations within hours of the incident, most likely by criminals aiming to trick the company’s customers.
Avoiding the type of error that caused Friday’s outages was “a matter of testing”, said Ian Batten, a lecturer in the School of Computer Science at the University of Birmingham. In this case it looked like someone simply “got a bit of code wrong”, he added.
Companies like CrowdStrike are under pressure to roll out new security updates as quickly as possible to defend against the latest cyber attacks.
“There’s a trade-off here between the speed of ensuring that systems get protected against new threats and the due diligence done to protect the system’s resilience and stop things like this incident from happening,” said Adam Leon Smith, a fellow of the British Computer Society, a professional IT body.
The damage caused by this week’s flawed software update “could take days and weeks” to repair, he said.