Trending Articles
Trending Topics
Trending
Now that the dust has settled on the CrowdStrike outage, IT manager Michael Dear delivers a set of valuable lessons that still need to be learned
19th July 2024, a Friday. A day to go down in infamy as the start of the biggest computer outage ever. A day that a big but largely unknown firm gained worldwide notoriety. I am, of course, talking about the CrowdStrike outage, when an update via its Falcon software took millions of Windows PCs offline.
The first I knew of this incident came via a tweet on X from Troy Hunt, who runs HaveIBeenPwd. It was soon evident that this was an Event with a capital E. I have previously written about the Y2K Millennium Bug and the work that went into the rectifying of the problem: this Event was of a scale that demonstrates what could likely have occurred if that work didn’t happen.
So what lessons can we learn from the CrowdStrike outage? Now that we’re a few months down the line, it turns out quite a few.
Before I start, I should point out that if your company isn’t running something similar to CrowdStrike Falcon (the tool in question), you should be. Also, the answer is not to uninstall CrowdStrike: the risk of this event happening is much less than the risk posed by uninstalling the tool and losing the visibility and protection it provides.
There are many tools like CrowdStrike Falcon. They fall under the category of EDR, standing for Endpoint Detection and Response. What happened to CrowdStrike could affect any of them. And indeed any software that updates Windows. Please don’t think I’m bashing CrowdStrike in this; everyone makes mistakes.
CrowdStrike Falcon is like an advanced antivirus tool. Kind of. To quote from Terry Pratchett’s character Lobsang, “That is a lie, but a useful and helpful lie,” because Falcon isn’t an antivirus tool – but people know what one of those is and understand how it works in principle. In short, it looks for threats and responds to them by sending updates to your PCs.
One day last July, CrowdStrike updated the rules for the Windows version of Falcon, but in doing so it caused any connected computer to crash. The fix was to delete a single file that contained the update and restart the computer, when everything would be fine.
The fly in this particular ointment was that the password was stored in the computer’s disk, and these days that disk tends to be encrypted. To access that password meant decrypting the storage and accessing it directly – and unless you used a tool such as Intel’s vPro the only way to do so was to physically sit in front of the computer, type in the decryption password, and then dig the Falcon password out of the storage.
Only then could you delete the file.
I am sure that the legal cases will go on for years, as large firms sue for their computer systems going down – Delta Airlines being one obvious example – but I will also bet that most of the affected firms have fully signed off disaster recovery plans in place. Each plan would have been tested, but for some reason – it seems – this particular scenario wasn’t planned for.
So why did we have such a worldwide meltdown of computers?
First, tools. If I were to ask you which job requires metal, a forge and a hammer, I hope you would think of a blacksmith. There is no way that you would say insurance or banking, for example. However, if I was to ask, “what are an airline company’s equivalent tools?” most people would say planes. Delta can’t make money if it isn’t flying people around the world.
It turns out that if the computers aren’t running then planes don’t fly. So whilst the computer system isn’t generating revenue, it is a requirement to making the money. Much like my blacksmith needs the hammer.
Which brings me to the topic of dependencies. As a society, we have built lots of tools that look good until you peer under the hood. Here is my favourite from online cartoon XKCD.
It shows that most software is a collection of stuff made before, and that it’s a tower structure. It also highlights the unintentional incorporation of a key but overlooked part.
Then we have cloud computing and the internet. Cloud computing has allowed people to say “we are a [insert what they think they do] firm and we just buy the computing resources we need and pass that responsibility on to others, as they are a tool”.
Blacksmiths don’t hire tools. In fact, most people who require tools for a job will own them, and they don’t skimp on them. For some reason, when it comes to computing this is not the way things work. Perhaps it’s due to the continuous upgrade cycle, which obscures the value these tools provide to a business, and turn them into a cost.
If anything, our switch to cloud computing has made things worse. Not that things are any more or less likely to break, more that each of us is reliant on the exact same tool, so when that breaks the impact is on a far larger scale.
This brings me back to CrowdStrike. The reason this was such as massive outage is that it made a mistake, and it simultaneously pushed that mistake to every computer that was switched on at that time. CRASH!
In computing, we’ve known for years about canary builds. With these, you make a change and push it to a small number of systems. You check these computers and, if all is well, push out to more computers and repeat – upping the numbers that get the update each time. This is what Microsoft and Apple do with their updates.
Of course, it’s expensive to do and mostly nothing will happen, so canary builds could be viewed as adding an unnecessary cost. However, if CrowdStrike had used this approach, a couple of thousand random computers would have died, it would have quietly apologised to those firms and fixed the problem.
You won’t be shocked to learn that CrowdStrike has since shared that is moving to fast canary builds, and that you as a customer can select what level of risk you want in terms of using these early builds.
Why take the risk? Because CrowdStrike is in the business of reacting to cyberthreats, and the earlier you’re company is protected the better.
In the months since the “Event”, a number of things have happened that involve CrowdStrike.
In August 2024 at DEFCON (a security and hacking show), the prize of Most Epic Fail was awarded to CrowdStrike. However, unlike previous winners like Microsoft or Twitter, the President of CrowdStrike Michael Sentonas accepted the prize in person and followed it up with a speech about how CrowdStrike should own its own mistakes. And that the trophy will be displayed in the Texas headquarters as a reminder of that.
In early September, CrowdStrike said it expected to lose about $60 million of revenue following the outage, although it didn’t say how many accounts that meant it had lost. Other firms in the EDR space said they were fielding calls from CrowdStrike customers, but that doesn’t mean the customers are leaving.
Incidentally, on 26 November 2024 CrowdStrike announced its quarterly financial results. For the first time ever, it exceeded $1 billion in quarterly revenue, grows 29% year on year. It will be interesting to see if that trend continues when it announces its next results in March.
The lesson we can learn? That by owning your mistake, your customers forgive you. They may even think better of you. And that could help you find new customers too.
The really interesting thing that has followed the CrowdStrike outage – and other recent questions around AI – is that the EU wants to make software have liabilities.
Presently, in that dense legalise we all click through but don’t read when installing a program, there’s a part that basically says: “if this doesn’t work or does something unexpected it’s not our fault”. The EU wants that to change, for the producer of the software to have a liability if something goes wrong.
That, in my opinion, is long overdue. I’m amazed that software developers have got away with this for so long.
The EU decision is just one of many positives to emerge from the CrowdStrike outage.
Another, I suspect, is that CrowdStrike’s legal battles will act as encouragement to other software producers to ensure they don’t find themselves in the same spot. In particular, others in the EDR space will have sharpened their game.
We have also seen Intel announce that it’s now far easier for companies to activate vPro on their machines, which would have allowed admins to remotely access PCs’ storage rather than visit every machine physically.
But what about you? How can you and your business/employer know that you won’t be affected by something like this is the future? Well, I have bad news. There is no way: all the due diligence in the world isn’t going to save you from something like this.
That’s because so many programs that can cause this type of issue are installed on your computers. Your antivirus, endpoint management system, patching tools and remote access apps have similar levels of access to Falcon that allow them to break your computer.
What you do need is a disaster recovery plan that accepts this could happen – that assumes it will happen – and a method of rescuing your systems that doesn’t involve manually visiting broken PCs. If you haven’t put that plan in place, then you really haven’t learned the lesson of CrowdStrike at all.
Luckily, my company wasn’t affected by CrowdStrike. I considered working long hours over that weekend as a contractor to help out affected companies, but ended up sharing memes on social media instead. Including the one below.
