So, I was rather unplugged on Friday… anything happen in the world of IT?
Stupid jokes aside, the top story was the Crowdstrike update. In the early hours of Friday morning, Crowdstrike released an software update to their Falcon platform that caused Windows machines to enter a boot loop, ending in the Blue Screen of Death. Crowdstrike has kernel level access, allowing the system level failure. The fix was released, and booting to safe mode is required to either workaround the issue or apply the fix. Microsoft confirmed that the issue also impacted Windows 365 PCs.
That’s the technical side of the story. The impact was significant. According to a Microsoft blog post, around 8.5 million Windows devices, less than 1 percent of Windows machines globally. The impact was widespread, affecting banks, retailers, brokerage companies, airlines and airports and rail networks globally. Because the fix is so manual, despite its simplicity, it is a massive manual labor at scale.
Microsoft has released a recovery tool to help IT admins repair Windows machines that were impacted by CrowdStrike’s faulty update. The tool creates a bootable USB drive that allows IT admins to quickly recover affected machines by automatically deleting the problematic CrowdStrike file. Separate recovery steps are available for Windows Virtual Machines running on Azure, and Microsoft has published recovery steps for all Windows 10 and Windows 11 devices on its support site.
Shares of the company closed down 11% Friday, but are up over 93% over the last 12 months. Crowdstrike CEO George Kurtz issued an apology for the incident. Kurtz was also involved in a similar tech debacle as the CTO of McAfee in 2010, and for those saying it didn’t happen on Linux, it did – back in the June, with a different kernel system crash.
Why do we care?
Breaking format, I want to highlight two perspectives. Megan McArdle in The Washington Post had an opinion piece about the blame – our drive for efficiency. Quoting the article:
“It’s natural to ask how the heck this could have happened, and how we can make sure it never does again. But another question needs to come first: How much might we be willing to sacrifice to protect against the risk? At the moment, this might seem absurd. To people stranded by canceled flights or missing doctor’s appointments, it’s worth spending almost anything to avert the chaos that has beset airlines, hospitals, first responders and other critical services. But this is like saying it’s worth insuring your car to the absolute maximum — after you’ve gotten into a big accident. Many car owners decide not to buy the most generous possible coverage because that coverage also carries higher premiums.
So, too, with maximally insuring against meltdowns like the CrowdStrike mess, because doing so would require paying a price in economic efficiency. So we have to decide what trade-offs we’re willing to make.
Averting this particular disaster might not have cost a ton of money, to be clear. But CrowdStrike is only one of approximately one zillion points of possible failure in our thoroughly networked and globalized economy. Over the past fifty years, the market’s relentless drive for efficiency and reach has made such mass failure nodes more numerous, more potentially catastrophic and harder to see before they fail — while also giving us instant access to all the world’s culture and most of its information, plus more, cheaper and better goods and services, and a global economy that every year lifts tens of millions more people out of poverty.”
And another key perspective – Jen Easterly, Director of CISA from a post on LinkedIn.
“While it wasn’t malicious, it was a serious mistake, one for which CEO George Kurtz took full responsibility, apologized, and committed to resolving collaboratively. Needless to say, in a world of opportunistic punditry and schadenfreude, George and his team came under pretty heavy fire. I’ll just say two things on that. First, from (very) early morning until late in the evening, George and his entire team were transparent, responsive, and professional with my team. Second, anyone who has ever been part of managing a major incident knows that gleefully throwing metaphorically flaming piles of poo at anyone, particularly in the middle of a massive crisis response, is just generally a$$hole behavior.”
And this second passage:
“To channel my alter ego Bob Lord, we don’t have a cybersecurity problem, we have a software quality problem. Now before you start throwing flaming poo at me, yes, I further recognize the irony of a cybersecurity vendor creating a defective update that temporarily crippled the operations of the world’s biggest software company. And to be clear, this was not a Microsoft issue. As I said at the top, we don’t know yet fully know what happened or why, but one thing I do know is that any company that builds any kind of software should design, test, and deliver it with a priority on dramatically driving down the number of flaws—flaws which can be intentionally exploited by bad actors or flaws that can unintentionally take down critical services across the globe. The other thing I know is that anyone who consumes tech (yup—that’s basically all of us) should demand that those technology and software manufacturers do exactly that. Why we’ve been working with technology companies large and small, including Crowdstrike and Microsoft, to voluntarily commit to the Secure by Design pledge.”
Why do we really care?
Now, I want to push back on some buzz I saw online, where the call was if you’re not supporting to be silent. Empathy is not the only response to the incident, and constructive criticism is also required. I want to applaud the technology community, both MSP and security, for working together to solve customers’ problems.
But you shouldn’t have to bear all the weight of incidents such as this. Easterly is right – we do have a software quality problem. And as such, it is appropriate to be asking the question “How did this happen… again?” and “What can we do differently to prevent it?”
Multiple security experts are noting that the product wasn’t sufficiently tested. Or that it should have been rolled out to a limited pool first. And this CEO himself has experience with outages, and like everyone, memory fades.
Consider McArdle’s argument – how many customers will actually spend the money to create the necessary redundancies across those one zillion points of failure. Asking the customer to spend more because systems are fragile accepts that software vendors can take all the value – and the money – and the downside is pushed outward. Placing the entire workload on the implementors — IT Departments and service providers is ridiculous.
Software vendors should have a non-zero level of responsibility for the products they release. Easterly is right – anyone who consumes tech should demand that those technology and software manufacturers design, test, and deliver with a priority on driving down the number of flaws. And there should be consequences beyond the stock price. Compare this to other industries – aviation, automobile, legal, and medical – where there are real consequences when things go wrong.
IT departments and service providers must advocate for more stringent testing protocols and phased rollouts from their vendors. This includes demanding transparency in testing procedures and results. Encourage and support regulatory frameworks that hold software vendors accountable for the reliability and security of their products.
And speaking of consequences…..