The Day the Digital World Stood Still: Lessons from the Largest IT Outage in History

What if a single update could bring the world to a standstill? On July 19th, that hypothetical nightmare became a reality when a CrowdStrike sys file update led to the largest IT outage in history. In this episode, we unravel the catastrophic chain of events that left 8.5 million Windows systems crippled and incurred an estimated $15 billion in global damages. From grounded flights to disrupted healthcare services, the fallout was immense and far-reaching. But that wasn’t the end—the chaos also opened the door for opportunistic hackers, posing as CrowdStrike support, to exploit the situation further.

Join us as we dissect the lessons learned from this unprecedented incident. We’ll explore the critical importance of rigorous testing, robust backup plans, and zero trust technologies in maintaining system integrity. Companies must prioritize compliance and vendor vetting to mitigate risks. Through CrowdStrike’s transparent response, we see a roadmap for handling such crises. As consumers of cybersecurity, the onus is on us to demand more stringent development and testing processes from our providers. Tune in to get the insights you need to stay secure and informed in an ever-evolving threat landscape.

Support the show

Episode Transcript

Matthew Chapman (00:10):
I'm back.
It's been a long time I thinkit's actually almost been 10
months so I'm going to make apromise now that I'm going to
start doing these on the moreoften, on them more often, and
just wanted to check in a littlebit, since the CrowdStrike
thing has been such a big thing.
So, first off, I'm not going torag on CrowdStrike, so if
you're looking for that you cango away now.

(00:32):
But just some of my thoughts.
So, as we all know, it happenedon July 19th.
It was an update to a sys fileTook down about eight and a half
million Windows systems aroundthe world.
It's being called the largestIT outage in history.
Insurers are estimating thatthe outage will cost at least
just the US Fortune 500 companyis about $5.4 billion, and

(00:57):
estimates now for worldwide arereaching somewhere around $15
billion.
The blue screen of deathrequired a reboot into safe mode
to remove the offending file.
That caused an increaseddowntime because of the reboots,
especially for machines thathad to go into the bitlocker and
deal with that first.
Roughly 10,000 flights weredelayed or canceled or affected

(01:20):
in some way Healthcaredisruptions, clinics, hospitals.
Some states even reported that911 services had outages.
Banks and media outletssuffered outages as well.
And, of course, the hackers aretaking advantage of the
situation, sending phishingemails saying that they're from
CrowdStrike and here to supportyou and help you, and some are
even going so far as callingcustomers and posing as

(01:42):
CrowdStrike support.
Some are even selling scriptsthat are telling you that they
can automate the process torecover your system, when
actually they're installing somemalware and backdoors.
So what are the lessons?
Have a test and release planand I love this phrase.
I've always loved this phrasetrust but verify.
Have a backup recovery plan,but just don't have one tested

(02:06):
regularly.
At least twice a year, I tendto do quarterlies when I was
doing things like CISO work.
Consider deploying a zero trusttechnology to help keep changes
from happening to criticalsystems.
Some of my other thoughts onthis.
Companies, depending on theindustry and sector, are
required to follow complianceregulations.
The vendor they choose fortools and services either have

(02:30):
to help them be compliant and orbe compliant themselves.
As someone that has been aroundlong enough in this industry to
see most cybersecuritycompanies suffer some level of
issue impacting customers.
Testing and more testing beforepressing the button to release
needs to be the standardRegression testing.
Eat your own dog food Test.

(02:51):
Roll it out to yourself, yourown company, when satisfied and
diligence has been accomplished.
Do a staggered rollout to thecustomer base, not all at once.
Follow the pipeline for testing, not just of core code, but
also your drivers, your patchesand any changes to the state of
known good.
Any change needs to be tested.

(03:11):
That's the lesson.
Good change management needs tobe followed and software
development processes need to befollowed more stringently.
We all know that, even with allthat, mistakes or anomalies
will occur and the reaction ofsupport to CrowdStrike because
they have been very forthcomingand thus far have shown nothing.
But transparency is also neededand respected in the industry.

(03:34):
As a customer and a consumer,we now need to ask questions
about how is your code developedand tested, not just how does
it work and keep me secure andin compliance, we need to do
more than just check the boxes.
So there's my thoughts onCrowdStrike and I hope all of

(03:55):
you are having a better daytoday.
It's been a little while now,so hopefully things are starting
to get a little better outthere and I'll see you next time
.