July 31, 2024 • 29 mins
Good morning and welcome to another episode of Breakfast Bytes. I'm your host, Felicia King, and today, I'm joined by my colleague, Jeff Birner, hailing from Florida. Our riveting discussion centers around the recent CrowdStrike incident that has sent shockwaves through the cybersecurity community and beyond. This episode promises to offer insights and perspectives you won't find in the typical news coverage.
As we delve into the conversation, Jeff and I explore the core issues surrounding CrowdStrike, including its lack of trustworthiness as a counterparty and the legal implications of delayed security updates. We discuss the broader impacts of the incident, such as the staggering $5.8 billion in losses faced by companies worldwide, and discuss how technology decisions could have eliminated the impact.
Through engaging storytelling, Jeff and I break down the complexities of cybersecurity, offering practical solutions and strategies for organizations to consider. From the importance of testing updates to the choice of operating systems for critical infrastructure, this episode is packed with valuable takeaways for IT professionals and business leaders alike.
Join us as we navigate the nuances of the CrowdStrike controversy, highlight the lessons learned, and provide actionable advice to help you safeguard your organization against similar pitfalls. Whether you're a seasoned cybersecurity veteran or just starting your journey, this episode of Breakfast Bytes is a must-listen.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
Good morning. You're listening to Breakfast Bites, and I'm Felicia King.
And joining me today is Jeff Berner, who's a colleague of mine in the cybersecurity
space, and he hails from Florida.
And Jeff contacted me about this CrowdStrike thing, just to get my opinion.
We had one heck of a good conversation about it.
(00:21):
And so I said, wow, considering the impact that this has made,
I figured it would be good to have a discussion and share it with a wider audience.
And before we get started on that, I want to comment that I saw plenty of people
writing articles about CrowdStrike.
And the thing that I found was that nobody has yet written an article about
(00:47):
the stuff associated with CrowdStrike talking about really two key elements.
One was around the fact that CrowdStrike just wasn't a trustworthy counterparty
to begin with, so we can talk about that.
The second one really had to do with the legal defensibility of delayed security updates.
(01:11):
There. So let's talk about that in more detail.
So, you know, Jeff, do you want to start us off with a little bit about you
and then, you know, hover into like some questions for me about CrowdStrike?
Absolutely. And thank you for the introduction and having me on your podcast.
No, my name is Jeff Berner. I am the CEO of IT Consulting St.
(01:32):
Petersburg, recently certified as a virtual chief information security officer.
And what I've learned in the short time that I, And when I say short time,
the six or seven years that I've been researching the cybersecurity and information
security is that it's a daily learning experience.
And one of the ways that I learn the best is asking colleagues,
(01:53):
asking those who I believe have more information and getting their opinions
on certain subjects that allow me to learn and grow myself.
So thank you very much for the camaraderie, the collaboration,
the discussions that we've had.
And when it came to Crouch, right, one of the things that I had first asked
(02:16):
you was, how could this have been prevented from the, just in the realms of Delta Airlines? lines.
Because here we are five, six days into this thing,
and there are news reports everywhere about luggage being here and not being
(02:36):
able to reach pilots and flight attendants.
And I looked at it from a security standpoint of, wait a minute,
wouldn't Delta test this update prior prior to deploying it,
but in my head, I thought, you know, well, if there's anyone who I'm going to trust,
(02:57):
it's going to be my cybersecurity vendor with the reputation of CrowdStrike.
And that's kind of where you led into Node.js.
That's not the way I would have a third-party security vendor rely on me to do their testing.
Can you elaborate a little bit more on that? Because I thought it was extremely
(03:18):
interesting to hear your opinion as far as the trustability of your upstream
vendor when it comes to security or anyone else for that matter.
No, thank you for that. Yes, I will absolutely answer that question.
And I think that this matter elucidates the necessity of having a ridiculously
experienced technology executive at your fingertips,
(03:42):
which is what I think most organizations lack,
because you can't just get the details that you need by reading bleeping computer every day.
It just doesn't work that way.
So I think a lot of people thought this concept of testing the updates before they get deployed.
(04:05):
And they're like, oh, well, this would have avoided the entire thing.
And that's true to the degree that the company testing the updates is actually
the software vendor themselves.
Where I think the distinction has to be made is that it's not only not reasonable for a customer,
(04:29):
so I'm going to call a customer as an end user or the IT department of the customer
as the end user of that software. work.
It's not only not reasonable for them to be doing test deployments,
but it also isn't legally defensible.
And I think that's the piece that nobody's really talking about.
So whenever you're picking a software package or you're picking a paradigm or
(04:53):
a strategy like this is going to be our process, there's always this push, push, push to try to.
To balance high security and operational functionality with low total cost of ownership.
So if you're going to have really high security and you're going to have high
(05:14):
operational functionality, there's a lot of babysitting involved in that.
But yet they want low total cost of ownership.
So part of the theory is, okay, well, where can we get our economies of scale?
Theoretically, you would expect that the software manufacturer themselves would
have the capabilities to have that economies of scale.
(05:35):
It would be basically a simple thing for them to be able to have a test lab
and then to have their own internal team where it's their own internal employees doing that testing.
And in fact, that is the exact process that WatchGuard uses.
One of the things that I saw that was really interesting that came out of this
whole debacle was that a number of software companies literally said,
(05:57):
hey, this is our process.
And in all these years, WatchGuard, which now owns Panda, Panda Security.
And certainly now the WatchGuard team has been doing that sort of staged deployment,
which is completely economical for the software vendor to do.
It is not economical at all for
(06:17):
the end user of
the software to do and if you're like the it manager
it director the cso for delta
airlines for example and then if you
decided oh well we're going to delay the deployment of that update of our business
critical endpoint protection platform and then the delay of that update the
(06:40):
deployment of that is the thing that actually leads to a breach that's not a
legally defensible approach, right?
In fact, you could make it, the insurance company could make an argument that
says, you didn't deploy this update in a timely fashion.
So therefore, we're going to now deny your insurance coverage.
And there's something really important that people need to understand when it
(07:03):
comes to technologies that provide security, whether you're talking about endpoint
protection, web content filtering, whatever.
It's technologies that deliver security filtration of some shape, size, size or flavor.
Two main categories of that type of stuff. It's the stuff that updates once
an hour, and it's the stuff that updates real time.
(07:25):
So it's important to understand that those two categories,
it updates in real time, it updates once an hour, neither one of those are conducive
to a scenario that says I'm going to have a delayed update schedule, right?
That's kind of like, we want the security protections as quickly as that data is available.
(07:49):
I mean, the entire concept of SOC and MDR and XDR and all of those things is
this entire concept that says,
we're going to take Intel that we see over here in this one little area and
extrapolate that into protections across a much wider audience,
whether that's inside the same company or if a company is leveraging the security
(08:12):
technologies from an upstream vendor who is aggregating all of that live Intel data.
And in fact, that is one of the big pieces that Microsoft uses as their argumentation
for why they think everybody should use Windows Defender.
And their argument is that, we have all of this live real-time data from millions
(08:35):
of endpoints around the entire world,
So that's why people should use Microsoft Defender is because it has that ability to share the Intel,
the data feed for that security intelligence,
which frankly, they all work that way.
So that's where that bifurcation has to exist, where the software vendor,
(08:56):
they can do the delayed testing and the delayed deployment and be okie-dokie.
And in fact, they need to because they're the only ones that can economically
do it due to the economies of scale.
But the end customer, they legally should not be doing that.
Well, let me respond to that and maybe a couple of follow-up questions because
(09:16):
it seems to me that the director of IT or the CTO or the CISO at Delta is probably,
or I think there's a likelihood that the figure is being pointed at them to
say, how could you not test?
I would assume that, just from my experience alone,
that when it comes to the functionality of a business and when technology is
(09:41):
interfering with revenue and profit,
and the report I read this morning was it's up to $5.8 billion in losses worldwide,
which is a staggering number.
And I'm sure you know this already, but that the customer agreement with CrowdStrike
is that CrowdStrike can't be held liable for any of that. Of course it is.
(10:03):
I mean, we're laughing, but we both know that that is very common, right?
Because it's, of course, they're not going to be held liable.
But this is not a cybersecurity attack.
But at the same time, it shows the world. Well, it sure as hell was a denial
of service attack, wasn't it?
(10:24):
Right. It shows the world what technology can do, even if it was just a human
error. Now, you had mentioned.
Well, so it was denial of service in that it was a denial of service from what
people had mistakenly made the decision was a trustworthy counterparty.
(10:45):
When I knew years ago, CrowdStrike was not a trustworthy counterparty.
And we'll circle back to that point.
Okay. Yeah, I would like you to. But now we have a company like CrowdStrike
who, like you said, it's feasible, economically feasible for them to do the testing.
Yet, well, I even read they test and test and test.
(11:06):
But if that were the case, how in the world could this have happened?
So, okay. So, I mean, like, let's just take this from like an IT service provider perspective.
If I was going to, let's say I had clients that had SQL 2014, SQL 2016, SQL 2022.
And if I'm going to air quote test patches or test updates, then I have to basically
(11:31):
maintain the expense of that entire lab.
Okay, so now in the Wayback Machine, I used to be one of the senior security
architects for a 13,000 user company.
And a 13,000 user company did not have a test lab for that kind of stuff.
Okay so i don't know
how many users exist in delta but what i can say is that their test lab cannot
(11:58):
possibly emulate what would actually happen in a production environment so whenever
people are like test lab test lab test lab,
I'm telling you from my 30 years of experience in the technology industry and
having been the person who deploys updates and deploys patches for almost that
(12:20):
entire duration of time, the best testing is the testing that's done in production.
And one of the things that WatchGuard brought to light when they talked about
their process for testing, they do a lovely job of it where they have their test lab,
but then they do kind of a slow roll deploy to the people who are going to best
(12:42):
be able to handle the business interruption.
It's kind of this like staged deployment. So it could be that Delta was doing something like that.
But from like a CTO perspective and a strategy perspective,
I think that the concept that says that they should be doing this kind of staged
(13:05):
deployment, that's not even how I would address this.
I would address it fundamentally at a much more core systemic concept of just
simply saying, why would you
take critical infrastructure and run it on an end-user operating system?
And that's my position on this because Windows is inherently an operating system
(13:27):
that, let's just take Patch Tuesday as an example.
If I'm going to have an endpoint that's on the network, work,
I need to make sure that that thing is patched, that the threat landscape or
the threat surface of that computer is fairly reduced pretty substantially.
So in order to do that, if it's patched Tuesday, and let's say there's a Chrome
vulnerability and a Firefox vulnerability,
(13:48):
and then there's a .NET Framework update, and then there's a patch for the operating
system, and then there's a cumulative update, I've now basically said there's
X number of updates that I need to get all done in the same day.
Now, in order to do that, I'm looking at what?
Five reboot windows. I swear, this is literally like you have to have this patch
(14:09):
reboot, this patch reboot, this patch reboot, this patch reboot.
So literally, this is what actually happens in the Windows world when you have
a Windows operating system.
So let's say you were running your phone system on that Windows operating system.
You can have your phone system down five times that day. I mean,
this is bonkers. This is why you run your phone system on a Linux operating system.
(14:30):
This is why your network layer security appliances don't run on Windows.
I personally do not think that you should ever run critical infrastructure on
anything that is an end user operating system, which is fundamentally what Windows is.
It is fundamentally something that it is not designed to have the kind of uptime
(14:52):
that an AIX server or AS400 would have.
And if I was the CTO of Delta, what I would be doing is I'd be saying,
why on earth are we using Windows computers?
At the checkout stations, at the flight lines, you know, the lines that people
(15:13):
walk up to and they want to talk to a, you know, flight boarding assistant,
whatever their title is.
Why do I want to use a Windows computer to project to the display in the airport.
What my boarding times are for anything?
Why would I do that?
I would absolutely not do that. I mean, if anything, what I would be doing is
(15:36):
I'd either be running these people on Chromebooks,
I'd be running everything off of an iPad to a website or to an app or,
you know, Android tablets to, you know, a website,
you know, but fundamentally what I'm talking about here is the back end system
is something that can be accessed with an app or a website.
(15:57):
Full stop. That's it. Webify the whole thing. And in the event that you're trying
to project a display to these like, you know, boarding displays,
here's our boarding times.
Don't use Windows for that. Like, fundamentally, as soon as you say,
I want to use Windows for that, you're now saying, I want to have a fairly substantial
(16:19):
high total cost of ownership for something. And notice, I'm not anti-Windows on everything.
What I'm saying is that Windows is a Swiss army knife operating system,
right? It can do your accounting.
It can do your business critical software applications.
You know, it can run desktop apps and do email and do web browsing and you can
shop on it and you can, I mean, it's a Swiss army knife operating system.
(16:42):
So it's fantastic for an end user platform where you have a knowledge worker
who needs to be able to do a whole bunch of things.
What it is not a good fit for is an operating system where you need it to do this one thing.
Because as soon as you say, I'm going to use a Windows operating system now
from a legally defensible perspective.
(17:03):
Now, let's look at it from the insurance standpoint.
Many of these companies had CrowdStrike exclusively just simply because of the
fact that their insurance company said they had to have it.
And then, so as soon as you have this Windows endpoint,
you're having to fill out this answer on your insurance application that says,
what's the percentage of your endpoints that have endpoint protection on,
(17:26):
you know, that are covered by your endpoint protection platform?
I would rather as the CISO, I would prefer to be able to produce a report that
says these are our point of sale systems and they're basically a bunch of Android tablets.
And we have these in a mobile device management platform and we have them locked down, whatever.
(17:46):
Or I'd like to say, nope, we just have a bunch of Chromebooks and this is what
we use for it. And then we have the Chromebooks locked down and blah, blah.
I mean, I don't need to run an antivirus on a Chromebook.
I don't need to run an endpoint protection on a Chromebook.
And I can put the Chromebooks all on their own little isolated VLAN and I can
(18:09):
make the Chromebooks only talk to the certain things that I want them to talk to.
So you're talking about, I'm going to to create a security zone profile that
says, these are business critical functions.
These are not the computers that people at headquarters are using to do email
and accounting from and whatnot.
(18:31):
You don't try and take a knowledge worker like myself and stick them on a Chromebook.
That's a recipe for like a fork in the eye.
But the Chromebook might be the right solution
or i'd even go so
far as to say if i was the cto i would
probably be using raspberry pies to run
those displays of the flight times seriously
(18:55):
dead serious it would probably be ubuntu and a raspberry pie there you go you
know no i think your your perspective is absolutely spot on because you're you're
talking about, you and I discussed this before,
you know, windows being an end user operating system, it can do a lot of things.
(19:17):
We're not trashing windows here.
What we are questioning is, you know, just the, what I had first called you
about, how could this have been prevented?
And I say testing and you say testing.
Windows. How is this possible? So this is kind of where, for everyone who's listening,
(19:40):
the perspective that I get from Felicia is that when I am thinking one way,
Felicia can absolutely say,
Jeff, hold on to that thought for a moment, then let me explain to you where
my experience and my knowledge base will show you that in this simple process,
and maybe it's not simple, bomb use the net word loosely,
utilizing a different operating system to do exactly what you were just saying
(20:05):
would have prevented this meltdown of an airline.
Other airlines have come back online. Other companies are back online.
There are some fixes. There are some manual requirements to get this back online.
Can you tell me about that a little bit?
Yeah. Yeah. And I just want to add to something that you just talked about as
(20:26):
well, which is that, so beyond the whole, we got to get back online piece,
there's like two core things that you have to think about when you're going
to select a technology and a strategy.
And you're saying, what's my current total cost of ownership,
right? What's my burden rate, my run rate for that solution?
And then what's my red line case? So in this instance,
(20:48):
the burden rate or total cost of ownership of a lot of these other technologies that I'm talking about,
the burden rate or the the total cost of ownership for just the running of that
in those kind of tighter security zone profile use cases is a lot less than
if Windows would have used.
Certainly in the red line case, it's a lot less because what happens if your
(21:11):
Raspberry Pi is a naughty biscuit?
Well, I mean, I can tell you, I would have been able to go into the switch.
And just been like, shut down these ports and reboot the Raspberry Pi.
If you take the power away from a Raspberry Pi remotely, you can power cycle.
It's the same as a wireless access point or a desk phone or whatever, right?
(21:32):
So now you're talking about real economies of scale in that redline case versus
what these companies had to do in their redline case, TCO,
was somebody had to physically, manually do something to each individual busted computer.
I mean, it was like apocalypse now.
(21:54):
Right, right. You know, put it into safe mode and then do some manual stuff
on it. So it was really one of two things.
Either an IT person had to manually go visit every single computer and manually
take action on it, or the IT person had to somehow support an end user doing that themselves.
(22:14):
In both cases, this is apocalyptically expensive and not something that can just be done remotely.
There is a very sweetness sometimes associated with something where you're looking
at like a Google Chromebook or you're maybe talking about an Apple iPad or again, an Android tablet.
(22:36):
In all of those cases, there is a factory reset option.
So if poo-poo hits a fan, like this red line case we're talking about.
You could just tell the end user, factory reset that turkey, right?
And then it talks to the MDM and it gets its new stuff and it does its little magical dance.
And, you know, like you could make an argument that theoretically you could
(22:58):
use Windows Autopilot to do that, but not so much, right?
It's like, again, Windows is a really, really powerful operating system.
And one of the things that's powerful about it is that it'll run on a vast,
diverse set of hardware.
Versus like an Apple operating system is only going to run on this hardware, right?
(23:21):
A Chrome OS is only going to run on a Chromebook, you know, like,
and so those other things, those other devices, I kind of look at them as like
the IOT stuff, right? The internet of things.
And so it's like a desk phone where you have this like factory reset button.
And so in the case where you're sending out a device where you need its recovery
(23:41):
time period to be pretty short from a massive poo-poo hits the fan scenario,
then again, Windows isn't exactly the right choice for that.
We have an article that we provide to our customers as part of our knowledge base.
And that article basically says, if you're looking to have business continuity
for your core machine, let's say you're an independent architect or an independent
(24:05):
tax preparer or something like that.
If you don't If you don't have two computers providing your own level of business continuity,
then you shouldn't have any expectation that when something goes wrong with
that one computer that you're going to continue to be able to work because Windows
doesn't have a factory reset button and like everything just magically starts working again.
(24:26):
Agreed, agreed. And this is kind of the hard lessons that everyone's learning.
And from, you know, my initial thought was there was some sort of cyber attack.
And you and I kind of laughed at that. And you're like, no, Jeff, this isn't an update.
And, you know, and that's kind of what the world is looking at is,
you know, how can something like this happen?
But this is a prime example of a conversation you and I have had that technology
(24:53):
drives revenue and profit.
And without it, what will you have?
So this is a, you know, whether this is the worst day of someone's career or life at,
you know, CrowdStrike because, you know, folder was not properly tested and
(25:13):
a file was sent out to nine and a half million machines that disabled half the earth.
I mean, Jesus, right? This is kind of one of those learning lessons of this
is just an update. You know, we are battling cyber issues every single day.
We see it every single day.
(25:34):
And again, this was a mistake compared to what what people could be facing,
you know, in the upcoming years. Yeah.
So let's pivot to the whole counterparty risk thing, because I have to admit
that I was over here like the cat that ate the bird because I'm like,
I told you so, everybody.
CrowdStrike, in my view, was never a trustworthy counterparty.
(25:58):
And I would have never used CrowdStrike under any circumstances because CrowdStrike
was infamous for falsely certifying that it was a Russian hack of DNC computers
that actually led to the leaking of emails and documents to WikiLeaks.
When in fact, it was actually Seth Rich who leaked the incriminating emails.
(26:19):
Okay, so this is CrowdStrike. Right. They said that.
And then and then, OK, CrowdStrike was also guilty of falsely confirming conspiracy
theories around Russian collusion around the, you know, two elections ago. Right.
And and then and then here's the the crumb of it is that CrowdStrike refused
(26:41):
to give data to the FBI. And it's like, oh, get out of here with that.
So, I mean, CrowdStrike publicly,
internationally in two major, massive, high-profile events has fully discredited
themselves as basically being a tool for the deep state.
(27:04):
And it's like, no, I don't want that as a counterparty. I mean,
I feel like as a business leader, as a business decision maker, there is so much.
Of the the the covert
economy and the covert risks
that you have to attempt to manage that
(27:28):
you have to be focused on that reality because that's
actually real as opposed to the propaganda that exists in the overt economy
which for the most part is just nothing but mendacity after mendacity after
mendacity you know and so you can't pay attention to those things and use those
as your decision-making pieces.
(27:48):
And so when I see a company like CrowdStrike being completely obvious with what
is at the core of their decision-making practices and what they...
Those two things basically revealed key pieces about their character and integrity
and how they were going to make decisions.
And so that flows down to everything. That's how you hire people,
(28:11):
What's the culture of your company?
What's the tone that you set? What do you prioritize? Are you prioritizing a
meritocracy or are you creating a culture that tolerates mendacity and not only
tolerates it, but actually promotes it systemically?
So the last 30 seconds is for you, Jeff.
Well, I'll tell you what, Felicia, it's an absolute pleasure every time we get
(28:33):
to talk to you, being on your podcast, we're learning something every day.
And for everyone who's listening, You know, we all have to, no matter what your
experience level is, no matter how you consider yourself as an expert on a certain
subject, you know, we have to understand where our lane is.
And my lane is this. It is straightforward. forward. And when there's some sort
(28:53):
of roadblock or I don't understand something, I know that Felicia King will
answer the call, will guide me through any of my misconceptions,
any of my misinformation, or
at least certify or validate what I'm feeling towards any specific events.
Felicia, having a relationship with you is invaluable, just absolutely priceless.
And I appreciate answering all my questions. Thank you much, Jeff.
Good morning. You're listening to Breakfast Bites, and I'm Felicia King.
And joining me today is Jeff Berner, who's a colleague of mine in the cybersecurity
space, and he hails from Florida.
And Jeff contacted me about this CrowdStrike thing, just to get my opinion.
We had one heck of a good conversation about it.
(00:21):
And so I said, wow, considering the impact that this has made,
I figured it would be good to have a discussion and share it with a wider audience.
And before we get started on that, I want to comment that I saw plenty of people
writing articles about CrowdStrike.
And the thing that I found was that nobody has yet written an article about
(00:47):
the stuff associated with CrowdStrike talking about really two key elements.
One was around the fact that CrowdStrike just wasn't a trustworthy counterparty
to begin with, so we can talk about that.
The second one really had to do with the legal defensibility of delayed security updates.
(01:11):
There. So let's talk about that in more detail.
So, you know, Jeff, do you want to start us off with a little bit about you
and then, you know, hover into like some questions for me about CrowdStrike?
Absolutely. And thank you for the introduction and having me on your podcast.
No, my name is Jeff Berner. I am the CEO of IT Consulting St.
(01:32):
Petersburg, recently certified as a virtual chief information security officer.
And what I've learned in the short time that I, And when I say short time,
the six or seven years that I've been researching the cybersecurity and information
security is that it's a daily learning experience.
And one of the ways that I learn the best is asking colleagues,
(01:53):
asking those who I believe have more information and getting their opinions
on certain subjects that allow me to learn and grow myself.
So thank you very much for the camaraderie, the collaboration,
the discussions that we've had.
And when it came to Crouch, right, one of the things that I had first asked
(02:16):
you was, how could this have been prevented from the, just in the realms of Delta Airlines? lines.
Because here we are five, six days into this thing,
and there are news reports everywhere about luggage being here and not being
(02:36):
able to reach pilots and flight attendants.
And I looked at it from a security standpoint of, wait a minute,
wouldn't Delta test this update prior prior to deploying it,
but in my head, I thought, you know, well, if there's anyone who I'm going to trust,
(02:57):
it's going to be my cybersecurity vendor with the reputation of CrowdStrike.
And that's kind of where you led into Node.js.
That's not the way I would have a third-party security vendor rely on me to do their testing.
Can you elaborate a little bit more on that? Because I thought it was extremely
(03:18):
interesting to hear your opinion as far as the trustability of your upstream
vendor when it comes to security or anyone else for that matter.
No, thank you for that. Yes, I will absolutely answer that question.
And I think that this matter elucidates the necessity of having a ridiculously
experienced technology executive at your fingertips,
(03:42):
which is what I think most organizations lack,
because you can't just get the details that you need by reading bleeping computer every day.
It just doesn't work that way.
So I think a lot of people thought this concept of testing the updates before they get deployed.
(04:05):
And they're like, oh, well, this would have avoided the entire thing.
And that's true to the degree that the company testing the updates is actually
the software vendor themselves.
Where I think the distinction has to be made is that it's not only not reasonable for a customer,
(04:29):
so I'm going to call a customer as an end user or the IT department of the customer
as the end user of that software. work.
It's not only not reasonable for them to be doing test deployments,
but it also isn't legally defensible.
And I think that's the piece that nobody's really talking about.
So whenever you're picking a software package or you're picking a paradigm or
(04:53):
a strategy like this is going to be our process, there's always this push, push, push to try to.
To balance high security and operational functionality with low total cost of ownership.
So if you're going to have really high security and you're going to have high
(05:14):
operational functionality, there's a lot of babysitting involved in that.
But yet they want low total cost of ownership.
So part of the theory is, okay, well, where can we get our economies of scale?
Theoretically, you would expect that the software manufacturer themselves would
have the capabilities to have that economies of scale.
(05:35):
It would be basically a simple thing for them to be able to have a test lab
and then to have their own internal team where it's their own internal employees doing that testing.
And in fact, that is the exact process that WatchGuard uses.
One of the things that I saw that was really interesting that came out of this
whole debacle was that a number of software companies literally said,
(05:57):
hey, this is our process.
And in all these years, WatchGuard, which now owns Panda, Panda Security.
And certainly now the WatchGuard team has been doing that sort of staged deployment,
which is completely economical for the software vendor to do.
It is not economical at all for
(06:17):
the end user of
the software to do and if you're like the it manager
it director the cso for delta
airlines for example and then if you
decided oh well we're going to delay the deployment of that update of our business
critical endpoint protection platform and then the delay of that update the
(06:40):
deployment of that is the thing that actually leads to a breach that's not a
legally defensible approach, right?
In fact, you could make it, the insurance company could make an argument that
says, you didn't deploy this update in a timely fashion.
So therefore, we're going to now deny your insurance coverage.
And there's something really important that people need to understand when it
(07:03):
comes to technologies that provide security, whether you're talking about endpoint
protection, web content filtering, whatever.
It's technologies that deliver security filtration of some shape, size, size or flavor.
Two main categories of that type of stuff. It's the stuff that updates once
an hour, and it's the stuff that updates real time.
(07:25):
So it's important to understand that those two categories,
it updates in real time, it updates once an hour, neither one of those are conducive
to a scenario that says I'm going to have a delayed update schedule, right?
That's kind of like, we want the security protections as quickly as that data is available.
(07:49):
I mean, the entire concept of SOC and MDR and XDR and all of those things is
this entire concept that says,
we're going to take Intel that we see over here in this one little area and
extrapolate that into protections across a much wider audience,
whether that's inside the same company or if a company is leveraging the security
(08:12):
technologies from an upstream vendor who is aggregating all of that live Intel data.
And in fact, that is one of the big pieces that Microsoft uses as their argumentation
for why they think everybody should use Windows Defender.
And their argument is that, we have all of this live real-time data from millions
(08:35):
of endpoints around the entire world,
So that's why people should use Microsoft Defender is because it has that ability to share the Intel,
the data feed for that security intelligence,
which frankly, they all work that way.
So that's where that bifurcation has to exist, where the software vendor,
(08:56):
they can do the delayed testing and the delayed deployment and be okie-dokie.
And in fact, they need to because they're the only ones that can economically
do it due to the economies of scale.
But the end customer, they legally should not be doing that.
Well, let me respond to that and maybe a couple of follow-up questions because
(09:16):
it seems to me that the director of IT or the CTO or the CISO at Delta is probably,
or I think there's a likelihood that the figure is being pointed at them to
say, how could you not test?
I would assume that, just from my experience alone,
that when it comes to the functionality of a business and when technology is
(09:41):
interfering with revenue and profit,
and the report I read this morning was it's up to $5.8 billion in losses worldwide,
which is a staggering number.
And I'm sure you know this already, but that the customer agreement with CrowdStrike
is that CrowdStrike can't be held liable for any of that. Of course it is.
(10:03):
I mean, we're laughing, but we both know that that is very common, right?
Because it's, of course, they're not going to be held liable.
But this is not a cybersecurity attack.
But at the same time, it shows the world. Well, it sure as hell was a denial
of service attack, wasn't it?
(10:24):
Right. It shows the world what technology can do, even if it was just a human
error. Now, you had mentioned.
Well, so it was denial of service in that it was a denial of service from what
people had mistakenly made the decision was a trustworthy counterparty.
(10:45):
When I knew years ago, CrowdStrike was not a trustworthy counterparty.
And we'll circle back to that point.
Okay. Yeah, I would like you to. But now we have a company like CrowdStrike
who, like you said, it's feasible, economically feasible for them to do the testing.
Yet, well, I even read they test and test and test.
(11:06):
But if that were the case, how in the world could this have happened?
So, okay. So, I mean, like, let's just take this from like an IT service provider perspective.
If I was going to, let's say I had clients that had SQL 2014, SQL 2016, SQL 2022.
And if I'm going to air quote test patches or test updates, then I have to basically
(11:31):
maintain the expense of that entire lab.
Okay, so now in the Wayback Machine, I used to be one of the senior security
architects for a 13,000 user company.
And a 13,000 user company did not have a test lab for that kind of stuff.
Okay so i don't know
how many users exist in delta but what i can say is that their test lab cannot
(11:58):
possibly emulate what would actually happen in a production environment so whenever
people are like test lab test lab test lab,
I'm telling you from my 30 years of experience in the technology industry and
having been the person who deploys updates and deploys patches for almost that
(12:20):
entire duration of time, the best testing is the testing that's done in production.
And one of the things that WatchGuard brought to light when they talked about
their process for testing, they do a lovely job of it where they have their test lab,
but then they do kind of a slow roll deploy to the people who are going to best
(12:42):
be able to handle the business interruption.
It's kind of this like staged deployment. So it could be that Delta was doing something like that.
But from like a CTO perspective and a strategy perspective,
I think that the concept that says that they should be doing this kind of staged
(13:05):
deployment, that's not even how I would address this.
I would address it fundamentally at a much more core systemic concept of just
simply saying, why would you
take critical infrastructure and run it on an end-user operating system?
And that's my position on this because Windows is inherently an operating system
(13:27):
that, let's just take Patch Tuesday as an example.
If I'm going to have an endpoint that's on the network, work,
I need to make sure that that thing is patched, that the threat landscape or
the threat surface of that computer is fairly reduced pretty substantially.
So in order to do that, if it's patched Tuesday, and let's say there's a Chrome
vulnerability and a Firefox vulnerability,
(13:48):
and then there's a .NET Framework update, and then there's a patch for the operating
system, and then there's a cumulative update, I've now basically said there's
X number of updates that I need to get all done in the same day.
Now, in order to do that, I'm looking at what?
Five reboot windows. I swear, this is literally like you have to have this patch
(14:09):
reboot, this patch reboot, this patch reboot, this patch reboot.
So literally, this is what actually happens in the Windows world when you have
a Windows operating system.
So let's say you were running your phone system on that Windows operating system.
You can have your phone system down five times that day. I mean,
this is bonkers. This is why you run your phone system on a Linux operating system.
(14:30):
This is why your network layer security appliances don't run on Windows.
I personally do not think that you should ever run critical infrastructure on
anything that is an end user operating system, which is fundamentally what Windows is.
It is fundamentally something that it is not designed to have the kind of uptime
(14:52):
that an AIX server or AS400 would have.
And if I was the CTO of Delta, what I would be doing is I'd be saying,
why on earth are we using Windows computers?
At the checkout stations, at the flight lines, you know, the lines that people
(15:13):
walk up to and they want to talk to a, you know, flight boarding assistant,
whatever their title is.
Why do I want to use a Windows computer to project to the display in the airport.
What my boarding times are for anything?
Why would I do that?
I would absolutely not do that. I mean, if anything, what I would be doing is
(15:36):
I'd either be running these people on Chromebooks,
I'd be running everything off of an iPad to a website or to an app or,
you know, Android tablets to, you know, a website,
you know, but fundamentally what I'm talking about here is the back end system
is something that can be accessed with an app or a website.
(15:57):
Full stop. That's it. Webify the whole thing. And in the event that you're trying
to project a display to these like, you know, boarding displays,
here's our boarding times.
Don't use Windows for that. Like, fundamentally, as soon as you say,
I want to use Windows for that, you're now saying, I want to have a fairly substantial
(16:19):
high total cost of ownership for something. And notice, I'm not anti-Windows on everything.
What I'm saying is that Windows is a Swiss army knife operating system,
right? It can do your accounting.
It can do your business critical software applications.
You know, it can run desktop apps and do email and do web browsing and you can
shop on it and you can, I mean, it's a Swiss army knife operating system.
(16:42):
So it's fantastic for an end user platform where you have a knowledge worker
who needs to be able to do a whole bunch of things.
What it is not a good fit for is an operating system where you need it to do this one thing.
Because as soon as you say, I'm going to use a Windows operating system now
from a legally defensible perspective.
(17:03):
Now, let's look at it from the insurance standpoint.
Many of these companies had CrowdStrike exclusively just simply because of the
fact that their insurance company said they had to have it.
And then, so as soon as you have this Windows endpoint,
you're having to fill out this answer on your insurance application that says,
what's the percentage of your endpoints that have endpoint protection on,
(17:26):
you know, that are covered by your endpoint protection platform?
I would rather as the CISO, I would prefer to be able to produce a report that
says these are our point of sale systems and they're basically a bunch of Android tablets.
And we have these in a mobile device management platform and we have them locked down, whatever.
(17:46):
Or I'd like to say, nope, we just have a bunch of Chromebooks and this is what
we use for it. And then we have the Chromebooks locked down and blah, blah.
I mean, I don't need to run an antivirus on a Chromebook.
I don't need to run an endpoint protection on a Chromebook.
And I can put the Chromebooks all on their own little isolated VLAN and I can
(18:09):
make the Chromebooks only talk to the certain things that I want them to talk to.
So you're talking about, I'm going to to create a security zone profile that
says, these are business critical functions.
These are not the computers that people at headquarters are using to do email
and accounting from and whatnot.
(18:31):
You don't try and take a knowledge worker like myself and stick them on a Chromebook.
That's a recipe for like a fork in the eye.
But the Chromebook might be the right solution
or i'd even go so
far as to say if i was the cto i would
probably be using raspberry pies to run
those displays of the flight times seriously
(18:55):
dead serious it would probably be ubuntu and a raspberry pie there you go you
know no i think your your perspective is absolutely spot on because you're you're
talking about, you and I discussed this before,
you know, windows being an end user operating system, it can do a lot of things.
(19:17):
We're not trashing windows here.
What we are questioning is, you know, just the, what I had first called you
about, how could this have been prevented?
And I say testing and you say testing.
Windows. How is this possible? So this is kind of where, for everyone who's listening,
(19:40):
the perspective that I get from Felicia is that when I am thinking one way,
Felicia can absolutely say,
Jeff, hold on to that thought for a moment, then let me explain to you where
my experience and my knowledge base will show you that in this simple process,
and maybe it's not simple, bomb use the net word loosely,
utilizing a different operating system to do exactly what you were just saying
(20:05):
would have prevented this meltdown of an airline.
Other airlines have come back online. Other companies are back online.
There are some fixes. There are some manual requirements to get this back online.
Can you tell me about that a little bit?
Yeah. Yeah. And I just want to add to something that you just talked about as
(20:26):
well, which is that, so beyond the whole, we got to get back online piece,
there's like two core things that you have to think about when you're going
to select a technology and a strategy.
And you're saying, what's my current total cost of ownership,
right? What's my burden rate, my run rate for that solution?
And then what's my red line case? So in this instance,
(20:48):
the burden rate or total cost of ownership of a lot of these other technologies that I'm talking about,
the burden rate or the the total cost of ownership for just the running of that
in those kind of tighter security zone profile use cases is a lot less than
if Windows would have used.
Certainly in the red line case, it's a lot less because what happens if your
(21:11):
Raspberry Pi is a naughty biscuit?
Well, I mean, I can tell you, I would have been able to go into the switch.
And just been like, shut down these ports and reboot the Raspberry Pi.
If you take the power away from a Raspberry Pi remotely, you can power cycle.
It's the same as a wireless access point or a desk phone or whatever, right?
(21:32):
So now you're talking about real economies of scale in that redline case versus
what these companies had to do in their redline case, TCO,
was somebody had to physically, manually do something to each individual busted computer.
I mean, it was like apocalypse now.
(21:54):
Right, right. You know, put it into safe mode and then do some manual stuff
on it. So it was really one of two things.
Either an IT person had to manually go visit every single computer and manually
take action on it, or the IT person had to somehow support an end user doing that themselves.
(22:14):
In both cases, this is apocalyptically expensive and not something that can just be done remotely.
There is a very sweetness sometimes associated with something where you're looking
at like a Google Chromebook or you're maybe talking about an Apple iPad or again, an Android tablet.
(22:36):
In all of those cases, there is a factory reset option.
So if poo-poo hits a fan, like this red line case we're talking about.
You could just tell the end user, factory reset that turkey, right?
And then it talks to the MDM and it gets its new stuff and it does its little magical dance.
And, you know, like you could make an argument that theoretically you could
(22:58):
use Windows Autopilot to do that, but not so much, right?
It's like, again, Windows is a really, really powerful operating system.
And one of the things that's powerful about it is that it'll run on a vast,
diverse set of hardware.
Versus like an Apple operating system is only going to run on this hardware, right?
(23:21):
A Chrome OS is only going to run on a Chromebook, you know, like,
and so those other things, those other devices, I kind of look at them as like
the IOT stuff, right? The internet of things.
And so it's like a desk phone where you have this like factory reset button.
And so in the case where you're sending out a device where you need its recovery
(23:41):
time period to be pretty short from a massive poo-poo hits the fan scenario,
then again, Windows isn't exactly the right choice for that.
We have an article that we provide to our customers as part of our knowledge base.
And that article basically says, if you're looking to have business continuity
for your core machine, let's say you're an independent architect or an independent
(24:05):
tax preparer or something like that.
If you don't If you don't have two computers providing your own level of business continuity,
then you shouldn't have any expectation that when something goes wrong with
that one computer that you're going to continue to be able to work because Windows
doesn't have a factory reset button and like everything just magically starts working again.
(24:26):
Agreed, agreed. And this is kind of the hard lessons that everyone's learning.
And from, you know, my initial thought was there was some sort of cyber attack.
And you and I kind of laughed at that. And you're like, no, Jeff, this isn't an update.
And, you know, and that's kind of what the world is looking at is,
you know, how can something like this happen?
But this is a prime example of a conversation you and I have had that technology
(24:53):
drives revenue and profit.
And without it, what will you have?
So this is a, you know, whether this is the worst day of someone's career or life at,
you know, CrowdStrike because, you know, folder was not properly tested and
(25:13):
a file was sent out to nine and a half million machines that disabled half the earth.
I mean, Jesus, right? This is kind of one of those learning lessons of this
is just an update. You know, we are battling cyber issues every single day.
We see it every single day.
(25:34):
And again, this was a mistake compared to what what people could be facing,
you know, in the upcoming years. Yeah.
So let's pivot to the whole counterparty risk thing, because I have to admit
that I was over here like the cat that ate the bird because I'm like,
I told you so, everybody.
CrowdStrike, in my view, was never a trustworthy counterparty.
(25:58):
And I would have never used CrowdStrike under any circumstances because CrowdStrike
was infamous for falsely certifying that it was a Russian hack of DNC computers
that actually led to the leaking of emails and documents to WikiLeaks.
When in fact, it was actually Seth Rich who leaked the incriminating emails.
(26:19):
Okay, so this is CrowdStrike. Right. They said that.
And then and then, OK, CrowdStrike was also guilty of falsely confirming conspiracy
theories around Russian collusion around the, you know, two elections ago. Right.
And and then and then here's the the crumb of it is that CrowdStrike refused
(26:41):
to give data to the FBI. And it's like, oh, get out of here with that.
So, I mean, CrowdStrike publicly,
internationally in two major, massive, high-profile events has fully discredited
themselves as basically being a tool for the deep state.
(27:04):
And it's like, no, I don't want that as a counterparty. I mean,
I feel like as a business leader, as a business decision maker, there is so much.
Of the the the covert
economy and the covert risks
that you have to attempt to manage that
(27:28):
you have to be focused on that reality because that's
actually real as opposed to the propaganda that exists in the overt economy
which for the most part is just nothing but mendacity after mendacity after
mendacity you know and so you can't pay attention to those things and use those
as your decision-making pieces.
(27:48):
And so when I see a company like CrowdStrike being completely obvious with what
is at the core of their decision-making practices and what they...
Those two things basically revealed key pieces about their character and integrity
and how they were going to make decisions.
And so that flows down to everything. That's how you hire people,
(28:11):
What's the culture of your company?
What's the tone that you set? What do you prioritize? Are you prioritizing a
meritocracy or are you creating a culture that tolerates mendacity and not only
tolerates it, but actually promotes it systemically?
So the last 30 seconds is for you, Jeff.
Well, I'll tell you what, Felicia, it's an absolute pleasure every time we get
(28:33):
to talk to you, being on your podcast, we're learning something every day.
And for everyone who's listening, You know, we all have to, no matter what your
experience level is, no matter how you consider yourself as an expert on a certain
subject, you know, we have to understand where our lane is.
And my lane is this. It is straightforward. forward. And when there's some sort
(28:53):
of roadblock or I don't understand something, I know that Felicia King will
answer the call, will guide me through any of my misconceptions,
any of my misinformation, or
at least certify or validate what I'm feeling towards any specific events.
Felicia, having a relationship with you is invaluable, just absolutely priceless.
And I appreciate answering all my questions. Thank you much, Jeff.
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
The Breakfast Club
The World's Most Dangerous Morning Show, The Breakfast Club, With DJ Envy And Charlamagne Tha God!
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.