March 2, 2023 • 47 mins
This is the second part of the interview with Soheil Katal, the CIO for LAUSD. In Episode 108 Interview Part One, we set the stage and talk through the first few days of the cyber incident that took place in September 2022. Go back and listen to Part One if you have not already.
In this episode Soheil talks about bringing in the FBI and how their goal was not necessarily the same goal that Soheil and his team had. We also crack a few jokes about the news stories that said LAUSD "picked up the phone and called the Whitehouse" asking for more assistance from the Federal Government, and yes, it pretty much happened that way.
We would like to thank Soheil for taking the time out of his schedule to grant us this interview. Like we said before, we are pretty certain this is the first and only one on one interview that Soheil has granted since the event took place, and we are VERY grateful that he was willing to talk to us.
LISTEN HERE (and on all major podcast platforms).
Hang out with us at K12TechPro.com
Fortinet - Email fortinetpodcast@fortinet.com
SomethingCool.com - Cybersecurity Solutions
Extreme Networks - Email dmayer@extremenetworks.com
Oh, and...
Email us at k12techtalk@gmail.com
Tweet us @k12techtalkpod
Visit our LinkedIn page HERE
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:24):
live from thesomethingcool.com Studios
this is the K-12 Tech Talk podcast this
I am Josh
with me as always even when he's sicker
than a dog I wish you could see his face
mark
it's pretty bad
uh-huh Chris is not with us tonight due
(00:45):
to a prior commitment
um so we will take every chance we get
to make fun of him
Oh I thought I thought we put Chris on
suspension because he screwed up the
visor
that that is what happened I didn't get
that memo from HR but you're right I
think that's right so Chris did really
screw up with visor
um so for our listeners and for visor we
(01:07):
apologize we're just gonna say no
visor.cloud v i z o r dot c-l-o-u-d
don't listen to Chris's last ad
hopefully he fixes it tonight
um but yeah we we're really sorry visor
visor.cloud we're proud to have you on
here as a sponsor listeners go show them
some love
um Chris really screwed up send us hate
(01:27):
mail send Chris hate mail because you
know you screwed up a a sponsorship
um the other request that I have for
listeners tonight is uh I had an
interesting conversation with I believe
it was a listener from California this
week
um and I asked that the one favor that
we ask in return for doing this well two
(01:48):
favors one visit our sponsors
um and and listen to their sales pitches
to and potentially buy product two share
this podcast with your friends with your
meeting groups that's how this spreads
Chris and Mark had a wonderful time in
Austin Texas at k-126
um Mark you just want to we're going to
(02:08):
talk about that in the next episode
because this is part of the um suhil
katal interview from LAUSD but do you
want to quickly just talk about k-126
Mark yeah we're going to do a little
Deep dive in the next episode we have an
interview with Doug Levin uh we're going
to talk all about some of the sessions
that were there it was a really really
fun conference it was the first one for
(02:29):
K-12 six so definitely a lot more to
come and we'll see that on the next
episode and we might talk about Chris
and Mark's relationship like their
friendship slash you know
they all met in person per person oh my
goodness
um Okay so
(02:50):
this is going to be the follow-up
episode part two part Dua however you
want to say it uh of the interview with
the CIO from LAUSD uh he is a friend of
of marks he is a um kind of cohort they
you know they're in this the secret
society together
um katal he was awesome enough to let us
(03:13):
interview him a couple weeks ago
um Mark why don't you if you feel well
enough tell us about episode one and
kind of where we left off yeah so when
we left off Sohail was giving us kind of
the Deep dive of what happened the night
of the incident uh a little bit of their
initial recovery uh and then where we're
gonna start tonight is when we pick up
(03:34):
the conversation is is going into the
recovery going into the decisions that
need to be made in the days leading
afterwards the political the uh the
media kind of Firestorm that came out as
a result of it
and uh uh this is a really really cool
episode because it's it's just kind of a
a look under the hood uh and definitely
(03:55):
you know when you're talking about a big
organization as large as LAUSD to hear
about what an incident like this is like
at that large of a scale is just
fascinating so
hope you'll enjoy it yeah so uh this
will close out our interview with uh Sue
Hill and we really appreciate the time
that he spent with us he didn't have to
(04:16):
do that it ever from everything we've
seen
um we were
the first one of the first if not the
first one-on-one interview that he
granted about this specific topic since
it has happened so we're super
appreciative of him spending time with
us and and letting us have first crack
at that
(04:37):
um I know you guys talked about it down
at k126 in your in your address I think
the feedback there was some excitement
it seemed like the building around that
and um that clearly shows with our
listens so
again here we go into the interview but
share this with your friends visit our
sponsors and hopefully Chris gets
visor.cloud right at the end of the
(04:59):
episode thanks for listening so all
right so you've got all these people in
the room this is the first 72 hours and
just to kind of paint the picture this
is Friday night Saturday Sunday and then
Monday was Labor Day
the feds are there and it sounds like
this is great you got the feds in there
but their strategy is not to help you
get to school on Tuesday right their
(05:19):
strategy is to what oh so remember
that's where you need to be able to
manage everything yourself if nobody is
going to manage anything for you you are
doing multiple things first you want to
prevent
um this incident uh to crippling the
system at the same time as a CIO you are
(05:40):
still responsible to come to operation
so people think these stages of the
cyber security response uh they're
coming in sequence oh it you did the
detection now you get to the response
now you get to eradication and Recovery
these are steps you need to follow like
the way that they show it in the chart
probably inside the cycle oh yeah yeah
(06:01):
yeah truck it's not that there's no way
you can hold everybody back and do
something you need to prioritize that
you need to section systems based on
what is top priority to what area and
then sequence them to operation so when
we did the initial detection we went in
we will start doing the radical patient
(06:22):
we needed to prioritize the system to
bring to production one of our main
priority was to be operational by the
next day which was the Tuesday morning
to be able to get kids to the school the
goal did this was a clear goal by our
superintendent with the board we were
briefing board every three four hours so
(06:43):
it wasn't like hey every day so we were
briefing board we were briefing
superintendent and working with the team
great closely and directly I was
Hands-On work for me uh it was restoring
the system so
um that's why FBI and the team they were
working their work but at the same time
we had our own team to work on the
(07:04):
recovery uh of critical system
some ways you can divide and conquer and
that's the way that we decided to do we
pulled our resources from any area
around that and focused on recovery and
Recovery uh sometimes if you have a
proper recovery but you can take some
action with a modified version
especially when your system could be
(07:25):
potentially compromised you do is
standing against those critical system
make sure there is no residue there is
no touch on those systems if it is or if
you're not you don't have time to do
that you come you activate your Disaster
Recovery plan you bring up the
transportation Food Services
um that is a school district these are
critical system for us we want to make
(07:46):
sure we fit the kit we want to make sure
we transport because we want to make
sure we teach the kid when you stay
teaching the kid means counting the kids
that they go to the school in IIT
function
so
for me the goal was is my transportation
system going to be up and running for
the next day with my food services are
going to be up and running by the next
(08:06):
day and if my student information system
be able to take attendance and
ultimately my LMS is going to be
available with by the way LMS now added
to the mixed post pandemic no it's a
critical system it wasn't as good right
well no it is therefore you need to make
sure these four system the day of your
operation are active and we were able to
(08:28):
successfully with the day and night
working and splitting different tips to
different section
to restore all of those systems uh to
operation and but in a different
modified version remember it was working
with the ongoing investigation and
incident what was that
we made the decision to reset
(08:49):
everybody's password that's um remember
when you have a compromise system you
don't know what question you don't want
to take the risk open a system and
ultimately somebody be able to get in so
if I'm making a rational decision to
open a system in a short period of time
I need to get the some
some type of a pre-assessed measure to
(09:10):
protect my system uh one of them was our
system should not be accessible from
internet we blocked quarries we checked
all the internet access block from all
over assistant except you need to be at
the school to do it you want to reset
your password go to your campus you want
to do this go to your campus all the
system was restricted from outside
(09:31):
and that password reset probably was uh
very impactful especially when you have
that many users that we were talking
about right now at 1 million user 1.5 by
the way I forget to say I have my
retirees in my active directory because
they are receiving the uh there are
about 200 retires in our systems that we
(09:52):
are managing because they're accessing
the requirement plan and benefits and
everything else but managing all of
those account resetting their password
communicating with that and be able to
restore the services
it's funny because as you're saying this
the pat when you brought up the password
reset thing I remember reading the three
(10:16):
of us were following your I think it was
the Twitter account and and at some
point you guys spun up a website where
you were super transparent about what
the current activities were what the
current status of systems were and I
remember reading about the password
reset and the numbers the sheer numbers
of people's passwords that were reset
(10:36):
thinking my God the number of calls that
that is probably generating to support
staff has got to be astronomical
yeah I just couldn't imagine
it it was but you know what the good
thing uh one thing we learned from
pandemi and that probably is still gonna
stay with us for a long time until
(10:57):
people forget about everything uh was we
learn uh how to work and collaborate
with each other when the incident
happened every Department every rule uh
put the support beyond Behind I.T I mean
it was an I.T running everything
operation schools field teachers there
(11:18):
was no separation and again we little
we're learning about this lesson during
the uh during the Kobe that we need to
work together and that was the same
scenario so when we were releasing this
information read the really
as an ITV source of the decision making
we want to do this we want to
communicate this
(11:38):
um and then communication would go out
and come obviously if there was a
mistake would be our responsibility to
make sure we correct it and uh yes there
is always a glitches when you do
something that fast if I have 2 million
user want to reset the password to the
same password reset portal I need to
size it properly believe it or not you
have it yes yeah so that's what so
(12:01):
something funny you should know so we
have the biggest sis system oh no at the
time we'll custom the World by us and
everything else and it's massive being
for this Hardware that's running this
system is sometimes I can say is unheard
of number of course CPU and everything
so when we when we went through this uh
(12:22):
password reset process we needed a high
compute icq High memory to handle the
size of the password reset that is tummy
we took the resources away from sis
system we put it for password reset
literally it was that huge of the system
to process that many passport reset
request I can tell you exactly it took
(12:43):
me 250 CPU Gore and I bought three
terabyte of memory to process and again
because I'm using technical term but you
guys yeah here he goes bragging again
[Laughter]
wow uh three terabytes of memory I have
(13:04):
white six terabytes of sand storage for
my entire environment so that just blows
me away but that's amazing great so so
you talked about the different systems
that you had to get up and running
Transportation Food Services the LMS and
by the way the LMS is not just a
classroom tool you have students who are
fully online so it's there's password
resets for students who are fully online
(13:25):
and their LMS
but one thing I've always been
fascinated by is there has to be this
conversation between you and the
department heads transportation for
example to say
here's the status of your system
at at some point I'm going to give you a
go no go meaning you have access to your
system or you don't can you describe a
(13:46):
little bit about the conversations you
had with department heads around you
need to start to activate your
contingency plans for Tuesday
yeah so obviously everything runs on the
plot so when we were talking about
restoring this system it's not like oh
two days before we opened the school the
system was up and running now literally
(14:06):
like by minutes by minutes we were
counting time to do the stress test load
test activate the system do the scanning
and everything else that goes with it
and at the same time
um have the transportation take to
access to their system so they can put
the route put the student roster and
everything else to be able to activate
so it was a constant communication and
(14:28):
release of information to individual
Department we were releasing data
information every two three hours there
was information actually I was even
standing to superintendo every 15
minutes literally every 15 minutes so
the communication was very rapid and
very occurring because if your
information wouldn't get to them on time
(14:50):
it has no value the information for
yesterday
it's out of date it's like a six months
old information when you are in the
middle of rapid recovery therefore those
information was for example we were
tracking password reset why was
important because the field and the
principle they were all engaged to make
sure the kids get their password reset
(15:12):
so what did we do we said okay we
divided again divide and conquer is
strategy
if it's Elementary a student it is hard
for elementary student to reset the
password but it is uh simpler for uh
secondary student to reset the password
what did we say we said if you are a
secondary a student we send you to the
(15:33):
portal directly to reset the password
job everything was sent schools
communicated we communicated we put it
in so many different ways for the school
to get access to that one way one thing
we didn't do uh as much as we were
transparent in our communication what
are we doing we didn't uh we didn't put
it in Twitter or public forum which we
(15:53):
could to tell oh how are we doing it
yeah we never said publicly that we
closed our Network except to internal
school why you wouldn't say that because
the bad actor reads your Twitter same
way that you're reading so you don't
want to over communicate it right you
want to communicate enough to get to the
user the transparency was mostly around
(16:14):
and was about what we are doing but not
whole year and that's where the separate
the communication criteria but to
continue what I was saying separating
the Elementary from or from secondary
was crucial because the secondary day we
set their password through the portal
Elementary kit we did the password set
(16:35):
because we want the always mitigate the
situation if the bad actor has access to
the password of somebody it doesn't
repeat the same incident again so what
we did we set the password but we give
them longer period of time for Teacher
to work with the kid to reset the
password rather than watch that so the
first half the instructions still
started with the new password there was
(16:57):
no old password when we did it because
the password was set by us and sent to
the teacher to help the student log it
when they log in in the first layer of
instruction the teacher was able to help
the student to reset the password with
preset password that we configured it is
very important to you know this mechanic
around it to mitigate otherwise later on
(17:19):
is not lied to you and probably that's
some of the key areas they both need to
pay attention remember I told you K12 is
and you know it K12 is different than
anybody else when I'm doing the password
policy or anything else some of those uh
need to be different when you deal with
the elementary kids yeah yeah so at one
(17:42):
point uh through this process were you
making a decision or or was it from the
beginning you're saying we're not going
to close school was there ever a point
during the weekend where you had to make
a decisions to town with a
superintendent and say it's a go no go
for school on Tuesday it was my decision
it was my decision I was oh God
(18:04):
can we open this school on Tuesday
and I couldn't say no literally I
couldn't say no I could uh if I wanted
to take it easy and take the pressure
off the team and everybody else probably
I could have said no and delayed opening
up the stool but the impact of the
(18:25):
school district and our side and the
disruption that can create under the
city it was massive that we couldn't
afford that
um it was crucial for me to commit and
deliver what I promised I promise I'm
gonna open this school my kid goes to
the same school yeah I want the kids to
learn and then nowhere uh and at the
(18:47):
same time I I hate to say this but you
wanna say uh that you are not yet played
by the bad actor and criminal that
easily yeah you would Mark you work hard
and you respond hard and at the same
time you can recover the system on time
well I think if you look at it from a
humanitarian standpoint too you know and
(19:08):
we learned this during covid a lot of
students receive or the only place a
student really gets a good meal is at
school in some cases so if you're that
had to play in a huge factor in whether
or not you could open that Tuesday as
though if we don't open how many of our
students aren't going to have a decent
meal on Tuesday because we're not we're
(19:30):
not providing that meal for them I'm
sure that was a huge factor of that too
it was you're right during the pandemic
that was uh that was one of and you know
that La is uh in Los Angeles Unified we
are not about 85 to 90 percent free
reuse which that tells you how many
low-income kids go through our schools
(19:50):
right because that's uh easy to make the
decision like that I'm not going to open
business right
so it's hard to believe that we've only
we've been talking for about 45 minutes
and we're only up to day three or day
four in your in your response so now
let's talk about Tuesday School opens
and you now have to make sure that
(20:12):
school opens safely and continue to with
your with your work what's going on now
so um that's part of the the keep
obviously you'll bring the Key Systems
off you know they are available you need
to chip them up at the same time and as
you're working to the rest of your
system you know in Ln USC we have about
(20:34):
200 applications and prioritizing those
applications depending on the
sensitivity about those applications
become critical one thing you're
probably gonna learn there are some
compliance system like you need to send
some report to this state something here
and there and they'd have like a
deadline you need to send this by such a
date you need to send that twice run
(20:54):
your payroll you need to do this you
need to do that and that's become
important why because uh to some extent
you can get the exemption from your
estate and we were in that such a
situation because every again it was a
big incident
um and it was publicly communicated and
feds everybody the White House and
(21:15):
everybody they were engaged and helping
us
um and at the same time communicated
with the state we need an exemption from
those requirements to be able to really
secure our system to be able to generate
those reports so uh it was important uh
some of the those decisions was made to
prioritize this system based on
compliance based on need based on
(21:38):
priorities to gradually so there are
different priorities usually you say
Okay tier one system critical bring it
out the rest of the system if you look
at uh through Disaster Recovery plan
most of the time you don't see clear
definition what are those systems and
the priorities that they need to come up
and I think that's become important
(21:59):
people usually ignore the tier 2 systems
priority but it is important it's not as
important year one it's not going to
prevent you to open your school but it's
gonna be your headache post-production
because you need to be uh bringing them
on time but you need to know which one
first and that was a part of our process
and we were able to actually as a result
(22:21):
of that there were a little bit proper
business continuity plan per Department
I'm not talking about I.T business plan
I'm talking about the departmental
business management class to be able to
have an updated access to their
technology at the time that they
hmm so let's talk a little bit about
um you know we've heard your
responsibility and and you were you were
(22:43):
making a lot of the decisions
um but I'm I'm sure there were still
some things that the superintendent had
to handle or the you know give
interviews make decisions make certain
calls
um can you talk a little bit about his
response to the situation his support
for you and your team
um buying pizza if he bought pizza a
night or two
(23:04):
um you know that that kind of that kind
of discussion around that
so definitely the main decision making
on all of this uh especially when you
look at the impact of the school and
incident ultimately was superintendent
yes he may ask my opinion do we need to
open this school or not ultimate beat
was his decision should we often this
(23:25):
will run back it was my opinion that I'm
gonna make it ready he realized he
trusted me that I can bring it up and I
believe her
uh the rest of it was his decision Army
operation only ready to open this school
after what happened what we went through
holiday schools the student and they're
gonna feel in the classroom and
everything else and that was a critical
decision I think that was one of the
(23:47):
most important decisions that he made
secondly uh I think from the day one uh
he confirmed and he committed he's not
gonna negotiate with the uh with the Bad
actors we never uh negotiated with them
to pay the ransom or anything that they
were requesting they were requesting we
(24:09):
never said even how much we never said
we uh that we're gonna pay them or
anything else and uh in his word uh and
I repeat uh his board make sure you
always know what are your uh wrist with
them so try to find out what data
because when you get talking about the
data exploration
(24:31):
um in that short period of time you may
have some data of some information some
logs knowing what they may have had
access but reality you don't know until
you get that uh you get something from
them and that's there's nothing wrong
with it but do not negotiate with them
because technically
they may give you a promise they may
(24:52):
give you your data they may give you
everything that uh you want if you pay
them to Ransom but they still have
access to your data their bad actor is
just like they have another copy they
publish it they sell it they do
everything so now that was a
recommendation from FBI as well that
essentially negotiating with the bad
actor is not the wise choice although
(25:14):
this bad actor apparently I didn't know
that FBI confirmed for me they usually
deliver on their promise in their
website they publish then your name and
they call you partner if you pay them
and they don't release your information
but your name is gonna be there to me
that's worse than if they publish my
(25:35):
data if they call me a partner sure
so technically your organization pay
them they publish your name they don't
lease your data they call you partner if
you don't pay them they just release
your data and they name you over there
so I again uh for us the rest of the
data was minimal uh I I believe I hardly
(25:58):
believe depending on the risk and
whatever it was we would never pay the
ransom uh and we would never negotiate
with the hard uh with this type of a
criminal and that was one of the
decisions to print and make from day one
to not to negotiate not to pay the rents
up and it's he's stood why
now it are so there's there's a threat
(26:20):
actor who's who's communicating things
to you are you communicating is the FBI
communicating I mean who is who's taking
care of that side of the the situation
you don't deal with that uh usually any
type of analysis about the bad actor
goes back to your insurance company they
have experts they know how to deal with
(26:41):
it to not engage that's what I'm saying
I repeat do not engage with the bad
actor because technically that's made
backfire politically they may back for
his security wise and everything else
living up to the expert they know what's
your risk obviously they ask you uh if
what's your risk tolerant and for us the
(27:02):
risk tolerance was we're not going to
pay
wow
you can see me as a CIO when
superintendent tells I'm not going to
pay the rent somewhere it's a good thing
I'm proud of him and I his decision but
it made my job more difficult means I
need to recover the system no matter
what there is no alternative I'm not
(27:23):
gonna be able to have a champion coming
and saving me I need to save restore
bring up the systems and and I think you
you alluded to this earlier though even
though most most of these threat actors
if you pay them on the idea that you're
going to get a decryption key sometimes
that decryption key doesn't work
sometimes it does work so you're you
(27:45):
know your 50 50 shot there and then
there's the double extortion of we're
going to release your data okay if you
pay them again that still really doesn't
mean they're not going to post your data
they still could post your data so I
mean his his stance of we're not going
to pay
to me it makes sense if if you if you
(28:06):
weigh that risk now again depending on
what kind of data they have but if you
weigh that risk I you know it's not a
guaranteed conclusion you know
true and and exactly that the European
it's a criminal you you can exactly
um you cannot trust you don't you never
know what you're gonna get uh and again
(28:26):
with the decryption key and everything
else again and by the way nowadays you
mostly deal with the double extortion
because most of the rans iso will stop
the rest and where tanks are failing
because every Everybody learn the lesson
keep your backup man keep your back up
don't worry about this that's why they
go with the attacks menstruation and
(28:46):
that's where his big game become tricky
how fast you can get this data
exfiltration and how can you stop them
from continually and that as early as
you can detect it is could be crucial
for the decision making around it uh
which goes back to uh what's your post
recovery plan what's yours um for
example for us uh it was implementing a
(29:08):
security Operation Center we had Network
Operation Center that was monitoring our
network but we never had the Security
offers which would be the results of
this incident as part of a
recommendation that came out of our ID
our I.T security task force
Okay so you just hit on it one of the
(29:29):
changes that you've made since the
incident is is a security operations
center
um can you give us an idea of uh some of
the other changes that you guys have put
in place post incident to improve either
operationally I mean I know you probably
can't get into what you've changed for
security posture wise but well what are
(29:49):
some of those those small tweaks that
you've made operationally maybe that
could enhance a future incident
Sean
um
I think it is important of going back to
the incident I think was in the first
week of incident uh when superintendent
gave his first uh briefing to the public
(30:10):
and to the board and everybody that he
announced that he's gonna have a 90 days
plan and that 90 days plan goal was not
only address things in the admittedly
mitigate the current situation but lay
on the pathway for us to build the most
uh secure and reliable
(30:33):
um it infrastructure indicate love
industry Across the Nation that was our
goal our commitment to essentially come
up with the recommendation to build this
as the model that everybody else can
take advantage that's where a
conversation with white house has
started because White House wanted to
develop a model for others education
entities
(30:54):
and the same the conversation expanded
to seesaw Cesar is the architect for
this security architecture across the
government and private sector
so well what we did uh we created the
very rapid task force we call it it
security password that their their
(31:15):
function was it's just going to come for
90 days
the people that they were invited to be
part of that task force
probably a school district like us or
like yours or anybody else they're not
going to be able to afford to pay the
salary of these people I named them for
you there were people from uh Wells
Fargo Chief level uh there were people
(31:38):
from uh Intel Chief level there were
people from
from uh I would say
um Microsoft and other areas Apple uh
again not the business side don't look
at the business side because business
sites always they try to sell you stuff
stay away from them
(31:58):
when you deal with the task force you
want operations like when I'm uh when
I'm saying when I have a ciso from uh
let's say apple
is not nothing to do with the government
sales or anything else these are the
people they're protecting Apple I.T or
apple and Lewis from Bad actors or same
(32:19):
uh other areas so these were the um even
we got the resources from uh
uh retired General that was working with
the Air Force cyber command so these
people with a lot of experience in the
Cyber industry that they are more
advanced than K-12 we know that K
through 12 in cyber security we aren't
behind the rest of the industry even in
(32:42):
it forget about security
um I believe one of our analysis showed
the K the education sector
um the uh the entire project of the I.T
Department in education sector was
um about five to ten percent of the
entire organization budget
believe it or not are entirely spending
(33:04):
I'm not taking the apartment budget
buying devices the school buying
technology everything we summarized it
we were 2.5 percent so the reality below
industry average when we talk about
because K-1 literally did dwarf us and
spending and this type of technology so
when you don't have those investing plat
(33:25):
you're relying on the resources that you
have
the ball up that task force was to make
a recommendation
um and review our strategies what we
have what we implemented the audit the
pen test the incident that added and uh
based on that they make a recommendation
for course correction and those are the
(33:47):
ones right now we are uh we build the
three years plan actually it's a
four-year plan the goal is to implement
it in three years
um the three years plan to revamp the
security infrastructure and footprint of
the LA USD to become the one of the most
secure and reliable infrastructure in
(34:07):
the country so from an outcome
standpoint okay you you've done this
90-day task force with these csos and
and other Chief level
um operational folks from from major
industry
um you have your report now are you
going to be a good and this is selfish
I'm just going to say this is purely
selfish on my part
(34:29):
um are you is La willing to be a good
neighbor in in your work with the White
House and this is a
um will you guys uh publish maybe
confidentially to to k-12s a framework
that that we can follow that okay here
here's kind of a checklist of things
(34:49):
that you need to be thinking about this
is these are the the higher priority
items that you need to be addressing as
soon as possible but this needs to be on
your roadmap as well
um is the plan for you guys to be kind
of a good neighbor in that method do you
know that that's actually one of the
things in partha were three years plan
and actually that was one of the ask
(35:10):
from the White House and from actually
the
um from uh cyber command as well that we
need to educate other uh K-12 and
literally there was a discussion about
traveling going to different School
District sharing our experience and
everything going on a podcast right yeah
(35:32):
did you think he was going to say no
we're gonna be a bad neighbor Josh well
I mean I figured I'd ask that question
hey you just said YT I will be there
what number did you call the White House
maybe we can start there is there a
phone number
yeah he they he's got a red phone on his
desk we can't see it in the picture but
(35:54):
there's a red phone that he picks up
yeah you adopt I guarantee you contact
your ogr office of government relation
they have all of those phone number for
you to be able to expedite the calls but
definitely the superintendent always uh
have an escalation access to the
Department of Education or from there
you get to the White House so that's the
easiest way to tell you how to get to
(36:16):
the White House yeah now one thing I
think you haven't mentioned here is that
your superintendent as as helpful as
he's been he he's been on the job for a
month two months maybe at the time
I didn't know that
(36:37):
yeah so he's he's brand new to LA and
he's probably looking you'd be like is
this is this part of the course is this
happened all the time
oh it was he was surprised it was a
surprise for us as well yeah in L.A we
never had the Cyber incident like that
but if one thing can tell you uh if you
are not safe nobody's safe uh especially
(36:58):
in education I think this is something
can happen it can happen to us as well I
don't actually the man there are a lot
of statistic that tells you if you have
an incident usually there is a 60 70
percent chance and they have another one
in the same year oh we are very
sensitive to be able to protect
ourselves uh one thing I would tell you
(37:20):
uh you this bad actor is specifically
they are famous or going after the low
profile Target they would barely go
after the law I provide like us
so there are debates why they picked us
um are they changing their uh structure
are they want to become a more uh
(37:42):
involved in this type of incident or no
they just made a mistake it was a stupid
mistake they didn't know who we are or
what size we are and the reality is they
spend money it wasn't like they did this
for free uh when did you spend when you
want to do this type of thing as a cyber
criminal organization
you have resources you have they have
(38:04):
Financial analysts they study
organization
and at the same time they do a lot of
reconnaissance they go through your
system they go to the market they buy
things and they try to break into your
systems and in our case uh it didn't pay
off so if you call it Roi their Roi
(38:26):
so uh
from a feedback standpoint from from uh
parents and students I saw a post on
Twitter uh a couple months ago I believe
it was I'm not going to name a name
because I'll I'll probably get it wrong
and I don't want to mess up but there
was a school district a relatively large
school district that had a ransomware
(38:46):
incident and they recovered pretty quick
but there were a couple days down time
and one of I think it was their CIO
posted on Twitter that when kids came
back they started receiving hand-drawn
pictures and notes and and uh thank you
more or less thank you notes from
students saying thank you it department
for working so hard on getting getting
(39:08):
the school back and getting us back here
at school and and fixing things
um what what was your experience like
from a from a parent aspect from a
student aspect from a from a faculty
standpoint I know you said that everyone
was pitching in and assisting
um if you could talk about that that
feedback or that buy-in support from
(39:28):
those those key critical folks
uh thank you I I received a lot of
support I mean uh there were text email
communication walking even in the
elevator with other people uh that they
were impacted with the Cyber attack it
was not a stop I mean people that they
left or retired now I mean people that
(39:50):
they went to other organization people
that they were not even with us anymore
they it was uh it was very support uh
very very well supported um the public
comment but yes when you read the you
know one thing I would tell you when
you're in the middle this type of
incident
it's easy for people to point a finger
(40:11):
and blame oh you did this you did that
you didn't pay attention to this you
didn't pay attention to that but the
reality is uh we survived number one if
we would have failed that would have
been the totally different conversation
uh and probably blame blaming pointing
figure would be much more than what we
received after this label with that we
(40:33):
still receive uh pointing finger and
blaming and everything up but the
reality is we need to survive we recover
we're restored and the student back in
the classroom that's number one and we
learned a lesson uh and we want to share
that lesson with everybody to be able to
recover and make a resilient system not
(40:54):
to be really uh
compromise with this type of a Bad
actors uh because it eventually it's
gonna happen and it just matter how you
mitigate the situation and how you
contain it to not to become a
catastrophic like some example of other
school district they ended up shutting
(41:15):
down and you think shutting down the is
is a critical part
I think that there could be much worse
when you lose 30 years or 20 years 30
years old critical information that
you're keeping in your system to do AI
analytic projection
(41:35):
or graduation for a student that they
are on track and everything else losing
that information you are it's a harm for
those students that essentially know
they're just writing in the rail without
the conductor and that's that's not easy
yes if same year same teacher as those
(41:56):
information you that the student
graduate to next grade what do you have
to support the next teacher to support
that the student is nothing when your
information now off in the air and
decrypt
yeah I I think we were kind of off the
side watching uh the situation in LA and
I think there's the public reaction and
(42:17):
there is you know when there's tweets
and social media around what's going on
there's also kind of the the behind the
scenes conversation right the folks like
us who are in the industry watching
things and I think generally speaking
those of us who are in the industry were
both very impressed that a LAUSD did not
close
and and you showed very very high
(42:39):
expectations for how this should be
handled and how you should respond to
this uh and B the other response was if
this can happen to La this can happen to
anybody uh so it's it's very
heartwarming to hear that you're you're
taking your roadmap and you are uh
sharing this with the community you are
coming on to our our podcast today to
talk about what you experienced to share
(43:00):
the lesson and hopefully uh help another
just prepare for for something that can
happen so so thank you so much to Hale
for uh for for joining us today
so thank you Mark and thank you all of
you guys uh you're doing a good job
sharing this information hopefully one
day we can go more detail about the uh
(43:20):
areas of the security and probably in
the confidential way or aren't we gonna
share that with other school district to
make sure again
when we share we want to make sure we
don't uh give information out to the Bad
actors but it's being used by the good
actors like you and the rest of your
team of course yeah yeah we need there
(43:41):
needs to be a a website that
like vets users so that you can share
you you know you're sharing that data or
that information with vetted individuals
who are in the industry they're not a
threat actor I look that's something
that's needed in this in this industry I
(44:01):
think that would make that that job a
little bit easier and uh to some extent
um I uh probably need to share with you
guys
um FBI has a structure called infra
guard and in Orange you you will be a
member of the info guard because that's
where uh to some extent is protected
(44:22):
it's more vetted people are part of that
um and mainly because uh you need to do
the full background check to be part of
that conversation and share
yeah we appreciate your time we don't
want to keep you any longer
um hopefully the fire department put out
the fire that they were going to uh we
we do really appreciate you being uh as
(44:44):
honest as you were tonight and being a
good neighbor in the future with uh you
know sharing that framework with
everybody when it's done hopefully in a
confidential manner
um any any closing questions mark Chris
favorite pizza favorite pizza
you want to go I don't know wow Boston
(45:05):
this stuff you have peace and it's in
Boston Mark yeah we've had we we've got
some pizza here yeah you gotta you gotta
come over here we'll we'll take you I'll
take you the north end it's okay the
Italian section of Boston you get the
best pizza you'll ever have okay I
didn't know that but if it is I go with
pepperon that's right okay all right all
right
thank you thank you very much thank you
(45:28):
but as we wrap up part two we want to
thank our sponsors for making this
happen let's talk about visor that's
v-i-z-o-r a recent NBC report revealed
that a school district in Virginia lost
1800 Chromebooks that's quote unquote
lost uh for a total loss of over five
hundred thousand dollars auditor said
(45:49):
that the district failed to have a
process to monitor device collection
when a student was withdrawing that was
a costly mistake of course so visor
v-i-z-o-r helps districts manage
Chromebooks and other it assets by
automating Best Practices within their
schools to avoid those kinds of
situations if you're interested in
(46:09):
learning more about that you're going to
go to this website
visor.cloud that's
v-i-z-o-r dot Cloud slash K-12 Tech talk
that's us visor.cloud K12 Tech talk to
get up to 20 off you can learn more
about them other sponsors we got
Fortinet you can email fortinetpodcast
(46:31):
fornet.com they got your Florida gate
your Ford EDR your afford a token and
whatever else Florida they are the
leader in cyber Security Solutions and
services also extreme networks are proud
sponsored the K-12 Tech Talk podcast you
can email Dominic there D mayor that's
d-m-a-y-e-r at extremenetworks.com for
(46:53):
your networking needs he can help you
with specs also look up their work with
the Super Bowl
provision Data Solutions you got a
Windows server issue you got some
networking that needs to be upgraded a
problem you can't figure out get a hold
of provision Data Solutions and then
somethingcool.com you can reach out to
Jeremy there that's Jeremy
somethingcool.com and you can check out
(47:15):
their cyber security offerings in the
link description
MFA
live from thesomethingcool.com Studios
this is the K-12 Tech Talk podcast this
I am Josh
with me as always even when he's sicker
than a dog I wish you could see his face
mark
it's pretty bad
uh-huh Chris is not with us tonight due
(00:45):
to a prior commitment
um so we will take every chance we get
to make fun of him
Oh I thought I thought we put Chris on
suspension because he screwed up the
visor
that that is what happened I didn't get
that memo from HR but you're right I
think that's right so Chris did really
screw up with visor
um so for our listeners and for visor we
(01:07):
apologize we're just gonna say no
visor.cloud v i z o r dot c-l-o-u-d
don't listen to Chris's last ad
hopefully he fixes it tonight
um but yeah we we're really sorry visor
visor.cloud we're proud to have you on
here as a sponsor listeners go show them
some love
um Chris really screwed up send us hate
(01:27):
mail send Chris hate mail because you
know you screwed up a a sponsorship
um the other request that I have for
listeners tonight is uh I had an
interesting conversation with I believe
it was a listener from California this
week
um and I asked that the one favor that
we ask in return for doing this well two
(01:48):
favors one visit our sponsors
um and and listen to their sales pitches
to and potentially buy product two share
this podcast with your friends with your
meeting groups that's how this spreads
Chris and Mark had a wonderful time in
Austin Texas at k-126
um Mark you just want to we're going to
(02:08):
talk about that in the next episode
because this is part of the um suhil
katal interview from LAUSD but do you
want to quickly just talk about k-126
Mark yeah we're going to do a little
Deep dive in the next episode we have an
interview with Doug Levin uh we're going
to talk all about some of the sessions
that were there it was a really really
fun conference it was the first one for
(02:29):
K-12 six so definitely a lot more to
come and we'll see that on the next
episode and we might talk about Chris
and Mark's relationship like their
friendship slash you know
they all met in person per person oh my
goodness
um Okay so
(02:50):
this is going to be the follow-up
episode part two part Dua however you
want to say it uh of the interview with
the CIO from LAUSD uh he is a friend of
of marks he is a um kind of cohort they
you know they're in this the secret
society together
um katal he was awesome enough to let us
(03:13):
interview him a couple weeks ago
um Mark why don't you if you feel well
enough tell us about episode one and
kind of where we left off yeah so when
we left off Sohail was giving us kind of
the Deep dive of what happened the night
of the incident uh a little bit of their
initial recovery uh and then where we're
gonna start tonight is when we pick up
(03:34):
the conversation is is going into the
recovery going into the decisions that
need to be made in the days leading
afterwards the political the uh the
media kind of Firestorm that came out as
a result of it
and uh uh this is a really really cool
episode because it's it's just kind of a
a look under the hood uh and definitely
(03:55):
you know when you're talking about a big
organization as large as LAUSD to hear
about what an incident like this is like
at that large of a scale is just
fascinating so
hope you'll enjoy it yeah so uh this
will close out our interview with uh Sue
Hill and we really appreciate the time
that he spent with us he didn't have to
(04:16):
do that it ever from everything we've
seen
um we were
the first one of the first if not the
first one-on-one interview that he
granted about this specific topic since
it has happened so we're super
appreciative of him spending time with
us and and letting us have first crack
at that
(04:37):
um I know you guys talked about it down
at k126 in your in your address I think
the feedback there was some excitement
it seemed like the building around that
and um that clearly shows with our
listens so
again here we go into the interview but
share this with your friends visit our
sponsors and hopefully Chris gets
visor.cloud right at the end of the
(04:59):
episode thanks for listening so all
right so you've got all these people in
the room this is the first 72 hours and
just to kind of paint the picture this
is Friday night Saturday Sunday and then
Monday was Labor Day
the feds are there and it sounds like
this is great you got the feds in there
but their strategy is not to help you
get to school on Tuesday right their
(05:19):
strategy is to what oh so remember
that's where you need to be able to
manage everything yourself if nobody is
going to manage anything for you you are
doing multiple things first you want to
prevent
um this incident uh to crippling the
system at the same time as a CIO you are
(05:40):
still responsible to come to operation
so people think these stages of the
cyber security response uh they're
coming in sequence oh it you did the
detection now you get to the response
now you get to eradication and Recovery
these are steps you need to follow like
the way that they show it in the chart
probably inside the cycle oh yeah yeah
(06:01):
yeah truck it's not that there's no way
you can hold everybody back and do
something you need to prioritize that
you need to section systems based on
what is top priority to what area and
then sequence them to operation so when
we did the initial detection we went in
we will start doing the radical patient
(06:22):
we needed to prioritize the system to
bring to production one of our main
priority was to be operational by the
next day which was the Tuesday morning
to be able to get kids to the school the
goal did this was a clear goal by our
superintendent with the board we were
briefing board every three four hours so
(06:43):
it wasn't like hey every day so we were
briefing board we were briefing
superintendent and working with the team
great closely and directly I was
Hands-On work for me uh it was restoring
the system so
um that's why FBI and the team they were
working their work but at the same time
we had our own team to work on the
(07:04):
recovery uh of critical system
some ways you can divide and conquer and
that's the way that we decided to do we
pulled our resources from any area
around that and focused on recovery and
Recovery uh sometimes if you have a
proper recovery but you can take some
action with a modified version
especially when your system could be
(07:25):
potentially compromised you do is
standing against those critical system
make sure there is no residue there is
no touch on those systems if it is or if
you're not you don't have time to do
that you come you activate your Disaster
Recovery plan you bring up the
transportation Food Services
um that is a school district these are
critical system for us we want to make
(07:46):
sure we fit the kit we want to make sure
we transport because we want to make
sure we teach the kid when you stay
teaching the kid means counting the kids
that they go to the school in IIT
function
so
for me the goal was is my transportation
system going to be up and running for
the next day with my food services are
going to be up and running by the next
(08:06):
day and if my student information system
be able to take attendance and
ultimately my LMS is going to be
available with by the way LMS now added
to the mixed post pandemic no it's a
critical system it wasn't as good right
well no it is therefore you need to make
sure these four system the day of your
operation are active and we were able to
(08:28):
successfully with the day and night
working and splitting different tips to
different section
to restore all of those systems uh to
operation and but in a different
modified version remember it was working
with the ongoing investigation and
incident what was that
we made the decision to reset
(08:49):
everybody's password that's um remember
when you have a compromise system you
don't know what question you don't want
to take the risk open a system and
ultimately somebody be able to get in so
if I'm making a rational decision to
open a system in a short period of time
I need to get the some
some type of a pre-assessed measure to
(09:10):
protect my system uh one of them was our
system should not be accessible from
internet we blocked quarries we checked
all the internet access block from all
over assistant except you need to be at
the school to do it you want to reset
your password go to your campus you want
to do this go to your campus all the
system was restricted from outside
(09:31):
and that password reset probably was uh
very impactful especially when you have
that many users that we were talking
about right now at 1 million user 1.5 by
the way I forget to say I have my
retirees in my active directory because
they are receiving the uh there are
about 200 retires in our systems that we
(09:52):
are managing because they're accessing
the requirement plan and benefits and
everything else but managing all of
those account resetting their password
communicating with that and be able to
restore the services
it's funny because as you're saying this
the pat when you brought up the password
reset thing I remember reading the three
(10:16):
of us were following your I think it was
the Twitter account and and at some
point you guys spun up a website where
you were super transparent about what
the current activities were what the
current status of systems were and I
remember reading about the password
reset and the numbers the sheer numbers
of people's passwords that were reset
(10:36):
thinking my God the number of calls that
that is probably generating to support
staff has got to be astronomical
yeah I just couldn't imagine
it it was but you know what the good
thing uh one thing we learned from
pandemi and that probably is still gonna
stay with us for a long time until
(10:57):
people forget about everything uh was we
learn uh how to work and collaborate
with each other when the incident
happened every Department every rule uh
put the support beyond Behind I.T I mean
it was an I.T running everything
operation schools field teachers there
(11:18):
was no separation and again we little
we're learning about this lesson during
the uh during the Kobe that we need to
work together and that was the same
scenario so when we were releasing this
information read the really
as an ITV source of the decision making
we want to do this we want to
communicate this
(11:38):
um and then communication would go out
and come obviously if there was a
mistake would be our responsibility to
make sure we correct it and uh yes there
is always a glitches when you do
something that fast if I have 2 million
user want to reset the password to the
same password reset portal I need to
size it properly believe it or not you
have it yes yeah so that's what so
(12:01):
something funny you should know so we
have the biggest sis system oh no at the
time we'll custom the World by us and
everything else and it's massive being
for this Hardware that's running this
system is sometimes I can say is unheard
of number of course CPU and everything
so when we when we went through this uh
(12:22):
password reset process we needed a high
compute icq High memory to handle the
size of the password reset that is tummy
we took the resources away from sis
system we put it for password reset
literally it was that huge of the system
to process that many passport reset
request I can tell you exactly it took
(12:43):
me 250 CPU Gore and I bought three
terabyte of memory to process and again
because I'm using technical term but you
guys yeah here he goes bragging again
[Laughter]
wow uh three terabytes of memory I have
(13:04):
white six terabytes of sand storage for
my entire environment so that just blows
me away but that's amazing great so so
you talked about the different systems
that you had to get up and running
Transportation Food Services the LMS and
by the way the LMS is not just a
classroom tool you have students who are
fully online so it's there's password
resets for students who are fully online
(13:25):
and their LMS
but one thing I've always been
fascinated by is there has to be this
conversation between you and the
department heads transportation for
example to say
here's the status of your system
at at some point I'm going to give you a
go no go meaning you have access to your
system or you don't can you describe a
(13:46):
little bit about the conversations you
had with department heads around you
need to start to activate your
contingency plans for Tuesday
yeah so obviously everything runs on the
plot so when we were talking about
restoring this system it's not like oh
two days before we opened the school the
system was up and running now literally
(14:06):
like by minutes by minutes we were
counting time to do the stress test load
test activate the system do the scanning
and everything else that goes with it
and at the same time
um have the transportation take to
access to their system so they can put
the route put the student roster and
everything else to be able to activate
so it was a constant communication and
(14:28):
release of information to individual
Department we were releasing data
information every two three hours there
was information actually I was even
standing to superintendo every 15
minutes literally every 15 minutes so
the communication was very rapid and
very occurring because if your
information wouldn't get to them on time
(14:50):
it has no value the information for
yesterday
it's out of date it's like a six months
old information when you are in the
middle of rapid recovery therefore those
information was for example we were
tracking password reset why was
important because the field and the
principle they were all engaged to make
sure the kids get their password reset
(15:12):
so what did we do we said okay we
divided again divide and conquer is
strategy
if it's Elementary a student it is hard
for elementary student to reset the
password but it is uh simpler for uh
secondary student to reset the password
what did we say we said if you are a
secondary a student we send you to the
(15:33):
portal directly to reset the password
job everything was sent schools
communicated we communicated we put it
in so many different ways for the school
to get access to that one way one thing
we didn't do uh as much as we were
transparent in our communication what
are we doing we didn't uh we didn't put
it in Twitter or public forum which we
(15:53):
could to tell oh how are we doing it
yeah we never said publicly that we
closed our Network except to internal
school why you wouldn't say that because
the bad actor reads your Twitter same
way that you're reading so you don't
want to over communicate it right you
want to communicate enough to get to the
user the transparency was mostly around
(16:14):
and was about what we are doing but not
whole year and that's where the separate
the communication criteria but to
continue what I was saying separating
the Elementary from or from secondary
was crucial because the secondary day we
set their password through the portal
Elementary kit we did the password set
(16:35):
because we want the always mitigate the
situation if the bad actor has access to
the password of somebody it doesn't
repeat the same incident again so what
we did we set the password but we give
them longer period of time for Teacher
to work with the kid to reset the
password rather than watch that so the
first half the instructions still
started with the new password there was
(16:57):
no old password when we did it because
the password was set by us and sent to
the teacher to help the student log it
when they log in in the first layer of
instruction the teacher was able to help
the student to reset the password with
preset password that we configured it is
very important to you know this mechanic
around it to mitigate otherwise later on
(17:19):
is not lied to you and probably that's
some of the key areas they both need to
pay attention remember I told you K12 is
and you know it K12 is different than
anybody else when I'm doing the password
policy or anything else some of those uh
need to be different when you deal with
the elementary kids yeah yeah so at one
(17:42):
point uh through this process were you
making a decision or or was it from the
beginning you're saying we're not going
to close school was there ever a point
during the weekend where you had to make
a decisions to town with a
superintendent and say it's a go no go
for school on Tuesday it was my decision
it was my decision I was oh God
(18:04):
can we open this school on Tuesday
and I couldn't say no literally I
couldn't say no I could uh if I wanted
to take it easy and take the pressure
off the team and everybody else probably
I could have said no and delayed opening
up the stool but the impact of the
(18:25):
school district and our side and the
disruption that can create under the
city it was massive that we couldn't
afford that
um it was crucial for me to commit and
deliver what I promised I promise I'm
gonna open this school my kid goes to
the same school yeah I want the kids to
learn and then nowhere uh and at the
(18:47):
same time I I hate to say this but you
wanna say uh that you are not yet played
by the bad actor and criminal that
easily yeah you would Mark you work hard
and you respond hard and at the same
time you can recover the system on time
well I think if you look at it from a
humanitarian standpoint too you know and
(19:08):
we learned this during covid a lot of
students receive or the only place a
student really gets a good meal is at
school in some cases so if you're that
had to play in a huge factor in whether
or not you could open that Tuesday as
though if we don't open how many of our
students aren't going to have a decent
meal on Tuesday because we're not we're
(19:30):
not providing that meal for them I'm
sure that was a huge factor of that too
it was you're right during the pandemic
that was uh that was one of and you know
that La is uh in Los Angeles Unified we
are not about 85 to 90 percent free
reuse which that tells you how many
low-income kids go through our schools
(19:50):
right because that's uh easy to make the
decision like that I'm not going to open
business right
so it's hard to believe that we've only
we've been talking for about 45 minutes
and we're only up to day three or day
four in your in your response so now
let's talk about Tuesday School opens
and you now have to make sure that
(20:12):
school opens safely and continue to with
your with your work what's going on now
so um that's part of the the keep
obviously you'll bring the Key Systems
off you know they are available you need
to chip them up at the same time and as
you're working to the rest of your
system you know in Ln USC we have about
(20:34):
200 applications and prioritizing those
applications depending on the
sensitivity about those applications
become critical one thing you're
probably gonna learn there are some
compliance system like you need to send
some report to this state something here
and there and they'd have like a
deadline you need to send this by such a
date you need to send that twice run
(20:54):
your payroll you need to do this you
need to do that and that's become
important why because uh to some extent
you can get the exemption from your
estate and we were in that such a
situation because every again it was a
big incident
um and it was publicly communicated and
feds everybody the White House and
(21:15):
everybody they were engaged and helping
us
um and at the same time communicated
with the state we need an exemption from
those requirements to be able to really
secure our system to be able to generate
those reports so uh it was important uh
some of the those decisions was made to
prioritize this system based on
compliance based on need based on
(21:38):
priorities to gradually so there are
different priorities usually you say
Okay tier one system critical bring it
out the rest of the system if you look
at uh through Disaster Recovery plan
most of the time you don't see clear
definition what are those systems and
the priorities that they need to come up
and I think that's become important
(21:59):
people usually ignore the tier 2 systems
priority but it is important it's not as
important year one it's not going to
prevent you to open your school but it's
gonna be your headache post-production
because you need to be uh bringing them
on time but you need to know which one
first and that was a part of our process
and we were able to actually as a result
(22:21):
of that there were a little bit proper
business continuity plan per Department
I'm not talking about I.T business plan
I'm talking about the departmental
business management class to be able to
have an updated access to their
technology at the time that they
hmm so let's talk a little bit about
um you know we've heard your
responsibility and and you were you were
(22:43):
making a lot of the decisions
um but I'm I'm sure there were still
some things that the superintendent had
to handle or the you know give
interviews make decisions make certain
calls
um can you talk a little bit about his
response to the situation his support
for you and your team
um buying pizza if he bought pizza a
night or two
(23:04):
um you know that that kind of that kind
of discussion around that
so definitely the main decision making
on all of this uh especially when you
look at the impact of the school and
incident ultimately was superintendent
yes he may ask my opinion do we need to
open this school or not ultimate beat
was his decision should we often this
(23:25):
will run back it was my opinion that I'm
gonna make it ready he realized he
trusted me that I can bring it up and I
believe her
uh the rest of it was his decision Army
operation only ready to open this school
after what happened what we went through
holiday schools the student and they're
gonna feel in the classroom and
everything else and that was a critical
decision I think that was one of the
(23:47):
most important decisions that he made
secondly uh I think from the day one uh
he confirmed and he committed he's not
gonna negotiate with the uh with the Bad
actors we never uh negotiated with them
to pay the ransom or anything that they
were requesting they were requesting we
(24:09):
never said even how much we never said
we uh that we're gonna pay them or
anything else and uh in his word uh and
I repeat uh his board make sure you
always know what are your uh wrist with
them so try to find out what data
because when you get talking about the
data exploration
(24:31):
um in that short period of time you may
have some data of some information some
logs knowing what they may have had
access but reality you don't know until
you get that uh you get something from
them and that's there's nothing wrong
with it but do not negotiate with them
because technically
they may give you a promise they may
(24:52):
give you your data they may give you
everything that uh you want if you pay
them to Ransom but they still have
access to your data their bad actor is
just like they have another copy they
publish it they sell it they do
everything so now that was a
recommendation from FBI as well that
essentially negotiating with the bad
actor is not the wise choice although
(25:14):
this bad actor apparently I didn't know
that FBI confirmed for me they usually
deliver on their promise in their
website they publish then your name and
they call you partner if you pay them
and they don't release your information
but your name is gonna be there to me
that's worse than if they publish my
(25:35):
data if they call me a partner sure
so technically your organization pay
them they publish your name they don't
lease your data they call you partner if
you don't pay them they just release
your data and they name you over there
so I again uh for us the rest of the
data was minimal uh I I believe I hardly
(25:58):
believe depending on the risk and
whatever it was we would never pay the
ransom uh and we would never negotiate
with the hard uh with this type of a
criminal and that was one of the
decisions to print and make from day one
to not to negotiate not to pay the rents
up and it's he's stood why
now it are so there's there's a threat
(26:20):
actor who's who's communicating things
to you are you communicating is the FBI
communicating I mean who is who's taking
care of that side of the the situation
you don't deal with that uh usually any
type of analysis about the bad actor
goes back to your insurance company they
have experts they know how to deal with
(26:41):
it to not engage that's what I'm saying
I repeat do not engage with the bad
actor because technically that's made
backfire politically they may back for
his security wise and everything else
living up to the expert they know what's
your risk obviously they ask you uh if
what's your risk tolerant and for us the
(27:02):
risk tolerance was we're not going to
pay
wow
you can see me as a CIO when
superintendent tells I'm not going to
pay the rent somewhere it's a good thing
I'm proud of him and I his decision but
it made my job more difficult means I
need to recover the system no matter
what there is no alternative I'm not
(27:23):
gonna be able to have a champion coming
and saving me I need to save restore
bring up the systems and and I think you
you alluded to this earlier though even
though most most of these threat actors
if you pay them on the idea that you're
going to get a decryption key sometimes
that decryption key doesn't work
sometimes it does work so you're you
(27:45):
know your 50 50 shot there and then
there's the double extortion of we're
going to release your data okay if you
pay them again that still really doesn't
mean they're not going to post your data
they still could post your data so I
mean his his stance of we're not going
to pay
to me it makes sense if if you if you
(28:06):
weigh that risk now again depending on
what kind of data they have but if you
weigh that risk I you know it's not a
guaranteed conclusion you know
true and and exactly that the European
it's a criminal you you can exactly
um you cannot trust you don't you never
know what you're gonna get uh and again
(28:26):
with the decryption key and everything
else again and by the way nowadays you
mostly deal with the double extortion
because most of the rans iso will stop
the rest and where tanks are failing
because every Everybody learn the lesson
keep your backup man keep your back up
don't worry about this that's why they
go with the attacks menstruation and
(28:46):
that's where his big game become tricky
how fast you can get this data
exfiltration and how can you stop them
from continually and that as early as
you can detect it is could be crucial
for the decision making around it uh
which goes back to uh what's your post
recovery plan what's yours um for
example for us uh it was implementing a
(29:08):
security Operation Center we had Network
Operation Center that was monitoring our
network but we never had the Security
offers which would be the results of
this incident as part of a
recommendation that came out of our ID
our I.T security task force
Okay so you just hit on it one of the
(29:29):
changes that you've made since the
incident is is a security operations
center
um can you give us an idea of uh some of
the other changes that you guys have put
in place post incident to improve either
operationally I mean I know you probably
can't get into what you've changed for
security posture wise but well what are
(29:49):
some of those those small tweaks that
you've made operationally maybe that
could enhance a future incident
Sean
um
I think it is important of going back to
the incident I think was in the first
week of incident uh when superintendent
gave his first uh briefing to the public
(30:10):
and to the board and everybody that he
announced that he's gonna have a 90 days
plan and that 90 days plan goal was not
only address things in the admittedly
mitigate the current situation but lay
on the pathway for us to build the most
uh secure and reliable
(30:33):
um it infrastructure indicate love
industry Across the Nation that was our
goal our commitment to essentially come
up with the recommendation to build this
as the model that everybody else can
take advantage that's where a
conversation with white house has
started because White House wanted to
develop a model for others education
entities
(30:54):
and the same the conversation expanded
to seesaw Cesar is the architect for
this security architecture across the
government and private sector
so well what we did uh we created the
very rapid task force we call it it
security password that their their
(31:15):
function was it's just going to come for
90 days
the people that they were invited to be
part of that task force
probably a school district like us or
like yours or anybody else they're not
going to be able to afford to pay the
salary of these people I named them for
you there were people from uh Wells
Fargo Chief level uh there were people
(31:38):
from uh Intel Chief level there were
people from
from uh I would say
um Microsoft and other areas Apple uh
again not the business side don't look
at the business side because business
sites always they try to sell you stuff
stay away from them
(31:58):
when you deal with the task force you
want operations like when I'm uh when
I'm saying when I have a ciso from uh
let's say apple
is not nothing to do with the government
sales or anything else these are the
people they're protecting Apple I.T or
apple and Lewis from Bad actors or same
(32:19):
uh other areas so these were the um even
we got the resources from uh
uh retired General that was working with
the Air Force cyber command so these
people with a lot of experience in the
Cyber industry that they are more
advanced than K-12 we know that K
through 12 in cyber security we aren't
behind the rest of the industry even in
(32:42):
it forget about security
um I believe one of our analysis showed
the K the education sector
um the uh the entire project of the I.T
Department in education sector was
um about five to ten percent of the
entire organization budget
believe it or not are entirely spending
(33:04):
I'm not taking the apartment budget
buying devices the school buying
technology everything we summarized it
we were 2.5 percent so the reality below
industry average when we talk about
because K-1 literally did dwarf us and
spending and this type of technology so
when you don't have those investing plat
(33:25):
you're relying on the resources that you
have
the ball up that task force was to make
a recommendation
um and review our strategies what we
have what we implemented the audit the
pen test the incident that added and uh
based on that they make a recommendation
for course correction and those are the
(33:47):
ones right now we are uh we build the
three years plan actually it's a
four-year plan the goal is to implement
it in three years
um the three years plan to revamp the
security infrastructure and footprint of
the LA USD to become the one of the most
secure and reliable infrastructure in
(34:07):
the country so from an outcome
standpoint okay you you've done this
90-day task force with these csos and
and other Chief level
um operational folks from from major
industry
um you have your report now are you
going to be a good and this is selfish
I'm just going to say this is purely
selfish on my part
(34:29):
um are you is La willing to be a good
neighbor in in your work with the White
House and this is a
um will you guys uh publish maybe
confidentially to to k-12s a framework
that that we can follow that okay here
here's kind of a checklist of things
(34:49):
that you need to be thinking about this
is these are the the higher priority
items that you need to be addressing as
soon as possible but this needs to be on
your roadmap as well
um is the plan for you guys to be kind
of a good neighbor in that method do you
know that that's actually one of the
things in partha were three years plan
and actually that was one of the ask
(35:10):
from the White House and from actually
the
um from uh cyber command as well that we
need to educate other uh K-12 and
literally there was a discussion about
traveling going to different School
District sharing our experience and
everything going on a podcast right yeah
(35:32):
did you think he was going to say no
we're gonna be a bad neighbor Josh well
I mean I figured I'd ask that question
hey you just said YT I will be there
what number did you call the White House
maybe we can start there is there a
phone number
yeah he they he's got a red phone on his
desk we can't see it in the picture but
(35:54):
there's a red phone that he picks up
yeah you adopt I guarantee you contact
your ogr office of government relation
they have all of those phone number for
you to be able to expedite the calls but
definitely the superintendent always uh
have an escalation access to the
Department of Education or from there
you get to the White House so that's the
easiest way to tell you how to get to
(36:16):
the White House yeah now one thing I
think you haven't mentioned here is that
your superintendent as as helpful as
he's been he he's been on the job for a
month two months maybe at the time
I didn't know that
(36:37):
yeah so he's he's brand new to LA and
he's probably looking you'd be like is
this is this part of the course is this
happened all the time
oh it was he was surprised it was a
surprise for us as well yeah in L.A we
never had the Cyber incident like that
but if one thing can tell you uh if you
are not safe nobody's safe uh especially
(36:58):
in education I think this is something
can happen it can happen to us as well I
don't actually the man there are a lot
of statistic that tells you if you have
an incident usually there is a 60 70
percent chance and they have another one
in the same year oh we are very
sensitive to be able to protect
ourselves uh one thing I would tell you
(37:20):
uh you this bad actor is specifically
they are famous or going after the low
profile Target they would barely go
after the law I provide like us
so there are debates why they picked us
um are they changing their uh structure
are they want to become a more uh
(37:42):
involved in this type of incident or no
they just made a mistake it was a stupid
mistake they didn't know who we are or
what size we are and the reality is they
spend money it wasn't like they did this
for free uh when did you spend when you
want to do this type of thing as a cyber
criminal organization
you have resources you have they have
(38:04):
Financial analysts they study
organization
and at the same time they do a lot of
reconnaissance they go through your
system they go to the market they buy
things and they try to break into your
systems and in our case uh it didn't pay
off so if you call it Roi their Roi
(38:26):
so uh
from a feedback standpoint from from uh
parents and students I saw a post on
Twitter uh a couple months ago I believe
it was I'm not going to name a name
because I'll I'll probably get it wrong
and I don't want to mess up but there
was a school district a relatively large
school district that had a ransomware
(38:46):
incident and they recovered pretty quick
but there were a couple days down time
and one of I think it was their CIO
posted on Twitter that when kids came
back they started receiving hand-drawn
pictures and notes and and uh thank you
more or less thank you notes from
students saying thank you it department
for working so hard on getting getting
(39:08):
the school back and getting us back here
at school and and fixing things
um what what was your experience like
from a from a parent aspect from a
student aspect from a from a faculty
standpoint I know you said that everyone
was pitching in and assisting
um if you could talk about that that
feedback or that buy-in support from
(39:28):
those those key critical folks
uh thank you I I received a lot of
support I mean uh there were text email
communication walking even in the
elevator with other people uh that they
were impacted with the Cyber attack it
was not a stop I mean people that they
left or retired now I mean people that
(39:50):
they went to other organization people
that they were not even with us anymore
they it was uh it was very support uh
very very well supported um the public
comment but yes when you read the you
know one thing I would tell you when
you're in the middle this type of
incident
it's easy for people to point a finger
(40:11):
and blame oh you did this you did that
you didn't pay attention to this you
didn't pay attention to that but the
reality is uh we survived number one if
we would have failed that would have
been the totally different conversation
uh and probably blame blaming pointing
figure would be much more than what we
received after this label with that we
(40:33):
still receive uh pointing finger and
blaming and everything up but the
reality is we need to survive we recover
we're restored and the student back in
the classroom that's number one and we
learned a lesson uh and we want to share
that lesson with everybody to be able to
recover and make a resilient system not
(40:54):
to be really uh
compromise with this type of a Bad
actors uh because it eventually it's
gonna happen and it just matter how you
mitigate the situation and how you
contain it to not to become a
catastrophic like some example of other
school district they ended up shutting
(41:15):
down and you think shutting down the is
is a critical part
I think that there could be much worse
when you lose 30 years or 20 years 30
years old critical information that
you're keeping in your system to do AI
analytic projection
(41:35):
or graduation for a student that they
are on track and everything else losing
that information you are it's a harm for
those students that essentially know
they're just writing in the rail without
the conductor and that's that's not easy
yes if same year same teacher as those
(41:56):
information you that the student
graduate to next grade what do you have
to support the next teacher to support
that the student is nothing when your
information now off in the air and
decrypt
yeah I I think we were kind of off the
side watching uh the situation in LA and
I think there's the public reaction and
(42:17):
there is you know when there's tweets
and social media around what's going on
there's also kind of the the behind the
scenes conversation right the folks like
us who are in the industry watching
things and I think generally speaking
those of us who are in the industry were
both very impressed that a LAUSD did not
close
and and you showed very very high
(42:39):
expectations for how this should be
handled and how you should respond to
this uh and B the other response was if
this can happen to La this can happen to
anybody uh so it's it's very
heartwarming to hear that you're you're
taking your roadmap and you are uh
sharing this with the community you are
coming on to our our podcast today to
talk about what you experienced to share
(43:00):
the lesson and hopefully uh help another
just prepare for for something that can
happen so so thank you so much to Hale
for uh for for joining us today
so thank you Mark and thank you all of
you guys uh you're doing a good job
sharing this information hopefully one
day we can go more detail about the uh
(43:20):
areas of the security and probably in
the confidential way or aren't we gonna
share that with other school district to
make sure again
when we share we want to make sure we
don't uh give information out to the Bad
actors but it's being used by the good
actors like you and the rest of your
team of course yeah yeah we need there
(43:41):
needs to be a a website that
like vets users so that you can share
you you know you're sharing that data or
that information with vetted individuals
who are in the industry they're not a
threat actor I look that's something
that's needed in this in this industry I
(44:01):
think that would make that that job a
little bit easier and uh to some extent
um I uh probably need to share with you
guys
um FBI has a structure called infra
guard and in Orange you you will be a
member of the info guard because that's
where uh to some extent is protected
(44:22):
it's more vetted people are part of that
um and mainly because uh you need to do
the full background check to be part of
that conversation and share
yeah we appreciate your time we don't
want to keep you any longer
um hopefully the fire department put out
the fire that they were going to uh we
we do really appreciate you being uh as
(44:44):
honest as you were tonight and being a
good neighbor in the future with uh you
know sharing that framework with
everybody when it's done hopefully in a
confidential manner
um any any closing questions mark Chris
favorite pizza favorite pizza
you want to go I don't know wow Boston
(45:05):
this stuff you have peace and it's in
Boston Mark yeah we've had we we've got
some pizza here yeah you gotta you gotta
come over here we'll we'll take you I'll
take you the north end it's okay the
Italian section of Boston you get the
best pizza you'll ever have okay I
didn't know that but if it is I go with
pepperon that's right okay all right all
right
thank you thank you very much thank you
(45:28):
but as we wrap up part two we want to
thank our sponsors for making this
happen let's talk about visor that's
v-i-z-o-r a recent NBC report revealed
that a school district in Virginia lost
1800 Chromebooks that's quote unquote
lost uh for a total loss of over five
hundred thousand dollars auditor said
(45:49):
that the district failed to have a
process to monitor device collection
when a student was withdrawing that was
a costly mistake of course so visor
v-i-z-o-r helps districts manage
Chromebooks and other it assets by
automating Best Practices within their
schools to avoid those kinds of
situations if you're interested in
(46:09):
learning more about that you're going to
go to this website
visor.cloud that's
v-i-z-o-r dot Cloud slash K-12 Tech talk
that's us visor.cloud K12 Tech talk to
get up to 20 off you can learn more
about them other sponsors we got
Fortinet you can email fortinetpodcast
(46:31):
fornet.com they got your Florida gate
your Ford EDR your afford a token and
whatever else Florida they are the
leader in cyber Security Solutions and
services also extreme networks are proud
sponsored the K-12 Tech Talk podcast you
can email Dominic there D mayor that's
d-m-a-y-e-r at extremenetworks.com for
(46:53):
your networking needs he can help you
with specs also look up their work with
the Super Bowl
provision Data Solutions you got a
Windows server issue you got some
networking that needs to be upgraded a
problem you can't figure out get a hold
of provision Data Solutions and then
somethingcool.com you can reach out to
Jeremy there that's Jeremy
somethingcool.com and you can check out
(47:15):
their cyber security offerings in the
link description
MFA
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Therapy Gecko
An unlicensed lizard psychologist travels the universe talking to strangers about absolutely nothing. TO CALL THE GECKO: follow me on https://www.twitch.tv/lyleforever to get a notification for when I am taking calls. I am usually live Mondays, Wednesdays, and Fridays but lately a lot of other times too. I am a gecko.