Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
On this week's episode of the K-12 Tech Talk podcast, we interviewed Charlie
(00:04):
Crash, founder and CEO of Infinite Campus, about their recent Salesforce breach and
what this means for school districts. Charlie shares how the threat actors got
in, what they gained access to, and how the company responded. Thanks for
listening. Live at the NTP studios, this is the K-12 Tech Talk podcast. I am Josh.
(00:25):
This is episode, did I say this? This is episode 257. Wasn't that a TV show in the
80s? 250? No. What? 557. All the old people like me will remember it. I think it was
557. Anyway, episode 257. It's been a day. I looked up at the clock like it was
raining cats all morning and looked up at the clock and it was 1230. That's the
(00:49):
kind of day I have been and it feels like I think K-12 ed tech in general has
had that kind of week, right guys? I agree with that. So lots on the news. Fill us in,
Mark. You're the news guy. I know this is gonna be a long episode, I think, so
let's just get into it. Well, we've got a lot on the news. We're gonna save some of
(01:13):
the big stuff for next week. We've got some stuff going on at the federal
government, some things at the White House. Melania Trump came out with her
new robot friend. We're gonna we're gonna table that one. Replace teachers.
I'm excited about that.
Mark, reel us back in quickly. All right, we did have a couple of big events hit
(01:35):
the news around cybersecurity. So the first one, there's a company called
Navigate360. They have a few products but they did announce a breach or
actually the hackers announced a breach of a specific product which is an
anonymous tip line. So there's multiple companies out there that do this kind of
service where students or parents can report an anonymous threat or concern
(01:55):
and then that will be routed to schools to take care of, Navigate360 being one
of them. We were hit with the news earlier this week that that may have
suffered a breach. If that is true, obviously we've got a lot of concerning
data out there and I know that a few of the districts are still trying to wrap
their head around what is going on and what has been breached. So this is very
early but it does go to show that, you know, no platform is obviously immune to
(02:20):
cybersecurity attacks and when you are putting something like that amount of
sensitive data into a system that just ramps up the risk. So I was gonna say is
the breach the anonymous stuff? Yes, yeah, yeah. That's devastating. Could be
devastating. Could be, yeah. I mean the company is still wrapping their heads
around what's going on at this time. At the time we're recording on Thursday
afternoon, the hackers are the ones who are claiming what they have. So the
(02:44):
proof will be in the pudding over the next few days to find out exactly what
was breached and if that information hits the dark web. Pudding or do you say
pudding? Proof is in the pudding. Yep. Okay. I thought you're saying putting it out on
the dark web. Well, that's another way to, yeah, that's P-U-T-T-I-N. It ticks me off. I
don't know what I'm saying here but I want bad guys to go after not that data.
(03:09):
Like, can we pick better data to chase after? Why are we gonna pick data of the
middle schooler that it took a lot for them to actually get the courage to do
the anonymous report thing? We're gonna expose that data? What, you want them to
go after Salesforce data? No, but if I'm picking between the things I guess I am
(03:30):
saying that. No, I get what you're saying completely, yeah. Bummer. Yeah, very, very
unfortunate. The hackers in a statement did say that they have claimed or they're
claiming they have stolen 93 gigabytes of data from this anonymous tip which is
about 8 million confidential tips. Very troubling to see that they would go
(03:52):
after that data but as you can imagine that kind of data some people would pay
a pretty penny to have that not wind up in the in the public area. So, very
unfortunate but we'll see what happens with that data. Second breach that we did
hear about and the two of you are in the hot seat on this one is Infinite
Campus sent out a notification on Sunday about a confirmed breach. This is of
(04:14):
their Salesforce instance which is their ticketing system and customer
relationship management system. Josh, Chris, what was your initial reaction
when you got that email? Gave me the Sunday scaries. I like that phrase. Yeah.
I don't like experiencing that phrase but I like the phrase. No, it definitely, I
sent email to my central office senior leadership team said, hey, just got this
(04:36):
email. I don't think it's hugely impactful yet but just to be aware. Did
you reach out and communicate to your families? No, not yet. No. Okay. Holding the
line. All right. Well, we were trying to figure out how do we get more information
about this breach or this incident and Chris, you said I got an idea. Let's reach
(04:56):
out and get a guest on the episode. Yeah, we do what we always do. You guys talk
about somebody and then I like to go ahead and send the email. So we reached
out to Charlie Kratch, the CEO of Infinite Campus. I did some Google
searches and some looking around and found his email address exposed on the
web. So I reached out to Charlie and he replied back and graciously said that he
(05:20):
would come and hang out with us for a few minutes to unpack the incident and
to talk about where things are today. So Charlie is in the room. Bring him in.
We are welcoming Charlie Kratch, the CEO of Infinite Campus to the studio. Charlie,
how's it going? I'm doing well. It's warming up in Minnesota so I'm doing a
(05:44):
lot better. So you're hanging out. Yeah, what's warm to you? It was 42
degrees and that was fantastic. Although we have had a 70 degree day here but
spring is strong. We're feeling good. The ice has left the state. Okay, I think it's
gonna be 90 here today. Yeah, you're hanging out with two guys in Missouri. It
(06:04):
was maybe it was in the 40s this morning. Now it's in the 90s. I wore a jacket to
school, took my jacket off and it's just that kind of roller coaster season for
us. And then Mark is coming from Boston. Mark, how's the weather in Boston?
The snow is about an hour north of us so we're still in pretty cold
weather. Same as Minnesota. Yeah, on the other side I also have a place down in
(06:25):
Phoenix, Arizona. So, you know, we've been up 107 down there, 108, so that's a whole
different world. All right, enough about the weather talk. Let's get into the
incident. Yeah, let's turn the temperature up here. Charlie, we are, so of course, we
banter every single week. We pay attention to what's trending, what news
is going on. We use, at the districts we work for, we use Infinite Campus. So we
(06:49):
pay attention to those emails that come out too. We, I don't think we even laugh
anymore. We talk about, hey, we should send an email to so-and-so. The guys know
that I will typically actually follow up with that. So we emailed you hoping to
see that you might respond and you did. So really appreciate you responding and
then following up to actually join us today. It is a big deal for the CEO of
(07:15):
Infinite Campus dealing with an incident to come on and hang out with us for a
few minutes. So I want to say thank you to you for doing that and kudos to you
for doing that. So thank you. Yeah, I appreciate that. I also appreciate, you
know, being customers. I appreciate that as well. It's always good to talk to
folks that use the product, so. And you used to be a K-12 tech yourself. I did.
(07:37):
When I founded the company, I spent the first three years also acting as a
district technology director and that's really what guided the development of
the whole system. When I started out, I wasn't even going to do an SIS, but that
experience of just seeing all the different data silos and the state of
affairs, that's what really drew me into this. And that was 33 years ago. Awesome.
(07:58):
That's awesome. Okay, so let's dig into this. Yeah, so Sunday
night, I, like however many other IT directors across the country, were
sitting on the couch enjoying a frosty adult beverage and I get an email alert
and the email came from Infinite Campus. And it had the subject of Infinite
(08:25):
Campus cyber incident, I believe is what the the subject said. So Charlie, I hate
to say it, but you ruined my Sunday night. Unbelievable. And you, I think, gave a
number of districts the Sunday scaries going into Monday morning. But your email
said that you guys were taking action against a cyber incident. And I believe
(08:46):
it even said that it was related to your Salesforce instance. So can you kind of
give us a little bit of background as to what the heck happened and up and to
that point of that email going out? Sure. And first of all, what I'll say, I
apologize for ruining your Sunday night. But, you know, it's, it's, it's always a
(09:08):
judgment call on a lot of these things, whether we should notify districts or
not, we are always under attack. We are a high profile target. And when something
rises to the point where we need to make a notification, that's when you see
things like this. And fortunately, that's few and far between. You're absolutely
right. Well, we put out in that notification and, and this is a thing
(09:31):
that try to be clear with anyone. This was not a threat against the infinite
campus database, our student systems. It was an attack against Salesforce and we
use Salesforce for our customer support ticketing system. So when you submit a
ticket or you call in, we'd log those tickets using Salesforce. There is no
(09:55):
connection between Salesforce and our campus instances for individual school
districts or States that we work with. Now, that being said, you know, there's a
couple things that I would highlight with this. The first is exactly that is
Salesforce and not campus. Salesforce was their target. And this is kind of what I
(10:19):
refer to as a smash and grab that the folks went in, they threw a brick through
the window and they went after the cash in the register. They weren't interested
in the jewels back in the vault. And the jewels is all the student data,
administrative data and stuff we have. That's what we're always paranoid about.
And so we take every unauthorized access seriously, including the Salesforce
(10:43):
access. And so the way things unfolded, and this would be the second thing that I
would point out is how fast these events move. These bad actors aren't, you know,
the hacker kids of old. These are professional, international criminal
games.
Right? Yeah. And sometimes state funded.
(11:03):
Absolutely. And just I've got to, if I can take like a minute here, just to give
you a little tick tock of how fast this stuff went. On the 18th, that was March
18th of Wednesday, there was this threat actor at 2.11 in the afternoon put up a domain, an
(11:27):
imposter domain. And that's what they would direct their prey to, to log in. That's how
they capture their credentials. So that started at 2.11. At 2.20, our security systems
began to alert us that it was sensing something was going on out there. So that was nine
(11:50):
minutes after this domain was created. And then at the same time, our employees on their
office phones started receiving voice phishing or phishing calls.
Wow.
I mean, so this is how fast it's done, right? 2.20. At 2.38, the targeted employee received a
(12:12):
call. And this is an unfortunate thing. We have a handful of employees that do not work on
site here. So this was an offsite employee. So that was kind of the start of a
vulnerability that, you know, most people internally, somebody, you know, they're going to
talk to somebody next to them. Hey, have you heard this thing? Is this real? Or they might
know other, you know, our IT people more personally. This employee offsite was targeted at
(12:35):
2.38. That was literally 10 or 18 minutes after this started. They got hit and they logged into
that imposter site. And that was it. You know, that's the start of the event. And the thing that
I want to point out for everybody out there, we do quarterly security training for all our
employees, they have to go through all the simulations. And what our folks would receive
(12:59):
these calls, what they said to a person was the person on the other end of the phone sounded like,
you know, a real American. You know, it wasn't like some foreigner calling in, it could have
been, you know, Bob down at IT. And these are foreign attackers. But the thing I want to point
out is they're recruiting kind of real Americans to do the legwork on these calls. And that's
(13:21):
pretty sneaky. So this is very organized. So 2.38, you've got our person, you know, logs into
that site, they're able to capture the credentials, you know, through MFA and everything else at 2.44.
So this is now six minutes after that, the unauthorized actor access the employee Salesforce
(13:42):
account, that's where we saw the first hit into that account. And they started just basically
running reports, customer lists, that kind of stuff, they knew exactly what they were doing. Once
they got into Salesforce, this was nobody that needed some training on how to use Salesforce, they
knew exactly what they were going after. Because at 2.48, four minutes later, we had already blocked
(14:06):
that imposter site at 3.10. You know, because our employee didn't know they had been hacked at this
point. But we were already going through and auditing all of our Salesforce accounts, all the
access by 3.10, we'd identified that one of the accounts this person accounts was had unauthorized
(14:27):
access. And by 3.16, we'd shut off that employees access to everything. So this entire event from,
you know, the initial contact of that threat actor coming after us, all the way to the point that we
had shut everything down, blocked access, that was 38 minutes.
(14:48):
It's pretty fantastic. Yeah,
that is, these are pros. I mean, it's not some kid getting in and playing around or somebody else
trying to traverse through systems and find their way through. They know exactly what they're looking
for. They got in and they got out.
Well, in this, this group is, I guess, known for Salesforce attacks, right? Like, yeah, it's pretty
(15:08):
well documented that this group that did this to you guys is, that's what they do is they go after
Salesforce.
Yeah. And we've been working with the FBI on this and the FBI is like, yep. Hundreds and hundreds
simultaneously of this is, and I'm not giving anything away here, but the group calls themselves
Shiny Hunters. If you Google Shiny Hunter Salesforce, you're going to get a whole screen full of this
(15:33):
stuff. And it's not even really, and the thing that I'd point out here too, it's in Salesforce's
defense, they're not exploiting shortcomings in Salesforce security. You know, this is social
engineering. This is getting an end user to turn over their credentials. And at the same time,
they're getting through MFA by putting up these bogus websites.
(15:55):
Yeah. And we're seeing the same thing with Google account credential harvesting. And
immediately there's a tool that logs in with those credentials so that the user gets prompted for that
MFA and then the bad guy has the session cookie. Like that is a very common thing right now with Google
account compromise as well. Mark, you had a question?
Yeah. So going before, so the domain was set up and the vishing started right away. How long do you
(16:21):
think they were reconning Infinite Campus in order to prep for this attack?
Oh, to prep ahead of doing all this?
Yeah.
I have no idea. You know, that's something that, you know, we're investigating. They were, they were
clearly going through our call tree. You know, they had our numbers, although anybody, you know, you
(16:42):
could probably guess what our phone numbers are going to be. We're also investigating in that
system, you know, what kind of origin number spoofing they might have been doing. You know,
that's an ongoing kind of thing. Now that we're really past this incident, we're doing the forensics
now going through and trying to reconstruct all the parts and then we'll pass that on to the FBI.
(17:03):
Okay. The vishing resulted in one employee giving their credentials away. Can I ask, did this
employee have MFA on their account?
Oh yeah. That's required. Everything we do is MFA.
So any idea or how they were able to get through or is there, or efforts that you guys can do to
strengthen the type of MFA that you're using? I guess all of us are saying, well, we got MFA. Is
(17:28):
are we at risk of the same attack?
The problem with these kind of social engineered attacks is once you have the trust of an end user,
they're walking you through it. You know, it just, yeah, you can make it even more difficult, but
now you're getting to the point where the system becomes unusable to a point. This, and this is,
(17:50):
you know, our training covers it every single quarter, everybody goes through it. You know, our
IT will not ask you to do these things. You know, it just, and unfortunately what the bad actors do
is they, they go after the trusting people, you know, and somebody that, that is just willing,
(18:12):
you know, to, to follow orders, you know, directions through this. You know, the silver
lining in this, we never want to see unauthorized access like this. But the, and we can talk about
more of the kinds of things they got, but you know, it's just, it's basically public directory
information. It is, it is a good fire drill for us, walking us through our systems, our responses,
(18:38):
but also for employees. This, just our response to this was expensive for us. You know, the time
and energy bringing in, we have outside organizations that came through and also audited
all the access and everything else. Folks that have experience with these Salesforce kind of attacks,
(18:59):
that costs money. It's a good thing, you know, for all our employees and anybody who accesses
any system to understand this is the form of a cyber attack now. It's not the kind of traditional
hack you think of where they're getting into backend systems. They're walking through the
front door and people are letting them in. And that worries me, you know, school districts,
(19:22):
I think are, are very open to this kind of thing. Yeah. I would, I've heard of at least three or
four districts in the last month and a half that have had this happen. Similar, not Salesforce
related, but other systems. Very, very similar. Yeah. Salesforce is not really used by K-12
districts. So I think a lot of our listeners may not understand what Salesforce is, but we do know
what a ticking system is. Can you describe how Infinite Campus uses Salesforce? There's a variety
(19:47):
of uses. In this case, what this user had access to is just the, the our support information and
customer management information. So if a customer calls in or contacts them through a ticket and
says, I've got a problem, you can process that ticket, forward it on to somebody else.
The our client resource managers use it to just review those tickets, make sure everything's being
(20:14):
taken care of. The customers get what they want. Our marketing people will use that for messaging
campaigns. We do our Salesforce, our salespeople will use it to qualify leads and follow up on
them. And that's interesting. One of the things that this particular threat actor does is they
threaten to release your customer list to the public. And we're like, fine. You know, everybody
(20:39):
knows what that is. I mean, they know who our customers are. Well, they'll see all your contacts.
Yeah. You know, these are school district employees, things like, yeah. It, it, it is, it's
interesting that, and this is the thing we were more paranoid about. It wasn't the fact that they
were able to access this stuff, which was bad, you know, obviously in itself, but our paranoia is if
(20:59):
there's any content in those support tickets that would include, you know, any secrets, any PII or
anything. Now our staff is trained not to put it in there, but fairly often a district employee,
when they're submitting a ticket to us, we'll put that information in and then we promptly delete it.
So a lot of what we were doing here was right after, you know, after we shut down the accounts
(21:23):
and locked everything out, we then ran an audit to find all the reports that they had run.
And we ran those reports and then began going through those, checking all the files for,
for anything, you know, that, that we wouldn't want out there to just, you know, kind of get our arms
around if there was any potential content. And that's what we spent most of our time doing,
(21:49):
was going through, I think it was, I don't know, it was about 20 gigabytes of files or something.
I mean, it's a lot, but there's really, you know, the vast majority of it is just a lot of nothing.
So this one account was compromised. Did this single account have access to the entire
ticketing database as far as all customers open and close tickets?
(22:10):
A good portion of it. However, it really depends on what reports were run. And that was key to
our forensic work was seeing the reports that were run and going back through and rerunning those.
And they had, they ran a report that really just dumped a small subset of data.
(22:31):
And it was an interesting report they ran. But it was, like I said, it wasn't, they weren't able to
get all of our Salesforce data. This employee didn't have access to it. And they even got a
subset of what this employee had just because of the reports they ran.
So we know that that original email went out to tech contacts, be that superintendents or
(22:57):
tech directors, Sunday evening. Are you, or have you already contacted the districts that were
subject in those reports that were ran and exported out? Has there been an alert sent to
those districts that, Hey, this is the scope of data that was taken. And yes, it, your data was
(23:19):
included in that report. Right. Well, without going into any specifics, there was no data in
any of these, there were no attachment files or anything. We were able to check that. And it's,
it's our policy that if there was any PII or any other information that was released specific to a
district, we would contact that district directly immediately. But the districts that just their
(23:43):
general ticket information, there's no, you don't have a policy to contact them at that point.
No, no. Other than what we sent out that this directory information had been released and,
you know, our assumption then, and that holds to this point, that was, you know, there was probably
a ticket in for nearly every district. Oh, sure. They were all notified, but if there's anything
(24:06):
further than that, anything that would require additional notification, our policy is that
district would be notified immediately and we would work with them to remediate whatever
damage was done. Have you had to make any of those notifications where there was a district
where PII or confidential data was compromised? That's something that we're not getting into
(24:29):
specifics on, but I can say that this was not a major incident with related
relation to that. And so whether or not there was any of that data in there, we're
again, just notifying individual districts. The trick with this is, and especially when we were
(24:50):
notified by the threat actor and the incident is how much information you want to get into the
hands of both that threat actor and other threat actors. You don't want to make yourself a willing
target. And this also goes to the point of the, you know, the extortion. They're looking for money
and it is our policy again in these exercises that we don't even interact with the threat actors.
(25:17):
And so we were notified and then we were notified again. They were letting us know they were really
serious. They put a sample file out. We didn't even look at it because we didn't want them to
know that we're, you know, even reacting to this thing. It's just give them the mushroom treatment,
keep them in the dark. And meanwhile, you're buying time to go through and do assessments.
(25:40):
And then at some point, you know, in these types of situations where once you've got your arms
around it, you've dealt with the immediate threat, you know, then you move on. And I think that's
probably Sunday when I interrupted your beer with our notice and we can do the general notification
and that's when that went out. And that was after, I think, Saturday was like their first deadline.
(26:04):
I know on Kato Tech Pro and we run the Kato Assist Admin subreddit, several folks gave you
some kudos on not being willing to engage because we've seen a lot of companies that will do that.
So has that always been what, I mean, was that a planned stance or would you ever consider
(26:25):
engagement? No, that's our policy. And that's why I put an email because, you know, in this world,
it's lessons learned from other hacks. I mean, just look at like, everybody's familiar with
the PowerSchool hack from, you know, a couple of years ago where they went in and paid the
threat actor and the threat actor took the money and just released it anyhow or not released it,
(26:45):
but went out and started, you know, these are criminals, you know, and they can't be trusted.
And it's a collective action problem. When you pay these ransoms, that just funds their next
attack. And as the FBI, you know, will tell you that when you pay a ransom, you're now 80% more
likely to be attacked again. You need to make that investment on the front end, not only in hardening
(27:09):
your systems, but also with these policies and procedures. So our policy is, not only do we not
engage or not pay the ransoms, we don't even engage with the attacker. There is a school of
thought that says you engage with the attacker to buy time. But what you see with this is this
whole thing was over in about an hour. Our forensics, you know, were done within the next
(27:31):
day. And that was really waiting for them just to verify that what we thought they got, they actually
got. So I think the tough part about the initial message that you get from a vendor saying that,
hey, we had a security incident is always the, well, what about me? What ifs questions that a
lot of districts might have? And I think a lot of districts are still wondering, am I going to get
(27:52):
another message? And so you did mention that, you know, if there was a district that was deeply
impacted or impacted more than others, that you would reach out and talk to them directly. Is
there going to be an all clear message where you say to districts at this point, if you have not
heard from us otherwise, this incident is closed, or are you still communicating with districts that
were impacted more than others? Excellent point. That's kind of the phase we're in right now,
(28:16):
that it is my understanding as of this morning, this threat actor did release all the files they
got. And so I know our security team is doing a final analysis of that versus what we believe
they received. And so we're just verifying that. And if we're able to verify that file, then you're
absolutely right. We can give the all clear. And we haven't seen anything yet that would make us
(28:39):
believe they got anything we didn't anticipate about an hour after we thought they got.
Okay. So when do you anticipate having a final message to districts about this particular
incident? That'll depend on, we have both our security team that's gone through that. We also
have a third party resource that's doing the same thing that came through the outside counsel we
(29:04):
have on this. And the safest thing would be to wait until that third party who does this for a
living goes through it and does that final all clear. And so, like I said, our security team
moves a lot faster than them. We feel pretty good about where we're at, but I think before we close
(29:27):
the book on this thing, I want to have that outside firm give us the thumbs up. Understood.
Yeah. So districts that are listening to this are in one of two categories. One, when am I going to
hear from Infinite Campus, which we've talked about. There's not necessarily a hard timeline
on that one. And two is, what should I be telling my community? For everybody, all Infinite Campus
(29:49):
customers or even non-customers who may have been incorporated or a part of this, should there be
messages that go out to families around this particular incident? And if so, what should those
messages to families say? Yeah. It's a tough one because on one hand, you don't want to cry wolf.
Every time something happens, you don't want to send a notice out to your parents because then
(30:13):
when something actually really happens, they may not pay attention to it. And that's kind of been
our approach of really emphasizing that this was Salesforce information. It was publicly available
directory information. And quite honestly, if I was still a tech director in the district where I
was at, I wouldn't even notify parents of this. Now, if it is escalated to the point where it is
(30:38):
an unauthorized access of the student information system, absolutely. That's a whole different kettle
of fish. But they're not at that point, but it's their call. I mean, it's up to them and what they
feel their comfort level is. I think that, again, I talk about this. We're always under attack.
(31:03):
Districts more and more are under attack. It's a great example. My message to school districts is
try to develop a policy and actions to that policy. What are the scenarios? Absolutely. If
there's been a breach of your student information system and student data has gotten out, boom,
(31:24):
red alert. This is how you do it. If it's low level stuff where you have a third party like us,
where a supporting system that's not connected to anything you have in your district,
you know, had an unauthorized access, it's like, huh, that's interesting. I'm going to continue
to monitor that. You guys know this. I mean, we have dozens and dozens and dozens of third-party
(31:44):
products around here, not only with our ticketing system like this, but systems that help with our
software development, that help with monitoring systems and everything else. We protect them all.
But in this day and age, again, it's not a technical hack, but you get especially,
you know, some less sophisticated user, they give their credentials out. I mean, this is going to
(32:07):
happen. Charlie? So for a district that's saying, hey, I submitted a ticket last week and I had
this student name and student phone number and all this information in that ticket,
should they be reaching out to Infinite Campus for more information?
Yeah. I mean, if they know of that, and again, our support techs, when they see that,
they will immediately, you know, delete that information and work with the district to
(32:33):
understand why to do that. Or if that information needs to be included, we have a standard operating
procedure for encrypting that data, which we, you know, use all the time. And again, when we
send anything out, we always encrypt it. It's just usually in plain text. Somebody will just,
in a note, will, and a lot of times it's just a student name, you know. And that's even a tough
(32:57):
one because the student name per se is not, you know, under the definition of FERPA PII,
you know, that's technically directory information. And this is a lot of the training that we do for
districts to understand what directory data is. Now, we're not out giving it out to people,
but a lot of your district staff is, you know, when you post a list of your football team,
(33:17):
that's directory information. You know, when you put things in the newspaper, that's directory
information, you know, understand what that is. For districts going forward, if they're curious,
I was in the support portal the other day, I guess Monday morning after all this went down,
and I saw that I can export all of my tickets out of the support portal. And if I export that out
(33:41):
and load that into my favorite LLM that I've got a privacy agreement with and say, show me any
incidents where I might have a student name or any other PII, and it spits back five or six
instances that could be considered PII, should a school district reach out to the help desk or
(34:04):
create a ticket about that? Well, okay. So the first thing I would say is, I mean, you're
technically sophisticated, so you understand when you load... Don't talk up Josh like that, Charlie.
Don't do that. I just, I want to be clear that the average end user, when you say load this into an
LLM and have it scanned, that if you're just putting that in chat GPT, that becomes publicly
(34:26):
available information. Right, and then that's why I added that you have a privacy agreement with, yeah.
I just want to make that clear for anybody else there. Please, if you don't know what you're doing,
please don't do that. But effectively, that is a version of what we did
with everything we thought that was released. So would I recommend a district do that?
(34:51):
Probably not, unless you're super paranoid and super technical. We haven't met in person, Charlie,
but I am very paranoid. Yeah, you just, if you talk to Josh's closest people, they would literally
call him paranoid and tech savvy. And the reason for that is, I can't stop a district from doing
that. Sure, yeah. You know, but on the other hand, you can easily do more harm than good with that.
(35:13):
And with all of this security and everything else, and there was another thing we did,
which I want to bring up in our reaction to this, was one of the services that we provide,
just in the background for our hosting, is we're always doing threat assessment and scanning of
the district instances of our product. Because districts are doing connections to our product to,
(35:35):
you know, God knows what that's out there. So we scan them for security vulnerabilities.
And our security team will contact the local school district and, you know, walk them through
how to, you know, lock down IP address ranges and things like that. In response to this,
we re-scanned our entire network. And we found, you know, again, they always pop up. We found some
(35:57):
of those. But instead of, again, these things move fast. Instead of contacting the district
and working with them to fix those things, we just shut them down ourselves. And then,
you know, as the dust settled, worked with the districts to re-establish those connections and
lock them down properly. But, you know, it's under, you know, an abundance of caution. Again,
(36:18):
you know, going back to the first thing I said, these things move fast. And our biggest paranoia
was not that the unauthorized access could be used to laterally move across our system through
Salesforce because there is no connection. It was that there may have been content of a message
that they could have then used, you know, to access the system in a different way.
(36:40):
So if a district is paranoid and tech-savvy like I, and they create a ticket that says,
hey, I have these ticket numbers with this information in it. Was this in the release?
Is the help desk going to be able to pick that up and run with it?
That would be forwarded to our security team. And then they would respond to that.
Okay.
And then a district like myself is assuming that I'm going to be contacted if there was
(37:05):
some extra data in those tickets and I was impacted by those reports that were ran.
Absolutely.
And it just takes time for us to go through everything. You well know,
even AI systems that go through can make mistakes. And so we crank it up to false positives.
Sure.
So a lot of what we're dealing with is like, oh, that's not something. But, you know, when you have
(37:26):
many, many documents, it just, it takes time.
Going forward, would you recommend that districts stop including student names and tickets and move
to referencing them as their person ID out of their campus instance?
You know, oddly enough, you're better off. I mean, it depends on the context, but the
(37:48):
student ID is more damaging than the student's name.
That's why I said person ID, not student ID.
Person ID is even more. It's just how unique it is.
Our help desk has a protocol for this. And typically what will happen is in instances like
that, the, you know, a ticket will come in and the followup will be made via voice call, you know,
(38:16):
and that information will be then transmitted via voice, which is of course much more difficult
to capture that data.
Sure. Mark, go ahead.
Or it'll be encrypted.
I think we're all hyper-focused on student and staff data, PII, and obviously that's going to
be the first lens that you look at. But there's also some concerns around, you know, support
(38:37):
ticket data can expose potential vulnerabilities in your configuration, in your district, in your
setup. We did hear from one district who said that they got a phishing email referencing an
infinite campus ticket. Now the timeline may just be coincidental because it doesn't necessarily
line up with the release of the data. But, you know, is there a greater risk to the community
(38:58):
now that districts' inside information and confidential kind of district configuration
data that's exposed within the ticket could be used for greater harm?
Perhaps.
It's a tough thing to answer. Is that possibility there? Of course.
(39:19):
This kind of, our experience has been this, you know, very professional actor going after
Salesforce accounts with the smash and grab approach is different than the kind of actor
is different than the kind of actor that's really targeting a single district and trying to drill
into them. And we haven't seen any crossover between the two. And we, I have seen no evidence
(39:45):
that any of our support tickets contain anything that would help the latter. That being said,
that threat is still there. That, you know, a lot of times I hate to say it, but the threat actors
are kids in your district. They're doing port scans of all your servers. They're, you know,
I used to say this when I was a tech director that the enemy's inside the wire.
(40:08):
You know, the kids were putting key loggers on the computers, you know, to get the teacher's
username and password off of that. And there's a lot of things we do internally with our design
to thwart that. The threats are everywhere. This is the world that in which we live.
It's, you know, I do a lot of work with politicians and whatnot, and quite honestly,
(40:30):
I'm disappointed in the lack of action that the federal government's taken to head things like
this off. That, I mean, the FBI takes, you know, these cases and things, and they help to be
proactive. But at the end of the day, they know where these threat actors are coming from. North
Korea, St. Petersburg, places like that. They can see these attacks coming. And I believe this is
(40:58):
my personal opinion. They let, you know, they let the puck pass a goalie because they don't want to
use the tools they have to shut those guys down because they're waiting for the real big ones when
they're going after the power grid and everything. So we're just, you know, we're, you know, these
kind of things, the end of the world is not somebody getting these support tickets. And so
they let it go. And it just becomes an annoyance. But with these threat actors, and even with
(41:22):
districts and whatnot, it's seen as a victimless crime. And that's not so. I mean, the cost and
the potential damage to students and parents. This is a real deal. I think it needs to be
taken seriously. I think that, you know, the federal government's the only one that can really
do something about this. And if it's left up to vendors like us and school districts like yours,
(41:44):
we're outgunned. I mean, the tools, like I said, 38 minutes, this thing was over.
How long is it going to take for them to just use AI to do all this?
Yeah.
Where our phones are constantly ringing with AI vishing attacks. I mean, who's going to stop that?
That's got to be the federal government, you know, and until they get serious about this,
this is the world in which we live. It's a sad state of affairs. The time we spend
(42:10):
dealing with this is time we're not spend innovating our product. It's time we're not
spending helping school districts do their jobs. It's money that, you know, we could spend on other
things. It's, you know, it's a tough world we live in right now. It's got to get better.
Yeah, we certainly agree. And I think it's been hard as a district feeling a little bit more and
more alone every day. But we do want to try to turn every crisis into action. On your side,
(42:36):
Charlie, what is happening with an infant campus as a result of this to make sure that this doesn't
happen again? Or what lessons can districts learn from this?
Yeah, well, let's start with us internally. On one hand, on the negative side,
that one employee, you know, of the 650 employees and all the times we get hit that one employee,
that one minute, that one lapse of judgment, set this off. And it's a reminder to everybody
(43:00):
in the company, this is real. This is what we train for. On the positive side, our systems
for detecting this, our system for dealing with it, our policies and procedures for reacting to it.
I thought worked quite well. And we've also been complimented by the outside council and
others we work with where they were shocked at just our game plan. And they're like, this must
(43:20):
have happened to you before. And we said, no, this is really the first time we've been hit with
something like this. But we've been planning for this. We've had, you know, a lot of our security
team work in long hours just to make sure this doesn't get worse. There is some things now you
extend that out to districts, you know, silver linings right now for our district customers on
(43:44):
our district application, the stuff you guys use is, is multi-factor authentication is an option
that we strongly encourage be enabled. I can already tell you that MFA will no longer be an
option next year that every district will have to have that. And we get a lot of pushback on that
because of the inconvenience that teachers yada, yada, yada, you know, they don't want to do this.
(44:09):
That's just in this day and age, no longer an option. And so Samuel accounts as well as local
off accounts are just, well, if the district has a, an, a, an authentication mechanism outside
our product, then, you know, we'll basically grant a waiver on that. We just want to make sure
that proper security is in place all around our product. And then the next big thing, and we
(44:29):
haven't done this yet, but I get asked a lot in this call is kind of, you know, what should we be
worried about? I'm not real worried, you know, the front door, because even if a teacher account or
even some administrator accounts are compromised, we've got a lot of things that can contain that
damage. The biggest thing is the backend access, you know, and that's what your sophisticated
(44:53):
actors are going after. The unfettered, you know, ODBC access or whatever. And that's, you know,
that's the third-party vulnerability where if a district is giving out, you know, an admin privilege
to, you know, an ODBC access to a third-party vendor that's providing a connection to other
(45:13):
products, I have no control over that. I don't know what's going through that pipe. I can't secure it.
Our roadmap is to end-of-life ODBC access at some point in the future.
And that'll go, everything will be going through our API, which is much more controlled.
It's a different world, but again, it's the world we're living in. And it's going to be
(45:38):
inconvenient, especially a lot of smaller vendors, that when you're working with a vendor like
Infinite Campus, we have a security team, we have all these things in place. There's a lot of small
companies out there that a lot of districts use. They like that little IEP system from the person
down the street. They don't have any of this. Now, we're a target that folks go after, but those
(45:59):
little companies survive, you know, security by obscurity, right? Nobody knows they're there.
Well, when somebody knows they're there, it's actually very easy to traverse through their
system right into our product. That's what keeps me up at night. That's what worries me more than
what we're doing. It's what those, you know, small, especially local vendors aren't doing.
You know, have they got, you know, do they have SOC 2 Type 2 certification? They don't even know
(46:22):
what that is. Are they doing regular security audits? Why would we do that? We're just,
you know, a little company that does, you know, a nice little food service app.
You know, that's the kind of the reckoning I think K12 is going to get into is this whole
kind of EdTech ecosystem and how secure the ecosystem is at all. Even things like a number
(46:46):
of states, I know like Missouri, where you guys are talking about it, there's a big movement,
like with all the EdFi systems and all of that. Well, you know, that's a wide open door who's
managing all those keys. And how is that, you know, it's again, security by obscurity. Most
people don't know what it is or how it works. But if you really look into some of these things,
(47:09):
there are vulnerabilities there. And we try to manage as much as we can on our end. But,
you know, there needs to be, I think, some serious thinking done about how this whole
ecosystem works. Well, I think you need to buy your security team pizza, Charlie.
Lots of pizza. I saw a lot of donuts up there this morning.
(47:33):
That employee that gave their credentials away, getting Hawaiian pizza.
Yeah, because that's the worst. Yeah. Charlie, I have one question that we were talking about it
on K12 Tech Pro, and it's interesting timing. So I do want to ask. So Missouri, we got the
announcement about CIC and that campus support is shifting away from CIC for Missouri folks to
(47:58):
Infinite Campus. Is the timing of that related to this? Was that already in motion? We knew
that billing and things changed. No, that was in motion. This was just a coincidence in timing.
And that question did come up, but the issue was all the billing shifts and all that had already
been in play. And so the two events are independent of each other. We like to wear a tinfoil hat
(48:23):
sometimes. So that email coming today was really interesting. I know. It's just, you know,
there's a lot of operational and legal issues that go along with the timing of that and
billing cycles and all of that. And I can assure you that there's absolutely nothing with that CIC
(48:44):
support shift, no link whatsoever to the unauthorized access of the Salesforce system.
That one I can tell you for sure. Okay. That's good to hear. Yes. Charlie, we thank you so much
for hanging out with us. You have hung out with us longer than we thought you would hang out with
us. So we do appreciate you. I've been on with you guys before. I know you go along, so I buffered it.
(49:08):
It's all Chris's fault. It's all Chris's fault. Yeah. And then sometime we can have a talk. I
would love at some point to talk about that future of the edtech ecosystem. We're doing
some great things with the architecture of our product and really where we see edtech going
with getting away from, I mean, still working with administrative processes, streamlining and
(49:31):
those and everything, but really focusing more on educational outcomes. That's where technology
needs to go. And that's really where I started working at Mac with things like Oregon Trail and
number bunches and all that. We really need to say, how does this technology both help learning,
but also when you look at a lot of generative AI that's out there, how can it hurt? And I'd love to
(49:51):
have that conversation because that's what I spend most of my day working on. Awesome. I get
interrupted by these things and I don't drink beer. So, you know, whatever. Awesome. Well, thank you
so much. Have a good time. Thank you. All right. So that's the episode and episodes like this are
special episodes. So I'm going to say to our sponsors, I'm sorry that I'm going to kind of
(50:15):
put you guys all together, but sponsors like this make these episodes happen. So in particular,
meter has been hanging out with us for several weeks, meter.com slash K-12 tech talk. Go to that
URL to book a demo with them. They can do your network infrastructure, your full stack network,
your internet, wired, wifi, cellular, all that good stuff. Check out meter at meter.com slash
(50:38):
K-12 tech talk. We also want to thank visor, visor.cloud slash K-12 tech talk. They just came
out with a huge release of over 70 new features. One of their biggest updates is a redesigned bar
code. Check in, check out interface. It's faster. It's more intuitive and takes out some unnecessary
(50:59):
clicks. So check out visor.cloud slash K-12 tech talk. Thanks to Eaton. They've been hanging out
with us for a while now too. They can do your UPSs. I'm going to put a link in the podcast
description to Eaton's products. Learn more about their cloud solution as well. So check out Eaton,
their different UPS models. They can hook you up. All that also to say, we are talking about
(51:24):
Infinite Campus on K-12 tech pro. Join that community. It's a vetted private community just
for K-12 techs and sponsors like NTP, Fortinet, chromebookparts.com and managed methods are on
K-12 tech pro. Again, helping make this thing possible. When you go to join that thing, you can
click membership. You can either pay or just click sponsorship and NTP, Fortinet, chromebookparts.com
(51:47):
and managed methods will hook you up with a free sponsorship for K-12 tech pro. Once again, thanks
for listening.
The views and opinions expressed on the K-12 tech talk podcast are the personal opinions of Josh,
(52:10):
Chris, and Mark, and do not represent the views or opinions of our sponsors or other organizations
that we're affiliated with. The material and information presented here is for general
information and entertainment purposes only. Thanks for listening and we'll see you next week.