December 4, 2023 • 9 mins
Are you ready to unravel the mysteries of cybersecurity and exploit prediction? We're joined by our esteemed guest, Jay Jacobs. Today's episode promises to be a thrilling ride into the world of data insecurity as we welcome the innovator behind the groundbreaking EPSS (exploit prediction scoring system). This ingenious machine learning model, launched in 2021, predicts the likelihood of a vulnerability being exploited. Not only does it help us zero in on potential threats, but it also offers us the luxury of knowing which ones can be put on the back-burner for patching. With daily updates and a scoring system that ranges from 0 to 1, we'll discuss how this system has become indispensable in the realm of cybersecurity.
Josh Bruyning LinkedIn
Jay Jacobs LinkedIn
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:02):
This exploit prediction scoring system, EPSS
exploit prediction scoringsystem, and this is the first
time I've ever heard about this,so let us know what is the
exploit prediction scoringsystem EPSS?
Speaker 2 (00:17):
Yeah, let me go back to your first thing.
So you're talking about datainsecurity, and that is a huge
and wonderful topic to get into,and essentially, data is just a
form of feedback, right, it isobserving and recording
something that happened and orlabeling or something, and that
is huge in cybersecurity becausewe struggle with this feedback
mechanism, right, trying tounderstand what is a great
(00:39):
decision, what's a good decision, what should we do?
In this case, Getting data andgetting that feedback is
essential, and so carrying thatover into EPSS.
So EPSS is the exploitprediction scoring system, and I
remember, probably 15 years ago, looking at CVSS and looking at
what they were doing forvulnerabilities and I thought
that's it, like they get somepeople and they talk about what
(01:02):
they think is important and thenthey give it some weights and
then there's some magic math andthen you get a score and I
thought, wow, we have all ofthis information, there's all of
these things going on.
We have so many devices tocatch things being exploited.
Why don't we look at those andtry to create that feedback loop
?
And that is exactly what EPSSstarted to do.
(01:23):
So, through my day job atSanctea, we were getting.
We have a bunch of datapartnerships with different
companies to do research ontheir data, and one of those
companies was Fortinet, andFortinet has an enormous network
of devices out there blockingand reporting on exploit
attempts.
And so we grabbed that data.
And we're also partnering witha company called Kena Security
(01:45):
who's now part of Cisco, andthey're doing vulnerability
management.
So we had all this data onvulnerabilities and we had all
this data on exploits and wejust brought it together and we
said, all right, we know thatthese things have been exploited
.
What about thesevulnerabilities, or make them
more or less likely to do that?
And so we built a machinelearning model and it's updated
daily.
So like if something happens inthe landscape let's say, an
(02:09):
exploit module gets published tothe internet, like a lot of
well-intentioned people will putthings out there with exploits
or they're trying to get a sliceof fame so they put it on a
GitHub or something like that.
When that happens, we know thatthere's actually an increase in
the chance of something usingthat and exploiting it in the
wild.
And so we update.
We've got well over I thinkwe're at 1500 different
(02:31):
variables that we're looking atfor vulnerabilities, trying to
predict when something or howlikely something is to be
exploited.
Speaker 1 (02:39):
That's insane, and we find that people are moving
more towards, especially CISOs.
They wanna match what's goingon in the real world with what's
going on in their dashboardsand in their ecosystem.
So what's the reception like?
I mean, is this somethingthat's brand new?
Is it out in the wild?
Are people using it?
Speaker 2 (03:00):
Absolutely yeah, and it's free.
You can go grab these scoresright now.
It's at firstorg slash, EPSS,and the scores are out there.
You've got an API.
You can download data on adaily basis, but the reception
so like it's been really, reallygood and we launched it.
I think we started publishingthe first scores in 2021, the
initial research was 2019.
(03:21):
We've gone through threeiterations since then, so we're
on a third version and we'rejust getting better and better
as we iterate.
And the big thing you'd thinkthat it was like we can tell
people really what to focus on.
That's not the real benefit.
The real benefit is that we canactually identify things that
you probably don't have to focuson, and that's been a huge
(03:41):
thing for people, because whatwe found through some of our
research with Kenna is that mostcompanies are fixing between 10
and 15% of the openvulnerabilities in their
environment a month.
So any given month, they'reonly able to fix 10 to 15%.
That was about the median.
Some are higher, some are lower, of course, but that's about
the average, and so what thatmeans is that you need
(04:01):
prioritization.
You need to know what youshould focus on versus what you
can safely delay and work into alonger patch cycle, and so
that's really the value thatwe're bringing.
We're telling people thingsthat they could safely more
safely delay and being a lotmore accurate than the
prioritization that they aredoing.
Speaker 1 (04:19):
Excellent.
So it's not just what shouldyou be doing, but what to not
waste your time on.
You should have.
Okay, great, what is thescoring system like?
Is it sort of a you know one to10?
Like what's a good score?
Speaker 2 (04:32):
So it's between zero and one and it is a probability
like an actual trained, you knowcalibrated probability.
And so what you can do, if youhave one machine with 10
vulnerabilities, you could saywhat is the probability that any
one of these will be exploited,and you can just, as you know,
if you understand probability,mathematics takes a little
(04:52):
learning curve there, but youcan do that combination, which
is very, very hard, as you know.
If you have like two highs anda medium and a low, how do you
combine that?
You know when you have thatordinal value, so the
probability, you can combine itand it is just between zero or
one, or we'll display it, as youknow, zero to 100%, yes, and
nothing ever gets to zero,nothing is ever at 100%, and
that's just the way the model'sworking.
(05:13):
Yeah, and we have a whole.
We've got several papers thatwe've published on the research
we put behind this and we'lltalk about the calibration.
So when we say 50% chance ofexploitation in the next 30 days
, we put that time window on it,that about 50% of those will
actually be exploited in thenext 30 days.
Speaker 1 (05:30):
Can you produce a bell curve with that data?
I guess that you guys have alot of you know there's a
distribution in the score.
Speaker 2 (05:37):
Yeah, but it's not a bell curve.
It's.
You know, bell curve issymmetric and nice and pretty.
This is really heavily weightedon the low end.
So I think like the top 15% waslike 2% probability and above.
So, like 85% of the data isbelow 2% chance of exploitation,
and that goes back to allowingpeople to say what should I not
(05:58):
focus on right away?
You know there's a whole bunchof things that you don't have to
focus on right away.
Speaker 1 (06:03):
Okay, yeah, so just don't focus on those outliers.
It's probably a really longtail and all these things.
So this is a really thepractical application is here.
Here is if you're in, you'rescrolling through the news and
you see all these buzzwords andyou're seeing all these look,
the media has been hijacked andyou know cybersecurity folks
(06:24):
don't know what's importantanymore.
You know, I mean, if you'rereally in the trenches, you know
, but this seems really usefulin being able to say, okay, well
, yeah, we heard that thisperson got breached because XYZ,
maybe they're not in myindustry, Maybe this doesn't
really matter to me, and so yoursystem would be able to say,
yeah, this thing is an outlier.
(06:44):
It gets a lot of buzz, Peopletalk about it a whole lot, but
guess what?
It doesn't really happen thatoften.
What about if it?
If it's an outlier, it onlyhappens.
You know it's very rare, but doyou have a way of capturing the
impact of that vulnerability?
Speaker 2 (07:01):
No, and that's a great question, because we very
explicitly did not look atimpact.
And so if you, if you want totake like a risk based approach,
there's two other things toconsider other than the score.
So the score is basically howlikely is someone to try and
exploit this?
The other thing is what sort ofcompetent and controls do I
have?
So if you know if thevulnerability is like 15 layers
(07:24):
deep in your network or if it'sexternally facing you know, on
the on your perimeter, those aretwo totally different scenarios
from your perspective.
The other thing is the impact.
Right, we have no idea what itis.
But if you can look at a systemand say, hey, this is really
really important stuff, we wantto be careful with the
vulnerabilities on here, versushey, this is just, you know,
basically a brochure that couldgo away, we could reboot it, we
(07:45):
could rebuild it, whatever, wedon't care.
All of those are going toimpact the overall decision.
So EPSS is just producing thatthreat, if you will.
And then you need to understandthe environment and the impact
which, of course, we have noidea from a centralized location
how everybody, how differentvulnerabilities are going to
have an impact on theenvironment.
Speaker 1 (08:05):
Excellent.
Well, I'm going to stay on topof this and I hope that you come
back and drop in another timeto tell us what's going on in
the EPSS space exploitprediction scoring man.
Well, I just learned somethingtoday.
So if this is interesting toyou, if you're listening to this
or watching watching this, ifthis is interesting to you, drop
(08:26):
us a note, leave us a comment.
And, jay, I don't know if youlike people reaching out to you,
but if somebody wants to reachout to you, how can they find
you?
Speaker 2 (08:34):
Boy I.
I don't know anymore.
You know it used to be Twitter.
I'm on LinkedIn, jay Jacobs onLinkedIn at at Santhea, and
that's probably the best way toreach out to LinkedIn.
I put occasional few posts datadriven posts out there, so all
right, jay.
Speaker 1 (08:49):
Well, thank you for being so gracious with your time
.
Thanks for dropping in andthank you for listening to this.
We still got to find a name.
Maybe we'll take somesuggestions, but right now you
know, we'll call this securitymarket watch drop ins.
So thanks again.
Thanks, jess, all right.
Bye, all right.
Bye, all right.
This exploit prediction scoring system, EPSS
exploit prediction scoringsystem, and this is the first
time I've ever heard about this,so let us know what is the
exploit prediction scoringsystem EPSS?
Speaker 2 (00:17):
Yeah, let me go back to your first thing.
So you're talking about datainsecurity, and that is a huge
and wonderful topic to get into,and essentially, data is just a
form of feedback, right, it isobserving and recording
something that happened and orlabeling or something, and that
is huge in cybersecurity becausewe struggle with this feedback
mechanism, right, trying tounderstand what is a great
(00:39):
decision, what's a good decision, what should we do?
In this case, Getting data andgetting that feedback is
essential, and so carrying thatover into EPSS.
So EPSS is the exploitprediction scoring system, and I
remember, probably 15 years ago, looking at CVSS and looking at
what they were doing forvulnerabilities and I thought
that's it, like they get somepeople and they talk about what
(01:02):
they think is important and thenthey give it some weights and
then there's some magic math andthen you get a score and I
thought, wow, we have all ofthis information, there's all of
these things going on.
We have so many devices tocatch things being exploited.
Why don't we look at those andtry to create that feedback loop
?
And that is exactly what EPSSstarted to do.
(01:23):
So, through my day job atSanctea, we were getting.
We have a bunch of datapartnerships with different
companies to do research ontheir data, and one of those
companies was Fortinet, andFortinet has an enormous network
of devices out there blockingand reporting on exploit
attempts.
And so we grabbed that data.
And we're also partnering witha company called Kena Security
(01:45):
who's now part of Cisco, andthey're doing vulnerability
management.
So we had all this data onvulnerabilities and we had all
this data on exploits and wejust brought it together and we
said, all right, we know thatthese things have been exploited
.
What about thesevulnerabilities, or make them
more or less likely to do that?
And so we built a machinelearning model and it's updated
daily.
So like if something happens inthe landscape let's say, an
(02:09):
exploit module gets published tothe internet, like a lot of
well-intentioned people will putthings out there with exploits
or they're trying to get a sliceof fame so they put it on a
GitHub or something like that.
When that happens, we know thatthere's actually an increase in
the chance of something usingthat and exploiting it in the
wild.
And so we update.
We've got well over I thinkwe're at 1500 different
(02:31):
variables that we're looking atfor vulnerabilities, trying to
predict when something or howlikely something is to be
exploited.
Speaker 1 (02:39):
That's insane, and we find that people are moving
more towards, especially CISOs.
They wanna match what's goingon in the real world with what's
going on in their dashboardsand in their ecosystem.
So what's the reception like?
I mean, is this somethingthat's brand new?
Is it out in the wild?
Are people using it?
Speaker 2 (03:00):
Absolutely yeah, and it's free.
You can go grab these scoresright now.
It's at firstorg slash, EPSS,and the scores are out there.
You've got an API.
You can download data on adaily basis, but the reception
so like it's been really, reallygood and we launched it.
I think we started publishingthe first scores in 2021, the
initial research was 2019.
(03:21):
We've gone through threeiterations since then, so we're
on a third version and we'rejust getting better and better
as we iterate.
And the big thing you'd thinkthat it was like we can tell
people really what to focus on.
That's not the real benefit.
The real benefit is that we canactually identify things that
you probably don't have to focuson, and that's been a huge
(03:41):
thing for people, because whatwe found through some of our
research with Kenna is that mostcompanies are fixing between 10
and 15% of the openvulnerabilities in their
environment a month.
So any given month, they'reonly able to fix 10 to 15%.
That was about the median.
Some are higher, some are lower, of course, but that's about
the average, and so what thatmeans is that you need
(04:01):
prioritization.
You need to know what youshould focus on versus what you
can safely delay and work into alonger patch cycle, and so
that's really the value thatwe're bringing.
We're telling people thingsthat they could safely more
safely delay and being a lotmore accurate than the
prioritization that they aredoing.
Speaker 1 (04:19):
Excellent.
So it's not just what shouldyou be doing, but what to not
waste your time on.
You should have.
Okay, great, what is thescoring system like?
Is it sort of a you know one to10?
Like what's a good score?
Speaker 2 (04:32):
So it's between zero and one and it is a probability
like an actual trained, you knowcalibrated probability.
And so what you can do, if youhave one machine with 10
vulnerabilities, you could saywhat is the probability that any
one of these will be exploited,and you can just, as you know,
if you understand probability,mathematics takes a little
(04:52):
learning curve there, but youcan do that combination, which
is very, very hard, as you know.
If you have like two highs anda medium and a low, how do you
combine that?
You know when you have thatordinal value, so the
probability, you can combine itand it is just between zero or
one, or we'll display it, as youknow, zero to 100%, yes, and
nothing ever gets to zero,nothing is ever at 100%, and
that's just the way the model'sworking.
(05:13):
Yeah, and we have a whole.
We've got several papers thatwe've published on the research
we put behind this and we'lltalk about the calibration.
So when we say 50% chance ofexploitation in the next 30 days
, we put that time window on it,that about 50% of those will
actually be exploited in thenext 30 days.
Speaker 1 (05:30):
Can you produce a bell curve with that data?
I guess that you guys have alot of you know there's a
distribution in the score.
Speaker 2 (05:37):
Yeah, but it's not a bell curve.
It's.
You know, bell curve issymmetric and nice and pretty.
This is really heavily weightedon the low end.
So I think like the top 15% waslike 2% probability and above.
So, like 85% of the data isbelow 2% chance of exploitation,
and that goes back to allowingpeople to say what should I not
(05:58):
focus on right away?
You know there's a whole bunchof things that you don't have to
focus on right away.
Speaker 1 (06:03):
Okay, yeah, so just don't focus on those outliers.
It's probably a really longtail and all these things.
So this is a really thepractical application is here.
Here is if you're in, you'rescrolling through the news and
you see all these buzzwords andyou're seeing all these look,
the media has been hijacked andyou know cybersecurity folks
(06:24):
don't know what's importantanymore.
You know, I mean, if you'rereally in the trenches, you know
, but this seems really usefulin being able to say, okay, well
, yeah, we heard that thisperson got breached because XYZ,
maybe they're not in myindustry, Maybe this doesn't
really matter to me, and so yoursystem would be able to say,
yeah, this thing is an outlier.
(06:44):
It gets a lot of buzz, Peopletalk about it a whole lot, but
guess what?
It doesn't really happen thatoften.
What about if it?
If it's an outlier, it onlyhappens.
You know it's very rare, but doyou have a way of capturing the
impact of that vulnerability?
Speaker 2 (07:01):
No, and that's a great question, because we very
explicitly did not look atimpact.
And so if you, if you want totake like a risk based approach,
there's two other things toconsider other than the score.
So the score is basically howlikely is someone to try and
exploit this?
The other thing is what sort ofcompetent and controls do I
have?
So if you know if thevulnerability is like 15 layers
(07:24):
deep in your network or if it'sexternally facing you know, on
the on your perimeter, those aretwo totally different scenarios
from your perspective.
The other thing is the impact.
Right, we have no idea what itis.
But if you can look at a systemand say, hey, this is really
really important stuff, we wantto be careful with the
vulnerabilities on here, versushey, this is just, you know,
basically a brochure that couldgo away, we could reboot it, we
(07:45):
could rebuild it, whatever, wedon't care.
All of those are going toimpact the overall decision.
So EPSS is just producing thatthreat, if you will.
And then you need to understandthe environment and the impact
which, of course, we have noidea from a centralized location
how everybody, how differentvulnerabilities are going to
have an impact on theenvironment.
Speaker 1 (08:05):
Excellent.
Well, I'm going to stay on topof this and I hope that you come
back and drop in another timeto tell us what's going on in
the EPSS space exploitprediction scoring man.
Well, I just learned somethingtoday.
So if this is interesting toyou, if you're listening to this
or watching watching this, ifthis is interesting to you, drop
(08:26):
us a note, leave us a comment.
And, jay, I don't know if youlike people reaching out to you,
but if somebody wants to reachout to you, how can they find
you?
Speaker 2 (08:34):
Boy I.
I don't know anymore.
You know it used to be Twitter.
I'm on LinkedIn, jay Jacobs onLinkedIn at at Santhea, and
that's probably the best way toreach out to LinkedIn.
I put occasional few posts datadriven posts out there, so all
right, jay.
Speaker 1 (08:49):
Well, thank you for being so gracious with your time
.
Thanks for dropping in andthank you for listening to this.
We still got to find a name.
Maybe we'll take somesuggestions, but right now you
know, we'll call this securitymarket watch drop ins.
So thanks again.
Thanks, jess, all right.
Bye, all right.
Bye, all right.
Popular Podcasts
United States of Kennedy
United States of Kennedy is a podcast about our cultural fascination with the Kennedy dynasty. Every week, hosts Lyra Smith and George Civeris go into one aspect of the Kennedy story.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com