June 16, 2025 • 39 mins
Think you can manage industrial systems like your IT infrastructure? Think again. In this episode of The Audit, Dino Busalachi unpacks the high-stakes complexity of OT-IT convergence—and why your trusty IT playbook flatlines on the plant floor.
Join the IT Audit Labs crew as we dive into the chaos of managing 10,000+ industrial assets across a sprawling landscape of vendors, protocols, and operational rules that laugh in the face of standardization. From Siemens to Rockwell to Honeywell, Dino draws sharp parallels to hospital systems juggling specialized third-party contractors—because in the world of OT, consistency is a luxury and adaptability is survival.
🔧 Key Topics Covered:
• Why OT environments resist IT standardization efforts
• Managing thousands of industrial assets from multiple vendors
• The hospital analogy: treating OT specialists like medical contractors
• Building effective partnerships between OT and IT teams • Real-world challenges of securing industrial control systems
#OTSecurity #ITConvergence #IndustrialCybersecurity #SCADA #PLC #CriticalInfrastructure
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:04):
All right, you are listening to the Audit presented
by IT Audit Labs.
I'm Joshua Schmidt, yourco-host and producer Today.
We're joined by Nick Mellom andEric Brown of IT Audit Labs,
and our guest today is DinoBusolacchi.
He is currently in St Louis,missouri, and he's coming to us
today to talk about OT-ITconvergence.
And, dino, thanks for joiningus.
Speaker 2 (00:25):
Thanks for inviting me.
Speaker 1 (00:28):
Absolutely.
I know this isn't your firstpodcast, but we're excited to
talk about a topic that you'refamiliar with today that OT
meets IT and a lot of otherthings that go hand in glove
with that, so thanks for joiningus.
Can you give us a littlebackground?
You have so many titles here onLinkedIn I didn't know where to
start, so I'll hand that overto you.
Speaker 2 (00:49):
Yeah, that comes with the gray hair right.
So I'm currently a directorhere at the Barry Wimmler Design
Group.
The Design Group is amanufacturing engineering
organization been around for 100years and they acquired my firm
last year.
So I've been a design group nowsince July of 2024.
(01:12):
Prior to that, I owned andoperated a OT cybersecurity
systems integration company.
I was focused just with theseOT cyber securities.
Prior to that, I spent severalyears working for Rockwell
automation and their networksecurity services group.
Prior to that, I was in theRockwell channel working with
their distributors, a lot ofsystem integrators and OEMs, and
(01:34):
I'll touch a little bit on thatthen.
Prior to that, I spent 20 yearsworking for one of the world's
largest adult beveragemanufacturers based out of st
Louis you can probably guess whothat is working building out
breweries of the future and so Ispent a lot of time around
control systems, implementingwhat I want to call IT
technologies on the plant floor,which is why, when I was at
(01:55):
Rockwell when some of these OTcybersecurity technologies
started coming to the market, Iknew there was a niche in
driving cybersecurity down onthat plant floor the myth of
those machines being air-gappedit just doesn't exist anymore.
Prior to that.
I worked for General Motorsunder Ross Perot back in the 80s
.
I worked for Monsanto andvarious others.
(02:16):
I actually started in thebanking industry in the early
80s, working in data centers andgot involved with building out
networks to banks and putting inATM machines back in the day
when the ATM machine wasactually at the bank.
So that's a condensed versionof my background.
Speaker 1 (02:32):
That's awesome.
Do you get a lifetime supply offree beer then?
Does that come along with theterritory?
Speaker 2 (02:38):
If you know the right people.
The answer is yes.
Speaker 1 (02:42):
One of the things that stuck with me is you shared
that manufacturing is one ofthe largest attack vectors and
maybe you could explain why thatis and how that has met with
your experience working in OT.
Speaker 2 (02:54):
Attacking a manufacturing facility and
getting to the point where youhave a material breach and
you're able to shut down thatplant is very costly to shut
down that plant is very costly,right.
So when I worked for that largeadult beverage manufacturer, if
we just had one packaging linenot putting beer in a can or a
bottle that was $80,000 an hourof loss, right.
And so when you think about amanufacturing environment, you
(03:18):
know their plants aren'toperating because it's very
expensive, very quickly, right.
And cybersecurity insurancedoesn't cover business
disruption?
Right, it'll cover the aspectsof doing the remediation,
mitigation work that you got todo, but they're not going to pay
for the loss that you incurredfrom being down.
Speaker 3 (03:35):
So then, dino, probably the most famous example
of that air gap breach, so tospeak, is probably Stuxnet right
, right With the sandworm.
So, in your experience, haveyou seen other attacks like that
when you were working at thoseprevious organizations, and how
(03:59):
did you combat it?
Speaker 2 (04:00):
Not that targeted.
I mean, it was a highlyspecialized, developed malware,
you know, attacking a specificSiemens set of control systems
that, whatever nation stateswere involved with it, had an
inherent knowledge of what theywere using within that nuclear
facility.
But what I will say is that alot of the manufacturing
(04:21):
environments are impacted bybreaches that come through the
IT environment.
Right, because we use a lot ofthe same technologies on the
plant floor that you have in theenterprise or in the office
space.
Right, windows, for example,that are running HMIs.
You know your human machineinterfaces, your historians,
your application servers, yourengineering programming
terminals those are all Windowsmachines.
(04:43):
And you also have a lot ofnetworking equipment out there
too.
Right, you've got all stripesand types out there inside that
manufacturing facility and theyall have vulnerabilities.
So if you're accustomed andused to what I still call Patch
Tuesday for Microsoft, right, wedon't patch very often on the
plant floor because it's verydisruptive.
(05:04):
The life cycle of controlsystems is measured in decades.
Right, it's not measured in athree to five or seven year
window.
So when control systems, youknow you're putting in a $20
million packaging line, theintent is that packaging line is
going to be out there for 20years doing what it does, right,
and so, and even then, you know, the frames of that equipment
(05:27):
may still remain the same.
But you might do a recontrolright and recontrol efforts.
That means that you'rereplacing the PLCs, you're
replacing the SCADA.
Software might be doing someupgrade on the applications,
maybe the network.
It's very, very costly, right,and those are usually capital
projects.
They're not OPEX projects,they're capital projects.
And so the dynamics aredifferent in that particular
(05:49):
space.
And if you're buying from anOEM, right, they may not stay as
current on patching for theircontrol systems, right, they
just again, because it'sdisruptive and there's costs
involved with it, you know, andand so we don't see a lot of
that.
And so then you gotta, you knowwe subscribe to, you guys are
familiar with, you know, thesans institute, right, so think
(06:11):
of um the the sans.
There's five critical controlpoints for ot cyber security
right in the it world there'slike a dozen or more, but on the
ot side we can kind of focus onfive with incident response,
defensible architecture,vulnerability management, remote
access, which is really big,and we talk about that in a
second because COVID kicked thedoor open on that and then
(06:34):
continuous monitoring right,which is not something that
groups are used to doing.
Because most engineering teams,when you start talking to them
about putting technology insidetheir control system network to
monitor it, the first thing theythink of is that you're
scanning and dumping a lot ofdata on that control system
network and it can be verydisruptive.
And so they've lived throughthat because IT even though
(06:57):
sometimes they're really tryingto do their best on due
diligence you know putting in ITcybersecurity tools and they
start scanning that network orimplementing EDR on those HMIs
you know, like a CrowdStrike orSentinel-1, and all of a sudden
you're inducing delay aroundthose machine centers that
require very specific you knowtime sinks and communications
(07:20):
for a couple of reasons, safetybeing one right and depending on
the nature of whatever you'redoing, you know if you're not
watching that stuff it can bevery impactful and sometimes IT
doesn't even realize they'redoing it right and the plant
floor folks don't necessarilyhave the technology as a place
to see this stuff.
We have that technology now,right when you think of the
(07:41):
Armises and the Clarities, andthen the Zomis and Cisco Cyber
Vision and Dragos, if you'refamiliar with those types of
intrusion detection systems thatwere specifically built and
geared for the OT space, right,and they're passive, right.
They sit there and listen tothe network.
They do do packet inspection tobuild out that baseline of
(08:02):
those assets, to build a profileon that PLC, to tell you the
make and the model and thefirmware on it, and based on the
firmware version, now I knowthe vulnerabilities that are
associated with it, just likeyou know the operating system
level on a Windows machine I canstart building that CVE list,
right, and then from there I canstart building a strategy on
how I can improve mycybersecurity posture around
(08:23):
those control systems.
Because, back to my earlierpoint, the stuff is connected,
right, it's just, whether it'sfor support purposes or feeding
MES systems or manufacturingexecution systems or your ERPs
or other enterprise applications, they're pulling data up and
off that plant floor and thenyou have people showing up to
(08:44):
the plant floor.
So, and then you have peopleshowing up to the plant.
You know, if I go and if I showup to a plant, eight out of 10
of them will let us go in therewith our own laptops or
computers and plug into theirenvironment, right, and they
have no idea what's on thesemachines, right, and then a lot
of OEMs.
When you buy a machine fromthem, so think about buying a
new vehicle, right?
And it's got, you know, onstaron, it's got satellite, it's got
(09:06):
cellular, it's got Bluetooth,it's got Wi-Fi.
Well, these machines aren't anydifferent on the plant floor,
and we'll find cellular cradlepoint modems in there.
We'll find VPN concentratorsfrom SACOMI or EWON or
Toshiboxes, right, becausesomebody is connecting in to
provide support of thatequipment down on the plant
floor when it's not runningcorrectly.
(09:27):
Or if they want to do some typeof enhancement, maybe they're
moving to a different product,so they're doing some
programming, and so there'salways connectivity in those
environments.
They're very dynamic, they arenot static at all.
Speaker 1 (09:39):
That was going to be one of my questions and you've
pretty much illustrated it.
But for the people that aren't,you know, familiar with
manufacturing environments or ormaybe just you know are tuning
in for the tech aspects of thisshow, can you give us kind of
this like a general sense ofwhat these manufacturing floors
look like?
Speaker 2 (09:55):
yeah, and it's significant right, when you
think about the number of youknow you've probably heard the
term iot or iiot for theindustrial Internet of Things.
We obviously have the term OT.
But yeah, when you think aboutmotors and drives and robots and
pumps and sensors and fillersand conveyors and just the
(10:19):
presses and all you know dryersand fans, that can go on and on
and on and all of the differenttypes of connected physical
systems, that's what we kind ofcall it, you know, I think of
connected physical systems andin the IT world you have data
driving data outcomes.
Right.
In the OT world you've got datadriving physical outcomes.
And so you know the twopriorities.
(10:42):
In any manufacturing facility,safety is always job one.
Right, there's no doubt aboutthat.
You got to operate safe andit's not negotiable, right.
So you have to be a safeenvironment.
You have to demonstrate thatyou're doing it.
Number two is what I spoke aboutearlier.
It's unplanned, unscheduleddowntime, because it's expensive
and it's costly to demonstratewhat the target is for this
Right, because it's usually notin their purview, meaning that
(11:04):
IT doesn't own those assets.
You know.
Nowhere else in the business,especially on the enterprise
side, it has a responsibility ofthe technology that the company
is using to run their businessright.
But on the OT side it's not somuch.
You know the decision to put ina new packaging line and what's
going to consist on the controlsystems around that packaging
(11:26):
line, and what computers are wegoing to choose and what
networking equipment are wegoing to choose, and operating
systems and firmware andapplications.
It's an entirely differentgroup of people, right?
So I worked at Rockwell.
Even when I worked forAnheuser-Busch, it wasn't
sitting in the room when we weregetting ready to determine, you
know, to build a new brew housefor $500 million, should they
(11:49):
have been sitting in the room.
Speaker 3 (11:51):
Ideally, Eric.
Speaker 2 (11:52):
yes, we want that.
Yes, here's the way I kind ofdescribe it, Eric.
I want them on the field, Iwant them in the huddle.
They just can't be thequarterback, if that makes sense
right, but yeah, they play atremendous role right Governance
policy, technology resources,standards.
They bring a lot to the table,but they're not always invited
(12:17):
in through that CapEx project,on whatever it is that company's
doing an expansion, brownfield,greenfield, lifecycle
replacement, whatever there-control, whatever tech
project they're taking on wedon't typically see it sitting
in the room participating, youknow, in something that might
start two years prior to thatmachine being built and running.
And I'll give you another goodexample erica, we were just in
(12:39):
atlanta a couple weeks ago andthe client, the it, the
networking team, came on siteand met us there.
They were 20 minutes from thisplant and out of the three it
professionals out of thiscompany, only one of them had
been at the plant in the lastfive years.
The other two had never beenthere.
So you know it's very difficultif, from an it perspective, if
(13:01):
you're not engaged in what'sgoing on in that plant
operationally, from a processperspective, what technologies
are in there, it's verydifficult for you to start
putting a cybersecurity strategyaround that.
You've got to think globallybut you've got to act locally.
And if you've got 50 plants inyour fleet, no two are the same
(13:26):
fleet, no two are the same, andso you have to have some
inherent knowledge of what'sgoing on inside that facility
and what technologies do theyhave in there and how that stuff
is put together.
Speaker 4 (13:31):
You know we're seeing this too that IT is not invited
to the party.
I guess in your opinion, youknow, especially in your
industry.
Why do you think that is?
Why are they being left out ofthese meetings that are
happening, even if they'rehappening two years beforehand?
Why are they not being invited?
In your opinion?
Speaker 2 (13:49):
Yeah, good question.
You know, one of the thingsthat we look at when we start
these engagements is we look atthe organization.
One of the things that we lookat when we start these
engagements is we look at theorganization.
And sometimes the first timethat IT and OT have the same
manager leader is usually at thepresident or the CEO of the
(14:11):
company.
Think about that for a secondright and that's not necessarily
right either right.
And then we have to think aboutthe size and the scope of how
many people you need.
So if you're a CIO and let'ssay you got, you know, 200
people working for you, right,and you've got 100 plants, well,
how big is that that OTpractice need to be from a
technology standpoint?
And who's going to own thatRight?
Is it going to be the CIO?
(14:31):
Is the CIO going to become thenew plant manager or general
manager of the plant?
Because they operate as fiefdoms, these plants do their own
(14:52):
little kingdom.
You know meaning they decidewho they're going to work with
from an integrator perspectiveor whose machines they're going
to buy.
You know what their partnercommunity is.
On the OT ecosystem as I callit right, and the OT side is not
any better right, as I call itright, and the OT side is not
any better right.
The OT ecosystem, that supplychain on that side of the house,
needs to get better at thisright.
Because if you go in there andstart talking to them about
cybersecurity, who do you thinkthey're going to point their
fingers at and say isresponsible for cybersecurity?
That's IT.
They're going to come.
That's IT's problem, right.
And then we show these machinecenters and IT sitting in the
(15:15):
room.
They're going yeah, that's notus, I don't even know what that
stuff is Right.
And so there's a gap.
Speaker 3 (15:21):
We see it a lot where we'll go in to help an
organization maybe restructurehow they're doing information
security because they're at aninflection point, Right,
Whatever that is.
And a lot of theseorganizations have multiple
disciplines of OT, whether it'saround transportation or refuse
(15:42):
or control systems that aredoing something related to
critical infrastructure.
And I don't think I've goneinto an organization yet that
has had a what I would call evena standard plan for how they're
(16:02):
going to operate and secure thetechnology that the plant is
responsible for.
Like, at the end of the day,don't really care what's in the
plant, what the plant does, butit's about how are those systems
that are operating thatequipment being secured.
And, like you said, there'scommunication happening outside
(16:26):
of that.
Quote unquote, closed network.
I don't think I've ever seentruly closed network yet either,
and I've been doing this awhile.
But yet there's, you know you,some of them are museum pieces.
You go in there and there'swindows XP, there's windows
seven, there's, there's all ofthese um, none their their
desktop applications that, forwhatever reason, these OT
(16:50):
organizations have decided torun their infrastructure on, See
very little Unix, Linux flavors.
It's a lot of Windows flavorsand a lot of legacy Windows
flavors and you mentioned toolslike Dragos or specific to OT
(17:14):
that can help look passively forthings going on on that network
, which you know in some caseswhen the horsepower around the
technology was smaller or therewas less horsepower available to
do non-ILC processing.
Then nowadays, though there'stechnology that could sit
(17:38):
quietly on the network, couldlook, decrypt, whatever, for
things that might be calling outor attempting to call out or
attempting to move laterally,and unless that, that, that OT
part of the organization isbringing apart the discipline,
which it's really hard to have,the discipline to know how to
(18:00):
run a tier one cybersecurityshop and run a plant that is
responsible for operating thatline of business.
It's really hard to do both.
So you've got to kind ofoutsource the IT side to either
internal IT or a third party tobring about that best practice
and facilitate it.
(18:21):
And I don't think it'sacceptable to say, yeah, it's
okay to run on 20-year-oldtechnology just because the
plant itself is running on oldtechnology, because it can last
20, 30 years.
Those manufacturers need to beheld accountable to bringing the
(18:42):
technology along.
It's not a secret.
We all know that Windowsexpires over time.
We've got to stay on top of it.
So bringing governance in earlyinto that two-year process at
the contract level to hold thevendor accountable for making
sure that those updates there isat least a vehicle to do the
(19:03):
updates.
That's kind of where I am nowin my headspace, versus saying,
oh, it's an OT environment,let's just leave it, let it run
and hope nothing happens.
Speaker 2 (19:13):
Yeah Well, and the way what you're describing is
very expensive, right, sure Toreplace, to move from Windows 7
or XP and do a recontrol ormaybe move a piece of equipment.
You might be talking abouthundreds of thousands or
millions of dollars in order todo that right.
And so there has to be theclient has to recognize that.
Speaker 1 (19:34):
Where are you right?
Speaker 2 (19:35):
Because right now they don't even have a really
good idea where they are.
They don't have a good baseline.
So you got to get that figuredout first and there are.
There are technologies thathelp through that.
While they're building theircapital plan eric to your point
to be able to go after thatstuff um, like virtual patching
right or defensiblearchitectures around that
machine center so that you canat least improve your
(19:58):
cybersecurity posture, harden itwhile you're putting together
that multi-million dollar CapExplan to go after and replace
that.
The other thing that I'll tellyou is that I call it the OEM
blockade.
They won't let you touch theirstuff right.
If you want their support andyou want the warranty and
maintenance that comes with thatmachine center, they don't want
(20:19):
you touching it right, andthat'll because otherwise you
void the warranty.
So think about if you bought acar and you know you're getting
the warranty and support from GMand then you go in there and
you modify the exhaust systemand you decide to swap out the
transmission and you try to youknow, maybe do something with
fuel injection and you modify itand then you take it back to GM
to say fix it, they're going togo no right.
(20:40):
And these OEMs have thatmindset, because they don't even
understand that even if we justwant to put something in there
as simple as a sensor right tocollect the metadata within that
control system, they would say,if you touch that, you know.
If you add anything to thatswitch to do that, or change the
configuration of that switch tocreate a span port to pull that
(21:00):
stuff out of there, you voidthe warranty and a lot of the
clients are like they don't wantto buck the system with that
OEM.
And so now you've got to getahead of the game, right.
Now you've got to be proactiveand so if you're going to buy a
new piece of equipment, then youneed this and you're let's say
you're a drago shop or an armistshop or clarity shop, you
(21:21):
should be planning ahead to putin the right technology in that
machine center, right.
I always like using theautomobiles.
You don't put the safety, theseat belts, on after the fact,
or the anti-lock it breaks onafter the fact or the sensors of
the backup, backup cameras, theairbags.
You do it when you're buildingthe car, right.
And it's the same concept whichis one of the other big
disconnects that we see is thoseOEMs and a lot of these system
(21:44):
integrators, which the firm Iwork for is both of those trying
to change that narrative in themachines and the integrations
that you're doing andimplementing that up front,
right.
It's much more cost effectiveto do it then as it is to do it
on the retro, on the backside.
So two things it can beexpensive to try to do a
recontrol, replace equipment,and then you've got to deal with
the OEM.
(22:05):
That can be a little pig-headedsometimes, right, and you've
got to try to break through that.
Speaker 4 (22:08):
There's a lot of moving parts here.
I'm thinking back to one of myfirst IT jobs, relatively not
that long ago 10, 12 years agoand it was at a large
manufacturing company in theTwin Cities and, being a junior
IT, I was on a help desk thereand I'd run out and help all
these different lines get backup that were going down or do
(22:30):
any sort of tasks.
But there we had so much legacysoftware and I'll never forget
one of the more senior guys.
I was out running tasks withhim and we had one machine over
in this corner.
I don't remember what it wasdoing, I think it was printing
labels or something and he saidwe don't touch that thing
because if it turns off it mightnot ever come back on.
It had been there for maybe 30years out of support, obviously,
(22:52):
legacy software.
The vendor doesn't even existanymore.
But it was always so confusingto me why don't we just change
it out?
And I'm hearing from you now.
You know just the expense to it, but you know, is it a lack of
auditing?
Are we not asking thesequestions?
Is that?
Is that the root problem?
Are we not, as IT professionals, are they?
Are we not digging into thesand, letting them know these
(23:14):
issues.
I guess my question is are wegoing wrong somewhere or is it
just because we're not invitedto the table is my earlier
question.
But it's happening so oftenthat we're inviting the wolf
into the hen house, not beingcompliant or having these
discussions that we're havingnow, which it seems like we're
all on the same page, but itseems from a simple discussion
(23:37):
earlier on.
Like you said, putting theseatbelts in early, we could
solve a lot of these problemsdown the road.
Speaker 2 (23:45):
It's the timing of it , in my view, right.
How do you get that IT groupengaged and involved in that OT
environment right and becomepart of that ecosystem?
Speaker 3 (23:57):
What makes the most sense.
Speaker 2 (23:59):
So think of shadow IT right.
Eric, I know, you know what thatis, you've been probably we go
back a number of years.
Right, and that was one of thethings we were trying to chase
out was shadow IT.
But when it comes to that plantfloor, it's the one area that
they've just never been able totake on, and a lot of it is
because they don't own the asset, right.
(24:19):
Who owns that asset?
Like I said earlier, all of theenterprise stuff, there's no
question, it's it.
They own that right.
They make those decisions,whether using third parties or
it's an internal team or amixture of both, but at the end
of the day, they make thosedecisions.
They make that, those buyingdecisions.
They decide the resources andthe responsibility to take care
(24:39):
of it, just like that OEM doeson the plant floor.
But when it comes to that PLC,it's a programmable logic
controller, right, and so that'sthe instrument.
Speaker 3 (24:48):
And it's usually running on a.
Speaker 2 (24:50):
Linux-based operating system of some sort.
Right, it's not running on aWindows OS, it's usually a Linux
type of OS but it's the onethat's receiving signals and
making calls to make that motorgo faster or to turn the flames
up on that burner or to speed upthat filler or whatever that
might be.
That's what that controller isdoing, right, and IT just never
(25:12):
put their hands on it, even tothe extent that engineering
teams would design their controlsystem architecture to put
multiple network cards into thatPLC to make it a network
segmentation device.
Because one side if you'refamiliar with the Purdue model I
don't know if you guys arefamiliar with the Purdue model,
but think of a PLC with anEthernet card on it, because we
want information sent to ourhistorian, we need remote access
(25:35):
to it for programming and Iwant to send data up to my MES,
and so there's a connection towhat I want to call the
enterprise side of the network,because even though it's within
the plant, it's the side of thenetwork that IT usually manages.
But then there's other cards inthere for control that are on
that PLC and that's the spacethat you never ever find IT in,
(25:58):
because that's actual control.
That's the one that's sendingthe signals, you know, because
we're back to process integrityand we're back to safety, and if
you don't understand what thatthing is doing, it's very hard
for you to take on a role oftrying to be I call it, the
round peg square hole andapplying a security measures in
there that could be disruptiveto that environment.
(26:18):
How are you going to get agroup with the boots on the
ground knowledge of what's goingin there to implement?
All right, I got Siemens overhere, I got Rockwell over there,
I got Honeywell, I got GE, Igot Emerson, I got Yasago, I got
Mitsubishi.
They're not very standardized.
Right, it's done a really goodjob of standardizing everything
right Compute platform, networkplatform, databases.
(26:41):
you know ERP systems.
They standardize you get intothat plant and it's seven ways
to Sunday in there.
You know I'll find eightdifferent access methods into
the plant for remote access andI'll find several different
automation technology vendors inthat one facility.
Speaker 1 (26:59):
One thing you said that stood out to me, dino, is
that even just within a company,the plants are vastly different
from location to location aswell.
Not just the equipment that'srunning in them and how they get
things done and where they'repositioned, and all that.
Speaker 2 (27:13):
but yeah, the whole layout, the design, and
culturally and behaviorally Imean depending on what part of
the world they're in or whatpart of even in this country
they're in you have differentmindsets on how they operate
that are tough to get through.
And then you have a lot ofmergers and acquisitions, a lot
of these plants you know thathave been bought and sold
(27:33):
multiple times over the last 30or 40 years and you just don't
have a lot of good continuityand the way we pick up on that
really quickly is lack ofdocumentation.
They don't know what they got.
There's no network drawings,there's no good network drawings
.
Whatever documentation theyhave, you know 20%, 30%, 40% of
it's wrong, it hasn't beenmarked up, can't find it, don't
have it.
(27:53):
Right, it hasn't been marked up, can't find it, don't have it.
So we do a lot of investigatingand hunting.
When we're in there trying todetermine, we go in there and
drop one of these tools in tostart collecting a baseline, and
then we walk the plant and weopen up those panel doors to see
what's in there and we documentthat.
(28:16):
So I want to be able toreconcile what did I visually
see on the plant floor and it'snot in my baseline on my tool.
So I can figure out how do Iget that system over here on
this box Right, and make thosedeterminations of whether I got
to put a sensor out there or Igot to change the network to
some extent.
Do they even have cabling toget me out there?
Right, so, and so it's thedynamic of it.
Aspect of it is what can bechallenging Right, so, and so
it's the dynamic of it.
Aspect of it is what can bechallenging Right, and and not
(28:40):
everybody's prepared.
You know they're behind.
They're 20 years behind IT oncybersecurity 20 years.
Speaker 4 (28:46):
Dean, are you familiar with CMMC, the
cybersecurity maturity modelfrom Siemens, remodel from
Siemens?
This is from the Department ofDefense that you would have to
follow these guidelines to beable to be a contractor to work,
build a piece of a bomb, builda piece of an airplane or
something like that, and I thinkwe're all talking about the
(29:08):
same thing here, but with a setof standards for many different
organizations.
Now this would be very hard toimplement and govern.
But I think CMMC was one of thefirst entries into the space to
try to standardize this space.
But we're not seeing it fornon-federal contractors,
somebody like Anheuser-Busch.
(29:29):
We're not seeing it for thatspace at all.
Would implementing somethinglike that solve some of these?
Speaker 2 (29:34):
problems.
But that's one of the things welook for is are they following
a security framework, right?
You've got NIST.
It depends on the vertical one,right?
Some are highly, more highlyregulated than others, whether
using the IEC 62443 or MITRE orthe ISO 27002 or one.
And then you got NIST, whichfor the BOK for you know, the
(29:55):
Boko we do is food and chemical,life sciences, heavy industries
, semiconductor.
We do a lot of data center workaround power systems, and so
from a regulatory standpoint,not everybody has to even follow
regulation, right.
I mean, you think about ifyou're a publicly traded company
(30:16):
today and you get a materialbreach.
You've got like 48 hours, 72hours.
You got to tell your investorsthat you got pop, right, and
even then the government maygive you some leeway depending
on who you are like, if you'reBoeing, right, compared to you
know, some soda manufacturer,right, and the government may
(30:37):
even follow up on that.
But my point is is that we usethe security frameworks you know
to go after the identification,detection, you know, response,
recover, you know even anincident response plan.
Speaker 1 (30:49):
What is your?
Speaker 2 (30:50):
IR plan at this plant , right, and you'd be surprised
how many companies don't havethat.
They don't have an IR plan.
So then you get back to the ITconversations.
Well, how do you get IT todrive that?
Because IT's probably got an IRplan, you know, and they
probably practice it, you know.
I mean, I can remember the, youknow the IR plans.
You know, back in the day whereyou'd run off to some
third-party site to spin up yourbusiness somewhere else right,
(31:13):
but in a plant.
Spin up your business somewhereelse, right, but in a plant,
their only option is to eithermove their volumes to other
facilities to make up for theloss of this one if it's not up
and running right, but theydon't have one.
And I will tell you that if Igo into 10 plants, two of them
are going to have malware inthem.
Every one of them is going tohave a rogue asset or a series
of rogue assets in them.
(31:34):
You know even the guy that'sbeen there for 20 years and
we're telling him hey, there's aWAP sitting over here on your
packaging line.
He goes no, there's not, it'slike yes, there is we can see it
.
It's on the network.
Let's go hunt it down and findit.
And they have no idea how itgot there.
We found stuff in ceiling tilesand plants that somebody put in
(31:58):
there.
Speaker 4 (31:58):
There's all kinds of goofy things and they don't even
know that they're here.
Speaker 2 (31:59):
It was probably the server guy wearing umbros, just
to give you an idea betweenGartner and Cisco and Rockwell
and Siemens and these OT ideas.
Guys, 60% of manufacturers outthere are just.
They're either unaware or justbeginning to understand that OT
is a thing.
That's 60%.
You got 30% that have actuallystarted doing something.
(32:20):
Maybe they did a POC, Maybethey've had some demos.
Somebody came in and startedtalking about it.
Maybe they've actually got acouple of people who are in the
organization, have been taskedwith cybersecurity for the plant
floors.
And then you have 10% thatactually have a strategy and a
plan or starting to implement itand going down the line.
Speaker 1 (32:39):
That's where we are today.
I wanted to circle back reallyquick and kind of get Eric's
take on just integrating the ITin any kind of organization but
more specifically biggermanufacturing or larger entities
like Dino's talking aboutmostly.
What's your take on that?
Is it mostly just kind of justa knowledge base or does a
cultural thing of why there's adisconnect there?
(33:01):
And I know you spend a lot oftime on boards and things like
that.
You know showing people whatthe risk actually is, because no
one really sees it until ithappens.
So I was kind of curious to seewhat you thought about that.
Speaker 3 (33:15):
I've seen a couple of cases where more than a couple
and Dino alluded to this wherethe IT organization or the
leadership of the ITorganization has never actually
been into the plant.
Maybe they've toured the plant,been on the grounds but never
(33:35):
really looked at the technologyin the plant.
And, as as Dino was telling thestory about um rogue devices or
potentially malware in thesystems, I've walked these, um,
the, these plants or theseoperations and the.
We ought to do a pictureslideshow on this because
(33:56):
there's just some crazy pictureswhere it's in a hospital and
the amount of cabling that wasjust across the face of the rack
probably weighed more than me.
It was just this huge rat'snest and there was dead switches
that had just been cabledaround that were in that
(34:18):
environment.
There was dead rodents thatwere completely squished
underneath a piece of technologythat you know hadn't been moved
in 10 years.
It was way up in the rafters inthis production facility that is
responsible for making giftcards and we were doing this
(34:43):
network assessment and we werekind of tracing back where we
were seeing devices on thenetwork but we couldn't find
them.
And way up in the rafters on topof an I-beam was like some form
of a Linksys device that wascabled into the network.
There's no documentation, ofcourse, on it.
Nobody knew how it got there orwhat it was for, or if they
(35:06):
were trying to add a new machinethat they had long since taken
out and forgot about.
But the amount of undocumentedtechnology that you know
sometimes people that have beenaround the plant for years, they
may remember, or might be sometribal knowledge, but I think
part of it, josh, is if you'reresponsible for an environment
(35:28):
or you're providing some form ofresponsibility on the
technology side.
You got to know what's in yourenvironment, right Back to that
CIS one and two.
Know what the hardware is inthe environment, know what the
software is in the environment,because if you don't know what's
out there, you really can'teven begin to try to protect it.
Speaker 2 (35:51):
That's the first candidate in any cybersecurity
framework.
Is that right there?
Your asset inventory?
Yeah, the question is whetheryou want to get it in a
continuous fashion.
Like IT generally does iscollecting information
consistently, constantly, versusa snapshot in time.
But that's the number one tenetin any security framework is
asset inventory.
Even to the point wherecritical infrastructure
(36:13):
organizations, where the TSA andDepartment of Homeland Security
are coming to them, and evennow insurance Now the insurance
companies are coming in becausethey're getting smarter going
you need an accurate assetinventory and I want to know the
vulnerabilities and riskassociated with those things.
That's the first thing, rightBefore you get into anything
else.
Let's just do that Right.
Speaker 3 (36:32):
Do you know?
The first year that I had tofill out a cyber insurance form
it must have been like 2013 or2014 working with a customer, we
sat down over a lunch with theinsurance form and they're like
yep, you know, we have this formthat we need you to fill out.
It was half a page.
I've done a few this year andnone were less than like 20
(36:58):
pages of details around multipletabs.
It's all online now.
Right Of all of these thingsthat the insurance company wants
to know in order to rate you,and it seems every year there's
more and more questions goinginto it.
Speaker 4 (37:14):
Understand what's your network, but I think after
we do that, I thinkorganizations fall short of
documenting everything theydon't document.
They rely so heavily on tribalknowledge.
It's the operator that's beenin there for 20 years and then
they get a replacement, you getsomebody junior and there's that
gap of 10, 20 years.
But if you spend a little time,you know, bring a third party
(37:35):
in, go through these processes,build your policies and
procedures, document everythingfrom all these controls.
I think the maturity of anorganization would go through
the roof instantly if youstarted to document these
processes.
Speaker 3 (37:49):
Dino.
So how do you get them tomaintain their environment?
So you come in, you document it, you come back with a great
study.
How do you keep it so that youdon't show up five years later
and there hasn't been anyupdates to all that work that
you've done?
Speaker 2 (38:06):
So what we do is we don't sell technology and just
leave.
So I back up the truck, push itoff the dock and give it to
them, or just stand it up andget it running and leaving.
We pursue managed servicesbecause, you know, managed
service is big in the IT world,right, and most 80 percent or so
of the manufacturers todaydon't have a practice.
(38:27):
They don't have an OT practiceand they have an IT team, but
the IT team is is limited in thenumber of resources that they
have.
They've already got full-timejobs taking care of a whole
bunch of other stuff, much lessdumping this net new data stream
that looks like a fire hosepointed at their head.
Right, it's almost like theyget alert fatigue with the stuff
that comes flying out of there.
(38:48):
So our goal is to get a managedservices piece in place for
them, at least for the firstyear, eric, until they decide
that they want to build theirown practice.
Um or uh, some clients, justbecause they're in that managed
service mindset, um will flatout say, hey, I just want to
hire you guys for three years, Idon't even care what tool you
(39:09):
use.
You know here's, here's myrequirements, and if you can,
and then we just come in and runit and manage it for three
years for them, yeah, and wecatch new assets coming, because
these tools will catch newassets.
They'll catch new applications,they're going to catch new
protocols, they're going tocatch changes in the control
systems, new networking devices,it.
So we're constantly watchingand building to your point of
(39:31):
five years.
You, if I see a new asset comeout, well then that's going to
be a trouble ticket.
That's going to be created andsomebody's going to have to go
resolve.
It's going to get assigned andsomebody's got to go resolve
that.
Speaker 1 (39:42):
Bringing it back to the first question I asked at
the beginning of the podcastabout attack vectors, I think
we've thoroughly mapped thelandscape of how things are a
bit disjointed.
Maybe, Dino, you could speak towhy that is on just a practical
level, maybe coming from thehacker or malicious actor side
(40:03):
of things.
Why is this such a prime targetin manufacturing?
Speaker 2 (40:07):
Because they'll pay.
That manufacturer will write acheck and get it paid to get
back up and running.
Those that don't will spendmonths and months and lose
hundreds of millions of dollarstrying to unwind whatever got in
there.
Those that don't have theweather, all skill, resources to
deal with it will just write acheck.
Speaker 1 (40:25):
And that goes back to your earlier point about the
amount of money being lost perhour on the manufacturing floor.
And are you seeing just a lot?
Is that just a lot of malware?
Is there the ransomware andthings of that nature, or what's
the most common?
Speaker 2 (40:40):
Yeah, malware that comes in, ransomware that makes
its way in through some emailsystem that gets its hands on
those HMIs out there on theplant floor.
Because the HMI, that is ahuman machine interface and if
you can't gain access to thatthen you're blind to whatever
the machine's doing.
You can't control it, you haveto shut it down.
Speaker 1 (41:00):
And who's typically catching those things.
Speaker 2 (41:03):
Well, if you've got a clarity in the environment or
an armistice in the environment,it's going to catch it on that
side of the fence, or IT mayhave caught it on their side.
So think of Colonial Pipeline,right, caught it on their side.
So think of Colonial Pipeline,right?
You guys are probably familiarwith Colonial Pipeline.
You know, in their particularcase, you know when it came into
their environment, the reasonwhy they shut down on their side
, on the moving of the fuel side, on the plant side, because
(41:25):
they had no visibility, theydidn't know whether they were
safe or not, and because they'removing, you know petrol,
gasoline and diesel and allkinds of stuff, over 5,500 miles
of pipe.
They shut down on precautionbecause they didn't know the
extent of the breach.
They had no idea.
If you look at Clorox, theydecided to fight it themselves
(41:48):
and they lost several hundredsof millions of dollars trying to
eradicate it themselves.
I've got clients that have beenfighting stuff on their plant
floor for almost coming up ontwo years, right, because
they're trying to get itthemselves and try to eradicate
it.
And then you have those thatwill write the check.
But getting back to what Ericsaid at the very beginning, you
know in regards to somethinglike Stuxnet.
(42:08):
It was a very targeted controlsystem.
There's been a couple others.
There's one at Saudi Aramco hadone.
It was Red Typhoon or somethinglike that.
It was targeting controlsystems.
We don't see a lot of that asmuch as we see the malware
that's hitting the Windowssystem.
It bleeds over into the plantfloor where you have Windows
(42:31):
tools, right Windows systems.
But it doesn't mean that youmean that they're not out there.
They are, it's just a lot ofthem is not even advertised.
You've got clients out therethat just keep it quiet and find
it themselves.
Speaker 1 (42:44):
So we've touched on it briefly and, as we're looking
to wrap up this awesomeconversation, I'm just wondering
, as we look to the future, whatare some quick ideas or bullet
point thoughts that how we canimprove upon this in the future,
going forward both from the OTand the IT side?
Speaker 2 (43:01):
Well, obviously you want it to be collaborative,
right?
You want IT and OT to bealigned and collaborative and
recognize each other'sweaknesses in regards to dealing
with this problem.
You're probably going to need athird party to come in and help
you with some of that, andyou've got to develop.
You've got to bring in asecurity framework.
You need to pick one, whateverit is, if it's specific to your
(43:24):
vertical, if you're in criticalinfrastructure and IEC 62443 and
NERC separate, then adhere tothat.
A lot of regulatory stuff ismaking you do that anyway, but
you need to get a framework inplace and you need to get the
right tools and you got to bringthe right people to the table,
and you need to have people thathave experience doing this.
It's surprising to me how manypeople just try to fight through
(43:46):
it themselves, and I sawGartner came out with a report
at the end of this year thatsaid by 2027, was it?
A third or 25% of thecybersecurity professionals out
there are just going to quit andgo do something else.
My playbook would be my choice.
How come Did it say why?
Because of the stress, you know, and it was just too difficult.
(44:08):
You know, if you're a CISO.
The average tenure of a CISO islike less than two years,
something like that.
Speaker 4 (44:15):
You know that we better get our rakes.
We've got a lot of work to dowhen that comes up.
Speaker 1 (44:19):
I thought you were going to say they were going to
work on their golf game.
Speaker 3 (44:25):
Eric, how about you?
Yeah, yeah, sorry.
I think it's the organization'sunderstanding that they can't
do it themselves.
Dino, you alluded to a coupleof customers that you'd worked
with trying to do the cleanup ontheir own.
A lot of the customers thatwe've gone into recently kind of
(44:50):
had this.
I don't want to call it ananti-contractor mindset, but
it's almost like the us and themof like the internal full-time
employee staff versus thecontractors, who the contractors
are?
Essentially, they're just paidout of a different bucket.
(45:11):
So why do you care how they'repaid if they're there to help
you?
And I apply it to a hospitalmodel You're sick, you go to the
hospital.
The ER doctor is not anemployee of that hospital.
(45:31):
The ambulance that brought youthere is not an employee of that
hospital.
The anesthesiologist that ismaking sure you're comfortable
while you're under surgery, ifyou have to go through that,
it's not an employee of thathospital.
That cardiologist that's takingcare of you is not an employee
of that hospital.
(45:51):
So I look at organizations thathave been running.
You know, fat, dumb and happy.
They have an inflection point.
They need to bring in someoutside assistance to help them
get better, to help themoperationalize and continue to
run their business.
But really looking at thosethird-party contractors as
people that are reallyspecialized in an area, that can
(46:14):
come in and are honestly goingto be able to help you faster
than you could go out and try tostaff it on your own, you're
going to have to rely on thatexternal assistance and you're
going to have to pay for it.
So that's where I think thereis no free lunch in this whole
cybersecurity and even IT gamethat we're in.
(46:36):
Right, you just understand whatyou need to run your business
and if you don't have itinternally, look externally.
Speaker 2 (46:44):
Yeah, and the plant floor is not very much different
than what you just described,eric, because a lot of them
don't have their own engineeringteams.
They don't build those machines, right?
If you want a packaging line,they don't engineer and build
that packaging line.
They go out there and get thethree or four or five different
groups together to build thatpackaging line and the
engineering teams to do that,right.
It's the same concept, you know, just different.
But yeah, that's why you end upwith a lot of different
(47:05):
components on the plant floor ofSiemens and Rockwell and
Honeywell, and you know,hirschman switches versus
Ciscoisco switches versus, youknow, phoenix contacts or red
lions or whatever you know,because of the different groups
that are building that equipmentand just bringing it into your
plant.
So the very similar analogythat he gave, that's the plant
floor, um, and to your and it'svery standardized, right.
(47:28):
They think that they canstandardize everything and
they're just accustomed toworking from that premise and
you're just not going to getthat on the plant floor.
Sorry, it's just, it's not that, not that way.
Especially when you got 10,000assets out there right on and
just one plant, you got 10,000assets managed by gosh, who
knows who, right?
Speaker 1 (47:49):
it's really cool talking with you, dino.
Uh, it feels like you have areally good beat on what's going
on out there in themanufacturing world and and it's
really interesting for me, evenjust being in a completely
different industry in theentertainment and audio video it
makes a lot of sense and itjust impresses me on how these
things even get accomplished.
(48:10):
There's so many moving partsand there's so many intricacies
to all of it.
So hats off to you for being aleader in that space, and I'd
like to pass it around for anyfinal thoughts today before we
wrap up.
I think we could stay on for therest of the afternoon, but I
was going to say, like we didn'teven get into like incident
response and things like.
Speaker 4 (48:30):
Yeah, like emerging threats yeah.
Speaker 1 (48:38):
Maybe we'll have to have you back down the road,
dino, to talk about, uh, whatcould drill down on something
like incident response,particularly how how you handle
that.
But, um, at the very least,eric your question that you
always ask at the end of thepodcast.
Do you want to?
You want to do it?
Speaker 3 (48:47):
what's my question?
Speaker 1 (48:48):
okay.
Are you planning on going toany security conferences?
Do you know?
Do you go to?
Going to?
Speaker 3 (48:53):
any security conferences?
Dino, do you go to anyconferences?
Security conferences I do.
Speaker 2 (48:59):
I'm actually sitting on the steering committee for
Manusec that's coming up here inMay in Vegas.
I also go to RockwellAutomation Fair.
It's a big one that we attendS4.
I've been going to S4 in thelast few years down there.
I don't know if you're familiarwith Dale Peterson that runs
(49:20):
S4X out of Florida once a year.
It's a good one.
So from a cyber and it'sspecific on OT cybersecurity,
it's called S4.
He's been running it for over20 years.
Nice, that's a good one to lookup.
I haven't really hit RSA orBlack Hat.
To be honest with you, I tendto lean towards the ones where
(49:40):
I'm focused on the manufacturingorganizations.
Speaker 1 (49:45):
Do you do a booth at your conferences?
You go to Dino or do you justgo and do the cocktail hour?
I speak a lot of them.
Speaker 2 (49:51):
Obviously, we have clients, the cocktail hour, I
speak a lot of them.
So we take obviously we haveclients that are there partners
We've got a lot of partners, soyeah, Because I like your
musical.
I use the instrument a lot.
It's like I can give you aguitar.
The question is whether youknow how to play it or not.
Speaker 3 (50:08):
It comes with these tools Like you.
Speaker 2 (50:10):
Take a clarity, for example.
Right, there's those that knowhow to play that instrument and
then those uh, don't you know,and we've got.
People have been working withthis stuff for almost a decade,
so it's going to be hard to beatthem.
If you're just starting todaywith the tool you might as well
leverage, somebody's beenworking with it for for several
years you know that's a greatanalogy, yeah well.
Speaker 1 (50:32):
Thanks so much for your time today, Dino.
It's been a great conversation.
It's been very informative.
Yeah, Thanks for being ourguest.
I'm going to wrap it up here,guys, and then we can debrief
and, if you'd like, you've beenlistening to the Audit presented
by IT Audit Labs.
My name is Joshua Schmidt, yourco-host and producer.
Today, our guest has been DinoBusolacchi and he was speaking
about OT and IT and theconvergence with Nick Mellum and
(50:55):
Eric Brown from IT Audit Labs.
Please like, share andsubscribe wherever you source
your podcasts.
We have video every two weekson YouTube and Spotify and we'll
see you in a couple days.
Thanks so much for tuning in.
Don't forget to like, share andsubscribe.
If you have a moment, leave usa comment on a YouTube channel
(51:16):
or give us a review on an Applepodcast.
It really helps others find theshow.
Thanks so much for joining usand we'll see you in the next
one.
All right, you are listening to the Audit presented
by IT Audit Labs.
I'm Joshua Schmidt, yourco-host and producer Today.
We're joined by Nick Mellom andEric Brown of IT Audit Labs,
and our guest today is DinoBusolacchi.
He is currently in St Louis,missouri, and he's coming to us
today to talk about OT-ITconvergence.
And, dino, thanks for joiningus.
Speaker 2 (00:25):
Thanks for inviting me.
Speaker 1 (00:28):
Absolutely.
I know this isn't your firstpodcast, but we're excited to
talk about a topic that you'refamiliar with today that OT
meets IT and a lot of otherthings that go hand in glove
with that, so thanks for joiningus.
Can you give us a littlebackground?
You have so many titles here onLinkedIn I didn't know where to
start, so I'll hand that overto you.
Speaker 2 (00:49):
Yeah, that comes with the gray hair right.
So I'm currently a directorhere at the Barry Wimmler Design
Group.
The Design Group is amanufacturing engineering
organization been around for 100years and they acquired my firm
last year.
So I've been a design group nowsince July of 2024.
(01:12):
Prior to that, I owned andoperated a OT cybersecurity
systems integration company.
I was focused just with theseOT cyber securities.
Prior to that, I spent severalyears working for Rockwell
automation and their networksecurity services group.
Prior to that, I was in theRockwell channel working with
their distributors, a lot ofsystem integrators and OEMs, and
(01:34):
I'll touch a little bit on thatthen.
Prior to that, I spent 20 yearsworking for one of the world's
largest adult beveragemanufacturers based out of st
Louis you can probably guess whothat is working building out
breweries of the future and so Ispent a lot of time around
control systems, implementingwhat I want to call IT
technologies on the plant floor,which is why, when I was at
(01:55):
Rockwell when some of these OTcybersecurity technologies
started coming to the market, Iknew there was a niche in
driving cybersecurity down onthat plant floor the myth of
those machines being air-gappedit just doesn't exist anymore.
Prior to that.
I worked for General Motorsunder Ross Perot back in the 80s
.
I worked for Monsanto andvarious others.
(02:16):
I actually started in thebanking industry in the early
80s, working in data centers andgot involved with building out
networks to banks and putting inATM machines back in the day
when the ATM machine wasactually at the bank.
So that's a condensed versionof my background.
Speaker 1 (02:32):
That's awesome.
Do you get a lifetime supply offree beer then?
Does that come along with theterritory?
Speaker 2 (02:38):
If you know the right people.
The answer is yes.
Speaker 1 (02:42):
One of the things that stuck with me is you shared
that manufacturing is one ofthe largest attack vectors and
maybe you could explain why thatis and how that has met with
your experience working in OT.
Speaker 2 (02:54):
Attacking a manufacturing facility and
getting to the point where youhave a material breach and
you're able to shut down thatplant is very costly to shut
down that plant is very costly,right.
So when I worked for that largeadult beverage manufacturer, if
we just had one packaging linenot putting beer in a can or a
bottle that was $80,000 an hourof loss, right.
And so when you think about amanufacturing environment, you
(03:18):
know their plants aren'toperating because it's very
expensive, very quickly, right.
And cybersecurity insurancedoesn't cover business
disruption?
Right, it'll cover the aspectsof doing the remediation,
mitigation work that you got todo, but they're not going to pay
for the loss that you incurredfrom being down.
Speaker 3 (03:35):
So then, dino, probably the most famous example
of that air gap breach, so tospeak, is probably Stuxnet right
, right With the sandworm.
So, in your experience, haveyou seen other attacks like that
when you were working at thoseprevious organizations, and how
(03:59):
did you combat it?
Speaker 2 (04:00):
Not that targeted.
I mean, it was a highlyspecialized, developed malware,
you know, attacking a specificSiemens set of control systems
that, whatever nation stateswere involved with it, had an
inherent knowledge of what theywere using within that nuclear
facility.
But what I will say is that alot of the manufacturing
(04:21):
environments are impacted bybreaches that come through the
IT environment.
Right, because we use a lot ofthe same technologies on the
plant floor that you have in theenterprise or in the office
space.
Right, windows, for example,that are running HMIs.
You know your human machineinterfaces, your historians,
your application servers, yourengineering programming
terminals those are all Windowsmachines.
(04:43):
And you also have a lot ofnetworking equipment out there
too.
Right, you've got all stripesand types out there inside that
manufacturing facility and theyall have vulnerabilities.
So if you're accustomed andused to what I still call Patch
Tuesday for Microsoft, right, wedon't patch very often on the
plant floor because it's verydisruptive.
(05:04):
The life cycle of controlsystems is measured in decades.
Right, it's not measured in athree to five or seven year
window.
So when control systems, youknow you're putting in a $20
million packaging line, theintent is that packaging line is
going to be out there for 20years doing what it does, right,
and so, and even then, you know, the frames of that equipment
(05:27):
may still remain the same.
But you might do a recontrolright and recontrol efforts.
That means that you'rereplacing the PLCs, you're
replacing the SCADA.
Software might be doing someupgrade on the applications,
maybe the network.
It's very, very costly, right,and those are usually capital
projects.
They're not OPEX projects,they're capital projects.
And so the dynamics aredifferent in that particular
(05:49):
space.
And if you're buying from anOEM, right, they may not stay as
current on patching for theircontrol systems, right, they
just again, because it'sdisruptive and there's costs
involved with it, you know, andand so we don't see a lot of
that.
And so then you gotta, you knowwe subscribe to, you guys are
familiar with, you know, thesans institute, right, so think
(06:11):
of um the the sans.
There's five critical controlpoints for ot cyber security
right in the it world there'slike a dozen or more, but on the
ot side we can kind of focus onfive with incident response,
defensible architecture,vulnerability management, remote
access, which is really big,and we talk about that in a
second because COVID kicked thedoor open on that and then
(06:34):
continuous monitoring right,which is not something that
groups are used to doing.
Because most engineering teams,when you start talking to them
about putting technology insidetheir control system network to
monitor it, the first thing theythink of is that you're
scanning and dumping a lot ofdata on that control system
network and it can be verydisruptive.
And so they've lived throughthat because IT even though
(06:57):
sometimes they're really tryingto do their best on due
diligence you know putting in ITcybersecurity tools and they
start scanning that network orimplementing EDR on those HMIs
you know, like a CrowdStrike orSentinel-1, and all of a sudden
you're inducing delay aroundthose machine centers that
require very specific you knowtime sinks and communications
(07:20):
for a couple of reasons, safetybeing one right and depending on
the nature of whatever you'redoing, you know if you're not
watching that stuff it can bevery impactful and sometimes IT
doesn't even realize they'redoing it right and the plant
floor folks don't necessarilyhave the technology as a place
to see this stuff.
We have that technology now,right when you think of the
(07:41):
Armises and the Clarities, andthen the Zomis and Cisco Cyber
Vision and Dragos, if you'refamiliar with those types of
intrusion detection systems thatwere specifically built and
geared for the OT space, right,and they're passive, right.
They sit there and listen tothe network.
They do do packet inspection tobuild out that baseline of
(08:02):
those assets, to build a profileon that PLC, to tell you the
make and the model and thefirmware on it, and based on the
firmware version, now I knowthe vulnerabilities that are
associated with it, just likeyou know the operating system
level on a Windows machine I canstart building that CVE list,
right, and then from there I canstart building a strategy on
how I can improve mycybersecurity posture around
(08:23):
those control systems.
Because, back to my earlierpoint, the stuff is connected,
right, it's just, whether it'sfor support purposes or feeding
MES systems or manufacturingexecution systems or your ERPs
or other enterprise applications, they're pulling data up and
off that plant floor and thenyou have people showing up to
(08:44):
the plant floor.
So, and then you have peopleshowing up to the plant.
You know, if I go and if I showup to a plant, eight out of 10
of them will let us go in therewith our own laptops or
computers and plug into theirenvironment, right, and they
have no idea what's on thesemachines, right, and then a lot
of OEMs.
When you buy a machine fromthem, so think about buying a
new vehicle, right?
And it's got, you know, onstaron, it's got satellite, it's got
(09:06):
cellular, it's got Bluetooth,it's got Wi-Fi.
Well, these machines aren't anydifferent on the plant floor,
and we'll find cellular cradlepoint modems in there.
We'll find VPN concentratorsfrom SACOMI or EWON or
Toshiboxes, right, becausesomebody is connecting in to
provide support of thatequipment down on the plant
floor when it's not runningcorrectly.
(09:27):
Or if they want to do some typeof enhancement, maybe they're
moving to a different product,so they're doing some
programming, and so there'salways connectivity in those
environments.
They're very dynamic, they arenot static at all.
Speaker 1 (09:39):
That was going to be one of my questions and you've
pretty much illustrated it.
But for the people that aren't,you know, familiar with
manufacturing environments or ormaybe just you know are tuning
in for the tech aspects of thisshow, can you give us kind of
this like a general sense ofwhat these manufacturing floors
look like?
Speaker 2 (09:55):
yeah, and it's significant right, when you
think about the number of youknow you've probably heard the
term iot or iiot for theindustrial Internet of Things.
We obviously have the term OT.
But yeah, when you think aboutmotors and drives and robots and
pumps and sensors and fillersand conveyors and just the
(10:19):
presses and all you know dryersand fans, that can go on and on
and on and all of the differenttypes of connected physical
systems, that's what we kind ofcall it, you know, I think of
connected physical systems andin the IT world you have data
driving data outcomes.
Right.
In the OT world you've got datadriving physical outcomes.
And so you know the twopriorities.
(10:42):
In any manufacturing facility,safety is always job one.
Right, there's no doubt aboutthat.
You got to operate safe andit's not negotiable, right.
So you have to be a safeenvironment.
You have to demonstrate thatyou're doing it.
Number two is what I spoke aboutearlier.
It's unplanned, unscheduleddowntime, because it's expensive
and it's costly to demonstratewhat the target is for this
Right, because it's usually notin their purview, meaning that
(11:04):
IT doesn't own those assets.
You know.
Nowhere else in the business,especially on the enterprise
side, it has a responsibility ofthe technology that the company
is using to run their businessright.
But on the OT side it's not somuch.
You know the decision to put ina new packaging line and what's
going to consist on the controlsystems around that packaging
(11:26):
line, and what computers are wegoing to choose and what
networking equipment are wegoing to choose, and operating
systems and firmware andapplications.
It's an entirely differentgroup of people, right?
So I worked at Rockwell.
Even when I worked forAnheuser-Busch, it wasn't
sitting in the room when we weregetting ready to determine, you
know, to build a new brew housefor $500 million, should they
(11:49):
have been sitting in the room.
Speaker 3 (11:51):
Ideally, Eric.
Speaker 2 (11:52):
yes, we want that.
Yes, here's the way I kind ofdescribe it, Eric.
I want them on the field, Iwant them in the huddle.
They just can't be thequarterback, if that makes sense
right, but yeah, they play atremendous role right Governance
policy, technology resources,standards.
They bring a lot to the table,but they're not always invited
(12:17):
in through that CapEx project,on whatever it is that company's
doing an expansion, brownfield,greenfield, lifecycle
replacement, whatever there-control, whatever tech
project they're taking on wedon't typically see it sitting
in the room participating, youknow, in something that might
start two years prior to thatmachine being built and running.
And I'll give you another goodexample erica, we were just in
(12:39):
atlanta a couple weeks ago andthe client, the it, the
networking team, came on siteand met us there.
They were 20 minutes from thisplant and out of the three it
professionals out of thiscompany, only one of them had
been at the plant in the lastfive years.
The other two had never beenthere.
So you know it's very difficultif, from an it perspective, if
(13:01):
you're not engaged in what'sgoing on in that plant
operationally, from a processperspective, what technologies
are in there, it's verydifficult for you to start
putting a cybersecurity strategyaround that.
You've got to think globallybut you've got to act locally.
And if you've got 50 plants inyour fleet, no two are the same
(13:26):
fleet, no two are the same, andso you have to have some
inherent knowledge of what'sgoing on inside that facility
and what technologies do theyhave in there and how that stuff
is put together.
Speaker 4 (13:31):
You know we're seeing this too that IT is not invited
to the party.
I guess in your opinion, youknow, especially in your
industry.
Why do you think that is?
Why are they being left out ofthese meetings that are
happening, even if they'rehappening two years beforehand?
Why are they not being invited?
In your opinion?
Speaker 2 (13:49):
Yeah, good question.
You know, one of the thingsthat we look at when we start
these engagements is we look atthe organization.
One of the things that we lookat when we start these
engagements is we look at theorganization.
And sometimes the first timethat IT and OT have the same
manager leader is usually at thepresident or the CEO of the
(14:11):
company.
Think about that for a secondright and that's not necessarily
right either right.
And then we have to think aboutthe size and the scope of how
many people you need.
So if you're a CIO and let'ssay you got, you know, 200
people working for you, right,and you've got 100 plants, well,
how big is that that OTpractice need to be from a
technology standpoint?
And who's going to own thatRight?
Is it going to be the CIO?
(14:31):
Is the CIO going to become thenew plant manager or general
manager of the plant?
Because they operate as fiefdoms, these plants do their own
(14:52):
little kingdom.
You know meaning they decidewho they're going to work with
from an integrator perspectiveor whose machines they're going
to buy.
You know what their partnercommunity is.
On the OT ecosystem as I callit right, and the OT side is not
any better right, as I call itright, and the OT side is not
any better right.
The OT ecosystem, that supplychain on that side of the house,
needs to get better at thisright.
Because if you go in there andstart talking to them about
cybersecurity, who do you thinkthey're going to point their
fingers at and say isresponsible for cybersecurity?
That's IT.
They're going to come.
That's IT's problem, right.
And then we show these machinecenters and IT sitting in the
(15:15):
room.
They're going yeah, that's notus, I don't even know what that
stuff is Right.
And so there's a gap.
Speaker 3 (15:21):
We see it a lot where we'll go in to help an
organization maybe restructurehow they're doing information
security because they're at aninflection point, Right,
Whatever that is.
And a lot of theseorganizations have multiple
disciplines of OT, whether it'saround transportation or refuse
(15:42):
or control systems that aredoing something related to
critical infrastructure.
And I don't think I've goneinto an organization yet that
has had a what I would call evena standard plan for how they're
(16:02):
going to operate and secure thetechnology that the plant is
responsible for.
Like, at the end of the day,don't really care what's in the
plant, what the plant does, butit's about how are those systems
that are operating thatequipment being secured.
And, like you said, there'scommunication happening outside
(16:26):
of that.
Quote unquote, closed network.
I don't think I've ever seentruly closed network yet either,
and I've been doing this awhile.
But yet there's, you know you,some of them are museum pieces.
You go in there and there'swindows XP, there's windows
seven, there's, there's all ofthese um, none their their
desktop applications that, forwhatever reason, these OT
(16:50):
organizations have decided torun their infrastructure on, See
very little Unix, Linux flavors.
It's a lot of Windows flavorsand a lot of legacy Windows
flavors and you mentioned toolslike Dragos or specific to OT
(17:14):
that can help look passively forthings going on on that network
, which you know in some caseswhen the horsepower around the
technology was smaller or therewas less horsepower available to
do non-ILC processing.
Then nowadays, though there'stechnology that could sit
(17:38):
quietly on the network, couldlook, decrypt, whatever, for
things that might be calling outor attempting to call out or
attempting to move laterally,and unless that, that, that OT
part of the organization isbringing apart the discipline,
which it's really hard to have,the discipline to know how to
(18:00):
run a tier one cybersecurityshop and run a plant that is
responsible for operating thatline of business.
It's really hard to do both.
So you've got to kind ofoutsource the IT side to either
internal IT or a third party tobring about that best practice
and facilitate it.
(18:21):
And I don't think it'sacceptable to say, yeah, it's
okay to run on 20-year-oldtechnology just because the
plant itself is running on oldtechnology, because it can last
20, 30 years.
Those manufacturers need to beheld accountable to bringing the
(18:42):
technology along.
It's not a secret.
We all know that Windowsexpires over time.
We've got to stay on top of it.
So bringing governance in earlyinto that two-year process at
the contract level to hold thevendor accountable for making
sure that those updates there isat least a vehicle to do the
(19:03):
updates.
That's kind of where I am nowin my headspace, versus saying,
oh, it's an OT environment,let's just leave it, let it run
and hope nothing happens.
Speaker 2 (19:13):
Yeah Well, and the way what you're describing is
very expensive, right, sure Toreplace, to move from Windows 7
or XP and do a recontrol ormaybe move a piece of equipment.
You might be talking abouthundreds of thousands or
millions of dollars in order todo that right.
And so there has to be theclient has to recognize that.
Speaker 1 (19:34):
Where are you right?
Speaker 2 (19:35):
Because right now they don't even have a really
good idea where they are.
They don't have a good baseline.
So you got to get that figuredout first and there are.
There are technologies thathelp through that.
While they're building theircapital plan eric to your point
to be able to go after thatstuff um, like virtual patching
right or defensiblearchitectures around that
machine center so that you canat least improve your
(19:58):
cybersecurity posture, harden itwhile you're putting together
that multi-million dollar CapExplan to go after and replace
that.
The other thing that I'll tellyou is that I call it the OEM
blockade.
They won't let you touch theirstuff right.
If you want their support andyou want the warranty and
maintenance that comes with thatmachine center, they don't want
(20:19):
you touching it right, andthat'll because otherwise you
void the warranty.
So think about if you bought acar and you know you're getting
the warranty and support from GMand then you go in there and
you modify the exhaust systemand you decide to swap out the
transmission and you try to youknow, maybe do something with
fuel injection and you modify itand then you take it back to GM
to say fix it, they're going togo no right.
(20:40):
And these OEMs have thatmindset, because they don't even
understand that even if we justwant to put something in there
as simple as a sensor right tocollect the metadata within that
control system, they would say,if you touch that, you know.
If you add anything to thatswitch to do that, or change the
configuration of that switch tocreate a span port to pull that
(21:00):
stuff out of there, you voidthe warranty and a lot of the
clients are like they don't wantto buck the system with that
OEM.
And so now you've got to getahead of the game, right.
Now you've got to be proactiveand so if you're going to buy a
new piece of equipment, then youneed this and you're let's say
you're a drago shop or an armistshop or clarity shop, you
(21:21):
should be planning ahead to putin the right technology in that
machine center, right.
I always like using theautomobiles.
You don't put the safety, theseat belts, on after the fact,
or the anti-lock it breaks onafter the fact or the sensors of
the backup, backup cameras, theairbags.
You do it when you're buildingthe car, right.
And it's the same concept whichis one of the other big
disconnects that we see is thoseOEMs and a lot of these system
(21:44):
integrators, which the firm Iwork for is both of those trying
to change that narrative in themachines and the integrations
that you're doing andimplementing that up front,
right.
It's much more cost effectiveto do it then as it is to do it
on the retro, on the backside.
So two things it can beexpensive to try to do a
recontrol, replace equipment,and then you've got to deal with
the OEM.
(22:05):
That can be a little pig-headedsometimes, right, and you've
got to try to break through that.
Speaker 4 (22:08):
There's a lot of moving parts here.
I'm thinking back to one of myfirst IT jobs, relatively not
that long ago 10, 12 years agoand it was at a large
manufacturing company in theTwin Cities and, being a junior
IT, I was on a help desk thereand I'd run out and help all
these different lines get backup that were going down or do
(22:30):
any sort of tasks.
But there we had so much legacysoftware and I'll never forget
one of the more senior guys.
I was out running tasks withhim and we had one machine over
in this corner.
I don't remember what it wasdoing, I think it was printing
labels or something and he saidwe don't touch that thing
because if it turns off it mightnot ever come back on.
It had been there for maybe 30years out of support, obviously,
(22:52):
legacy software.
The vendor doesn't even existanymore.
But it was always so confusingto me why don't we just change
it out?
And I'm hearing from you now.
You know just the expense to it, but you know, is it a lack of
auditing?
Are we not asking thesequestions?
Is that?
Is that the root problem?
Are we not, as IT professionals, are they?
Are we not digging into thesand, letting them know these
(23:14):
issues.
I guess my question is are wegoing wrong somewhere or is it
just because we're not invitedto the table is my earlier
question.
But it's happening so oftenthat we're inviting the wolf
into the hen house, not beingcompliant or having these
discussions that we're havingnow, which it seems like we're
all on the same page, but itseems from a simple discussion
(23:37):
earlier on.
Like you said, putting theseatbelts in early, we could
solve a lot of these problemsdown the road.
Speaker 2 (23:45):
It's the timing of it , in my view, right.
How do you get that IT groupengaged and involved in that OT
environment right and becomepart of that ecosystem?
Speaker 3 (23:57):
What makes the most sense.
Speaker 2 (23:59):
So think of shadow IT right.
Eric, I know, you know what thatis, you've been probably we go
back a number of years.
Right, and that was one of thethings we were trying to chase
out was shadow IT.
But when it comes to that plantfloor, it's the one area that
they've just never been able totake on, and a lot of it is
because they don't own the asset, right.
(24:19):
Who owns that asset?
Like I said earlier, all of theenterprise stuff, there's no
question, it's it.
They own that right.
They make those decisions,whether using third parties or
it's an internal team or amixture of both, but at the end
of the day, they make thosedecisions.
They make that, those buyingdecisions.
They decide the resources andthe responsibility to take care
(24:39):
of it, just like that OEM doeson the plant floor.
But when it comes to that PLC,it's a programmable logic
controller, right, and so that'sthe instrument.
Speaker 3 (24:48):
And it's usually running on a.
Speaker 2 (24:50):
Linux-based operating system of some sort.
Right, it's not running on aWindows OS, it's usually a Linux
type of OS but it's the onethat's receiving signals and
making calls to make that motorgo faster or to turn the flames
up on that burner or to speed upthat filler or whatever that
might be.
That's what that controller isdoing, right, and IT just never
(25:12):
put their hands on it, even tothe extent that engineering
teams would design their controlsystem architecture to put
multiple network cards into thatPLC to make it a network
segmentation device.
Because one side if you'refamiliar with the Purdue model I
don't know if you guys arefamiliar with the Purdue model,
but think of a PLC with anEthernet card on it, because we
want information sent to ourhistorian, we need remote access
(25:35):
to it for programming and Iwant to send data up to my MES,
and so there's a connection towhat I want to call the
enterprise side of the network,because even though it's within
the plant, it's the side of thenetwork that IT usually manages.
But then there's other cards inthere for control that are on
that PLC and that's the spacethat you never ever find IT in,
(25:58):
because that's actual control.
That's the one that's sendingthe signals, you know, because
we're back to process integrityand we're back to safety, and if
you don't understand what thatthing is doing, it's very hard
for you to take on a role oftrying to be I call it, the
round peg square hole andapplying a security measures in
there that could be disruptiveto that environment.
(26:18):
How are you going to get agroup with the boots on the
ground knowledge of what's goingin there to implement?
All right, I got Siemens overhere, I got Rockwell over there,
I got Honeywell, I got GE, Igot Emerson, I got Yasago, I got
Mitsubishi.
They're not very standardized.
Right, it's done a really goodjob of standardizing everything
right Compute platform, networkplatform, databases.
(26:41):
you know ERP systems.
They standardize you get intothat plant and it's seven ways
to Sunday in there.
You know I'll find eightdifferent access methods into
the plant for remote access andI'll find several different
automation technology vendors inthat one facility.
Speaker 1 (26:59):
One thing you said that stood out to me, dino, is
that even just within a company,the plants are vastly different
from location to location aswell.
Not just the equipment that'srunning in them and how they get
things done and where they'repositioned, and all that.
Speaker 2 (27:13):
but yeah, the whole layout, the design, and
culturally and behaviorally Imean depending on what part of
the world they're in or whatpart of even in this country
they're in you have differentmindsets on how they operate
that are tough to get through.
And then you have a lot ofmergers and acquisitions, a lot
of these plants you know thathave been bought and sold
(27:33):
multiple times over the last 30or 40 years and you just don't
have a lot of good continuityand the way we pick up on that
really quickly is lack ofdocumentation.
They don't know what they got.
There's no network drawings,there's no good network drawings
.
Whatever documentation theyhave, you know 20%, 30%, 40% of
it's wrong, it hasn't beenmarked up, can't find it, don't
have it.
(27:53):
Right, it hasn't been marked up, can't find it, don't have it.
So we do a lot of investigatingand hunting.
When we're in there trying todetermine, we go in there and
drop one of these tools in tostart collecting a baseline, and
then we walk the plant and weopen up those panel doors to see
what's in there and we documentthat.
(28:16):
So I want to be able toreconcile what did I visually
see on the plant floor and it'snot in my baseline on my tool.
So I can figure out how do Iget that system over here on
this box Right, and make thosedeterminations of whether I got
to put a sensor out there or Igot to change the network to
some extent.
Do they even have cabling toget me out there?
Right, so, and so it's thedynamic of it.
Aspect of it is what can bechallenging Right, so, and so
it's the dynamic of it.
Aspect of it is what can bechallenging Right, and and not
(28:40):
everybody's prepared.
You know they're behind.
They're 20 years behind IT oncybersecurity 20 years.
Speaker 4 (28:46):
Dean, are you familiar with CMMC, the
cybersecurity maturity modelfrom Siemens, remodel from
Siemens?
This is from the Department ofDefense that you would have to
follow these guidelines to beable to be a contractor to work,
build a piece of a bomb, builda piece of an airplane or
something like that, and I thinkwe're all talking about the
(29:08):
same thing here, but with a setof standards for many different
organizations.
Now this would be very hard toimplement and govern.
But I think CMMC was one of thefirst entries into the space to
try to standardize this space.
But we're not seeing it fornon-federal contractors,
somebody like Anheuser-Busch.
(29:29):
We're not seeing it for thatspace at all.
Would implementing somethinglike that solve some of these?
Speaker 2 (29:34):
problems.
But that's one of the things welook for is are they following
a security framework, right?
You've got NIST.
It depends on the vertical one,right?
Some are highly, more highlyregulated than others, whether
using the IEC 62443 or MITRE orthe ISO 27002 or one.
And then you got NIST, whichfor the BOK for you know, the
(29:55):
Boko we do is food and chemical,life sciences, heavy industries
, semiconductor.
We do a lot of data center workaround power systems, and so
from a regulatory standpoint,not everybody has to even follow
regulation, right.
I mean, you think about ifyou're a publicly traded company
(30:16):
today and you get a materialbreach.
You've got like 48 hours, 72hours.
You got to tell your investorsthat you got pop, right, and
even then the government maygive you some leeway depending
on who you are like, if you'reBoeing, right, compared to you
know, some soda manufacturer,right, and the government may
(30:37):
even follow up on that.
But my point is is that we usethe security frameworks you know
to go after the identification,detection, you know, response,
recover, you know even anincident response plan.
Speaker 1 (30:49):
What is your?
Speaker 2 (30:50):
IR plan at this plant , right, and you'd be surprised
how many companies don't havethat.
They don't have an IR plan.
So then you get back to the ITconversations.
Well, how do you get IT todrive that?
Because IT's probably got an IRplan, you know, and they
probably practice it, you know.
I mean, I can remember the, youknow the IR plans.
You know, back in the day whereyou'd run off to some
third-party site to spin up yourbusiness somewhere else right,
(31:13):
but in a plant.
Spin up your business somewhereelse, right, but in a plant,
their only option is to eithermove their volumes to other
facilities to make up for theloss of this one if it's not up
and running right, but theydon't have one.
And I will tell you that if Igo into 10 plants, two of them
are going to have malware inthem.
Every one of them is going tohave a rogue asset or a series
of rogue assets in them.
(31:34):
You know even the guy that'sbeen there for 20 years and
we're telling him hey, there's aWAP sitting over here on your
packaging line.
He goes no, there's not, it'slike yes, there is we can see it
.
It's on the network.
Let's go hunt it down and findit.
And they have no idea how itgot there.
We found stuff in ceiling tilesand plants that somebody put in
(31:58):
there.
Speaker 4 (31:58):
There's all kinds of goofy things and they don't even
know that they're here.
Speaker 2 (31:59):
It was probably the server guy wearing umbros, just
to give you an idea betweenGartner and Cisco and Rockwell
and Siemens and these OT ideas.
Guys, 60% of manufacturers outthere are just.
They're either unaware or justbeginning to understand that OT
is a thing.
That's 60%.
You got 30% that have actuallystarted doing something.
(32:20):
Maybe they did a POC, Maybethey've had some demos.
Somebody came in and startedtalking about it.
Maybe they've actually got acouple of people who are in the
organization, have been taskedwith cybersecurity for the plant
floors.
And then you have 10% thatactually have a strategy and a
plan or starting to implement itand going down the line.
Speaker 1 (32:39):
That's where we are today.
I wanted to circle back reallyquick and kind of get Eric's
take on just integrating the ITin any kind of organization but
more specifically biggermanufacturing or larger entities
like Dino's talking aboutmostly.
What's your take on that?
Is it mostly just kind of justa knowledge base or does a
cultural thing of why there's adisconnect there?
(33:01):
And I know you spend a lot oftime on boards and things like
that.
You know showing people whatthe risk actually is, because no
one really sees it until ithappens.
So I was kind of curious to seewhat you thought about that.
Speaker 3 (33:15):
I've seen a couple of cases where more than a couple
and Dino alluded to this wherethe IT organization or the
leadership of the ITorganization has never actually
been into the plant.
Maybe they've toured the plant,been on the grounds but never
(33:35):
really looked at the technologyin the plant.
And, as as Dino was telling thestory about um rogue devices or
potentially malware in thesystems, I've walked these, um,
the, these plants or theseoperations and the.
We ought to do a pictureslideshow on this because
(33:56):
there's just some crazy pictureswhere it's in a hospital and
the amount of cabling that wasjust across the face of the rack
probably weighed more than me.
It was just this huge rat'snest and there was dead switches
that had just been cabledaround that were in that
(34:18):
environment.
There was dead rodents thatwere completely squished
underneath a piece of technologythat you know hadn't been moved
in 10 years.
It was way up in the rafters inthis production facility that is
responsible for making giftcards and we were doing this
(34:43):
network assessment and we werekind of tracing back where we
were seeing devices on thenetwork but we couldn't find
them.
And way up in the rafters on topof an I-beam was like some form
of a Linksys device that wascabled into the network.
There's no documentation, ofcourse, on it.
Nobody knew how it got there orwhat it was for, or if they
(35:06):
were trying to add a new machinethat they had long since taken
out and forgot about.
But the amount of undocumentedtechnology that you know
sometimes people that have beenaround the plant for years, they
may remember, or might be sometribal knowledge, but I think
part of it, josh, is if you'reresponsible for an environment
(35:28):
or you're providing some form ofresponsibility on the
technology side.
You got to know what's in yourenvironment, right Back to that
CIS one and two.
Know what the hardware is inthe environment, know what the
software is in the environment,because if you don't know what's
out there, you really can'teven begin to try to protect it.
Speaker 2 (35:51):
That's the first candidate in any cybersecurity
framework.
Is that right there?
Your asset inventory?
Yeah, the question is whetheryou want to get it in a
continuous fashion.
Like IT generally does iscollecting information
consistently, constantly, versusa snapshot in time.
But that's the number one tenetin any security framework is
asset inventory.
Even to the point wherecritical infrastructure
(36:13):
organizations, where the TSA andDepartment of Homeland Security
are coming to them, and evennow insurance Now the insurance
companies are coming in becausethey're getting smarter going
you need an accurate assetinventory and I want to know the
vulnerabilities and riskassociated with those things.
That's the first thing, rightBefore you get into anything
else.
Let's just do that Right.
Speaker 3 (36:32):
Do you know?
The first year that I had tofill out a cyber insurance form
it must have been like 2013 or2014 working with a customer, we
sat down over a lunch with theinsurance form and they're like
yep, you know, we have this formthat we need you to fill out.
It was half a page.
I've done a few this year andnone were less than like 20
(36:58):
pages of details around multipletabs.
It's all online now.
Right Of all of these thingsthat the insurance company wants
to know in order to rate you,and it seems every year there's
more and more questions goinginto it.
Speaker 4 (37:14):
Understand what's your network, but I think after
we do that, I thinkorganizations fall short of
documenting everything theydon't document.
They rely so heavily on tribalknowledge.
It's the operator that's beenin there for 20 years and then
they get a replacement, you getsomebody junior and there's that
gap of 10, 20 years.
But if you spend a little time,you know, bring a third party
(37:35):
in, go through these processes,build your policies and
procedures, document everythingfrom all these controls.
I think the maturity of anorganization would go through
the roof instantly if youstarted to document these
processes.
Speaker 3 (37:49):
Dino.
So how do you get them tomaintain their environment?
So you come in, you document it, you come back with a great
study.
How do you keep it so that youdon't show up five years later
and there hasn't been anyupdates to all that work that
you've done?
Speaker 2 (38:06):
So what we do is we don't sell technology and just
leave.
So I back up the truck, push itoff the dock and give it to
them, or just stand it up andget it running and leaving.
We pursue managed servicesbecause, you know, managed
service is big in the IT world,right, and most 80 percent or so
of the manufacturers todaydon't have a practice.
(38:27):
They don't have an OT practiceand they have an IT team, but
the IT team is is limited in thenumber of resources that they
have.
They've already got full-timejobs taking care of a whole
bunch of other stuff, much lessdumping this net new data stream
that looks like a fire hosepointed at their head.
Right, it's almost like theyget alert fatigue with the stuff
that comes flying out of there.
(38:48):
So our goal is to get a managedservices piece in place for
them, at least for the firstyear, eric, until they decide
that they want to build theirown practice.
Um or uh, some clients, justbecause they're in that managed
service mindset, um will flatout say, hey, I just want to
hire you guys for three years, Idon't even care what tool you
(39:09):
use.
You know here's, here's myrequirements, and if you can,
and then we just come in and runit and manage it for three
years for them, yeah, and wecatch new assets coming, because
these tools will catch newassets.
They'll catch new applications,they're going to catch new
protocols, they're going tocatch changes in the control
systems, new networking devices,it.
So we're constantly watchingand building to your point of
(39:31):
five years.
You, if I see a new asset comeout, well then that's going to
be a trouble ticket.
That's going to be created andsomebody's going to have to go
resolve.
It's going to get assigned andsomebody's got to go resolve
that.
Speaker 1 (39:42):
Bringing it back to the first question I asked at
the beginning of the podcastabout attack vectors, I think
we've thoroughly mapped thelandscape of how things are a
bit disjointed.
Maybe, Dino, you could speak towhy that is on just a practical
level, maybe coming from thehacker or malicious actor side
(40:03):
of things.
Why is this such a prime targetin manufacturing?
Speaker 2 (40:07):
Because they'll pay.
That manufacturer will write acheck and get it paid to get
back up and running.
Those that don't will spendmonths and months and lose
hundreds of millions of dollarstrying to unwind whatever got in
there.
Those that don't have theweather, all skill, resources to
deal with it will just write acheck.
Speaker 1 (40:25):
And that goes back to your earlier point about the
amount of money being lost perhour on the manufacturing floor.
And are you seeing just a lot?
Is that just a lot of malware?
Is there the ransomware andthings of that nature, or what's
the most common?
Speaker 2 (40:40):
Yeah, malware that comes in, ransomware that makes
its way in through some emailsystem that gets its hands on
those HMIs out there on theplant floor.
Because the HMI, that is ahuman machine interface and if
you can't gain access to thatthen you're blind to whatever
the machine's doing.
You can't control it, you haveto shut it down.
Speaker 1 (41:00):
And who's typically catching those things.
Speaker 2 (41:03):
Well, if you've got a clarity in the environment or
an armistice in the environment,it's going to catch it on that
side of the fence, or IT mayhave caught it on their side.
So think of Colonial Pipeline,right, caught it on their side.
So think of Colonial Pipeline,right?
You guys are probably familiarwith Colonial Pipeline.
You know, in their particularcase, you know when it came into
their environment, the reasonwhy they shut down on their side
, on the moving of the fuel side, on the plant side, because
(41:25):
they had no visibility, theydidn't know whether they were
safe or not, and because they'removing, you know petrol,
gasoline and diesel and allkinds of stuff, over 5,500 miles
of pipe.
They shut down on precautionbecause they didn't know the
extent of the breach.
They had no idea.
If you look at Clorox, theydecided to fight it themselves
(41:48):
and they lost several hundredsof millions of dollars trying to
eradicate it themselves.
I've got clients that have beenfighting stuff on their plant
floor for almost coming up ontwo years, right, because
they're trying to get itthemselves and try to eradicate
it.
And then you have those thatwill write the check.
But getting back to what Ericsaid at the very beginning, you
know in regards to somethinglike Stuxnet.
(42:08):
It was a very targeted controlsystem.
There's been a couple others.
There's one at Saudi Aramco hadone.
It was Red Typhoon or somethinglike that.
It was targeting controlsystems.
We don't see a lot of that asmuch as we see the malware
that's hitting the Windowssystem.
It bleeds over into the plantfloor where you have Windows
(42:31):
tools, right Windows systems.
But it doesn't mean that youmean that they're not out there.
They are, it's just a lot ofthem is not even advertised.
You've got clients out therethat just keep it quiet and find
it themselves.
Speaker 1 (42:44):
So we've touched on it briefly and, as we're looking
to wrap up this awesomeconversation, I'm just wondering
, as we look to the future, whatare some quick ideas or bullet
point thoughts that how we canimprove upon this in the future,
going forward both from the OTand the IT side?
Speaker 2 (43:01):
Well, obviously you want it to be collaborative,
right?
You want IT and OT to bealigned and collaborative and
recognize each other'sweaknesses in regards to dealing
with this problem.
You're probably going to need athird party to come in and help
you with some of that, andyou've got to develop.
You've got to bring in asecurity framework.
You need to pick one, whateverit is, if it's specific to your
(43:24):
vertical, if you're in criticalinfrastructure and IEC 62443 and
NERC separate, then adhere tothat.
A lot of regulatory stuff ismaking you do that anyway, but
you need to get a framework inplace and you need to get the
right tools and you got to bringthe right people to the table,
and you need to have people thathave experience doing this.
It's surprising to me how manypeople just try to fight through
(43:46):
it themselves, and I sawGartner came out with a report
at the end of this year thatsaid by 2027, was it?
A third or 25% of thecybersecurity professionals out
there are just going to quit andgo do something else.
My playbook would be my choice.
How come Did it say why?
Because of the stress, you know, and it was just too difficult.
(44:08):
You know, if you're a CISO.
The average tenure of a CISO islike less than two years,
something like that.
Speaker 4 (44:15):
You know that we better get our rakes.
We've got a lot of work to dowhen that comes up.
Speaker 1 (44:19):
I thought you were going to say they were going to
work on their golf game.
Speaker 3 (44:25):
Eric, how about you?
Yeah, yeah, sorry.
I think it's the organization'sunderstanding that they can't
do it themselves.
Dino, you alluded to a coupleof customers that you'd worked
with trying to do the cleanup ontheir own.
A lot of the customers thatwe've gone into recently kind of
(44:50):
had this.
I don't want to call it ananti-contractor mindset, but
it's almost like the us and themof like the internal full-time
employee staff versus thecontractors, who the contractors
are?
Essentially, they're just paidout of a different bucket.
(45:11):
So why do you care how they'repaid if they're there to help
you?
And I apply it to a hospitalmodel You're sick, you go to the
hospital.
The ER doctor is not anemployee of that hospital.
(45:31):
The ambulance that brought youthere is not an employee of that
hospital.
The anesthesiologist that ismaking sure you're comfortable
while you're under surgery, ifyou have to go through that,
it's not an employee of thathospital.
That cardiologist that's takingcare of you is not an employee
of that hospital.
(45:51):
So I look at organizations thathave been running.
You know, fat, dumb and happy.
They have an inflection point.
They need to bring in someoutside assistance to help them
get better, to help themoperationalize and continue to
run their business.
But really looking at thosethird-party contractors as
people that are reallyspecialized in an area, that can
(46:14):
come in and are honestly goingto be able to help you faster
than you could go out and try tostaff it on your own, you're
going to have to rely on thatexternal assistance and you're
going to have to pay for it.
So that's where I think thereis no free lunch in this whole
cybersecurity and even IT gamethat we're in.
(46:36):
Right, you just understand whatyou need to run your business
and if you don't have itinternally, look externally.
Speaker 2 (46:44):
Yeah, and the plant floor is not very much different
than what you just described,eric, because a lot of them
don't have their own engineeringteams.
They don't build those machines, right?
If you want a packaging line,they don't engineer and build
that packaging line.
They go out there and get thethree or four or five different
groups together to build thatpackaging line and the
engineering teams to do that,right.
It's the same concept, you know, just different.
But yeah, that's why you end upwith a lot of different
(47:05):
components on the plant floor ofSiemens and Rockwell and
Honeywell, and you know,hirschman switches versus
Ciscoisco switches versus, youknow, phoenix contacts or red
lions or whatever you know,because of the different groups
that are building that equipmentand just bringing it into your
plant.
So the very similar analogythat he gave, that's the plant
floor, um, and to your and it'svery standardized, right.
(47:28):
They think that they canstandardize everything and
they're just accustomed toworking from that premise and
you're just not going to getthat on the plant floor.
Sorry, it's just, it's not that, not that way.
Especially when you got 10,000assets out there right on and
just one plant, you got 10,000assets managed by gosh, who
knows who, right?
Speaker 1 (47:49):
it's really cool talking with you, dino.
Uh, it feels like you have areally good beat on what's going
on out there in themanufacturing world and and it's
really interesting for me, evenjust being in a completely
different industry in theentertainment and audio video it
makes a lot of sense and itjust impresses me on how these
things even get accomplished.
(48:10):
There's so many moving partsand there's so many intricacies
to all of it.
So hats off to you for being aleader in that space, and I'd
like to pass it around for anyfinal thoughts today before we
wrap up.
I think we could stay on for therest of the afternoon, but I
was going to say, like we didn'teven get into like incident
response and things like.
Speaker 4 (48:30):
Yeah, like emerging threats yeah.
Speaker 1 (48:38):
Maybe we'll have to have you back down the road,
dino, to talk about, uh, whatcould drill down on something
like incident response,particularly how how you handle
that.
But, um, at the very least,eric your question that you
always ask at the end of thepodcast.
Do you want to?
You want to do it?
Speaker 3 (48:47):
what's my question?
Speaker 1 (48:48):
okay.
Are you planning on going toany security conferences?
Do you know?
Do you go to?
Going to?
Speaker 3 (48:53):
any security conferences?
Dino, do you go to anyconferences?
Security conferences I do.
Speaker 2 (48:59):
I'm actually sitting on the steering committee for
Manusec that's coming up here inMay in Vegas.
I also go to RockwellAutomation Fair.
It's a big one that we attendS4.
I've been going to S4 in thelast few years down there.
I don't know if you're familiarwith Dale Peterson that runs
(49:20):
S4X out of Florida once a year.
It's a good one.
So from a cyber and it'sspecific on OT cybersecurity,
it's called S4.
He's been running it for over20 years.
Nice, that's a good one to lookup.
I haven't really hit RSA orBlack Hat.
To be honest with you, I tendto lean towards the ones where
(49:40):
I'm focused on the manufacturingorganizations.
Speaker 1 (49:45):
Do you do a booth at your conferences?
You go to Dino or do you justgo and do the cocktail hour?
I speak a lot of them.
Speaker 2 (49:51):
Obviously, we have clients, the cocktail hour, I
speak a lot of them.
So we take obviously we haveclients that are there partners
We've got a lot of partners, soyeah, Because I like your
musical.
I use the instrument a lot.
It's like I can give you aguitar.
The question is whether youknow how to play it or not.
Speaker 3 (50:08):
It comes with these tools Like you.
Speaker 2 (50:10):
Take a clarity, for example.
Right, there's those that knowhow to play that instrument and
then those uh, don't you know,and we've got.
People have been working withthis stuff for almost a decade,
so it's going to be hard to beatthem.
If you're just starting todaywith the tool you might as well
leverage, somebody's beenworking with it for for several
years you know that's a greatanalogy, yeah well.
Speaker 1 (50:32):
Thanks so much for your time today, Dino.
It's been a great conversation.
It's been very informative.
Yeah, Thanks for being ourguest.
I'm going to wrap it up here,guys, and then we can debrief
and, if you'd like, you've beenlistening to the Audit presented
by IT Audit Labs.
My name is Joshua Schmidt, yourco-host and producer.
Today, our guest has been DinoBusolacchi and he was speaking
about OT and IT and theconvergence with Nick Mellum and
(50:55):
Eric Brown from IT Audit Labs.
Please like, share andsubscribe wherever you source
your podcasts.
We have video every two weekson YouTube and Spotify and we'll
see you in a couple days.
Thanks so much for tuning in.
Don't forget to like, share andsubscribe.
If you have a moment, leave usa comment on a YouTube channel
(51:16):
or give us a review on an Applepodcast.
It really helps others find theshow.
Thanks so much for joining usand we'll see you in the next
one.
Popular Podcasts
Bookmarked by Reese's Book Club
Welcome to Bookmarked by Reese’s Book Club — the podcast where great stories, bold women, and irresistible conversations collide! Hosted by award-winning journalist Danielle Robay, each week new episodes balance thoughtful literary insight with the fervor of buzzy book trends, pop culture and more. Bookmarked brings together celebrities, tastemakers, influencers and authors from Reese's Book Club and beyond to share stories that transcend the page. Pull up a chair. You’re not just listening — you’re part of the conversation.
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com