August 31, 2024 • 34 mins
Have you ever wondered how time management can revolutionise Operational Technology (OT) in critical infrastructure? Join us for a compelling conversation with Brendan McCarthy, the Managing Director for Availability Systems Engineering, who brings a wealth of experience from his 25-year career, including 22 years at BHP. Brendan shares invaluable insights into managing operational systems across numerous mining assets, emphasising the importance of duration, frequency, opportunity, and sequencing in OT environments. Learn how leveraging opportunistic time can enhance system availability and minimise outages, setting a new standard for efficiency in critical infrastructure.
Transitioning from a cybersecurity-centric approach to a comprehensive risk management strategy is no small feat. Brendan delves into the significance of maintenance and transparency in building organisational trust, and the transformative impact of the Technology Remote Operation Center (TROC) in streamlining incident management. Discover how preventative maintenance plans, rebranded as "planned maintenance," can improve communication and effectiveness within operational teams. Brendan also sheds light on the crucial role of managing failure domains and understanding human factors to ensure smooth operations in industrial settings, making this chapter a must-listen for anyone passionate about operational technology.
What's the future of OT with advancements like Edge computing and AI? Brendan discusses the integration of IT and OT, the unique cybersecurity challenges faced by critical infrastructure, and the skill shortages in the cyber and OT sectors. He emphasises the importance of system thinkers who understand both hardware and software in production-critical environments. Reflect on Brendan’s advice for specialising early and expanding across multiple functions for faster problem-solving, and hear his thoughts on Australia's leading role in mining technology. Don't miss his insights on the transformative impact of LTE on autonomous fleet availability and the exciting future potential of 5G connectivity. This episode is packed with knowledge and foresight for advancing technology in critical infrastructure.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Brendan McCarthy (00:02):
With an IT environment, we've almost
generally tried to outsource thecustomer intimacy.
You may be integrating thecapability, the technology, but
how you communicate, how youengage, how you maintain
customer intimacy is different.
Michael van Rooyen (00:17):
Today I have the pleasure of interviewing
Brendan McCarthy, who is theManaging Director for
Availability Systems Engineering.
We're going to be talking allthings operational technology,
critical infrastructure, allcybersecurity, all things that
relate to that industry.
Welcome, Brendan.
Brendan McCarthy (00:32):
Thank you for the invite.
Michael van Rooyen (00:33):
Yeah, great, I appreciate you catching up
with me here in Sydney.
We're here the Vivid Festival'son and Brendan is based in
Perth but was able to actuallycome at a time to catch up with
him here in Sydney.
So again appreciate the time Iwas able to actually coordinate
a time to catch up with him herein Sydney.
So again appreciate the time.
Before we kick things off,brendan, do you mind just
sharing a little bit about yourjourney in the industry,
particularly criticalinfrastructure and the tech side
of that, and then what led youto your role today as the
(00:55):
managing director and founder ofAvailability Systems
Engineering.
Brendan McCarthy (00:58):
Yeah, look, it's been a 25-year journey, 22
years at BHP.
Wow a 25-year journey 22 yearsat BHP Wow, when it was Western
Mining.
Then it became BHP Billiton andeven passed that through when
we exited the South 32 assets.
So we spent a lot of timeacross all of the commodities
during multiple cycles in thecommodity price.
(01:20):
So obviously budget pressuresand all the rest of it.
But uranium, copper, nickel,iron ore and that progressed
around Australia in thoseengagements.
The last five or six years Iwas looking after all of the
operational systems across BHP'sassets.
So all of the rail, all of theports, all the control systems,
(01:41):
mining, autonomous mining,everything, anything that was
making money we were lookingafter.
Michael van Rooyen (01:46):
Wow, I mean that's an extensive amount of
infrastructure right across, nodoubt, multiple mining
facilities.
Brendan McCarthy (01:51):
Yes, 20 mines, five ports.
There was always somethinghappening.
Michael van Rooyen (01:55):
Oh, no doubt , no doubt.
There was 24-7 operations, 24-7, yeah, oh wow, Across 20
locations.
Did you ever get some sleep?
Brendan McCarthy (02:02):
Oh look, we did for a while we started to
look after our petroleum assetsout of Houston and our South
American assets.
Michael van Rooyen (02:09):
Wow.
Brendan McCarthy (02:09):
And at that point it just started to get to.
I couldn't scale to that pointwith the way we were operating.
So basically we built theequivalent of what I created in
Australia, the T-Rock TechnologyRemote Operation Centre for the
24-7 support.
We built those out in Houstonin South America along with the
(02:30):
IROCs that we had.
Michael van Rooyen (02:31):
Right and was some of that related to
autonomous vehicles and allthose systems as well?
Brendan McCarthy (02:37):
So autonomous control systems which is all
automated, conveyor thepetroleum assets and plants,
even the heat bleach facilities,etc.
Michael van Rooyen (02:48):
That's just fantastic.
I mean just considering theamount of infrastructure you've
seen no doubt after that manyyears as well at BHP, and seeing
that you were the head ofproduction systems in your last
six years there at BHP.
What were some of the keyinsights that you learned, or
learning experiences, from yourtime with BHP?
Brendan McCarthy (03:01):
In the move into the, the ot, and I'm
talking real ot.
So there's three layers of ot.
There's the it world, thenthere's the mes world, which is
people call level three stack.
Then you've got level two.
When we talk ot it, we miss theopportunity to really understand
the difference between real ot,which runs plants from
(03:21):
control've got MES, which is theoptimization layer compared to
the IT space.
So in terms of the insights inmanaging MES, especially with an
IROC where supply chainmanagement is really important,
that having that availabilitywhere we manage things within
the week is really important,and on the critical environments
(03:42):
, so things that run controlsystems, autonomous vehicles,
that's in the seconds, managingtime is really important.
And the four elements that Ireally focus on is duration,
frequency, then you've gotopportunity and then you've got
sequence.
So those four elements arereally important to understand.
So duration is how long is itand how do I reduce it?
(04:03):
Frequency the more thingshappen, the more inconsistent
you are, the less predictableyou are and, as a result, the
controllers do things that arepotentially abnormal to what
they normally should do.
And it generally creates more ofan inconsistency in the
production line.
Sequencing is really importantMaking sure the right people are
there at the time when you needto execute the change.
(04:23):
Making sure there's all of thecapability, the tools, the
systems, the components so thatwhen you are having an outage
it's reduced as much as possible.
And my favorite part is theopportunistic.
There is generally an eventwhether it be a shutdown or
weather event or even a safetystop for a couple of hours that
we get in the morning Having agroup of changes that were
(04:44):
pre-approved, pre-understood,pre-risk, managed, we could
execute.
We basically get an outage forfree.
Michael van Rooyen (04:50):
Right right.
Brendan McCarthy (04:50):
So our outage is not high, right, so that's
how you increase youravailability, right, right?
So all of those managing timeis probably the key lesson in
working in OT.
Michael van Rooyen (05:00):
Right, right , and look, you did touch on two
really good points there.
Time, absolutely.
I mean everything's timecritical.
You want to avoid frequentoutages, ships waiting, trains
waiting and all that.
The real opportunistic time, asyou said, is when you bank
these up ready to go and you geta time to do it rather than
have to wait for a window.
It really gives you a bit of anadvantage, right?
And then you touched on theother good point which I'm happy
you clarified, because the wordOT has been bounced around a
(05:22):
lot.
Right, and it's becoming verypopular IT for OT.
But I'm really happy you talkmore about real, specific mining
operations, criticalinfrastructure operations, as
opposed to people saying OT,which is considered just OT.
Right, if you think about yourtime 22 plus years at BHP,
particularly in the end, and Ithink about 20 mines, lots of
different systems I mean I wouldhate to think of the hundreds
of systems you'd be running Withthat extensive experience.
(05:44):
What were some of the mostsignificant changes you've
observed in criticalinfrastructure in that sector
over the years?
Brendan McCarthy (05:50):
One is the move from an appliance mindset
to commoditized IT hardware.
So, even though the systemshave not changed the utilization
of commoditized hardware itused to be a Schneider appliance
but now it's an HP server.
That you would find no differentin any other location
Commoditized hardware which thencreated its own worries around
(06:13):
operating systems.
So you have to patch it.
You have to do everything youhave to do for Microsoft in the
IT space, you have to try to dothat in the OT space, and so you
have to either put controls inplace in managing the fact that
you've got the samevulnerabilities in the IT as you
have now in the OT space.
That's one.
There's two parts of automation.
(06:33):
One is obviously autonomousvehicles becomes times critical.
In a mine site of 60-odd haultrucks, there is only two
reasons why all those haultrucks would stop.
In an operated fleet it's asafety event or a weather event,
generally, no other reason,whereas in an autonomous fleet
it could be a network error, itcould be a broken antenna, it
(06:54):
could be something simple, likesomeone doing a change which has
happened inadvertently, puttinga change across all systems,
maybe out of India, andbasically I think people recall
it was a 24th of December.
Oh, wow inadvertently putting achange across all systems, maybe
out of India and basically on aI think people recall it was a-
24th of December, oh wow.
Michael van Rooyen (07:13):
And patched everything in our call assets,
oh wow, and they're not onholiday because they're still
operating, right?
No, you wouldn't have been youwouldn't call back from holiday.
Brendan McCarthy (07:18):
No, it definitely wasn't on holiday.
Michael van Rooyen (07:20):
Yes.
Brendan McCarthy (07:20):
That was never the case in the past.
Things would just operate, evenif the systems went offline
right, just on that point.
Michael van Rooyen (07:27):
Two aspects I just want to tease out there.
You're saying, first of all,the the example there of a
vendor that was providing thesystem for you.
It was an appliance.
Yeah, it was just a piece ofhardware.
Brendan McCarthy (07:36):
No one cared about what was inside it,
because it was just not.
It wasn't a windows device, itwas probably linux or who knew.
I mean in fact was a black box.
So they moved to commoditizedhardware because it was much
cheaper, like a $3,000 serverversus a $40,000 appliance.
Michael van Rooyen (07:52):
Did you see some benefits and negatives?
So, first of all, costreduction great, but then you
had to then probably take thatoperational lens on it, which
obviously has a cost.
Was the other way a bettermodel, or it was more moving
from analog to kind of digital?
What are the pros and cons ofthose two models that you saw in
your time?
Brendan McCarthy (08:09):
So obviously the pros it was much easier to
refresh and move theenvironments on.
The advantages now is theyprobably wouldn't have the
compute capacity that's requiredto run advanced process control
or autonomous systems.
You need proper two to threetier solutions to run some
process control or autonomoussystems.
You need proper two to threetier solutions to run some of
these solutions.
Michael van Rooyen (08:28):
I guess the other lens on that and we'll
talk about cybersecurity lateron, I guess the other lens on
there is then you have to thinkabout that safety blanket of
separated networks isdisappearing more and more as we
see more interconnectivityhappening and not specifically
be in your scenario, butcertainly you know other
customers I've talked to Off theback of that.
What were some of the biggestchallenges you encountered in
operational technology and howdid you approach solving them?
Brendan McCarthy (08:49):
I think the biggest challenge is with regard
to what I was referring tobefore, where there was a point
in time when control systems OTlayer so not.
The MES layer was becoming tooadvanced for the control system
engineers who the best systemthinkers on the planet.
There was no longer acapability because you went to
storage, they went to sandstorage too advanced for the
control system engineers who thebest system thinkers on the
planet.
There was no longer acapability because you went to
storage, they went to sandstorage and all the rest of it.
(09:10):
They just weren't able tosupport when the problems
occurred because the the bigprojects came in and deployed
the hardware that they liked.
But the control systemengineers are quite generous in
in their technology skills, butthey're actually good process
thinkers and understand how theprocess is working and write the
code for the machines.
(09:30):
When that became too complex,they reached out to help and up
until that point we didn'treally have a leg in.
There was don't touch this, wedo not want to be touched,
you're not going to be comingnear here until that point.
And I suppose the challenge washow do we?
Once that challenge occurredthen it was like well, these
things do need to be maintained.
(09:50):
Just like you maintain tripleVF drives and trucks and engines
, et cetera, we do need to domaintenance both on
cybersecurity but also patchmanagement and application
management, and I suppose ittook a while for me to
transition my thinking from acyber this is what we do and
this is why we do it lens tojust purely a risk management
(10:12):
lens.
So production risk managementif you don't do this, there is a
chance, obviously from a cyberperspective, that you will be
impacted If you don't do thesechanges.
For example, an autonomousfleet if we didn't reboot every
two weeks, it actually causedtwo and a half hours of outage
because we had to go to eachtruck and restart one right at
the beginning of that journey.
So we actually had apreventative maintenance plan
(10:34):
and we got that approved.
Eventually.
That trust was built up becausewe could provide data and
evidence, which is actually howthey work.
We changed our methodology andour wording.
Instead of doing patching, it'scalled planned maintenance.
When we have a breakdown, it'scalled unplanned maintenance.
Basically, by being transparentabout what happened, doing our
root cause analysis andproviding that information back
and what we've learned and whatwe've changed.
(10:55):
That's how we gained that trust.
So gaining that trust was hardbecause we had to be different.
Michael van Rooyen (11:00):
Of course.
Of course I was going to thinkabout that.
I mean, again, I just thinkabout your experience over 20
minds, the pressure ofoperations, the dollar cost of
downtime, et cetera.
So even just hearing you comein with language that's easy to
digest for people to understandwhat the difference is, I mean
that must have been sometimes ahell of a lot of pressure on you
and your team.
But that's also an educationprocess.
(11:21):
It's how to improve process,learn from mistakes, all those
ongoing continuous improvements.
Brendan McCarthy (11:25):
Parts the biggest change that I was able
to execute with the team one ofthe floors on the building that
we had put in a nice fancyscreen.
But all I did is I put theserver, the network, the
wireless network, the databaseand the applications people all
in a pod Right and I said, and Iput a leader in front of them
to say a good, major incidentmanager.
(11:45):
So rather than the ticket goingfrom each group ticket go, went
to the center, the incidentmanager and I said this is the
problem.
They all look and they allidentify that.
That, for me, was the start ofwhat I called the t-rock right
so is that something you?
Michael van Rooyen (11:59):
you created that?
The t-rock wow, wow.
So I've heard you talk about it.
That's, that's, that's exciting.
We had the i-Roc Wow, wow.
So I've heard you talk about it.
That's exciting.
Brendan McCarthy (12:05):
We had the IROC downstairs which was the
Integrated Remote OperationCenter Right and I had the
Technology Remote OperationCenter.
Michael van Rooyen (12:11):
Fair enough.
Fair enough too.
That's great.
Something new I knew.
See, I'm learning somethingeach day.
So off the back of thatextensive experience, what then
motivated you to go out and bethe founder of Evaluability
System Engineering?
Brendan McCarthy (12:29):
And how does it align with your passion for
operational technology?
My passion for operationaltechnology actually is.
It's also about production.
It's also about safety.
It's that automation where youknow that when you're at an
autonomous site and thensomebody comes from an operated
site, they're scared.
Once they go into an automatedsite, they're scared to once
they go into an automated site,they're scared to go to a manned
site or an operated sitebecause they are so predictable,
(12:50):
because you're basically robots.
Yes, same thing with thecontrol systems.
The way, I think, is probablymore failure domain management
and it's all based on risk.
So what is the risk of it goingdown and what are all the
causes for it going down?
And how do I mitigate some ofthose risks?
Sometimes it's separating thehardware, sometimes it's putting
in other controls, like, forexample, I mentioned before on
(13:13):
the 24th of December issue, theperson was able to select all
and do a change.
Wow.
So what we did is we said, well, you know what the failure
domain then was, that you'veintegrated the solution, the
patch management solution.
We created a new one, an OT one, which had limited access and
limited reach into any otherenvironments.
So you actually create processdomains as well as failure
(13:35):
domains, so that you canactually limit the impact that
people have, because change is90% of the problem.
Generally Correct, generally.
Michael van Rooyen (13:42):
Yes.
Brendan McCarthy (13:42):
The amount of money we invested in resilient
hardware.
It should never have failed.
Michael van Rooyen (13:48):
Yes.
Brendan McCarthy (13:48):
But people.
Michael van Rooyen (13:50):
People.
Generally it was people.
We see it with cybersecurity aswell.
Normally the root cause isalways a human, but adding a
process layer sounds common whenyou talk about it.
Now you know how to limit theblast radius is what you're
saying, right.
How do we really limit that?
Brendan McCarthy (14:03):
But it's on production impact, not just, of
course, because they do use iton cyber, but we used it on
failure domains to limit changeimpact as well as failure impact
.
So in a mine, you've got mining, you've got processing and
you've got logistics, you know,for the price of I don't know,
(14:23):
it's half a or $600,000 ofhardware, because there's never
an environment generally wherethe mining and the processing
goes out.
Why would you need to do achange that took them all out?
Michael van Rooyen (14:29):
Yeah, fair enough too.
Brendan McCarthy (14:30):
Good change, bad change, planned change,
unplanned change.
So you never get a window ofopportunity to do any
maintenance.
So you asked before how do wedo patch management?
We split the value domain.
So when there is opportunity inthe mining space, there's a
safe start or there's weatherevents, that's when we did our
work Opportunistic maintenance,opportunistic maintenance, so we
always had a list of things wehad to do.
Michael van Rooyen (14:52):
Yes.
Brendan McCarthy (14:52):
And they're always approved.
They're all ready to go,they're all risk managed.
Yes, and as soon as we got thewindow, we could hit it.
Michael van Rooyen (14:58):
So then, you took those lessons and really
wanted to set up an organizationthat would help customers with
that journey, taking your yearsand years of experience.
Brendan McCarthy (15:06):
I've done 20 years of learning.
I've failed, I've succeeded andI've learned.
Michael van Rooyen (15:12):
Yes.
Brendan McCarthy (15:17):
And what I've found.
I've been working in othersmaller customers at the moment,
but it's amazing that learningdoesn't have to be paid for
anymore.
Right, it's ready to go, it'sunderstood and it's economic, so
they don't have to make thesame mistakes as we progress to
more automated, more includedand more optimization as we
start to kick in, taking thatdeep domain knowledge you've
built over many, many years,know what works, what doesn't.
Michael van Rooyen (15:36):
You're really just bringing that as a
methodology or way customers canengage on wanting to cross that
divide, maybe moving towardautomation, maybe trying to
cross some of these technologydomains.
You're really offering that asa service to those customers,
right?
Brendan McCarthy (15:49):
So yeah, I just had an experience in the
last, say, eight, nine months atan autonomous site that was in
trouble, Just walking in thereknowing hang on, I've been here
before.
Michael van Rooyen (15:58):
Yes.
Brendan McCarthy (15:58):
Have you looked at this, this, this and
this?
We increased from almost 65%availability to the truck to 99.
Michael van Rooyen (16:05):
Wow, Hang on , hang on 65% availability so
people listening to the truck.
Right, right, I mean thatsounds from a tech point of view
.
You know when you start talkingnumbers like that you think wow
, but I appreciate it's a littlebit different in that industry.
But for what you just said,they're taking it from mid-60s
to 90s.
Late 90s is phenomenal, withsome of the customers that
(16:26):
you're helping assist them withthis journey.
What are the commonmisconceptions about the role
technology has in the space thatyou encounter and how do you
address them?
Brendan McCarthy (16:33):
There's always the classic it's a wireless
network and when at home, mywireless network doesn't fail
like this, right, you get that.
People say that to you.
Oh, my goodness, yes, it's alittle bit more complicated,
complicated, and that's alsothat there is no downtime
permitted, right.
So there is no maintenance,there is no requirement.
I think those two, becauseeverything else leads to the
(16:54):
fact that they have apreconceived idea of what should
work, should how it should work, and two, that it should,
should always work yes whichmeans that there's no
expectation that at some pointyou have to refresh it after
four years, right, you don't getto maintain it as you do any
other piece of equipment,because they are machines.
Michael van Rooyen (17:11):
Yes.
Brendan McCarthy (17:11):
So they do need to be maintained, of course
, and they don't ever need anupgrade.
They don't need to be touched,and the way you change that is
through using their own language, which is risk management and
the controls around how.
So obviously there's a.
The risk is that you will havean unavailable solution.
What are the controls that youneed to mitigate each of those
risks?
The other one is that it's allmaintenance debt.
(17:32):
The way you avoid that thecontrol is you need to refresh
it when it's end of life, whenit's no longer a serviceable
life.
Call it five years.
You get that permission.
It's just built into their plannow.
Michael van Rooyen (17:42):
So it's the opposite view.
Brendan McCarthy (17:43):
Opposite view.
It's their language and the waythey operate, so we use
data-driven approach to say well, this is how it's the best way
to give you the most amount ofavailability.
Michael van Rooyen (17:53):
Leading on from that, so would you also say
over the years, with thisconsumerisation of new
technology, more sensor, do youget more sensor data than you
did in the past?
Brendan McCarthy (18:02):
Absolutely.
Michael van Rooyen (18:03):
And now taking all that sensor data part
of your availability systemengineering approach is really
data-driven insights, reallytaking that additional data
points to help customers reallymind that.
Look at it.
That all adds up to the pictureright.
Brendan McCarthy (18:16):
Absolutely, so we're always looking for
anomalous behavior.
I love the cyber guys as well,by the way, so we used to use
whenever we had anomalousbehavior, we'd bring the cyber
team in and say what are youguys seeing?
Because they're obviously verydeep packet inspections so they
can actually see what's going on.
It's really important in thatT-Rock facility everybody knew
what components affected whatpart of the process.
(18:39):
Even if you don't make veryhighly redundant components
across the board, but know yoursupply chain constraint,
Everyone else can be caught up.
But there are parts in mostprocesses that are their supply
chain constraint, so the highavailability on those components
is all you need to do.
So if you are under budgetpressure, just focus on that.
Michael van Rooyen (19:03):
Okay, that's a good piece of advice.
What are your long-term goalsfor ASE and how do you envisage
the future of criticalinfrastructure technology
evolving in the space?
Brendan McCarthy (19:12):
I'd love ASE to be involved in all things
that are critical and all thattroubleshooting and improvement
I just really really enjoy Interms of where it's going.
It's only getting more complex,it's only getting more critical
and I suppose the biggest thingfor me is we talked about the
three layers the IT, the MES andI'll call it the OT.
(19:32):
For me, my perspective, that'swhat I call OT.
There's the challenge of whatis sitting in the MES space and
what is sitting in the OT space,so it's really smart to do AI.
However, when I think about AIand it's being in the non-OT
space, there's a connectionbetween do I really want
something changing the algorithmon the PLCs?
(19:55):
And I'm thinking no.
I definitely think no.
But how do we still supportthat capability?
Because it's still required andthat's where Edge really comes
in and distributed Edge.
I think that capability nowit's more commoditized, still
keeping it secure in asegregated way, but having that
optimization is the next 10years.
Michael van Rooyen (20:15):
Yeah, do you mind, just for listeners, just
briefly touching on yourthoughts around Edge and the
importance of Edge for OT.
Brendan McCarthy (20:20):
Obviously, cloud has the software that does
AI and improvement and, lookingat the whole data suite, that
capability can be delivered atthe edge, can be delivered on
compute that's sitting at a siteand it is connected directly to
the PLC.
I suppose the important thingabout AIs and control systems is
you can actually get told by AIhow to improve your algorithm
(20:45):
and then you can execute thatand embed that algorithm because
it happens It'll continue to doit every day and every week and
every month, but I think havinga restriction where it's
sitting outside in the realcloud, which is somebody else's
infrastructure, but it's thesoftware.
The software is what's reallyimportant in cloud and it's how
do you bring that to the edge?
So you're not contributing anyof your controls around risk
(21:07):
management.
Michael van Rooyen (21:08):
You were just mentioning working with the
cyber guys, because it's reallyabout anomalous behaviour.
We know that a lot of threatactors are targeting critical
infrastructure.
What do you see in the timeyou're in the space and living
and breathing in the space?
What are the unique cybercybersecurity challenges that
the sector faces and how canthey be effectively be addressed
?
Brendan McCarthy (21:26):
So I think there are two real major
differences in the risk.
There's your geopoliticalactors versus your activists.
The geopolitical are basicallytrying to attain two things.
One is price point advantage,because they can see the
negotiation information, whatyou're prepared to offer in
terms of a price versus whatthey will offer.
(21:47):
The other one is IPintellectual property around how
you are doing what you're doingso that they can do it better
without having to go through the10-year journey of trying to
get their improvement.
Then you've got the activists,so uranium obviously will have
activism, coal has got a lot ofactivism at the moment.
One's stealing IP, so you'vegot a commercial disadvantage,
but the other one is actuallytrying to stop you.
(22:08):
You need to worry about both,because they're both a
significant impact, butobviously the activists as well
as obviously they're the onesthat are basically going to stop
your production.
I've seen attempts at both.
Michael van Rooyen (22:20):
Okay.
Brendan McCarthy (22:21):
Not just attempts.
Michael van Rooyen (22:22):
Right.
Okay, you've seen the coal faceof it, right.
Thanks for that.
That's great dimensions forpeople to understand.
You know what some of themotivators are behind it and you
know cybersecurity no doubtwill be such a large focus going
forward.
One of the things you touchedon in the earlier, this
convergence of IT and OT is kindof a real industry trend at the
moment.
How is this transitionimpacting operations and
(22:44):
cybersecurity in criticalinfrastructure?
Brendan McCarthy (22:46):
I think there's multiple facets of what
it really is IT and OTintegration, both in people and
process.
And then there's technology.
So we've talked about thecommoditization of control
system hardware.
That, for me, has been done andthe skill sets required to
deliver that.
It's very the same skill setsthat happen in the IT space, but
the process and the thinkingaround supporting an OT
(23:11):
environment is actually quitedifferent.
With an IT environment, we'vealmost generally tried to
outsource the customer intimacyHelp desks, chatbots, service
now and automation.
There's no intimacy from aperson who's got a problem, them
getting it resolved.
There's an assumption thatyou've got time, whereas a
controller who's looking after60 minds and they've got a
(23:34):
problem.
They want to know that they'recommunicating with somebody who
knows what they're talking about, they know who they are, they
know what the problem probablywhat the problem is and they
know how to fix it and they'regoing to fix it quickly because
they've got no other prioritiesthan that.
That's OT you may be integratingthe capability, the technology
but the skill sets are generallythe same.
But, how you communicate, howyou engage, how you become and
(23:55):
maintain customer intimacy isdifferent.
Michael van Rooyen (23:57):
Right, just leading on your point there
around skill sets, how are youguys dealing with, or how were
you dealing with, the skillshortage?
We see a massive skill shortagein cyber, but can you give some
thoughts around what we'redealing with the skill shortage,
because no doubt criticalinfrastructure is even more
important to get the right skillsets.
Brendan McCarthy (24:16):
With the T-Rock and obviously since then
with ASC, the equivalent of theT-Rock that we created.
The commoditized services canstill be adapted.
So I don't need an OT serverperson.
I don't need an OT networkperson, Fair enough.
What I do need is someone whowe communicate and we provide
what we call decision supportaround.
(24:36):
As soon as that you want tobring up server one, two, three.
You know, because that's how weprogram the system, that that's
part of a production criticalelement.
That key role is the systemthinker and then you can
actually start to go.
People learn to become systemthinkers.
Michael van Rooyen (24:53):
Yes.
Brendan McCarthy (24:54):
But when they are only getting the server
queue, they're not the systemthinkers.
You've got a MindStar or you'vegot Modular or you've got some
fleet management solution orsome autonomous.
That's the system.
Its component is part of asystem.
It's not a piece of hardware.
It is an integral part of thesystem and you're responsible
(25:15):
for looking after that piece.
It's just not a ticket forserver one.
Michael van Rooyen (25:20):
So break the solid view and really become
system oriented.
Brendan McCarthy (25:23):
That is the biggest issue, I think, with the
way IT groups have been set up,by design, it's volume.
It's driven by how many hoursor minutes I've got it in my
queue.
Yes, you're not solving theproblem, you're just moving it
out of your queue.
In IT companies, we createdthat.
Michael van Rooyen (25:38):
Yes, we did.
Brendan McCarthy (25:38):
Because we made them choose.
Do we want to do a server role?
True, Because we made themchoose, do we?
Want to do a server role, anetwork role or an applications
role, and so that's all you evergot.
You are now part of that teamand that's all the key you've
got.
Michael van Rooyen (25:49):
Did you see an impact on skill shortages in
that area as well?
Brendan McCarthy (25:52):
So obviously in the mining space there's a
big shortage of what we callwireless techs, the people that
cable correctly and you know theco-ex.
They don't bend it, the fibre,they don't twist it.
So they always come back inabout three or four months later
.
It's oh, hang on, that didn'twork.
So on the ground out, at siteis very, very hard.
If there are any parents outthere who are looking for their
(26:15):
kids that are wanting to do arole that is going to be in
demand for the next 20 years.
Michael van Rooyen (26:20):
Wow.
Brendan McCarthy (26:20):
Because automation they all need On the
trucks there's a rack of gear.
Michael van Rooyen (26:25):
Wow.
Brendan McCarthy (26:26):
They're all Comstex.
Michael van Rooyen (26:27):
In your environment.
You've got to do it properly,because a minor mistake can
cause lots of damage.
Brendan McCarthy (26:32):
Not just the mistake.
So it's availability.
The systems are generally soadvanced that our job was to
stop them stopping.
If they don't have connectivity, they would stop.
Michael van Rooyen (26:40):
Wow Off the back of that.
Where do you see the technologyindustry heading in terms of
supporting criticalinfrastructure and kind of?
What future trends should we bewatching out for?
Brendan McCarthy (26:50):
I think cyber for me and availability are
actually going to start tocombine.
So you know I'd be competingwith you at this point to get
some cyber people, becausethey're the best system thinkers
that we have.
I think With control systemengineers.
If you could have a personthat's a cyber person and a
control system engineer, thatwould be the gift.
Michael van Rooyen (27:08):
So that should be the new CCIE chase
that become one of them.
Brendan McCarthy (27:11):
Absolutely.
When we start to think aboutavailability of systems because
that's really what's going to beoccurring, because they're
going to become more critical,it's going to be more time
sensitive, there's availabilityat normal throughput and then
you'll have the optimizationthroughput.
So, whatever you put in AI, newsensors and new IT devices that
actually make an improveddecision, when you don't have
(27:31):
that, it's going to be back to asuboptimal outcome.
The first year it was a project.
The second year was anexpectation.
There was never a backtrack onthat.
Those become more advanced asthere's more advanced, as
there's more automation, asthere's more inter-automated
solutions.
So you've got vendor one on onepoint and vendor two on another
and the integration of thosetwo systems to talk to each
(27:52):
other generally through the MESlayer is going to be more
critical and important.
Michael van Rooyen (27:58):
Yeah, I think you've touched on some
really good points and the cyberprocess.
They're about being able to bebecause they have.
They follow the methodology,how they troubleshoot.
In fact, I read a articlearound a SOC organization you
know, providing secure operationcenter, and what one of these
organizations did overseas isfound that actually nurses who'd
resigned from the covid outfall, where they decided not to do
(28:20):
that, that role anymore, thatthey became the perfect SOC
operators because they're usedto following a process.
They could be instructed prettyclearly.
They understood a process, theyknew how to do it to a T Calm
under pressure, Calm underpressure, right, because that's
what they're used to doing.
Brendan McCarthy (28:34):
I employed a couple for my major incident
management.
Michael van Rooyen (28:36):
Well, there we go.
So here's me reading about itand you've done the same and it
seems like you had the sameresult.
Getting closer to the end ofour chat, considering all the
work you've done and continue todo, you know really help
customers with their journeywhat advice would you give
professionals working incritical infrastructure?
Brendan McCarthy (28:54):
Think system.
Specializing is helpful in thefirst three or four years of
your career.
It's easier to progress whenyou're really really good at
something.
I would start pushing formoving around a little bit.
It's more important that youare across multiple functions
rather than one.
There's always going to be the10 or 15% that are going to be
the absolute brilliant in theirfield.
They should continue to do that, because we do need brilliance
(29:17):
in those fields for the newdesigns, the new architectures,
the new solutions.
But the others we probably needyou to solve problems faster.
The way you solve problemsfaster is know everything.
Michael van Rooyen (29:28):
That's a good point.
Any additional advice you'dgive to technology leaders in
that space, you know, as theyplan their technology roadmap
and investments into newsolutions?
Brendan McCarthy (29:37):
If you're going through a path into OT, be
careful what you wish for interms of.
Be clear that just taking onthe hardware is half the problem
and you've got all of theresponsibility but not actually
the accountability for solvingthe problem Joining the hardware
, I worry, where's your window?
You'll have to restart yourproduction process much more
(29:59):
regularly because you have toadhere to the IT policy that you
create from a cyber perspective, or your IT will never get to
be patched.
Michael van Rooyen (30:06):
Right, right , right.
And if I think about youpersonally, reflecting on your
career right to date, you knowwhat has been the most rewarding
aspect of working in criticalinfrastructure.
Brendan McCarthy (30:15):
I think I led a team of just under 400
employees and 200 contractorsthrough this production critical
environment.
But what I think I'm most proudof is obviously, with the T-Rog
at the center, moving our ITmindset to that time, sensitive,
risk-based approach and movingto a situation where our time
(30:40):
and our management was budgetedin the process.
Our refresh cycles was justincluded.
It wasn't, we were no longer afunction and we were able to
effectively separate the OTcosts and the IT costs.
Because for a while there, whenwe didn't do that, we were just
becoming a function and our OTcosts were doing compressed,
just as our IT was.
And you can't do customerintimacy when you're trying to
(31:03):
outsource your front end.
Michael van Rooyen (31:05):
Great insights for how you feel
rewarded about it.
Considering the work that BHPdid, you get those proud moments
of walking around thinking youknow all those things you see
constructed, built, you knowthere's obviously been resources
.
Do you have that kind offeeling sometimes?
Brendan McCarthy (31:18):
The scale at which we're operating as a
country, maybe as a state WAI'll put a flag up for WA, but I
mean even out of New SouthWales and Queensland.
The scale at which these thingsoperate is just phenomenal.
Michael van Rooyen (31:32):
Yes.
Brendan McCarthy (31:33):
I hear constantly that Australia is at
the forefront of all thecapability around technology,
mining technology specificallyand we do.
We are way ahead of any othercountry.
I think there's only one or twopilot sites for autonomy in the
US for example Wow, even thoughwe've got Caterpillar and
Komatsu.
Wow, komatsu are actually basedin the US.
(31:53):
It's amazing that we're thatfar in front.
Michael van Rooyen (31:56):
That's a really interesting data point
there to think about that.
Would there be any other keytakeaways regarding the future
of current infrastructure andthe importance of operational
technology availability that youwant to share?
Brendan McCarthy (32:06):
If you're taking on the OT and the MES
scope, you are responsible tomake your company highly
available.
Fair enough, that's how theymake money, so it's a revenue
model.
You're not stop thinking.
You're a cost model, you're arevenue model.
Michael van Rooyen (32:20):
That's a really good key takeaway.
One thing I'd like to ask allthe people that attend the
podcast is think about yourwhole 20 plus years in the
industry.
What's kind of been the mostsignificant change or shift
you've seen in your time?
Brendan McCarthy (32:37):
Wow, that's a big one.
It is a big one.
20 years, that's a lot oftechnology, the biggest shift in
availability in our autonomousfleet was LTE Right.
Michael van Rooyen (32:40):
So you said LTE would be one of your biggest
changes.
You saw.
Brendan McCarthy (32:43):
It was overnight.
It was no longer a problem.
Michael van Rooyen (32:45):
Wow.
Brendan McCarthy (32:46):
Wireless connectivity was no longer a
problem.
Really Literally, like thatOvernight.
Michael van Rooyen (32:49):
So you planned it, built it, cut over
to it and the noise went away.
Effectively.
Noise went away Wow.
Brendan McCarthy (32:55):
I'm trying very hard right now to produce a
solution that can move in anaffordable way.
If autonomous vehicles can goon LTE, the less moving parts.
Everything works.
I love the fact that there's 5Gcoming, the spectrum that's now
available for companies in thearea-wide licensing.
You can actually have your ownspectrum and you can basically
(33:15):
eliminate all the IoT devicesyou like.
It's actually available aroundthe world.
That same frequency isavailable around the world in
that small context.
But 5g is the future forconnectivity and even inside or
outside well, fantastic.
Michael van Rooyen (33:30):
Look, there's so many more discussions
we could have had you.
We only were just touching onthe surfaces.
I'd love to love to have youback for another one of these
sessions in the near future,because I'd love to deep dive
into 5g, private lte, like wehaven't even touched on some of
the other thing.
I think there's many moreconversations we're going to
have and, if you're open to it,I'd love to have your back on
the show.
So, for those who areinterested in talking to Brendan
or having a look into more ofwhat he does, I think the
(33:53):
website from him isasetechnology.
That's correct and you know ifyou have a look at it, you know
Brendan's got a great storythere and it really is about
that.
As you said, people process andtechnology and how you can
really help them with theiravailability.
Brendan, again, reallyappreciate your time catching up
with me today and I lookforward to catching up with you
next season.
Brendan McCarthy (34:10):
Thank you and thanks for listening.
Michael van Rooyen (34:12):
Thank you.
With an IT environment, we've almost
generally tried to outsource thecustomer intimacy.
You may be integrating thecapability, the technology, but
how you communicate, how youengage, how you maintain
customer intimacy is different.
Michael van Rooyen (00:17):
Today I have the pleasure of interviewing
Brendan McCarthy, who is theManaging Director for
Availability Systems Engineering.
We're going to be talking allthings operational technology,
critical infrastructure, allcybersecurity, all things that
relate to that industry.
Welcome, Brendan.
Brendan McCarthy (00:32):
Thank you for the invite.
Michael van Rooyen (00:33):
Yeah, great, I appreciate you catching up
with me here in Sydney.
We're here the Vivid Festival'son and Brendan is based in
Perth but was able to actuallycome at a time to catch up with
him here in Sydney.
So again appreciate the time Iwas able to actually coordinate
a time to catch up with him herein Sydney.
So again appreciate the time.
Before we kick things off,brendan, do you mind just
sharing a little bit about yourjourney in the industry,
particularly criticalinfrastructure and the tech side
of that, and then what led youto your role today as the
(00:55):
managing director and founder ofAvailability Systems
Engineering.
Brendan McCarthy (00:58):
Yeah, look, it's been a 25-year journey, 22
years at BHP.
Wow a 25-year journey 22 yearsat BHP Wow, when it was Western
Mining.
Then it became BHP Billiton andeven passed that through when
we exited the South 32 assets.
So we spent a lot of timeacross all of the commodities
during multiple cycles in thecommodity price.
(01:20):
So obviously budget pressuresand all the rest of it.
But uranium, copper, nickel,iron ore and that progressed
around Australia in thoseengagements.
The last five or six years Iwas looking after all of the
operational systems across BHP'sassets.
So all of the rail, all of theports, all the control systems,
(01:41):
mining, autonomous mining,everything, anything that was
making money we were lookingafter.
Michael van Rooyen (01:46):
Wow, I mean that's an extensive amount of
infrastructure right across, nodoubt, multiple mining
facilities.
Brendan McCarthy (01:51):
Yes, 20 mines, five ports.
There was always somethinghappening.
Michael van Rooyen (01:55):
Oh, no doubt , no doubt.
There was 24-7 operations, 24-7, yeah, oh wow, Across 20
locations.
Did you ever get some sleep?
Brendan McCarthy (02:02):
Oh look, we did for a while we started to
look after our petroleum assetsout of Houston and our South
American assets.
Michael van Rooyen (02:09):
Wow.
Brendan McCarthy (02:09):
And at that point it just started to get to.
I couldn't scale to that pointwith the way we were operating.
So basically we built theequivalent of what I created in
Australia, the T-Rock TechnologyRemote Operation Centre for the
24-7 support.
We built those out in Houstonin South America along with the
(02:30):
IROCs that we had.
Michael van Rooyen (02:31):
Right and was some of that related to
autonomous vehicles and allthose systems as well?
Brendan McCarthy (02:37):
So autonomous control systems which is all
automated, conveyor thepetroleum assets and plants,
even the heat bleach facilities,etc.
Michael van Rooyen (02:48):
That's just fantastic.
I mean just considering theamount of infrastructure you've
seen no doubt after that manyyears as well at BHP, and seeing
that you were the head ofproduction systems in your last
six years there at BHP.
What were some of the keyinsights that you learned, or
learning experiences, from yourtime with BHP?
Brendan McCarthy (03:01):
In the move into the, the ot, and I'm
talking real ot.
So there's three layers of ot.
There's the it world, thenthere's the mes world, which is
people call level three stack.
Then you've got level two.
When we talk ot it, we miss theopportunity to really understand
the difference between real ot,which runs plants from
(03:21):
control've got MES, which is theoptimization layer compared to
the IT space.
So in terms of the insights inmanaging MES, especially with an
IROC where supply chainmanagement is really important,
that having that availabilitywhere we manage things within
the week is really important,and on the critical environments
(03:42):
, so things that run controlsystems, autonomous vehicles,
that's in the seconds, managingtime is really important.
And the four elements that Ireally focus on is duration,
frequency, then you've gotopportunity and then you've got
sequence.
So those four elements arereally important to understand.
So duration is how long is itand how do I reduce it?
(04:03):
Frequency the more thingshappen, the more inconsistent
you are, the less predictableyou are and, as a result, the
controllers do things that arepotentially abnormal to what
they normally should do.
And it generally creates more ofan inconsistency in the
production line.
Sequencing is really importantMaking sure the right people are
there at the time when you needto execute the change.
(04:23):
Making sure there's all of thecapability, the tools, the
systems, the components so thatwhen you are having an outage
it's reduced as much as possible.
And my favorite part is theopportunistic.
There is generally an eventwhether it be a shutdown or
weather event or even a safetystop for a couple of hours that
we get in the morning Having agroup of changes that were
(04:44):
pre-approved, pre-understood,pre-risk, managed, we could
execute.
We basically get an outage forfree.
Michael van Rooyen (04:50):
Right right.
Brendan McCarthy (04:50):
So our outage is not high, right, so that's
how you increase youravailability, right, right?
So all of those managing timeis probably the key lesson in
working in OT.
Michael van Rooyen (05:00):
Right, right , and look, you did touch on two
really good points there.
Time, absolutely.
I mean everything's timecritical.
You want to avoid frequentoutages, ships waiting, trains
waiting and all that.
The real opportunistic time, asyou said, is when you bank
these up ready to go and you geta time to do it rather than
have to wait for a window.
It really gives you a bit of anadvantage, right?
And then you touched on theother good point which I'm happy
you clarified, because the wordOT has been bounced around a
(05:22):
lot.
Right, and it's becoming verypopular IT for OT.
But I'm really happy you talkmore about real, specific mining
operations, criticalinfrastructure operations, as
opposed to people saying OT,which is considered just OT.
Right, if you think about yourtime 22 plus years at BHP,
particularly in the end, and Ithink about 20 mines, lots of
different systems I mean I wouldhate to think of the hundreds
of systems you'd be running Withthat extensive experience.
(05:44):
What were some of the mostsignificant changes you've
observed in criticalinfrastructure in that sector
over the years?
Brendan McCarthy (05:50):
One is the move from an appliance mindset
to commoditized IT hardware.
So, even though the systemshave not changed the utilization
of commoditized hardware itused to be a Schneider appliance
but now it's an HP server.
That you would find no differentin any other location
Commoditized hardware which thencreated its own worries around
(06:13):
operating systems.
So you have to patch it.
You have to do everything youhave to do for Microsoft in the
IT space, you have to try to dothat in the OT space, and so you
have to either put controls inplace in managing the fact that
you've got the samevulnerabilities in the IT as you
have now in the OT space.
That's one.
There's two parts of automation.
(06:33):
One is obviously autonomousvehicles becomes times critical.
In a mine site of 60-odd haultrucks, there is only two
reasons why all those haultrucks would stop.
In an operated fleet it's asafety event or a weather event,
generally, no other reason,whereas in an autonomous fleet
it could be a network error, itcould be a broken antenna, it
(06:54):
could be something simple, likesomeone doing a change which has
happened inadvertently, puttinga change across all systems,
maybe out of India, andbasically I think people recall
it was a 24th of December.
Oh, wow inadvertently putting achange across all systems, maybe
out of India and basically on aI think people recall it was a-
24th of December, oh wow.
Michael van Rooyen (07:13):
And patched everything in our call assets,
oh wow, and they're not onholiday because they're still
operating, right?
No, you wouldn't have been youwouldn't call back from holiday.
Brendan McCarthy (07:18):
No, it definitely wasn't on holiday.
Michael van Rooyen (07:20):
Yes.
Brendan McCarthy (07:20):
That was never the case in the past.
Things would just operate, evenif the systems went offline
right, just on that point.
Michael van Rooyen (07:27):
Two aspects I just want to tease out there.
You're saying, first of all,the the example there of a
vendor that was providing thesystem for you.
It was an appliance.
Yeah, it was just a piece ofhardware.
Brendan McCarthy (07:36):
No one cared about what was inside it,
because it was just not.
It wasn't a windows device, itwas probably linux or who knew.
I mean in fact was a black box.
So they moved to commoditizedhardware because it was much
cheaper, like a $3,000 serverversus a $40,000 appliance.
Michael van Rooyen (07:52):
Did you see some benefits and negatives?
So, first of all, costreduction great, but then you
had to then probably take thatoperational lens on it, which
obviously has a cost.
Was the other way a bettermodel, or it was more moving
from analog to kind of digital?
What are the pros and cons ofthose two models that you saw in
your time?
Brendan McCarthy (08:09):
So obviously the pros it was much easier to
refresh and move theenvironments on.
The advantages now is theyprobably wouldn't have the
compute capacity that's requiredto run advanced process control
or autonomous systems.
You need proper two to threetier solutions to run some
process control or autonomoussystems.
You need proper two to threetier solutions to run some of
these solutions.
Michael van Rooyen (08:28):
I guess the other lens on that and we'll
talk about cybersecurity lateron, I guess the other lens on
there is then you have to thinkabout that safety blanket of
separated networks isdisappearing more and more as we
see more interconnectivityhappening and not specifically
be in your scenario, butcertainly you know other
customers I've talked to Off theback of that.
What were some of the biggestchallenges you encountered in
operational technology and howdid you approach solving them?
Brendan McCarthy (08:49):
I think the biggest challenge is with regard
to what I was referring tobefore, where there was a point
in time when control systems OTlayer so not.
The MES layer was becoming tooadvanced for the control system
engineers who the best systemthinkers on the planet.
There was no longer acapability because you went to
storage, they went to sandstorage too advanced for the
control system engineers who thebest system thinkers on the
planet.
There was no longer acapability because you went to
storage, they went to sandstorage and all the rest of it.
(09:10):
They just weren't able tosupport when the problems
occurred because the the bigprojects came in and deployed
the hardware that they liked.
But the control systemengineers are quite generous in
in their technology skills, butthey're actually good process
thinkers and understand how theprocess is working and write the
code for the machines.
(09:30):
When that became too complex,they reached out to help and up
until that point we didn'treally have a leg in.
There was don't touch this, wedo not want to be touched,
you're not going to be comingnear here until that point.
And I suppose the challenge washow do we?
Once that challenge occurredthen it was like well, these
things do need to be maintained.
(09:50):
Just like you maintain tripleVF drives and trucks and engines
, et cetera, we do need to domaintenance both on
cybersecurity but also patchmanagement and application
management, and I suppose ittook a while for me to
transition my thinking from acyber this is what we do and
this is why we do it lens tojust purely a risk management
(10:12):
lens.
So production risk managementif you don't do this, there is a
chance, obviously from a cyberperspective, that you will be
impacted If you don't do thesechanges.
For example, an autonomousfleet if we didn't reboot every
two weeks, it actually causedtwo and a half hours of outage
because we had to go to eachtruck and restart one right at
the beginning of that journey.
So we actually had apreventative maintenance plan
(10:34):
and we got that approved.
Eventually.
That trust was built up becausewe could provide data and
evidence, which is actually howthey work.
We changed our methodology andour wording.
Instead of doing patching, it'scalled planned maintenance.
When we have a breakdown, it'scalled unplanned maintenance.
Basically, by being transparentabout what happened, doing our
root cause analysis andproviding that information back
and what we've learned and whatwe've changed.
(10:55):
That's how we gained that trust.
So gaining that trust was hardbecause we had to be different.
Michael van Rooyen (11:00):
Of course.
Of course I was going to thinkabout that.
I mean, again, I just thinkabout your experience over 20
minds, the pressure ofoperations, the dollar cost of
downtime, et cetera.
So even just hearing you comein with language that's easy to
digest for people to understandwhat the difference is, I mean
that must have been sometimes ahell of a lot of pressure on you
and your team.
But that's also an educationprocess.
(11:21):
It's how to improve process,learn from mistakes, all those
ongoing continuous improvements.
Brendan McCarthy (11:25):
Parts the biggest change that I was able
to execute with the team one ofthe floors on the building that
we had put in a nice fancyscreen.
But all I did is I put theserver, the network, the
wireless network, the databaseand the applications people all
in a pod Right and I said, and Iput a leader in front of them
to say a good, major incidentmanager.
(11:45):
So rather than the ticket goingfrom each group ticket go, went
to the center, the incidentmanager and I said this is the
problem.
They all look and they allidentify that.
That, for me, was the start ofwhat I called the t-rock right
so is that something you?
Michael van Rooyen (11:59):
you created that?
The t-rock wow, wow.
So I've heard you talk about it.
That's, that's, that's exciting.
We had the i-Roc Wow, wow.
So I've heard you talk about it.
That's exciting.
Brendan McCarthy (12:05):
We had the IROC downstairs which was the
Integrated Remote OperationCenter Right and I had the
Technology Remote OperationCenter.
Michael van Rooyen (12:11):
Fair enough.
Fair enough too.
That's great.
Something new I knew.
See, I'm learning somethingeach day.
So off the back of thatextensive experience, what then
motivated you to go out and bethe founder of Evaluability
System Engineering?
Brendan McCarthy (12:29):
And how does it align with your passion for
operational technology?
My passion for operationaltechnology actually is.
It's also about production.
It's also about safety.
It's that automation where youknow that when you're at an
autonomous site and thensomebody comes from an operated
site, they're scared.
Once they go into an automatedsite, they're scared to once
they go into an automated site,they're scared to go to a manned
site or an operated sitebecause they are so predictable,
(12:50):
because you're basically robots.
Yes, same thing with thecontrol systems.
The way, I think, is probablymore failure domain management
and it's all based on risk.
So what is the risk of it goingdown and what are all the
causes for it going down?
And how do I mitigate some ofthose risks?
Sometimes it's separating thehardware, sometimes it's putting
in other controls, like, forexample, I mentioned before on
(13:13):
the 24th of December issue, theperson was able to select all
and do a change.
Wow.
So what we did is we said, well, you know what the failure
domain then was, that you'veintegrated the solution, the
patch management solution.
We created a new one, an OT one, which had limited access and
limited reach into any otherenvironments.
So you actually create processdomains as well as failure
(13:35):
domains, so that you canactually limit the impact that
people have, because change is90% of the problem.
Generally Correct, generally.
Michael van Rooyen (13:42):
Yes.
Brendan McCarthy (13:42):
The amount of money we invested in resilient
hardware.
It should never have failed.
Michael van Rooyen (13:48):
Yes.
Brendan McCarthy (13:48):
But people.
Michael van Rooyen (13:50):
People.
Generally it was people.
We see it with cybersecurity aswell.
Normally the root cause isalways a human, but adding a
process layer sounds common whenyou talk about it.
Now you know how to limit theblast radius is what you're
saying, right.
How do we really limit that?
Brendan McCarthy (14:03):
But it's on production impact, not just, of
course, because they do use iton cyber, but we used it on
failure domains to limit changeimpact as well as failure impact
.
So in a mine, you've got mining, you've got processing and
you've got logistics, you know,for the price of I don't know,
(14:23):
it's half a or $600,000 ofhardware, because there's never
an environment generally wherethe mining and the processing
goes out.
Why would you need to do achange that took them all out?
Michael van Rooyen (14:29):
Yeah, fair enough too.
Brendan McCarthy (14:30):
Good change, bad change, planned change,
unplanned change.
So you never get a window ofopportunity to do any
maintenance.
So you asked before how do wedo patch management?
We split the value domain.
So when there is opportunity inthe mining space, there's a
safe start or there's weatherevents, that's when we did our
work Opportunistic maintenance,opportunistic maintenance, so we
always had a list of things wehad to do.
Michael van Rooyen (14:52):
Yes.
Brendan McCarthy (14:52):
And they're always approved.
They're all ready to go,they're all risk managed.
Yes, and as soon as we got thewindow, we could hit it.
Michael van Rooyen (14:58):
So then, you took those lessons and really
wanted to set up an organizationthat would help customers with
that journey, taking your yearsand years of experience.
Brendan McCarthy (15:06):
I've done 20 years of learning.
I've failed, I've succeeded andI've learned.
Michael van Rooyen (15:12):
Yes.
Brendan McCarthy (15:17):
And what I've found.
I've been working in othersmaller customers at the moment,
but it's amazing that learningdoesn't have to be paid for
anymore.
Right, it's ready to go, it'sunderstood and it's economic, so
they don't have to make thesame mistakes as we progress to
more automated, more includedand more optimization as we
start to kick in, taking thatdeep domain knowledge you've
built over many, many years,know what works, what doesn't.
Michael van Rooyen (15:36):
You're really just bringing that as a
methodology or way customers canengage on wanting to cross that
divide, maybe moving towardautomation, maybe trying to
cross some of these technologydomains.
You're really offering that asa service to those customers,
right?
Brendan McCarthy (15:49):
So yeah, I just had an experience in the
last, say, eight, nine months atan autonomous site that was in
trouble, Just walking in thereknowing hang on, I've been here
before.
Michael van Rooyen (15:58):
Yes.
Brendan McCarthy (15:58):
Have you looked at this, this, this and
this?
We increased from almost 65%availability to the truck to 99.
Michael van Rooyen (16:05):
Wow, Hang on , hang on 65% availability so
people listening to the truck.
Right, right, I mean thatsounds from a tech point of view
.
You know when you start talkingnumbers like that you think wow
, but I appreciate it's a littlebit different in that industry.
But for what you just said,they're taking it from mid-60s
to 90s.
Late 90s is phenomenal, withsome of the customers that
(16:26):
you're helping assist them withthis journey.
What are the commonmisconceptions about the role
technology has in the space thatyou encounter and how do you
address them?
Brendan McCarthy (16:33):
There's always the classic it's a wireless
network and when at home, mywireless network doesn't fail
like this, right, you get that.
People say that to you.
Oh, my goodness, yes, it's alittle bit more complicated,
complicated, and that's alsothat there is no downtime
permitted, right.
So there is no maintenance,there is no requirement.
I think those two, becauseeverything else leads to the
(16:54):
fact that they have apreconceived idea of what should
work, should how it should work, and two, that it should,
should always work yes whichmeans that there's no
expectation that at some pointyou have to refresh it after
four years, right, you don't getto maintain it as you do any
other piece of equipment,because they are machines.
Michael van Rooyen (17:11):
Yes.
Brendan McCarthy (17:11):
So they do need to be maintained, of course
, and they don't ever need anupgrade.
They don't need to be touched,and the way you change that is
through using their own language, which is risk management and
the controls around how.
So obviously there's a.
The risk is that you will havean unavailable solution.
What are the controls that youneed to mitigate each of those
risks?
The other one is that it's allmaintenance debt.
(17:32):
The way you avoid that thecontrol is you need to refresh
it when it's end of life, whenit's no longer a serviceable
life.
Call it five years.
You get that permission.
It's just built into their plannow.
Michael van Rooyen (17:42):
So it's the opposite view.
Brendan McCarthy (17:43):
Opposite view.
It's their language and the waythey operate, so we use
data-driven approach to say well, this is how it's the best way
to give you the most amount ofavailability.
Michael van Rooyen (17:53):
Leading on from that, so would you also say
over the years, with thisconsumerisation of new
technology, more sensor, do youget more sensor data than you
did in the past?
Brendan McCarthy (18:02):
Absolutely.
Michael van Rooyen (18:03):
And now taking all that sensor data part
of your availability systemengineering approach is really
data-driven insights, reallytaking that additional data
points to help customers reallymind that.
Look at it.
That all adds up to the pictureright.
Brendan McCarthy (18:16):
Absolutely, so we're always looking for
anomalous behavior.
I love the cyber guys as well,by the way, so we used to use
whenever we had anomalousbehavior, we'd bring the cyber
team in and say what are youguys seeing?
Because they're obviously verydeep packet inspections so they
can actually see what's going on.
It's really important in thatT-Rock facility everybody knew
what components affected whatpart of the process.
(18:39):
Even if you don't make veryhighly redundant components
across the board, but know yoursupply chain constraint,
Everyone else can be caught up.
But there are parts in mostprocesses that are their supply
chain constraint, so the highavailability on those components
is all you need to do.
So if you are under budgetpressure, just focus on that.
Michael van Rooyen (19:03):
Okay, that's a good piece of advice.
What are your long-term goalsfor ASE and how do you envisage
the future of criticalinfrastructure technology
evolving in the space?
Brendan McCarthy (19:12):
I'd love ASE to be involved in all things
that are critical and all thattroubleshooting and improvement
I just really really enjoy Interms of where it's going.
It's only getting more complex,it's only getting more critical
and I suppose the biggest thingfor me is we talked about the
three layers the IT, the MES andI'll call it the OT.
(19:32):
For me, my perspective, that'swhat I call OT.
There's the challenge of whatis sitting in the MES space and
what is sitting in the OT space,so it's really smart to do AI.
However, when I think about AIand it's being in the non-OT
space, there's a connectionbetween do I really want
something changing the algorithmon the PLCs?
(19:55):
And I'm thinking no.
I definitely think no.
But how do we still supportthat capability?
Because it's still required andthat's where Edge really comes
in and distributed Edge.
I think that capability nowit's more commoditized, still
keeping it secure in asegregated way, but having that
optimization is the next 10years.
Michael van Rooyen (20:15):
Yeah, do you mind, just for listeners, just
briefly touching on yourthoughts around Edge and the
importance of Edge for OT.
Brendan McCarthy (20:20):
Obviously, cloud has the software that does
AI and improvement and, lookingat the whole data suite, that
capability can be delivered atthe edge, can be delivered on
compute that's sitting at a siteand it is connected directly to
the PLC.
I suppose the important thingabout AIs and control systems is
you can actually get told by AIhow to improve your algorithm
(20:45):
and then you can execute thatand embed that algorithm because
it happens It'll continue to doit every day and every week and
every month, but I think havinga restriction where it's
sitting outside in the realcloud, which is somebody else's
infrastructure, but it's thesoftware.
The software is what's reallyimportant in cloud and it's how
do you bring that to the edge?
So you're not contributing anyof your controls around risk
(21:07):
management.
Michael van Rooyen (21:08):
You were just mentioning working with the
cyber guys, because it's reallyabout anomalous behaviour.
We know that a lot of threatactors are targeting critical
infrastructure.
What do you see in the timeyou're in the space and living
and breathing in the space?
What are the unique cybercybersecurity challenges that
the sector faces and how canthey be effectively be addressed
?
Brendan McCarthy (21:26):
So I think there are two real major
differences in the risk.
There's your geopoliticalactors versus your activists.
The geopolitical are basicallytrying to attain two things.
One is price point advantage,because they can see the
negotiation information, whatyou're prepared to offer in
terms of a price versus whatthey will offer.
(21:47):
The other one is IPintellectual property around how
you are doing what you're doingso that they can do it better
without having to go through the10-year journey of trying to
get their improvement.
Then you've got the activists,so uranium obviously will have
activism, coal has got a lot ofactivism at the moment.
One's stealing IP, so you'vegot a commercial disadvantage,
but the other one is actuallytrying to stop you.
(22:08):
You need to worry about both,because they're both a
significant impact, butobviously the activists as well
as obviously they're the onesthat are basically going to stop
your production.
I've seen attempts at both.
Michael van Rooyen (22:20):
Okay.
Brendan McCarthy (22:21):
Not just attempts.
Michael van Rooyen (22:22):
Right.
Okay, you've seen the coal faceof it, right.
Thanks for that.
That's great dimensions forpeople to understand.
You know what some of themotivators are behind it and you
know cybersecurity no doubtwill be such a large focus going
forward.
One of the things you touchedon in the earlier, this
convergence of IT and OT is kindof a real industry trend at the
moment.
How is this transitionimpacting operations and
(22:44):
cybersecurity in criticalinfrastructure?
Brendan McCarthy (22:46):
I think there's multiple facets of what
it really is IT and OTintegration, both in people and
process.
And then there's technology.
So we've talked about thecommoditization of control
system hardware.
That, for me, has been done andthe skill sets required to
deliver that.
It's very the same skill setsthat happen in the IT space, but
the process and the thinkingaround supporting an OT
(23:11):
environment is actually quitedifferent.
With an IT environment, we'vealmost generally tried to
outsource the customer intimacyHelp desks, chatbots, service
now and automation.
There's no intimacy from aperson who's got a problem, them
getting it resolved.
There's an assumption thatyou've got time, whereas a
controller who's looking after60 minds and they've got a
(23:34):
problem.
They want to know that they'recommunicating with somebody who
knows what they're talking about, they know who they are, they
know what the problem probablywhat the problem is and they
know how to fix it and they'regoing to fix it quickly because
they've got no other prioritiesthan that.
That's OT you may be integratingthe capability, the technology
but the skill sets are generallythe same.
But, how you communicate, howyou engage, how you become and
(23:55):
maintain customer intimacy isdifferent.
Michael van Rooyen (23:57):
Right, just leading on your point there
around skill sets, how are youguys dealing with, or how were
you dealing with, the skillshortage?
We see a massive skill shortagein cyber, but can you give some
thoughts around what we'redealing with the skill shortage,
because no doubt criticalinfrastructure is even more
important to get the right skillsets.
Brendan McCarthy (24:16):
With the T-Rock and obviously since then
with ASC, the equivalent of theT-Rock that we created.
The commoditized services canstill be adapted.
So I don't need an OT serverperson.
I don't need an OT networkperson, Fair enough.
What I do need is someone whowe communicate and we provide
what we call decision supportaround.
(24:36):
As soon as that you want tobring up server one, two, three.
You know, because that's how weprogram the system, that that's
part of a production criticalelement.
That key role is the systemthinker and then you can
actually start to go.
People learn to become systemthinkers.
Michael van Rooyen (24:53):
Yes.
Brendan McCarthy (24:54):
But when they are only getting the server
queue, they're not the systemthinkers.
You've got a MindStar or you'vegot Modular or you've got some
fleet management solution orsome autonomous.
That's the system.
Its component is part of asystem.
It's not a piece of hardware.
It is an integral part of thesystem and you're responsible
(25:15):
for looking after that piece.
It's just not a ticket forserver one.
Michael van Rooyen (25:20):
So break the solid view and really become
system oriented.
Brendan McCarthy (25:23):
That is the biggest issue, I think, with the
way IT groups have been set up,by design, it's volume.
It's driven by how many hoursor minutes I've got it in my
queue.
Yes, you're not solving theproblem, you're just moving it
out of your queue.
In IT companies, we createdthat.
Michael van Rooyen (25:38):
Yes, we did.
Brendan McCarthy (25:38):
Because we made them choose.
Do we want to do a server role?
True, Because we made themchoose, do we?
Want to do a server role, anetwork role or an applications
role, and so that's all you evergot.
You are now part of that teamand that's all the key you've
got.
Michael van Rooyen (25:49):
Did you see an impact on skill shortages in
that area as well?
Brendan McCarthy (25:52):
So obviously in the mining space there's a
big shortage of what we callwireless techs, the people that
cable correctly and you know theco-ex.
They don't bend it, the fibre,they don't twist it.
So they always come back inabout three or four months later
.
It's oh, hang on, that didn'twork.
So on the ground out, at siteis very, very hard.
If there are any parents outthere who are looking for their
(26:15):
kids that are wanting to do arole that is going to be in
demand for the next 20 years.
Michael van Rooyen (26:20):
Wow.
Brendan McCarthy (26:20):
Because automation they all need On the
trucks there's a rack of gear.
Michael van Rooyen (26:25):
Wow.
Brendan McCarthy (26:26):
They're all Comstex.
Michael van Rooyen (26:27):
In your environment.
You've got to do it properly,because a minor mistake can
cause lots of damage.
Brendan McCarthy (26:32):
Not just the mistake.
So it's availability.
The systems are generally soadvanced that our job was to
stop them stopping.
If they don't have connectivity, they would stop.
Michael van Rooyen (26:40):
Wow Off the back of that.
Where do you see the technologyindustry heading in terms of
supporting criticalinfrastructure and kind of?
What future trends should we bewatching out for?
Brendan McCarthy (26:50):
I think cyber for me and availability are
actually going to start tocombine.
So you know I'd be competingwith you at this point to get
some cyber people, becausethey're the best system thinkers
that we have.
I think With control systemengineers.
If you could have a personthat's a cyber person and a
control system engineer, thatwould be the gift.
Michael van Rooyen (27:08):
So that should be the new CCIE chase
that become one of them.
Brendan McCarthy (27:11):
Absolutely.
When we start to think aboutavailability of systems because
that's really what's going to beoccurring, because they're
going to become more critical,it's going to be more time
sensitive, there's availabilityat normal throughput and then
you'll have the optimizationthroughput.
So, whatever you put in AI, newsensors and new IT devices that
actually make an improveddecision, when you don't have
(27:31):
that, it's going to be back to asuboptimal outcome.
The first year it was a project.
The second year was anexpectation.
There was never a backtrack onthat.
Those become more advanced asthere's more advanced, as
there's more automation, asthere's more inter-automated
solutions.
So you've got vendor one on onepoint and vendor two on another
and the integration of thosetwo systems to talk to each
(27:52):
other generally through the MESlayer is going to be more
critical and important.
Michael van Rooyen (27:58):
Yeah, I think you've touched on some
really good points and the cyberprocess.
They're about being able to bebecause they have.
They follow the methodology,how they troubleshoot.
In fact, I read a articlearound a SOC organization you
know, providing secure operationcenter, and what one of these
organizations did overseas isfound that actually nurses who'd
resigned from the covid outfall, where they decided not to do
(28:20):
that, that role anymore, thatthey became the perfect SOC
operators because they're usedto following a process.
They could be instructed prettyclearly.
They understood a process, theyknew how to do it to a T Calm
under pressure, Calm underpressure, right, because that's
what they're used to doing.
Brendan McCarthy (28:34):
I employed a couple for my major incident
management.
Michael van Rooyen (28:36):
Well, there we go.
So here's me reading about itand you've done the same and it
seems like you had the sameresult.
Getting closer to the end ofour chat, considering all the
work you've done and continue todo, you know really help
customers with their journeywhat advice would you give
professionals working incritical infrastructure?
Brendan McCarthy (28:54):
Think system.
Specializing is helpful in thefirst three or four years of
your career.
It's easier to progress whenyou're really really good at
something.
I would start pushing formoving around a little bit.
It's more important that youare across multiple functions
rather than one.
There's always going to be the10 or 15% that are going to be
the absolute brilliant in theirfield.
They should continue to do that, because we do need brilliance
(29:17):
in those fields for the newdesigns, the new architectures,
the new solutions.
But the others we probably needyou to solve problems faster.
The way you solve problemsfaster is know everything.
Michael van Rooyen (29:28):
That's a good point.
Any additional advice you'dgive to technology leaders in
that space, you know, as theyplan their technology roadmap
and investments into newsolutions?
Brendan McCarthy (29:37):
If you're going through a path into OT, be
careful what you wish for interms of.
Be clear that just taking onthe hardware is half the problem
and you've got all of theresponsibility but not actually
the accountability for solvingthe problem Joining the hardware
, I worry, where's your window?
You'll have to restart yourproduction process much more
(29:59):
regularly because you have toadhere to the IT policy that you
create from a cyber perspective, or your IT will never get to
be patched.
Michael van Rooyen (30:06):
Right, right , right.
And if I think about youpersonally, reflecting on your
career right to date, you knowwhat has been the most rewarding
aspect of working in criticalinfrastructure.
Brendan McCarthy (30:15):
I think I led a team of just under 400
employees and 200 contractorsthrough this production critical
environment.
But what I think I'm most proudof is obviously, with the T-Rog
at the center, moving our ITmindset to that time, sensitive,
risk-based approach and movingto a situation where our time
(30:40):
and our management was budgetedin the process.
Our refresh cycles was justincluded.
It wasn't, we were no longer afunction and we were able to
effectively separate the OTcosts and the IT costs.
Because for a while there, whenwe didn't do that, we were just
becoming a function and our OTcosts were doing compressed,
just as our IT was.
And you can't do customerintimacy when you're trying to
(31:03):
outsource your front end.
Michael van Rooyen (31:05):
Great insights for how you feel
rewarded about it.
Considering the work that BHPdid, you get those proud moments
of walking around thinking youknow all those things you see
constructed, built, you knowthere's obviously been resources
.
Do you have that kind offeeling sometimes?
Brendan McCarthy (31:18):
The scale at which we're operating as a
country, maybe as a state WAI'll put a flag up for WA, but I
mean even out of New SouthWales and Queensland.
The scale at which these thingsoperate is just phenomenal.
Michael van Rooyen (31:32):
Yes.
Brendan McCarthy (31:33):
I hear constantly that Australia is at
the forefront of all thecapability around technology,
mining technology specificallyand we do.
We are way ahead of any othercountry.
I think there's only one or twopilot sites for autonomy in the
US for example Wow, even thoughwe've got Caterpillar and
Komatsu.
Wow, komatsu are actually basedin the US.
(31:53):
It's amazing that we're thatfar in front.
Michael van Rooyen (31:56):
That's a really interesting data point
there to think about that.
Would there be any other keytakeaways regarding the future
of current infrastructure andthe importance of operational
technology availability that youwant to share?
Brendan McCarthy (32:06):
If you're taking on the OT and the MES
scope, you are responsible tomake your company highly
available.
Fair enough, that's how theymake money, so it's a revenue
model.
You're not stop thinking.
You're a cost model, you're arevenue model.
Michael van Rooyen (32:20):
That's a really good key takeaway.
One thing I'd like to ask allthe people that attend the
podcast is think about yourwhole 20 plus years in the
industry.
What's kind of been the mostsignificant change or shift
you've seen in your time?
Brendan McCarthy (32:37):
Wow, that's a big one.
It is a big one.
20 years, that's a lot oftechnology, the biggest shift in
availability in our autonomousfleet was LTE Right.
Michael van Rooyen (32:40):
So you said LTE would be one of your biggest
changes.
You saw.
Brendan McCarthy (32:43):
It was overnight.
It was no longer a problem.
Michael van Rooyen (32:45):
Wow.
Brendan McCarthy (32:46):
Wireless connectivity was no longer a
problem.
Really Literally, like thatOvernight.
Michael van Rooyen (32:49):
So you planned it, built it, cut over
to it and the noise went away.
Effectively.
Noise went away Wow.
Brendan McCarthy (32:55):
I'm trying very hard right now to produce a
solution that can move in anaffordable way.
If autonomous vehicles can goon LTE, the less moving parts.
Everything works.
I love the fact that there's 5Gcoming, the spectrum that's now
available for companies in thearea-wide licensing.
You can actually have your ownspectrum and you can basically
(33:15):
eliminate all the IoT devicesyou like.
It's actually available aroundthe world.
That same frequency isavailable around the world in
that small context.
But 5g is the future forconnectivity and even inside or
outside well, fantastic.
Michael van Rooyen (33:30):
Look, there's so many more discussions
we could have had you.
We only were just touching onthe surfaces.
I'd love to love to have youback for another one of these
sessions in the near future,because I'd love to deep dive
into 5g, private lte, like wehaven't even touched on some of
the other thing.
I think there's many moreconversations we're going to
have and, if you're open to it,I'd love to have your back on
the show.
So, for those who areinterested in talking to Brendan
or having a look into more ofwhat he does, I think the
(33:53):
website from him isasetechnology.
That's correct and you know ifyou have a look at it, you know
Brendan's got a great storythere and it really is about
that.
As you said, people process andtechnology and how you can
really help them with theiravailability.
Brendan, again, reallyappreciate your time catching up
with me today and I lookforward to catching up with you
next season.
Brendan McCarthy (34:10):
Thank you and thanks for listening.
Michael van Rooyen (34:12):
Thank you.
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
The Breakfast Club
The World's Most Dangerous Morning Show, The Breakfast Club, With DJ Envy And Charlamagne Tha God!
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.