Are audits effective checks and verifications of our risk control systems?
Are they diving deep into the functionality and effectiveness of systems and practices, and evaluating actual daily, hazardous work?
Or, are they mostly rustling paperwork at the expense of real operational hazards?
Ref: Hutchinson, B., Dekker, S., & Rae, A. (2024). Audit masquerade: How audits provide comfort rather than treatment for serious safety problems. Safety science, 169, 106348.
Feel free to shout me a coffee to support my site &podcasts: https://buymeacoffee.com/benhutchinson
More research at SafetyInsights.Org
Intro/Output "Dark Synth Wave" by ElephantGreen(PixaBay.com)
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:07):
Imagine working somewhere you think has effective risk control
processes because the audit say so.
You see the reports, the sign offs, the procedures.
But what if those audits are actually missing the most
critical dangers? What if the story of auditing is
how checking the boxes can tragically over shadow what
(00:28):
truly keeps us safe, revealing ahidden disconnect between
paperwork and reality. Good day everyone.
I'm Ben Hutchinson and this is Safe as podcast dedicated to the
thrifty analysis of safety, riskand performance research.
Visit safetyinsights.org for more research.
So this is one of my papers Co authored with Drew Ray and
(00:50):
Sydney Decker in 2024 titled Audit Masquerade, How audits
provide comfort rather than Treatment for serious Safety
problems, published in Safety Science.
Essentially we unpacked 71 auditreports, a mixture of first,
second and third party reports from a large Australian design
and engineering, construction and maintenance company.
(01:13):
Over 16 separate and independentauditing firms were included in
the data and what we looked at was what were the audit reports
looking at. So we evaluated the corrective
actions and the observations from reports.
We wanted to know what the auditors look at, how do they
justify, what sort of evidence they use, All that sort of
questions, volume, background. While many different types of
(01:36):
audits exist, and they might have different goals and scopes,
one line of argumentation is that audit shouldn't be overly
concentrated on compliance against the administrative
aspects of the system and documentation.
Instead, audit should prioritiseaddressing tangible and
intangible factors linked to actual characteristics, states,
(01:58):
influences, objects or practisesin the workplace to enhance the
efficiency and the safety management system in achieving
the goals of the organisation. Hence we argue that a core goal
of audit really should be withinsafety critical context about
minimising decoupling. So decoupling is the distance
(02:19):
between the intended purpose or function of something like an A
safety plan versus the actual function in practise of the
artefact. So if we have a gas management
plan in a mine, decoupling wouldbe the actual management of gas
isn't being managed as expected by the plan.
Some criticism of audits is thatthey can focus on this thing
(02:40):
called surface compliance in thevery superficial elements of
their safety approaches and audits may over prioritise the
collection and review of documents.
Authors blew it in O'Keefe in their evaluation of industry
auditing, detailed how paperworkwas collected to to create an
auditable trial where the quality of information within
(03:01):
the documents was often a secondary consideration.
Also, they observed the disparity in auditing between
the paperwork that keeps people safe versus paperwork that helps
complete audits. Their words.
So let's jump in. What were the results?
Well, overall, we're looking at the corrective actions and
observations from these audit reports.
(03:22):
We found most of the findings related to solving or resolving
some sort of incomplete or missing documents, resolving
missing site signage, inspecting, placing, or
reviewing very sort of minor emergency equipment like fire
extinguishers and first aid kitsto submit or display documents
like posters or sending a register to somebody, or
(03:45):
resolving incorrect version thatwas on formatting.
We also observed that most of the findings were either only
moderately or weakly connected to some sort of physical issue
or hazard. In other words, only 16% of the
findings were strongly connectedto some sort of physical issue
or hazard. There was virtually no examples
of elimination or the higher order controls in the hierarchy.
(04:07):
And virtually all of the corrective actions which we said
there was a strong connection with a physical issue or has it
on site focused on rectifying sort of immediate or incidental
physical conditions like slips and trips.
What we did find though was audits with specific themes like
electrical or traffic audits tended to derive slightly better
quality corrective actions or atleast better quality in the
(04:30):
sense that they're more directlyconnected to some sort of
physical issue or physical hazard.
And not surprisingly, most of the corrective actions that were
rated as weak, weak is as far asthey had only a very indirect or
weak connection to solving some sort of physical issue or
hazard. Actually all of them were
administrative. So there was really wasn't a
(04:51):
clear link between the administrative action on solving
some sort of tangible issue or hazard.
We also looked at audits that unpacked communication on side
and what it was found is that even audits that were geared
towards auditing communication practises, they very rarely
evaluated actual communication on side.
And a total of 35 corrective actions or observations focused
(05:14):
on communication. Only seven of them commented on
either the content of the communication, the effectiveness
of the communication or the quality and retention of the
communicate information. Hence, most audit findings focus
solely on the outputs of communication like a signed
toolbox talk or pre start or prestart attendance sheets or that
(05:36):
the content was displayed on a notice board.
So actually all of them didn't assess the quality or content of
the communication. This suggests that audits might
prioritise verification of artefacts documents over
assessing the quality or contentof information.
Also, in this sample, we found that actions of communication
(05:57):
were used to address issues thatreally didn't relate to
communication. For instance, gaps in risk
assessments were addressed via toolbox talk rather than
addressing the risk assessment itself.
So the audits effectively rein in this decoupling effect.
Well, based on these findings, no, not really.
The audits rarely dug beneath very superficial matters of
(06:20):
documentation and system administration.
Even when auditing did focus on operational work and and
operational hazards, the emphasis really shifted to the
remediation of very trivial or incidental things like the slips
and trips and cables. And so it really appears that
audits myopically take a find and fix approach, focus on
(06:43):
immediate side issues rather than a deeper systematic focus
on learning and improvement. Also, the results suggest that
there may be a type of masquerade, a symbolic activity
configured for surface documentsand practises and is unable or
unwilling to ask substantive andcritical questions about risk
and work. Again, only 16% of the 327
(07:08):
corrective actions that were assigned to really improve
health and safety management were only linked to a physical
source of harm or a hazard in 16% of all the actions with the
remaining corrective actions really just wake or moderately
connected to physical issues. So in discussing the findings,
we point out that audits in thissample focused on surface
(07:30):
compliance and there was 2 variations of surface
compliance. 1 is we called an illusion of depth.
Audits were found to address immediate side issues you have.
We're implementing that missing lifting equipment register
without calling for the investigation of the underlying
causal factors. So why did that issue exist in
the 1st place? Didn't seem to ever get asked.
(07:51):
Additionally, no instances were observed of corrective actions
directing the oddities to systematically rectify a family
of issues. So how does the whole business
learn? We also propose that artefacts
take on the guise of the issue. So audits frequently focused on
revising some sort of document without accurately addressing
(08:12):
the underlying physical issues. So solving the paperwork was
conflated with solving the actual issue.
We also found that audits couldn't really identify
particular issues, couldn't or wouldn't.
There was a lack of connection between requirements and
specifications. So audits were found to focus a
lot on minor specifications likesignatures, templates, version
(08:34):
numbers, but weren't so well configured to focus on the
requirements. You know, what's the actual
purpose of that process? What is it trying to achieve?
But to be clear, we did find benefits of auditing.
It doesn't mean audits didn't have some sort of value or
purpose. Audits were demonstrably
effective at identifying readilyobservable site conditions and
(08:54):
lower tier hazards. They're also effective at
verifying the presence of documented systems and
deliverables. And they'll likely have other
benefits that we couldn't measure in this study.
So overall, this evidence highlights AT audits focused
largely on evaluating and modifying documents, conflating
the presence of a document as evidence of the effectiveness of
(09:16):
the process, even though they'renot the same thing.
So audits focused on surface compliance activities and lacked
a focus on critical components and critical risks.
These findings suggest that someauditing approaches may be at
risk of tweaking documents or addressing insignificant
physical issues at the expense of properly addressing critical
(09:37):
issues or hazards. So audits could be a false
veneer that the issues are beingproperly managed, the audit
masquerade. What can we make of these
findings? There's way too much to go
through here. What some simple clear
reflections for me is being clear about what we expect from
audits in the 1st place. For instance, is that audit,
(10:01):
that particular type of audit, purely a desktop documentation
check? If it is, there's likely to be
no real misalignment with goals.It's probably doing what we
expected to do. If you ask 20 of your leaders in
the business what they think audits are doing, I bet
someone's going to say they think the audits are verifying
the effectiveness of our risk systems.
(10:23):
If that's the case, then that needs to be firmly investigated
and challenged because your audits may not be doing that.
Now, of course, there's limitations, some of them, even
though we did have 16 separate and independent audit companies
as part of the data source, generalizability still has to be
considered. Also importantly, it's based on
(10:44):
document analysis by a person who wasn't present for those
audits, so we don't have all of the context discussed during the
audit. That's it on SAFE.
As I'm Ben Hutchinson, if you found this useful then please
share rate and review and checkoutsafetyinsights.org for
more research.
Imagine working somewhere you think has effective risk control
processes because the audit say so.
You see the reports, the sign offs, the procedures.
But what if those audits are actually missing the most
critical dangers? What if the story of auditing is
how checking the boxes can tragically over shadow what
(00:28):
truly keeps us safe, revealing ahidden disconnect between
paperwork and reality. Good day everyone.
I'm Ben Hutchinson and this is Safe as podcast dedicated to the
thrifty analysis of safety, riskand performance research.
Visit safetyinsights.org for more research.
So this is one of my papers Co authored with Drew Ray and
(00:50):
Sydney Decker in 2024 titled Audit Masquerade, How audits
provide comfort rather than Treatment for serious Safety
problems, published in Safety Science.
Essentially we unpacked 71 auditreports, a mixture of first,
second and third party reports from a large Australian design
and engineering, construction and maintenance company.
(01:13):
Over 16 separate and independentauditing firms were included in
the data and what we looked at was what were the audit reports
looking at. So we evaluated the corrective
actions and the observations from reports.
We wanted to know what the auditors look at, how do they
justify, what sort of evidence they use, All that sort of
questions, volume, background. While many different types of
(01:36):
audits exist, and they might have different goals and scopes,
one line of argumentation is that audit shouldn't be overly
concentrated on compliance against the administrative
aspects of the system and documentation.
Instead, audit should prioritiseaddressing tangible and
intangible factors linked to actual characteristics, states,
(01:58):
influences, objects or practisesin the workplace to enhance the
efficiency and the safety management system in achieving
the goals of the organisation. Hence we argue that a core goal
of audit really should be withinsafety critical context about
minimising decoupling. So decoupling is the distance
(02:19):
between the intended purpose or function of something like an A
safety plan versus the actual function in practise of the
artefact. So if we have a gas management
plan in a mine, decoupling wouldbe the actual management of gas
isn't being managed as expected by the plan.
Some criticism of audits is thatthey can focus on this thing
(02:40):
called surface compliance in thevery superficial elements of
their safety approaches and audits may over prioritise the
collection and review of documents.
Authors blew it in O'Keefe in their evaluation of industry
auditing, detailed how paperworkwas collected to to create an
auditable trial where the quality of information within
(03:01):
the documents was often a secondary consideration.
Also, they observed the disparity in auditing between
the paperwork that keeps people safe versus paperwork that helps
complete audits. Their words.
So let's jump in. What were the results?
Well, overall, we're looking at the corrective actions and
observations from these audit reports.
(03:22):
We found most of the findings related to solving or resolving
some sort of incomplete or missing documents, resolving
missing site signage, inspecting, placing, or
reviewing very sort of minor emergency equipment like fire
extinguishers and first aid kitsto submit or display documents
like posters or sending a register to somebody, or
(03:45):
resolving incorrect version thatwas on formatting.
We also observed that most of the findings were either only
moderately or weakly connected to some sort of physical issue
or hazard. In other words, only 16% of the
findings were strongly connectedto some sort of physical issue
or hazard. There was virtually no examples
of elimination or the higher order controls in the hierarchy.
(04:07):
And virtually all of the corrective actions which we said
there was a strong connection with a physical issue or has it
on site focused on rectifying sort of immediate or incidental
physical conditions like slips and trips.
What we did find though was audits with specific themes like
electrical or traffic audits tended to derive slightly better
quality corrective actions or atleast better quality in the
(04:30):
sense that they're more directlyconnected to some sort of
physical issue or physical hazard.
And not surprisingly, most of the corrective actions that were
rated as weak, weak is as far asthey had only a very indirect or
weak connection to solving some sort of physical issue or
hazard. Actually all of them were
administrative. So there was really wasn't a
(04:51):
clear link between the administrative action on solving
some sort of tangible issue or hazard.
We also looked at audits that unpacked communication on side
and what it was found is that even audits that were geared
towards auditing communication practises, they very rarely
evaluated actual communication on side.
And a total of 35 corrective actions or observations focused
(05:14):
on communication. Only seven of them commented on
either the content of the communication, the effectiveness
of the communication or the quality and retention of the
communicate information. Hence, most audit findings focus
solely on the outputs of communication like a signed
toolbox talk or pre start or prestart attendance sheets or that
(05:36):
the content was displayed on a notice board.
So actually all of them didn't assess the quality or content of
the communication. This suggests that audits might
prioritise verification of artefacts documents over
assessing the quality or contentof information.
Also, in this sample, we found that actions of communication
(05:57):
were used to address issues thatreally didn't relate to
communication. For instance, gaps in risk
assessments were addressed via toolbox talk rather than
addressing the risk assessment itself.
So the audits effectively rein in this decoupling effect.
Well, based on these findings, no, not really.
The audits rarely dug beneath very superficial matters of
(06:20):
documentation and system administration.
Even when auditing did focus on operational work and and
operational hazards, the emphasis really shifted to the
remediation of very trivial or incidental things like the slips
and trips and cables. And so it really appears that
audits myopically take a find and fix approach, focus on
(06:43):
immediate side issues rather than a deeper systematic focus
on learning and improvement. Also, the results suggest that
there may be a type of masquerade, a symbolic activity
configured for surface documentsand practises and is unable or
unwilling to ask substantive andcritical questions about risk
and work. Again, only 16% of the 327
(07:08):
corrective actions that were assigned to really improve
health and safety management were only linked to a physical
source of harm or a hazard in 16% of all the actions with the
remaining corrective actions really just wake or moderately
connected to physical issues. So in discussing the findings,
we point out that audits in thissample focused on surface
(07:30):
compliance and there was 2 variations of surface
compliance. 1 is we called an illusion of depth.
Audits were found to address immediate side issues you have.
We're implementing that missing lifting equipment register
without calling for the investigation of the underlying
causal factors. So why did that issue exist in
the 1st place? Didn't seem to ever get asked.
(07:51):
Additionally, no instances were observed of corrective actions
directing the oddities to systematically rectify a family
of issues. So how does the whole business
learn? We also propose that artefacts
take on the guise of the issue. So audits frequently focused on
revising some sort of document without accurately addressing
(08:12):
the underlying physical issues. So solving the paperwork was
conflated with solving the actual issue.
We also found that audits couldn't really identify
particular issues, couldn't or wouldn't.
There was a lack of connection between requirements and
specifications. So audits were found to focus a
lot on minor specifications likesignatures, templates, version
(08:34):
numbers, but weren't so well configured to focus on the
requirements. You know, what's the actual
purpose of that process? What is it trying to achieve?
But to be clear, we did find benefits of auditing.
It doesn't mean audits didn't have some sort of value or
purpose. Audits were demonstrably
effective at identifying readilyobservable site conditions and
(08:54):
lower tier hazards. They're also effective at
verifying the presence of documented systems and
deliverables. And they'll likely have other
benefits that we couldn't measure in this study.
So overall, this evidence highlights AT audits focused
largely on evaluating and modifying documents, conflating
the presence of a document as evidence of the effectiveness of
(09:16):
the process, even though they'renot the same thing.
So audits focused on surface compliance activities and lacked
a focus on critical components and critical risks.
These findings suggest that someauditing approaches may be at
risk of tweaking documents or addressing insignificant
physical issues at the expense of properly addressing critical
(09:37):
issues or hazards. So audits could be a false
veneer that the issues are beingproperly managed, the audit
masquerade. What can we make of these
findings? There's way too much to go
through here. What some simple clear
reflections for me is being clear about what we expect from
audits in the 1st place. For instance, is that audit,
(10:01):
that particular type of audit, purely a desktop documentation
check? If it is, there's likely to be
no real misalignment with goals.It's probably doing what we
expected to do. If you ask 20 of your leaders in
the business what they think audits are doing, I bet
someone's going to say they think the audits are verifying
the effectiveness of our risk systems.
(10:23):
If that's the case, then that needs to be firmly investigated
and challenged because your audits may not be doing that.
Now, of course, there's limitations, some of them, even
though we did have 16 separate and independent audit companies
as part of the data source, generalizability still has to be
considered. Also importantly, it's based on
(10:44):
document analysis by a person who wasn't present for those
audits, so we don't have all of the context discussed during the
audit. That's it on SAFE.
As I'm Ben Hutchinson, if you found this useful then please
share rate and review and checkoutsafetyinsights.org for
more research.
Popular Podcasts
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.
The Clay Travis and Buck Sexton Show
The Clay Travis and Buck Sexton Show. Clay Travis and Buck Sexton tackle the biggest stories in news, politics and current events with intelligence and humor. From the border crisis, to the madness of cancel culture and far-left missteps, Clay and Buck guide listeners through the latest headlines and hot topics with fun and entertaining conversations and opinions.