Spill the tea - we want to hear from you!
The biggest challenges in AI implementation aren't the headline-grabbing risks but the hidden ones stemming from human factors and organizational dynamics. Drawing parallels to aviation safety, successful AI deployment requires understanding how people interact with these tools in complex organizational settings.
• Hidden AI risks often come from human-system interactions rather than technical failures
• Real examples show how AI can create burnout, enable misinformation spread, or amplify existing biases
• Standard AI safety approaches often miss these subtle but critical issues
• The "Adopt, Sustain, Optimize" framework helps track the user journey through AI implementation
• Six categories of hidden risks include quality assurance, task-tool mismatch, and workflow challenges
• Proactive "pre-mortem" approaches are more effective than waiting for problems to emerge
• Human oversight only works when people have expertise, time, and authority to challenge AI outputs
• Successful implementation requires diverse teams, tailored training, and leadership understanding
• Measuring impact should go beyond efficiency to capture quality improvements and risk management
• Building resilient sociotechnical systems means designing for human realities, not just deploying technology
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
Okay, let's unpack this.
What if I told you that thebiggest challenges in
successfully bringing AI toolsinto your organization you know,
managing the genuinely riskystuff have more in common with,
well, aviation safety checksthan with, say, a headline about
a rogue chatbot.
It's not always the dramaticcrash.
Often it's about the equivalentof a slightly miscalibrated
(00:22):
instrument, or maybecommunication missed during
turbulence.
Speaker 2 (00:25):
That parallel is exactly where the sources we're
diving into today begin.
They argue that while we focuson the big visible AI risks you
know bias, deep fakes, aihallucinations the most
significant problems in realworld implementation often stem
from far more mundane human andorganizational factors, things
(00:45):
that remain hidden until theycause an issue.
Speaker 1 (00:47):
And these insights aren't coming from a tech giant
or some sci-fi author, are they?
Speaker 2 (00:51):
Correct.
No, we're working with a stackof material from the UK Cabinet
Office, specifically thesedetailed guides and resources
developed around two keypublications the People Factor a
human-centered approach toscaling AI tools, and the
Mitigating Hidden AI RisksToolkit.
Speaker 1 (01:06):
Okay, so our mission for this deep dive is basically
to strip down these governmentdocuments, pull out the
actionable insights.
We want to understand whygetting AI tools right in
practice is maybe less about thecode and more about the people
and processes Exactly.
And crucially, how you canproactively identify and maybe
head off the risks you probablyaren't even looking for yet.
Speaker 2 (01:29):
Precisely, it's about looking past the obvious
dangers to build systems thatare truly resilient in these
messy human environments.
Speaker 1 (01:37):
Okay, let's get into that core problem.
Then we're all kind of aware ofthe high profile AI risks
algorithmic bias, deep fakes, aigenerating false information
they're real.
Algorithmic bias, deep fakes,ai generating false information
they're real, they're importantand, yeah, they grab headlines.
Speaker 2 (01:49):
And the sources don't dismiss those, definitely not.
But they pivot pretty quicklyto argue that the most common
and damaging risks in actualorganizational use they come
from elsewhere.
Speaker 1 (01:58):
Right.
Speaker 2 (01:58):
Think back to that aviation parallel.
The majority of accidents arelinked to human factors
miscommunication, errors inprocedure, fatigue, maybe
organizational culture issues,not usually a sudden engine
failure.
These sources propose AIimplementation.
Risks are well likely to followa similar pattern.
Speaker 1 (02:17):
So it's not necessarily the AI failing, but
the system around the AI, thepeople, the processes, the
culture that introduces themajor vulnerabilities.
Speaker 2 (02:27):
That's the core argument, and these are what the
sources term hidden risks,basically unintended
consequences that aren'timmediately visible or salient.
They sort of remain below thesurface until they've
potentially snowballed.
Speaker 1 (02:39):
Can you give us some examples from the sources, ones
that really illustrate this?
What does a hidden riskactually look like on the ground
?
Speaker 2 (02:46):
Yeah, the sources use several helpful scenarios.
Consider Marco His organizationautomates his routine,
seemingly menial tasks.
Logically, this should free himup for higher value work.
Speaker 1 (02:57):
Sounds good on paper.
Speaker 2 (02:58):
But the hidden insight those tasks were also
cognitive breaks.
Now Marco's day is justrelentlessly demanding, leading
to fatigue, stress and actuallyreduced productivity.
The risk materialized not fromthe AI's technical function, but
from its unexamined impact onhis workflow and well-being.
Speaker 1 (03:15):
Wow, okay, that's insidious a benefit that creates
invisible costs.
Speaker 2 (03:19):
Totally.
Or Nithya, an analyst underintense deadline pressure, she
uses a gene AI tool for urgentresearch.
The output looks plausible,seems okay and, crucially, that
time pressure prevents her fromconducting proper due diligence,
like cross-referencing withverified sources, so she shares
inaccurate information that thenspreads.
(03:39):
The risk isn't just the AImight generate errors, but how a
human's operating conditions,the pressure, the lack of time
interact with the tool to enablethe spread of bad info.
Speaker 1 (03:50):
What about Jordan, the researcher, who already has
a strong opinion?
He prompts the AI reflectinghis pre-existing views.
Speaker 2 (03:57):
Yes, that's another good one.
He asks the AI to find evidencesupporting his hypothesis.
The AI, because it's designedto be helpful and provide
relevant responses, essentiallycaters to his perspective Right.
The sources point out this canreally reinforce confirmation
bias, leading decision makers toact on an incomplete or skewed
picture.
The AI didn't invent the bias,but the interaction amplified
(04:19):
the human's existing bias.
Speaker 1 (04:21):
And the Sarah example , the hiring manager using AI to
sift applications.
Speaker 2 (04:26):
That one is critical.
The tool gives plausiblesounding scores, maybe
percentages or ranks.
Sarah trusts them.
Maybe skip some quality checksoutlined in policy because she's
busy or the scores seemobjective OK.
While algorithmic bias mightexist in the tool, the actual
harmful outcome embeddingdiscrimination into recruitment
at scale happens because of thehuman decision to trust the
(04:49):
outputs without proper checksand maybe an organizational
failure to enforce those checks.
Speaker 1 (04:58):
It's the human system interaction where the risk
really hits.
These examples reallyunderscore that it's the human
and organizational layer wherethings get well complicated.
Score that it's the human andorganizational layer where
things get well complicated andthe sources are quite direct in
saying that standard AI safetyapproaches often miss these
issues right.
Speaker 2 (05:11):
They are very direct Technical safeguards, like
improving training data or usingadversarial testing, red
teaming.
They're necessary, sure, butthey're just not sufficient.
Speaker 1 (05:19):
They won't stop Marco's burnout.
Speaker 2 (05:21):
Exactly or address a manager's misplaced trust or
Nithya's time pressure problem.
Speaker 1 (05:26):
And the common phrase human in the loop, just having
a person involved in oversight.
What do the sources say aboutthat?
Speaker 2 (05:33):
They critique this quite heavily.
It's often presented as thesolution like a magic bullet,
but it's only effective if thehuman can actually provide
effective oversight.
Speaker 1 (05:42):
Like Sarah.
Speaker 2 (05:43):
Right.
Like in Sarah's example, peoplemight lack the necessary
expertise to judge the AI orsufficient time to do it
properly, or maybe thepsychological safety or the
actual authority to challengethe AI outputs or its use, even
if they spot a potential problem.
Research mentioned in thedocuments actually shows even
experienced professionals canstruggle with effective AI
(06:05):
oversight, sometimes evenintroducing new errors
themselves.
Speaker 1 (06:09):
We're just telling people you know, read the terms
and conditions and verifyoutputs yourself.
Speaker 2 (06:14):
Yeah, they liken that to the may contain nuts warning
on a food package.
Speaker 1 (06:17):
Yeah.
Speaker 2 (06:18):
It's necessary, legally maybe, but it's
insufficient on its own as asafety system.
People often don't read them orthey forget, or they don't act
on disclaimers when they'reunder pressure or just doing
something routinely.
It doesn't build real safetyculture.
Speaker 1 (06:31):
Okay, here's where it gets really interesting.
Then the sources argue therewas a critical gap.
Plenty of frameworks existedfor technical risks for the AI
itself, but none specificallyfor systematically identifying
and mitigating these subtlebehavioral and organizational
hidden risks before they blow up.
Speaker 2 (06:48):
Exactly, and that gap is precisely what their work
aims to fill.
Speaker 1 (06:52):
So how do they propose we fill it?
They offer this core frameworkcalled Adopt, Sustain, Optimize,
ASO.
What's the thinking behind thatstructure?
Speaker 2 (07:01):
Well, the fundamental idea is that successful AI
deployment isn't just atechnical rollout.
It's a social and behavioraltransition.
It's about people changing howthey work.
Speaker 1 (07:10):
Okay.
Speaker 2 (07:11):
The ASO framework maps this transition through
three distinct stages, reallyfrom the user's perspective.
Speaker 1 (07:16):
Adopt being the first stage makes sense.
Speaker 2 (07:18):
Right.
The goal here is simple, butabsolutely essential Encourage
people to start using the tool.
An unused tool provides zerobenefit, obviously, so this
phase focuses on understandingand addressing the barriers to
just getting started.
Things like do people thinkit's useful?
Is it easy enough to use?
Do they feel confident?
What are their colleagues doing?
Do they trust it?
(07:39):
Basic AI literacy stuff too.
Speaker 1 (07:41):
And they provide ways to figure that out.
Speaker 2 (07:43):
Yeah, the supplementary materials even
provide example survey questionsto help map user profiles and
identify those specific barrierseffectively.
Speaker 1 (07:51):
Okay, so you get people using it initially.
Then comes sustain.
Speaker 2 (07:55):
Right.
The goal of sustain is to embedthe tool into routine workflow,
making sure it's usedconsistently, long-term, not
just tried once and forgotten.
Speaker 1 (08:05):
Making it a habit.
Speaker 2 (08:06):
Exactly.
This phase considers conceptslike the habit loop needing a
trigger to use the tool,receiving some kind of benefit
or relief, the reward, and thenrepeating the action.
It's about integrating the AIinto people's daily tasks so it
becomes second nature.
Speaker 1 (08:21):
And you track that.
Speaker 2 (08:22):
Definitely Metrics are key here.
Things like login frequency,interaction patterns the
supplementary evaluation guidesmention these, Plus user
feedback to see if routine useis actually happening and spot
any hurdles.
Speaker 1 (08:34):
Got it From trying it once to using it regularly, and
that leads us to the thirdstage Optimize.
Speaker 2 (08:41):
Optimize.
This is where the focus reallysharpens on high quality,
effective and safe use.
It directly tackles the hiddenrisks we talked about earlier.
Speaker 1 (08:51):
Oh OK.
Speaker 2 (08:52):
The goal isn't just that people use the tool
consistently, but that they useit well, maximizing the intended
benefits while activelyavoiding or mitigating those
unintended negative consequences.
Speaker 1 (09:02):
So optimize is where the framework for tackling these
hidden risks really leavesintended benefits, while
actively avoiding or mitigatingthose unintended negative
consequences.
Speaker 2 (09:07):
So optimize is where the framework for tackling these
hidden risks really leavesPrecisely.
It's dedicated to that phasewhere usage is established.
But the subtle risks linked tothat ongoing human AI
interaction are most likely topop up if you're not proactively
managing them.
Speaker 1 (09:17):
And they break these potential hidden problems down
into six categories.
Let's unpack those lenses theysuggest using.
Speaker 2 (09:23):
Okay, first is quality assurance risks.
This covers problems that comefrom using outputs that are
inaccurate, maybe incomplete orjust sort of mediocre without
proper checking.
Speaker 1 (09:34):
Like Nithya under pressure.
Speaker 2 (09:36):
Exactly like Nithya Time pressure causing a human to
bypass quality checks on Gen AIoutput.
The risk is less about the AIerror rate itself and more about
how human factors make usaccept or miss errors.
Speaker 1 (09:49):
Okay, makes sense.
Speaker 2 (09:50):
Second Second task tool mismatch.
This is basically using an AItool for something it wasn't
designed for or just isn't verygood at.
Speaker 1 (09:56):
Right.
Speaker 2 (09:57):
The sources highlight risks like, say, a leader
insisting a tool is used forinappropriate task and maybe
staff feeling they can't pushback.
The outcome is often suboptimal, maybe even unsafe, because the
tool just isn't the right fitfor that particular job.
Speaker 1 (10:10):
Category three perceptions, emotions and
signaling.
This gets right into the humanside, doesn't it?
Speaker 2 (10:16):
Absolutely.
This is all about risks comingfrom how people feel about the
AI rollout, maybe anxiety aboutjob security, changes in morale,
resistance to change, or evenwhat the organization's approach
signals.
If the rollout is perceived aspurely about cutting costs or
efficiency, it might signal thathuman judgment or quality isn't
valued as much, which theninfluences how people use the
(10:41):
tool maybe cutting corners.
Fourth is workflow andorganizational challenges.
This covers risks from how AIchanges the actual structure of
work or team dynamics.
Marco's burnout is a classicexample here.
The AI changed his taskdistribution in a way that
negatively impacted hiswell-being.
Another example could be if lowadoption by some people impacts
(11:02):
the workflow of others who areusing the tool, maybe creating
friction or inefficiency in aprocess that needs everyone on
board.
Speaker 1 (11:08):
Fifth is ethics.
Speaker 2 (11:10):
Right.
This category, as the sourcesdefine it, addresses risks where
AI use could lead to unfair,inequitable or harmful outcomes
that violate organizationalstandards, norms or even legal
requirements like equality laws.
Speaker 1 (11:22):
Like the SARA recruitment example.
Speaker 2 (11:24):
Exactly the SARA example of biased recruitment,
where existing algorithmic biasleads to discriminatory hiring
outcomes because of the humanand process factors around the
tools use, like skipping checks,is a key illustration they use.
Speaker 1 (11:39):
And the final category, number six.
Speaker 2 (11:41):
The final one is human connection and
technological over-reliance.
This is about risks where AIreplaces essential human roles
or interactions.
This could lead to a loss ofcritical skills over time, maybe
reduced collaboration,decreased job satisfaction or
just diminished quality ofservice in areas where human
empathy or that nuanced humanjudgment is really crucial.
(12:02):
Relying only on automatedsystems when a human touch or
expert intervention is actuallyneeded is a big risk,
highlighted here.
Speaker 1 (12:11):
So these six categories give organizations a
structured way to think aboutwhere these hidden problems
might be lurking, and thesources strongly advocate for
being proactive, right, notwaiting for something to go
wrong.
Speaker 2 (12:21):
Completely.
Their core recommendation isbasically doing a pre-mortem,
not waiting for a post-mortem.
Don't wait for the crash toretrieve the black box, so to
speak.
Use the toolkit Exactly.
Use the structured, promptquestions provided in the
toolkit for each of these sixrisk categories to brainstorm
and anticipate what could gowrong before you even deploy or
scale the AI tool widely.
Speaker 1 (12:42):
And they provide a clear like five-step process for
actually doing this pre-mortem.
Speaker 2 (12:47):
Yeah, it's designed to be practical.
Step one assemble amultidisciplinary team.
Getting diverse perspectives isabsolutely crucial for spotting
these varied risks Makes sense.
Step two systematically surfacepotential hidden risks.
Use the prompt questions, lookat existing research, maybe
analyze early pilot use of thetool.
Step three review andprioritize the identified risks,
(13:10):
probably based on likelihoodand potential severity.
Speaker 1 (13:13):
You can't fix everything at once.
Speaker 2 (13:14):
Right.
Step four develop mitigationstrategies and plan how you're
going to monitor those risks.
And step five implement ongoingmonitoring and review
mechanisms, because these riskscan change over time as people
use the tool more.
Speaker 1 (13:26):
They mention a real example the assist team.
Speaker 2 (13:28):
Yeah, the sources mention the assist team.
Using this process, theyapparently identified over 100
potential risks related to theirtool and then prioritized 33 to
actively manage.
It shows it's doable.
Speaker 1 (13:40):
That provides a real roadmap.
And going back to that human inthe loop idea for a second, the
sources didn't just dismiss itentirely, did they?
They sort of refined how itneeds to work to actually be
effective within this optimizedphase.
Speaker 2 (13:52):
Exactly they didn't throw it out, but they put
serious conditions on it.
They argue human oversight isonly a valuable mitigation
strategy if the human involvedhas three critical things.
Speaker 1 (14:02):
Okay, what are they?
Speaker 2 (14:03):
One, sufficient relevant expertise to actually
evaluate the AI's outputcritically.
Two, adequate time allocatedfor the review process, not just
expected to squeeze it inbetween other tasks.
And three, the authority or,importantly, the psychological
safety to genuinely challengethe AI's output or its proposed
use, without fear of reprisal orbeing ignored.
Speaker 1 (14:25):
Psychological safety that's key being ignored.
Psychological safety.
Speaker 2 (14:26):
That's key.
Absolutely.
Without those conditions,expecting humans to reliably
catch AI errors or inappropriateuse is itself a significant
hidden risk.
Speaker 1 (14:35):
It's false reassurance.
It shifts the focus from justhaving a human there to ensuring
the human is actually empoweredand enabled to be effective.
Okay, so how do organizationsactually implement these
frameworks and mitigate thesediverse risks?
What kinds?
Speaker 2 (14:50):
of strategies do the sources suggest?
They offer various strategiesand they're woven throughout the
guides, always linking back tothat ASO framework and the six
risk categories.
It's less about finding onesingle technical fix and much
more about systemic changes.
Speaker 1 (15:02):
Like tailored training, maybe Better training.
Speaker 2 (15:05):
Yes, but moving beyond just generic AI literacy
101, the sources advocate fordeveloping actionable training
specific to the tool and thetask.
Training that includes guidanceon recognizing and mitigating
specific risks like bias, inpractice how it might show up in
this tool, rather than just ablanket warning about AI bias in
(15:25):
general.
Speaker 1 (15:25):
Okay, and clear guidelines on how to use the
tool.
Speaker 2 (15:28):
Crucial, especially for that task tool mismatch,
risk, Clearly defining the AItool's intended use cases, its
limitations and, reallyimportantly, how to handle
situations where it's notsuitable.
This involves understanding howpeople are actually using
general purpose tools like GenAIand steering them appropriately
.
Speaker 1 (15:46):
Leadership plays a big role too, I imagine A
massive one.
Speaker 2 (15:49):
Leaders need to understand the AI tool's
capabilities and limitationsthemselves.
You know, get hands-on maybe,so they can model and advocate
for responsible, effective use.
The sources even highlightgiving leaders early access as a
specific strategy to build thisunderstanding and encourage
them to lead by example.
Speaker 1 (16:07):
And measuring the right things, not just speed.
Speaker 2 (16:09):
Yes, rigorous impact measurement that goes beyond
simple efficiency gains orself-reported time savings,
which they know can beunreliable.
Are there actual qualityimprovements?
Are risks being managedeffectively?
The sources discuss usingdiverse method surveys,
structured interviews, maybeeven quasi-experimental
approaches where feasible.
Speaker 1 (16:28):
And getting feedback from everyone, not just the
enthusiasts.
Speaker 2 (16:31):
Absolutely vital, especially for addressing those
perception risks and ensuringequitable adoption.
Actively seeking input frompeople who aren't using the tool
or are hesitant helps uncoverthose hidden barriers, concerns
and potential inequities in whobenefits or who is exposed to
risks.
It requires a truly user andnon-user centered approach.
Speaker 1 (16:51):
And none of this works if it's done in silos
right, Like just the tech teamdoing it.
Speaker 2 (16:55):
Correct.
That's a recurring theme theneed for multidisciplinary teams
throughout the AI lifecycle,bringing together the technical
experts, sure, but alsobehavioral scientists, designers
, hr people, legal experts,ethicists and, crucially, the
end users themselves, to getthat holistic view of both the
risks and the potentialsolutions.
The structure and approach tothe ASSIST team, which is
(17:16):
referenced throughout thedocuments, seems to exemplify
this.
Speaker 1 (17:19):
It feels like the sources really ground these
frameworks in reality byconstantly referencing the
experiences and findings fromthat AS that assist tools
rollout within the UK government.
You mentioned tailoringtraining based on user prompts,
using surveys on confidencealongside login data,
specifically tracking equalityimpacts.
Speaker 2 (17:37):
Exactly.
It provides concrete examplesof the frameworks and strategies
in action, showing they aren'tjust abstract theories on a page
.
They've tried to apply thisstuff.
Speaker 1 (17:46):
So what does this all mean?
Theories on a page.
They've tried to apply thisstuff.
So what does this all mean?
If we boil down this whole deepdive into the UK Cabinet
Office's perspective on AIimplementation, what's the core
takeaway for you?
Speaker 2 (17:57):
I think the most important insight is this
Successful, safe and equitableAI adoption in organizations is
fundamentally a human andorganizational challenge, not
purely a technical one.
Focusing only on the AI modelitself while neglecting the
people factor and these hiddenrisks, the messy reality of how
humans and systems interact withthe technology, is probably the
(18:19):
most significant path toundermining the very benefits AI
promises in the first place,and these sources provide
practical tools to actuallyshift that focus, to manage that
complexity.
Exactly.
They offer the ASO framework asa guide for the entire user
journey.
They give you the sixcategories of hidden risks to
systematically look for problemsin the right places.
They provide prompt questionsto make that abstract search
(18:41):
more concrete.
And they lay out a structuredprocess for identifying,
prioritizing and mitigatingthose risks proactively process
for identifying, prioritizingand mitigating those risks
proactively.
It feels like a genuine roadmapfor moving beyond just hoping
for the best towards activelydesigning for resilience.
Amy.
Speaker 1 (18:56):
Quinton.
It really reframes thechallenge, doesn't it?
From sort of fearing somedramatic AI failure the Skynet
scenario to managing the muchmore likely messy reality of
human AI interaction withincomplex systems.
For anyone listening, who'sinvolved in leading or working
with AI in an organization,understanding these dynamics
seems absolutely essential forharnessing AI's potential while
(19:18):
avoiding those less obvious butpotentially more damaging
pitfalls.
Speaker 2 (19:22):
It's really about building a robust sociotechnical
system, not just deploying apiece of technology and walking
away.
Speaker 1 (19:27):
This has been incredibly valuable, thank you.
Speaker 2 (19:29):
My pleasure, good discussion.
Speaker 1 (19:31):
As we finish this deep dive, here's maybe a
thought to carry with you.
These sources compellinglyargue that the real risks in AI
implementation often stem fromhow we, the humans inside these
complex organizations, interactwith and integrate these
powerful tools, often with thebest intentions.
Even so, if we don't activelydesign our processes, our
workflows and empower our peopleto engage with AI effectively
(19:52):
and safely, really ensuring theyhave the necessary time, the
expertise and, crucially, theability to challenge what
critical human skills or maybevital organizational dynamics
are, we most subtly putting atrisk of erosion or even failure
in the long run?
Okay, let's unpack this.
What if I told you that thebiggest challenges in
successfully bringing AI toolsinto your organization you know,
managing the genuinely riskystuff have more in common with,
well, aviation safety checksthan with, say, a headline about
a rogue chatbot.
It's not always the dramaticcrash.
Often it's about the equivalentof a slightly miscalibrated
(00:22):
instrument, or maybecommunication missed during
turbulence.
Speaker 2 (00:25):
That parallel is exactly where the sources we're
diving into today begin.
They argue that while we focuson the big visible AI risks you
know bias, deep fakes, aihallucinations the most
significant problems in realworld implementation often stem
from far more mundane human andorganizational factors, things
(00:45):
that remain hidden until theycause an issue.
Speaker 1 (00:47):
And these insights aren't coming from a tech giant
or some sci-fi author, are they?
Speaker 2 (00:51):
Correct.
No, we're working with a stackof material from the UK Cabinet
Office, specifically thesedetailed guides and resources
developed around two keypublications the People Factor a
human-centered approach toscaling AI tools, and the
Mitigating Hidden AI RisksToolkit.
Speaker 1 (01:06):
Okay, so our mission for this deep dive is basically
to strip down these governmentdocuments, pull out the
actionable insights.
We want to understand whygetting AI tools right in
practice is maybe less about thecode and more about the people
and processes Exactly.
And crucially, how you canproactively identify and maybe
head off the risks you probablyaren't even looking for yet.
Speaker 2 (01:29):
Precisely, it's about looking past the obvious
dangers to build systems thatare truly resilient in these
messy human environments.
Speaker 1 (01:37):
Okay, let's get into that core problem.
Then we're all kind of aware ofthe high profile AI risks
algorithmic bias, deep fakes, aigenerating false information
they're real.
Algorithmic bias, deep fakes,ai generating false information
they're real, they're importantand, yeah, they grab headlines.
Speaker 2 (01:49):
And the sources don't dismiss those, definitely not.
But they pivot pretty quicklyto argue that the most common
and damaging risks in actualorganizational use they come
from elsewhere.
Speaker 1 (01:58):
Right.
Speaker 2 (01:58):
Think back to that aviation parallel.
The majority of accidents arelinked to human factors
miscommunication, errors inprocedure, fatigue, maybe
organizational culture issues,not usually a sudden engine
failure.
These sources propose AIimplementation.
Risks are well likely to followa similar pattern.
Speaker 1 (02:17):
So it's not necessarily the AI failing, but
the system around the AI, thepeople, the processes, the
culture that introduces themajor vulnerabilities.
Speaker 2 (02:27):
That's the core argument, and these are what the
sources term hidden risks,basically unintended
consequences that aren'timmediately visible or salient.
They sort of remain below thesurface until they've
potentially snowballed.
Speaker 1 (02:39):
Can you give us some examples from the sources, ones
that really illustrate this?
What does a hidden riskactually look like on the ground
?
Speaker 2 (02:46):
Yeah, the sources use several helpful scenarios.
Consider Marco His organizationautomates his routine,
seemingly menial tasks.
Logically, this should free himup for higher value work.
Speaker 1 (02:57):
Sounds good on paper.
Speaker 2 (02:58):
But the hidden insight those tasks were also
cognitive breaks.
Now Marco's day is justrelentlessly demanding, leading
to fatigue, stress and actuallyreduced productivity.
The risk materialized not fromthe AI's technical function, but
from its unexamined impact onhis workflow and well-being.
Speaker 1 (03:15):
Wow, okay, that's insidious a benefit that creates
invisible costs.
Speaker 2 (03:19):
Totally.
Or Nithya, an analyst underintense deadline pressure, she
uses a gene AI tool for urgentresearch.
The output looks plausible,seems okay and, crucially, that
time pressure prevents her fromconducting proper due diligence,
like cross-referencing withverified sources, so she shares
inaccurate information that thenspreads.
(03:39):
The risk isn't just the AImight generate errors, but how a
human's operating conditions,the pressure, the lack of time
interact with the tool to enablethe spread of bad info.
Speaker 1 (03:50):
What about Jordan, the researcher, who already has
a strong opinion?
He prompts the AI reflectinghis pre-existing views.
Speaker 2 (03:57):
Yes, that's another good one.
He asks the AI to find evidencesupporting his hypothesis.
The AI, because it's designedto be helpful and provide
relevant responses, essentiallycaters to his perspective Right.
The sources point out this canreally reinforce confirmation
bias, leading decision makers toact on an incomplete or skewed
picture.
The AI didn't invent the bias,but the interaction amplified
(04:19):
the human's existing bias.
Speaker 1 (04:21):
And the Sarah example , the hiring manager using AI to
sift applications.
Speaker 2 (04:26):
That one is critical.
The tool gives plausiblesounding scores, maybe
percentages or ranks.
Sarah trusts them.
Maybe skip some quality checksoutlined in policy because she's
busy or the scores seemobjective OK.
While algorithmic bias mightexist in the tool, the actual
harmful outcome embeddingdiscrimination into recruitment
at scale happens because of thehuman decision to trust the
(04:49):
outputs without proper checksand maybe an organizational
failure to enforce those checks.
Speaker 1 (04:58):
It's the human system interaction where the risk
really hits.
These examples reallyunderscore that it's the human
and organizational layer wherethings get well complicated.
Score that it's the human andorganizational layer where
things get well complicated andthe sources are quite direct in
saying that standard AI safetyapproaches often miss these
issues right.
Speaker 2 (05:11):
They are very direct Technical safeguards, like
improving training data or usingadversarial testing, red
teaming.
They're necessary, sure, butthey're just not sufficient.
Speaker 1 (05:19):
They won't stop Marco's burnout.
Speaker 2 (05:21):
Exactly or address a manager's misplaced trust or
Nithya's time pressure problem.
Speaker 1 (05:26):
And the common phrase human in the loop, just having
a person involved in oversight.
What do the sources say aboutthat?
Speaker 2 (05:33):
They critique this quite heavily.
It's often presented as thesolution like a magic bullet,
but it's only effective if thehuman can actually provide
effective oversight.
Speaker 1 (05:42):
Like Sarah.
Speaker 2 (05:43):
Right.
Like in Sarah's example, peoplemight lack the necessary
expertise to judge the AI orsufficient time to do it
properly, or maybe thepsychological safety or the
actual authority to challengethe AI outputs or its use, even
if they spot a potential problem.
Research mentioned in thedocuments actually shows even
experienced professionals canstruggle with effective AI
(06:05):
oversight, sometimes evenintroducing new errors
themselves.
Speaker 1 (06:09):
We're just telling people you know, read the terms
and conditions and verifyoutputs yourself.
Speaker 2 (06:14):
Yeah, they liken that to the may contain nuts warning
on a food package.
Speaker 1 (06:17):
Yeah.
Speaker 2 (06:18):
It's necessary, legally maybe, but it's
insufficient on its own as asafety system.
People often don't read them orthey forget, or they don't act
on disclaimers when they'reunder pressure or just doing
something routinely.
It doesn't build real safetyculture.
Speaker 1 (06:31):
Okay, here's where it gets really interesting.
Then the sources argue therewas a critical gap.
Plenty of frameworks existedfor technical risks for the AI
itself, but none specificallyfor systematically identifying
and mitigating these subtlebehavioral and organizational
hidden risks before they blow up.
Speaker 2 (06:48):
Exactly, and that gap is precisely what their work
aims to fill.
Speaker 1 (06:52):
So how do they propose we fill it?
They offer this core frameworkcalled Adopt, Sustain, Optimize,
ASO.
What's the thinking behind thatstructure?
Speaker 2 (07:01):
Well, the fundamental idea is that successful AI
deployment isn't just atechnical rollout.
It's a social and behavioraltransition.
It's about people changing howthey work.
Speaker 1 (07:10):
Okay.
Speaker 2 (07:11):
The ASO framework maps this transition through
three distinct stages, reallyfrom the user's perspective.
Speaker 1 (07:16):
Adopt being the first stage makes sense.
Speaker 2 (07:18):
Right.
The goal here is simple, butabsolutely essential Encourage
people to start using the tool.
An unused tool provides zerobenefit, obviously, so this
phase focuses on understandingand addressing the barriers to
just getting started.
Things like do people thinkit's useful?
Is it easy enough to use?
Do they feel confident?
What are their colleagues doing?
Do they trust it?
(07:39):
Basic AI literacy stuff too.
Speaker 1 (07:41):
And they provide ways to figure that out.
Speaker 2 (07:43):
Yeah, the supplementary materials even
provide example survey questionsto help map user profiles and
identify those specific barrierseffectively.
Speaker 1 (07:51):
Okay, so you get people using it initially.
Then comes sustain.
Speaker 2 (07:55):
Right.
The goal of sustain is to embedthe tool into routine workflow,
making sure it's usedconsistently, long-term, not
just tried once and forgotten.
Speaker 1 (08:05):
Making it a habit.
Speaker 2 (08:06):
Exactly.
This phase considers conceptslike the habit loop needing a
trigger to use the tool,receiving some kind of benefit
or relief, the reward, and thenrepeating the action.
It's about integrating the AIinto people's daily tasks so it
becomes second nature.
Speaker 1 (08:21):
And you track that.
Speaker 2 (08:22):
Definitely Metrics are key here.
Things like login frequency,interaction patterns the
supplementary evaluation guidesmention these, Plus user
feedback to see if routine useis actually happening and spot
any hurdles.
Speaker 1 (08:34):
Got it From trying it once to using it regularly, and
that leads us to the thirdstage Optimize.
Speaker 2 (08:41):
Optimize.
This is where the focus reallysharpens on high quality,
effective and safe use.
It directly tackles the hiddenrisks we talked about earlier.
Speaker 1 (08:51):
Oh OK.
Speaker 2 (08:52):
The goal isn't just that people use the tool
consistently, but that they useit well, maximizing the intended
benefits while activelyavoiding or mitigating those
unintended negative consequences.
Speaker 1 (09:02):
So optimize is where the framework for tackling these
hidden risks really leavesintended benefits, while
actively avoiding or mitigatingthose unintended negative
consequences.
Speaker 2 (09:07):
So optimize is where the framework for tackling these
hidden risks really leavesPrecisely.
It's dedicated to that phasewhere usage is established.
But the subtle risks linked tothat ongoing human AI
interaction are most likely topop up if you're not proactively
managing them.
Speaker 1 (09:17):
And they break these potential hidden problems down
into six categories.
Let's unpack those lenses theysuggest using.
Speaker 2 (09:23):
Okay, first is quality assurance risks.
This covers problems that comefrom using outputs that are
inaccurate, maybe incomplete orjust sort of mediocre without
proper checking.
Speaker 1 (09:34):
Like Nithya under pressure.
Speaker 2 (09:36):
Exactly like Nithya Time pressure causing a human to
bypass quality checks on Gen AIoutput.
The risk is less about the AIerror rate itself and more about
how human factors make usaccept or miss errors.
Speaker 1 (09:49):
Okay, makes sense.
Speaker 2 (09:50):
Second Second task tool mismatch.
This is basically using an AItool for something it wasn't
designed for or just isn't verygood at.
Speaker 1 (09:56):
Right.
Speaker 2 (09:57):
The sources highlight risks like, say, a leader
insisting a tool is used forinappropriate task and maybe
staff feeling they can't pushback.
The outcome is often suboptimal, maybe even unsafe, because the
tool just isn't the right fitfor that particular job.
Speaker 1 (10:10):
Category three perceptions, emotions and
signaling.
This gets right into the humanside, doesn't it?
Speaker 2 (10:16):
Absolutely.
This is all about risks comingfrom how people feel about the
AI rollout, maybe anxiety aboutjob security, changes in morale,
resistance to change, or evenwhat the organization's approach
signals.
If the rollout is perceived aspurely about cutting costs or
efficiency, it might signal thathuman judgment or quality isn't
valued as much, which theninfluences how people use the
(10:41):
tool maybe cutting corners.
Fourth is workflow andorganizational challenges.
This covers risks from how AIchanges the actual structure of
work or team dynamics.
Marco's burnout is a classicexample here.
The AI changed his taskdistribution in a way that
negatively impacted hiswell-being.
Another example could be if lowadoption by some people impacts
(11:02):
the workflow of others who areusing the tool, maybe creating
friction or inefficiency in aprocess that needs everyone on
board.
Speaker 1 (11:08):
Fifth is ethics.
Speaker 2 (11:10):
Right.
This category, as the sourcesdefine it, addresses risks where
AI use could lead to unfair,inequitable or harmful outcomes
that violate organizationalstandards, norms or even legal
requirements like equality laws.
Speaker 1 (11:22):
Like the SARA recruitment example.
Speaker 2 (11:24):
Exactly the SARA example of biased recruitment,
where existing algorithmic biasleads to discriminatory hiring
outcomes because of the humanand process factors around the
tools use, like skipping checks,is a key illustration they use.
Speaker 1 (11:39):
And the final category, number six.
Speaker 2 (11:41):
The final one is human connection and
technological over-reliance.
This is about risks where AIreplaces essential human roles
or interactions.
This could lead to a loss ofcritical skills over time, maybe
reduced collaboration,decreased job satisfaction or
just diminished quality ofservice in areas where human
empathy or that nuanced humanjudgment is really crucial.
(12:02):
Relying only on automatedsystems when a human touch or
expert intervention is actuallyneeded is a big risk,
highlighted here.
Speaker 1 (12:11):
So these six categories give organizations a
structured way to think aboutwhere these hidden problems
might be lurking, and thesources strongly advocate for
being proactive, right, notwaiting for something to go
wrong.
Speaker 2 (12:21):
Completely.
Their core recommendation isbasically doing a pre-mortem,
not waiting for a post-mortem.
Don't wait for the crash toretrieve the black box, so to
speak.
Use the toolkit Exactly.
Use the structured, promptquestions provided in the
toolkit for each of these sixrisk categories to brainstorm
and anticipate what could gowrong before you even deploy or
scale the AI tool widely.
Speaker 1 (12:42):
And they provide a clear like five-step process for
actually doing this pre-mortem.
Speaker 2 (12:47):
Yeah, it's designed to be practical.
Step one assemble amultidisciplinary team.
Getting diverse perspectives isabsolutely crucial for spotting
these varied risks Makes sense.
Step two systematically surfacepotential hidden risks.
Use the prompt questions, lookat existing research, maybe
analyze early pilot use of thetool.
Step three review andprioritize the identified risks,
(13:10):
probably based on likelihoodand potential severity.
Speaker 1 (13:13):
You can't fix everything at once.
Speaker 2 (13:14):
Right.
Step four develop mitigationstrategies and plan how you're
going to monitor those risks.
And step five implement ongoingmonitoring and review
mechanisms, because these riskscan change over time as people
use the tool more.
Speaker 1 (13:26):
They mention a real example the assist team.
Speaker 2 (13:28):
Yeah, the sources mention the assist team.
Using this process, theyapparently identified over 100
potential risks related to theirtool and then prioritized 33 to
actively manage.
It shows it's doable.
Speaker 1 (13:40):
That provides a real roadmap.
And going back to that human inthe loop idea for a second, the
sources didn't just dismiss itentirely, did they?
They sort of refined how itneeds to work to actually be
effective within this optimizedphase.
Speaker 2 (13:52):
Exactly they didn't throw it out, but they put
serious conditions on it.
They argue human oversight isonly a valuable mitigation
strategy if the human involvedhas three critical things.
Speaker 1 (14:02):
Okay, what are they?
Speaker 2 (14:03):
One, sufficient relevant expertise to actually
evaluate the AI's outputcritically.
Two, adequate time allocatedfor the review process, not just
expected to squeeze it inbetween other tasks.
And three, the authority or,importantly, the psychological
safety to genuinely challengethe AI's output or its proposed
use, without fear of reprisal orbeing ignored.
Speaker 1 (14:25):
Psychological safety that's key being ignored.
Psychological safety.
Speaker 2 (14:26):
That's key.
Absolutely.
Without those conditions,expecting humans to reliably
catch AI errors or inappropriateuse is itself a significant
hidden risk.
Speaker 1 (14:35):
It's false reassurance.
It shifts the focus from justhaving a human there to ensuring
the human is actually empoweredand enabled to be effective.
Okay, so how do organizationsactually implement these
frameworks and mitigate thesediverse risks?
What kinds?
Speaker 2 (14:50):
of strategies do the sources suggest?
They offer various strategiesand they're woven throughout the
guides, always linking back tothat ASO framework and the six
risk categories.
It's less about finding onesingle technical fix and much
more about systemic changes.
Speaker 1 (15:02):
Like tailored training, maybe Better training.
Speaker 2 (15:05):
Yes, but moving beyond just generic AI literacy
101, the sources advocate fordeveloping actionable training
specific to the tool and thetask.
Training that includes guidanceon recognizing and mitigating
specific risks like bias, inpractice how it might show up in
this tool, rather than just ablanket warning about AI bias in
(15:25):
general.
Speaker 1 (15:25):
Okay, and clear guidelines on how to use the
tool.
Speaker 2 (15:28):
Crucial, especially for that task tool mismatch,
risk, Clearly defining the AItool's intended use cases, its
limitations and, reallyimportantly, how to handle
situations where it's notsuitable.
This involves understanding howpeople are actually using
general purpose tools like GenAIand steering them appropriately
.
Speaker 1 (15:46):
Leadership plays a big role too, I imagine A
massive one.
Speaker 2 (15:49):
Leaders need to understand the AI tool's
capabilities and limitationsthemselves.
You know, get hands-on maybe,so they can model and advocate
for responsible, effective use.
The sources even highlightgiving leaders early access as a
specific strategy to build thisunderstanding and encourage
them to lead by example.
Speaker 1 (16:07):
And measuring the right things, not just speed.
Speaker 2 (16:09):
Yes, rigorous impact measurement that goes beyond
simple efficiency gains orself-reported time savings,
which they know can beunreliable.
Are there actual qualityimprovements?
Are risks being managedeffectively?
The sources discuss usingdiverse method surveys,
structured interviews, maybeeven quasi-experimental
approaches where feasible.
Speaker 1 (16:28):
And getting feedback from everyone, not just the
enthusiasts.
Speaker 2 (16:31):
Absolutely vital, especially for addressing those
perception risks and ensuringequitable adoption.
Actively seeking input frompeople who aren't using the tool
or are hesitant helps uncoverthose hidden barriers, concerns
and potential inequities in whobenefits or who is exposed to
risks.
It requires a truly user andnon-user centered approach.
Speaker 1 (16:51):
And none of this works if it's done in silos
right, Like just the tech teamdoing it.
Speaker 2 (16:55):
Correct.
That's a recurring theme theneed for multidisciplinary teams
throughout the AI lifecycle,bringing together the technical
experts, sure, but alsobehavioral scientists, designers
, hr people, legal experts,ethicists and, crucially, the
end users themselves, to getthat holistic view of both the
risks and the potentialsolutions.
The structure and approach tothe ASSIST team, which is
(17:16):
referenced throughout thedocuments, seems to exemplify
this.
Speaker 1 (17:19):
It feels like the sources really ground these
frameworks in reality byconstantly referencing the
experiences and findings fromthat AS that assist tools
rollout within the UK government.
You mentioned tailoringtraining based on user prompts,
using surveys on confidencealongside login data,
specifically tracking equalityimpacts.
Speaker 2 (17:37):
Exactly.
It provides concrete examplesof the frameworks and strategies
in action, showing they aren'tjust abstract theories on a page
.
They've tried to apply thisstuff.
Speaker 1 (17:46):
So what does this all mean?
Theories on a page.
They've tried to apply thisstuff.
So what does this all mean?
If we boil down this whole deepdive into the UK Cabinet
Office's perspective on AIimplementation, what's the core
takeaway for you?
Speaker 2 (17:57):
I think the most important insight is this
Successful, safe and equitableAI adoption in organizations is
fundamentally a human andorganizational challenge, not
purely a technical one.
Focusing only on the AI modelitself while neglecting the
people factor and these hiddenrisks, the messy reality of how
humans and systems interact withthe technology, is probably the
(18:19):
most significant path toundermining the very benefits AI
promises in the first place,and these sources provide
practical tools to actuallyshift that focus, to manage that
complexity.
Exactly.
They offer the ASO framework asa guide for the entire user
journey.
They give you the sixcategories of hidden risks to
systematically look for problemsin the right places.
They provide prompt questionsto make that abstract search
(18:41):
more concrete.
And they lay out a structuredprocess for identifying,
prioritizing and mitigatingthose risks proactively process
for identifying, prioritizingand mitigating those risks
proactively.
It feels like a genuine roadmapfor moving beyond just hoping
for the best towards activelydesigning for resilience.
Amy.
Speaker 1 (18:56):
Quinton.
It really reframes thechallenge, doesn't it?
From sort of fearing somedramatic AI failure the Skynet
scenario to managing the muchmore likely messy reality of
human AI interaction withincomplex systems.
For anyone listening, who'sinvolved in leading or working
with AI in an organization,understanding these dynamics
seems absolutely essential forharnessing AI's potential while
(19:18):
avoiding those less obvious butpotentially more damaging
pitfalls.
Speaker 2 (19:22):
It's really about building a robust sociotechnical
system, not just deploying apiece of technology and walking
away.
Speaker 1 (19:27):
This has been incredibly valuable, thank you.
Speaker 2 (19:29):
My pleasure, good discussion.
Speaker 1 (19:31):
As we finish this deep dive, here's maybe a
thought to carry with you.
These sources compellinglyargue that the real risks in AI
implementation often stem fromhow we, the humans inside these
complex organizations, interactwith and integrate these
powerful tools, often with thebest intentions.
Even so, if we don't activelydesign our processes, our
workflows and empower our peopleto engage with AI effectively
(19:52):
and safely, really ensuring theyhave the necessary time, the
expertise and, crucially, theability to challenge what
critical human skills or maybevital organizational dynamics
are, we most subtly putting atrisk of erosion or even failure
in the long run?
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.