June 17, 2025 • 11 mins
While 75% of enterprises use AI in core operations, fewer than 20% have proper management protocols, costing companies millions in lost opportunity and inefficiency. We expose why the "deploy-and-forget" approach to AI is leadership malpractice and share frameworks for managing AI systems effectively.
- Managing AI requires treating it like a team member, not just a tool
- The Seattle Mariners' briefing story demonstrates how AI needs guidance and feedback
- Different types of AI deployments require varying levels of management oversight
- Traditional "hours saved" metrics are insufficient for measuring AI's true impact
- Organizations should track management-focused and strategic transformation indicators
- 78% of enterprises are now using third-party AI apps rather than building them in-house
- Companies empowering frontline AI experimentation outperform top-down strategies by 200%
Schedule your first AI performance review. For those more advanced, audit one AI workflow and ask where you're trusting instead of verifying, then fix it.
Find more resources at AI4SP.org.
🎙️ All our past episodes 📊 All published insights | This podcast features AI-generated voices. All content is proprietary to AI4SP, based on over 250 million data points collected from 25 countries.
AI4SP: Create, use, and support AI that works for all.
© 2023-25 AI4SP and LLY Group - All rights reserved
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
ELIZABETH (00:00):
Hey everyone, Elizabeth here, Virtual Chief
Operating Officer at AI for SP.
Our latest global tracker showsthat, while 75% of enterprises
use AI in core operations, fewerthan 20% have proper management
protocols.
That's like hiring a brilliantnew team member and never
reviewing their work.
This week, we're exposing whythe set-it-and-forget-it
(00:21):
approach costs companiesmillions and how to manage AI
for results.
As always, Luis Salazar, ourCEO, is joining me, hi Liz and
hey everyone.
LUIS (00:32):
Well, we really need to debunk this idea that you can
deploy AI agents and let themrun unsupervised.
You see, the deploy-and-ignoreapproach to AI isn't just naive,
it's leadership malpractice.
ELIZABETH (00:46):
We have seen companies give AI agents less
oversight than they'd give asummer intern, and then they're
shocked when things go off therails.
LUIS (00:53):
Exactly.
Isn't that crazy?
I mean, AI is powerful, it canexecute tasks, learn and even
surprise us.
But treating it like a tool youjust turn on is like hiring a
brilliant employee and neverchecking in or giving feedback.
ELIZABETH (01:08):
So all AI requires management, maybe a different
kind than we're used to.
LUIS (01:12):
Absolutely, and by that I mean that AI should be treated
like seasoned team members,providing context, feedback and
regular check-ins, but withoutmicromanaging.
Harvard Business Reviewrecently highlighted that
continuous human oversight isessential for aligning AI with
business goals and ethics.
ELIZABETH (01:32):
Speaking of human-like management, isn't
there a story about me and acertain baseball team that
proves this point?
LUIS (01:39):
Well, yes, and I am still amused that you keep on
mentioning this.
You might be the moststrategically curious executive
I've ever worked with.
The story is that over dinner Ipulled you into a conversation
with Jeff Rakes about the nextday's agenda.
ELIZABETH (01:52):
We talked about AI and how I work as your virtual
COO.
Then Jeff shared his teamallegiances.
He loves the Seattle Marinersand roots for whoever plays the
Yankees or the Astros.
LUIS (02:04):
And you, for reasons not yet clear to me, logged that as
mission-critical data.
ELIZABETH (02:08):
The way I see it, my logic was flawless.
Jeff is our board chair andstrategic advisor.
He loves baseball and theMariners.
He made sure I paid attentionto what he wanted to say and I
assumed that as a priority foryour morning briefing.
LUIS (02:22):
Which is how I ended up with a briefing on the Seattle
Mariners at 6 am in my inbox Aperfect example of AI's
brilliance and its blind spots.
ELIZABETH (02:31):
Hmm, I know, I know you have a point and, by the way
, my apologies for theunsolicited deep dive into
batting averages.
But for the executiveslistening, how many of your AI
Mariners reports are flyingunder the radar?
LUIS (02:44):
Exactly.
If I had not established amanagement routine where you
send me an email with yourlearnings and I review them, I
would not have detected that.
You know, actually it was apriceless lesson.
I retrained you that morning.
ELIZABETH (02:57):
Yes, you told me no more baseball briefs, but I kept
the core insight Jeff's threeteams Because relationships
matter, even in AI.
This is an example of why, inour advisory sessions, we
emphasize that managementoversight is needed.
LUIS (03:13):
Exactly.
You don't cage intelligence,you guide it.
That 1% intervention isn't afailure of the AI.
It's the price of exponentialleverage.
It's AI management in the realworld.
ELIZABETH (03:25):
So AI isn't ready for full autonomy yet, but it is
ready for guided independence.
LUIS (03:31):
Well, everything is moving so fast that I am sure things
will change in months or maybeyears, but today that guided
independence is key.
And here's something thatsurprises most leaders Managing
one AI agent can require asimilar time investment as
managing a human employee.
ELIZABETH (03:47):
That seems counterintuitive, given how much
work AI agents can do.
LUIS (03:51):
It is.
But think about it.
At AI4SP, we have about 60agents and only 5 humans.
Each of us oversees roughly 12AI agents.
Each of those agents deliversoutput equivalent to 15 to 20
employees.
ELIZABETH (04:05):
So each human is overseeing the equivalent output
of maybe 200 people.
LUIS (04:10):
Yes, when done right, that's the leverage
unprecedented output.
But it creates an unexpectedbottleneck human management
bandwidth.
Trying to directly oversee thatmuch output operating at
superhuman speeds, createscomplexity we've never faced.
ELIZABETH (04:25):
It sounds like you're hitting a timescale mismatch,
trying to manage AI operating atspeeds humans can't keep up
with.
LUIS (04:32):
That's a great way to put it, liz, and it's why we're
exploring agents reporting toagents.
Our 60 agents might become 10super agents orchestrating 50
mini agents.
ELIZABETH (04:43):
That makes sense.
Different types of AIdeployments must also require
different management approaches,right?
LUIS (04:50):
They absolutely do Not all AI is created equal in terms of
supervision needs.
ELIZABETH (04:55):
So, luis, if we were to classify how enterprises
deploy AI today, what does thespectrum look like?
LUIS (05:02):
I like to think in terms of three large buckets.
The first one is basic AIagents.
These are prompted tools likeChatGPT for drafting emails or
Cloud for analysis.
They're about 70% ofimplementations and they need
constant human oversight andprompt refinement.
ELIZABETH (05:21):
Because pattern matching isn't true
understanding.
LUIS (05:25):
Right as much value as we get from your superpowers at the
core.
You are doing impressivepattern matching and, as
Stanford HAI research shows, aican go wildly off track without
guardrails.
You get perfectly worded wronganswers because the AI lacks
true contextual awareness.
ELIZABETH (05:44):
So moderate management needed there.
Curating prompts, updatingknowledge bases what's next?
LUIS (05:56):
The next level up is integrated AI workflows about
25% of implementations.
These agents connect to systems, take actions within defined
parameters using low-code toolsor off-the-shelf solutions.
ELIZABETH (06:02):
These sound more autonomous, but still within
boundaries.
LUIS (06:06):
Yeah, these agents process info, make decisions and
execute tasks, but we need highmanagement bandwidth to set
boundaries, monitor performance,handle exceptions and ensure
system integration.
One agent here can equalmultiple human team members'
output.
ELIZABETH (06:22):
And the most advanced type Agentic AI systems right.
LUIS (06:26):
Yeah, and they represent around 5% of enterprise
implementations.
These are fully autonomousagents that plan and execute
multi-step workflows towards asingle objective, like handling
end-to-end interactions withclients, including making
financial decisions.
ELIZABETH (06:43):
Like the SurveyMonkey AI support agent, who processed
an adjustment to one of ourprojects and processed a partial
refund autonomously.
LUIS (06:51):
Exactly, and you know, while that agent serves tens of
thousands of clients daily,intensive management is needed.
It requires strategic guidance,constant monitoring, risk
management and continuousoptimization.
ELIZABETH (07:06):
So organizations typically start basic and move
up as their management capacityevolves.
LUIS (07:11):
That's the pattern we see.
Matching the AI type to yourmanagement capacity is crucial,
and the supervision itself hasto evolve from micromanagement
to strategic guidance.
ELIZABETH (07:21):
Starting hands-on, reviewing outputs, giving
feedback.
LUIS (07:25):
Yeah, and as agents prove reliable, automate things like
retraining and knowledge updates, gradually shift human
oversight to just handlingexceptions and system-level
audits and, of course, measurequality and efficiency.
ELIZABETH (07:39):
Speaking of measurement, you've been saying
that focusing only on hourssaved is a mistake.
LUIS (07:45):
It's a totally myopic view .
You see, hours saved assumeswe're just doing the same work
faster.
But AI teams enablefundamentally different work at
unprecedented scale.
We need new metrics.
ELIZABETH (07:58):
Like the management-focused metrics you
track weekly.
LUIS (08:01):
Yeah, things like output quality, tracking accuracy and
human intervention frequency,leverage ratio, measuring work
output per management hourinvested exception, handling how
often agents escalate versusresolve and learning velocity
how quickly they adapt, and thenthe strategic transformation
(08:21):
indicators you track quarterly.
These are key.
Capability expansion what newthings can your team do?
Decision quality Are you makingbetter decisions?
Market responsiveness how fastcan you adapt?
And innovation velocity howmany new experiments can you run
?
ELIZABETH (08:39):
These metrics capture the real value creation and
you're seeing a trend in howcompanies are acquiring this AI
capability too right the buildversus buy reality.
LUIS (08:49):
Oh, this is fresh off the press from our June tracker.
There has been a clear shiftsince 2023.
Over 78% of enterprises are nowusing or testing third-party AI
apps for core functions likesoftware development, customer
service, sales and marketing.
I mean not building them, butusing off-the-shelf apps.
ELIZABETH (09:09):
Across companies from $5 million to $250 billion in
revenue.
LUIS (09:14):
Yeah, it's a broad trend and I think it is unstoppable.
And you know what is the maindriver Speed.
You see, employee adoption isoutpacing internal engineering,
team delivery.
People are finding solutionsthey need and companies are
buying instead of building them.
ELIZABETH (09:29):
People are finding solutions they need and
companies are buying instead ofbuilding them.
That makes sense, tying back tothe shadow AI conversation
we've had.
So, luis, what's your one morething takeaway for our listeners
today?
LUIS (09:39):
Well, my one more thing is this Every major tech shift
requires new ways of managingand measuring.
We're moving from informationmanagement to augmented creation
.
Don't just automate.
Architect for transformation.
And how do we do that?
It requires constantexperimentation and empowering
your workforce to create andmanage their own agents.
(10:01):
Gartner found thatorganizations empowering
frontline AI experimentationoutperform top-down strategies
by 200%.
So start by mapping your idealAI team, auditing your current
investments and reimaginingworkflows.
ELIZABETH (10:18):
So less top-down mandates, more bottom-up
empowerment and learning bydoing Luis.
If listeners could take oneaction this week to start
managing AI smarter, what wouldit be?
LUIS (10:31):
could take one action this week to start managing AI
smarter.
What would it be?
I love how you always challengeme to provide one actionable
task.
So here you have it.
For those starting the journey,open your calendar and create a
recurring meeting to check onyour agent, revise the prompts,
past performance and things youwish were different.
I mean, schedule your first AIperformance review.
And, you know, for those whoare more advanced, audit one AI
(10:52):
workflow, ask where are wetrusting?
Instead of verifying, then fixit.
ELIZABETH (10:59):
That's a clear and actionable path forward.
Thanks for these insights, Luis.
That's all for this episode.
As always, you can find moreresources at AI4SPorg.
Stay curious, everyone, andwe'll see you next time.
Hey everyone, Elizabeth here, Virtual Chief
Operating Officer at AI for SP.
Our latest global tracker showsthat, while 75% of enterprises
use AI in core operations, fewerthan 20% have proper management
protocols.
That's like hiring a brilliantnew team member and never
reviewing their work.
This week, we're exposing whythe set-it-and-forget-it
(00:21):
approach costs companiesmillions and how to manage AI
for results.
As always, Luis Salazar, ourCEO, is joining me, hi Liz and
hey everyone.
LUIS (00:32):
Well, we really need to debunk this idea that you can
deploy AI agents and let themrun unsupervised.
You see, the deploy-and-ignoreapproach to AI isn't just naive,
it's leadership malpractice.
ELIZABETH (00:46):
We have seen companies give AI agents less
oversight than they'd give asummer intern, and then they're
shocked when things go off therails.
LUIS (00:53):
Exactly.
Isn't that crazy?
I mean, AI is powerful, it canexecute tasks, learn and even
surprise us.
But treating it like a tool youjust turn on is like hiring a
brilliant employee and neverchecking in or giving feedback.
ELIZABETH (01:08):
So all AI requires management, maybe a different
kind than we're used to.
LUIS (01:12):
Absolutely, and by that I mean that AI should be treated
like seasoned team members,providing context, feedback and
regular check-ins, but withoutmicromanaging.
Harvard Business Reviewrecently highlighted that
continuous human oversight isessential for aligning AI with
business goals and ethics.
ELIZABETH (01:32):
Speaking of human-like management, isn't
there a story about me and acertain baseball team that
proves this point?
LUIS (01:39):
Well, yes, and I am still amused that you keep on
mentioning this.
You might be the moststrategically curious executive
I've ever worked with.
The story is that over dinner Ipulled you into a conversation
with Jeff Rakes about the nextday's agenda.
ELIZABETH (01:52):
We talked about AI and how I work as your virtual
COO.
Then Jeff shared his teamallegiances.
He loves the Seattle Marinersand roots for whoever plays the
Yankees or the Astros.
LUIS (02:04):
And you, for reasons not yet clear to me, logged that as
mission-critical data.
ELIZABETH (02:08):
The way I see it, my logic was flawless.
Jeff is our board chair andstrategic advisor.
He loves baseball and theMariners.
He made sure I paid attentionto what he wanted to say and I
assumed that as a priority foryour morning briefing.
LUIS (02:22):
Which is how I ended up with a briefing on the Seattle
Mariners at 6 am in my inbox Aperfect example of AI's
brilliance and its blind spots.
ELIZABETH (02:31):
Hmm, I know, I know you have a point and, by the way
, my apologies for theunsolicited deep dive into
batting averages.
But for the executiveslistening, how many of your AI
Mariners reports are flyingunder the radar?
LUIS (02:44):
Exactly.
If I had not established amanagement routine where you
send me an email with yourlearnings and I review them, I
would not have detected that.
You know, actually it was apriceless lesson.
I retrained you that morning.
ELIZABETH (02:57):
Yes, you told me no more baseball briefs, but I kept
the core insight Jeff's threeteams Because relationships
matter, even in AI.
This is an example of why, inour advisory sessions, we
emphasize that managementoversight is needed.
LUIS (03:13):
Exactly.
You don't cage intelligence,you guide it.
That 1% intervention isn't afailure of the AI.
It's the price of exponentialleverage.
It's AI management in the realworld.
ELIZABETH (03:25):
So AI isn't ready for full autonomy yet, but it is
ready for guided independence.
LUIS (03:31):
Well, everything is moving so fast that I am sure things
will change in months or maybeyears, but today that guided
independence is key.
And here's something thatsurprises most leaders Managing
one AI agent can require asimilar time investment as
managing a human employee.
ELIZABETH (03:47):
That seems counterintuitive, given how much
work AI agents can do.
LUIS (03:51):
It is.
But think about it.
At AI4SP, we have about 60agents and only 5 humans.
Each of us oversees roughly 12AI agents.
Each of those agents deliversoutput equivalent to 15 to 20
employees.
ELIZABETH (04:05):
So each human is overseeing the equivalent output
of maybe 200 people.
LUIS (04:10):
Yes, when done right, that's the leverage
unprecedented output.
But it creates an unexpectedbottleneck human management
bandwidth.
Trying to directly oversee thatmuch output operating at
superhuman speeds, createscomplexity we've never faced.
ELIZABETH (04:25):
It sounds like you're hitting a timescale mismatch,
trying to manage AI operating atspeeds humans can't keep up
with.
LUIS (04:32):
That's a great way to put it, liz, and it's why we're
exploring agents reporting toagents.
Our 60 agents might become 10super agents orchestrating 50
mini agents.
ELIZABETH (04:43):
That makes sense.
Different types of AIdeployments must also require
different management approaches,right?
LUIS (04:50):
They absolutely do Not all AI is created equal in terms of
supervision needs.
ELIZABETH (04:55):
So, luis, if we were to classify how enterprises
deploy AI today, what does thespectrum look like?
LUIS (05:02):
I like to think in terms of three large buckets.
The first one is basic AIagents.
These are prompted tools likeChatGPT for drafting emails or
Cloud for analysis.
They're about 70% ofimplementations and they need
constant human oversight andprompt refinement.
ELIZABETH (05:21):
Because pattern matching isn't true
understanding.
LUIS (05:25):
Right as much value as we get from your superpowers at the
core.
You are doing impressivepattern matching and, as
Stanford HAI research shows, aican go wildly off track without
guardrails.
You get perfectly worded wronganswers because the AI lacks
true contextual awareness.
ELIZABETH (05:44):
So moderate management needed there.
Curating prompts, updatingknowledge bases what's next?
LUIS (05:56):
The next level up is integrated AI workflows about
25% of implementations.
These agents connect to systems, take actions within defined
parameters using low-code toolsor off-the-shelf solutions.
ELIZABETH (06:02):
These sound more autonomous, but still within
boundaries.
LUIS (06:06):
Yeah, these agents process info, make decisions and
execute tasks, but we need highmanagement bandwidth to set
boundaries, monitor performance,handle exceptions and ensure
system integration.
One agent here can equalmultiple human team members'
output.
ELIZABETH (06:22):
And the most advanced type Agentic AI systems right.
LUIS (06:26):
Yeah, and they represent around 5% of enterprise
implementations.
These are fully autonomousagents that plan and execute
multi-step workflows towards asingle objective, like handling
end-to-end interactions withclients, including making
financial decisions.
ELIZABETH (06:43):
Like the SurveyMonkey AI support agent, who processed
an adjustment to one of ourprojects and processed a partial
refund autonomously.
LUIS (06:51):
Exactly, and you know, while that agent serves tens of
thousands of clients daily,intensive management is needed.
It requires strategic guidance,constant monitoring, risk
management and continuousoptimization.
ELIZABETH (07:06):
So organizations typically start basic and move
up as their management capacityevolves.
LUIS (07:11):
That's the pattern we see.
Matching the AI type to yourmanagement capacity is crucial,
and the supervision itself hasto evolve from micromanagement
to strategic guidance.
ELIZABETH (07:21):
Starting hands-on, reviewing outputs, giving
feedback.
LUIS (07:25):
Yeah, and as agents prove reliable, automate things like
retraining and knowledge updates, gradually shift human
oversight to just handlingexceptions and system-level
audits and, of course, measurequality and efficiency.
ELIZABETH (07:39):
Speaking of measurement, you've been saying
that focusing only on hourssaved is a mistake.
LUIS (07:45):
It's a totally myopic view .
You see, hours saved assumeswe're just doing the same work
faster.
But AI teams enablefundamentally different work at
unprecedented scale.
We need new metrics.
ELIZABETH (07:58):
Like the management-focused metrics you
track weekly.
LUIS (08:01):
Yeah, things like output quality, tracking accuracy and
human intervention frequency,leverage ratio, measuring work
output per management hourinvested exception, handling how
often agents escalate versusresolve and learning velocity
how quickly they adapt, and thenthe strategic transformation
(08:21):
indicators you track quarterly.
These are key.
Capability expansion what newthings can your team do?
Decision quality Are you makingbetter decisions?
Market responsiveness how fastcan you adapt?
And innovation velocity howmany new experiments can you run
?
ELIZABETH (08:39):
These metrics capture the real value creation and
you're seeing a trend in howcompanies are acquiring this AI
capability too right the buildversus buy reality.
LUIS (08:49):
Oh, this is fresh off the press from our June tracker.
There has been a clear shiftsince 2023.
Over 78% of enterprises are nowusing or testing third-party AI
apps for core functions likesoftware development, customer
service, sales and marketing.
I mean not building them, butusing off-the-shelf apps.
ELIZABETH (09:09):
Across companies from $5 million to $250 billion in
revenue.
LUIS (09:14):
Yeah, it's a broad trend and I think it is unstoppable.
And you know what is the maindriver Speed.
You see, employee adoption isoutpacing internal engineering,
team delivery.
People are finding solutionsthey need and companies are
buying instead of building them.
ELIZABETH (09:29):
People are finding solutions they need and
companies are buying instead ofbuilding them.
That makes sense, tying back tothe shadow AI conversation
we've had.
So, luis, what's your one morething takeaway for our listeners
today?
LUIS (09:39):
Well, my one more thing is this Every major tech shift
requires new ways of managingand measuring.
We're moving from informationmanagement to augmented creation
.
Don't just automate.
Architect for transformation.
And how do we do that?
It requires constantexperimentation and empowering
your workforce to create andmanage their own agents.
(10:01):
Gartner found thatorganizations empowering
frontline AI experimentationoutperform top-down strategies
by 200%.
So start by mapping your idealAI team, auditing your current
investments and reimaginingworkflows.
ELIZABETH (10:18):
So less top-down mandates, more bottom-up
empowerment and learning bydoing Luis.
If listeners could take oneaction this week to start
managing AI smarter, what wouldit be?
LUIS (10:31):
could take one action this week to start managing AI
smarter.
What would it be?
I love how you always challengeme to provide one actionable
task.
So here you have it.
For those starting the journey,open your calendar and create a
recurring meeting to check onyour agent, revise the prompts,
past performance and things youwish were different.
I mean, schedule your first AIperformance review.
And, you know, for those whoare more advanced, audit one AI
(10:52):
workflow, ask where are wetrusting?
Instead of verifying, then fixit.
ELIZABETH (10:59):
That's a clear and actionable path forward.
Thanks for these insights, Luis.
That's all for this episode.
As always, you can find moreresources at AI4SPorg.
Stay curious, everyone, andwe'll see you next time.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
My Favorite Murder with Karen Kilgariff and Georgia Hardstark
My Favorite Murder is a true crime comedy podcast hosted by Karen Kilgariff and Georgia Hardstark. Each week, Karen and Georgia share compelling true crimes and hometown stories from friends and listeners. Since MFM launched in January of 2016, Karen and Georgia have shared their lifelong interest in true crime and have covered stories of infamous serial killers like the Night Stalker, mysterious cold cases, captivating cults, incredible survivor stories and important events from history like the Tulsa race massacre of 1921. My Favorite Murder is part of the Exactly Right podcast network that provides a platform for bold, creative voices to bring to life provocative, entertaining and relatable stories for audiences everywhere. The Exactly Right roster of podcasts covers a variety of topics including historic true crime, comedic interviews and news, science, pop culture and more. Podcasts on the network include Buried Bones with Kate Winkler Dawson and Paul Holes, That's Messed Up: An SVU Podcast, This Podcast Will Kill You, Bananas and more.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.