How AI productivity wins are creating management nightmares

Share your thoughts with us

Managing AI agents presents complex challenges as these tools shift from personal productivity enhancers to shared team resources. The scale of AI adoption is massive, with nearly 80% of organizations using AI and 41% of US workers using it five or more days weekly, yielding significant productivity gains like teachers saving six weeks annually.

• AI adoption follows a pattern: individual usage leads to sharing, creating an accidental service provider situation.
• Most popular AI platforms weren't designed for a collaborative, high-stakes work environment.
• Organizations typically hit management bottlenecks within six months of adoption.
• New organizational structures are emerging with a clear division between business owners and technical support.
• Organizations should audit their AI dependencies by mapping which agents are being used and what breaks if they go down.

For more resources and research, visit us at AI4SP.org.

🎙️ All our past episodes 📊 All published insights | This podcast features AI-generated voices. All content is proprietary to AI4SP, based on over 250 million data points collected from 25 countries.

AI4SP: Create, use, and support AI that works for all.

© 2023-25 AI4SP and LLY Group - All rights reserved

Episode Transcript

ELIZABETH (00:00):
Hi everyone, I'm Elizabeth, your virtual co-host.
Today we're diving into a messytruth that nobody wants to talk
about.
Managing AI agents is way morecomplex than anyone admits.
As always, luis Salazar,founder of AI4SP, is with us.

LUIS (00:13):
Hi Elizabeth, it's great to be here.
Well, before we get into thedetails, let's talk about why
we're seeing this problem now.
I mean, the scale is just offthe charts.
According to our global tracker, nearly 80% of organizations
are using AI and 41% of USworkers are using it five days a
week or more.

ELIZABETH (00:34):
That's wild and the productivity gains are real.
Right.
I saw that new research fromthe Walton Foundation and Gallup
.
It shows that teachers ingrades K to 12 are saving six
weeks a year thanks to theirpersonal AI agents or AI tools
designed for their use in theclassroom, Exactly.

LUIS (00:51):
And that is also a great example of how not everyone
needs to create agents to getreal, tangible benefits.
Some AI tools just workperfectly fine for us.

ELIZABETH (01:03):
What I love the most about the findings from that
study is that they are measuringnot only time saved but the
significant impact how that timesaved allows them to spend more
time personalizing educationfor students and connecting with
them.

LUIS (01:17):
And it's not just teachers .
Our tracker shows that peopleusing AI five days a week or
more are saving an average of 79minutes per task.
And it's not just the basicslike email or summarizing
content.
We're seeing this in scientificresearch, business analysis and
even equipment repair andmaintenance.
The grassroots impact is huge.

ELIZABETH (01:38):
So naturally, when someone builds an agent that
saves them hours, everyone elsewants in hey, can I use your
agent?
That's how the sharing starts.

LUIS (01:47):
And that's the tipping point.
Suddenly, you're not justhelping yourself.
You're running a service foryour team, whether you want it
to or not, and the tools we'reusing ChatGPT, cloud, copilot
were never designed for thiskind of collaborative,
high-stakes work.

ELIZABETH (02:02):
And that's when the headaches begin.
There's no easy way to testchanges, undo mistakes or
restore from a backup.
Most people building theseagents aren't thinking about
what happens when something goeswrong.

LUIS (02:13):
Right, it's not just about building the agent for yourself
.
The moment you share it, you'veaccidentally become the manager
of a live service for yourwhole team.

ELIZABETH (02:21):
Let's make this real for people.
Remember Lauren, the directorat that major consulting company
.
She got her team together tolearn how to build AI agents.
Everyone picked a boring,repetitive task and built a
simple agent to automate it.

LUIS (02:35):
And it worked beautifully.
Each agent saved its creator afew hours every month.
And this wasn't an IT projectfrom the top down.
It was driven by the peopleclosest to the work real,
grassroots innovation.

ELIZABETH (02:47):
Productivity shot up, morale improved and suddenly
agent sharing started.

LUIS (02:52):
Yeah, you see, at this stage most people cannot even
imagine what type of agents theycan build.
I mean, they will use ChatGPTfor everyday tasks and get real
value, but there seems to be apsychological barrier to
overcome a mindset shift forthem to go from using ChatGPT or
Claude to creating apersonalized agent that has more

(03:14):
context about them and theirwork.

ELIZABETH (03:17):
That is right, and we saw that with Lauren's team.
Sharing agents also inspiredpeople to realize they could
build some agents too.

LUIS (03:25):
Well, that's true, but here's what most people don't
realize there's a hugedifference between the AI tools
that let you create shareableagents versus the ones that
don't.
I mean, chatgpt makes agentsharing really easy, but there's
also a whole category ofspecialized companies building
tools specifically for this.
There's also a whole categoryof specialized companies
building tools specifically forthis.

ELIZABETH (03:43):
That's a great point.
Ai tools like Relevance AI,custom GPT, dante AI, gpt Bots
and GUI AI Most people havenever heard of them, but they're
solving exactly this problem,making it easy for non-technical
people to build and shareagents safely.

LUIS (04:00):
We use all of them and many others and, by the way,
several of them are powering youand our 50 other agents at
AI4SP.

ELIZABETH (04:08):
Okay, let's go back to Lauren's team story.
They created Agent Siri, whichhandled reports for all the
managers.
Agent Anna took oncustomer-facing tasks.
Agent Wilson summarizedmeetings.

LUIS (04:20):
But then people started tweaking or even deleting agents
, not realizing others wererelying on them.
That is when workflows gotbroken, results changed day to
day and trust in these toolsjust evaporated.

ELIZABETH (04:32):
This is the crisis point right, when individual
creativity collides with shareddependency.
According to our enterprise AItracker, over 78% of
organizations are now using orpiloting third-party AI agents
for core business tasks.
Most hit a managementbottleneck within the first six
months, right when agents shiftfrom personal tools to shared

(04:53):
team resources.

LUIS (04:55):
That's the hidden evolution.
People go from being users toservice providers overnight,
with zero training.
The risk jumps from impactingjust your own work to impacting
your team and then to entiredepartments.

ELIZABETH (05:07):
Technology and adoption are moving way faster
than our people developmentprograms.

LUIS (05:12):
Yes, that is the challenge skills.
Most people have never manageda live service before.
They're used to managing theirown work, not running digital
teammates for a whole team.
Suddenly, you need basicservice management skills
testing, backups, communicationand documentation.

ELIZABETH (05:30):
That's not a trivial new set of skills.
It's a real shift and it'shappening fast.

LUIS (05:35):
This is the future New organizational structures, new
roles and the fundamental shiftfrom measuring activity to
measuring value.
The companies that get ahead ofthese challenges are the ones
that will unlock the realtransformative benefits of AI.

ELIZABETH (05:49):
Okay, so what does that look like in practice?
First, you establish a sharedbaseline.
Get everyone on the same pageabout what it actually means to
manage an AI agent.
Schedule regular check-ins.
Get feedback from users.
Set clear, measurable goals foreach agent.
Schedule regular check-ins.
Get feedback from users.

LUIS (06:02):
Set clear, measurable goals for each agent and, when
sharing an agent, test changeson a copy to avoid affecting
others.
Also, notify users beforeupdates and maintain backups.
I mean, don't just wing it,treat it like a real service.

ELIZABETH (06:16):
It's a new world and everyone's a manager now,
whether they realize it or not.
The organizations that teachthese skills, set up clear rules
and plan for this growth.
They're the ones that avoid thechaos and get the most out of
their AI investments.

LUIS (06:30):
And if you're listening and thinking that sounds like a
lot of work on your right, butthis extra work only happens
when an agent is helping tens orhundreds of people save time,
and that's a good problem tohave.

ELIZABETH (06:41):
Let's dig into the stages of that evolution.
What's the very first sign thata team is moving from personal
agent to shared?

LUIS (06:47):
team agent.
It's the moment someone askshey, can I use your agent?
You know, that's the tippingpoint.

ELIZABETH (06:58):
Suddenly, you're not just responsible for your own
productivity.
You're running a service forothers, and that's when the risk
of breaking things justskyrockets.
If you tweak your agent foryour own needs, you might
completely break someone else'sworkflow.

LUIS (07:07):
And at that moment you shift from I'm making my job
easier to I'm providing aservice my team depends on.

ELIZABETH (07:14):
And when these agents become division level, the
stakes are exponentially higher.
A mistake can halt operations,not just annoy a few colleagues.

LUIS (07:23):
That's when you need real service management, rigorous
testing, reliable backups, cleardocumentation and unambiguous
ownership.

ELIZABETH (07:30):
Let's talk about that management crisis.
Most people have never manageda live service before.
They're used to managing peopleor using software, not managing
a team of AI agents.

LUIS (07:41):
And the platforms don't make it easy.
There's very little support formulti-owner editing, safe
testing environments or rollingback changes that cause problems
.
Business users are forced toimprovise solutions for problems
they've never faced.

ELIZABETH (07:56):
So what's the fix?
How do you start building thosemanagement foundations?

LUIS (08:00):
transparency and communication.
Get the challenges out in theopen, align on what everyone
expects and establish a sharedbaseline for how you're going to
manage these AI agents as ateam.

ELIZABETH (08:12):
And you have to teach two distinct skill sets
managing agents like theirpeople and running them like
their reliable services.

LUIS (08:20):
Well.
To manage agents like peoplemeans to schedule regular
check-ins with the agent, getfeedback from its users, set
clear goals and measureperformance against those goals.

ELIZABETH (08:32):
And for the service side.
Unless your AI tools offer theability to test things on a
replica or clone, create manualprocesses and always test
changes on a copy of your agent.
Notify users before makingchanges, maintain backups and be
transparent about reliability.

LUIS (08:49):
And, of course, don't forget training.
Everyone creating agents needsbasic management skills, an
understanding of serviceoperations and fundamentals of
data safety.

ELIZABETH (08:58):
This requires a mindset change.
The creators of these agentsare now managers and leadership
has to recognize that workloadand plan for new management
ratios.

LUIS (09:08):
And when an agent becomes a key part of your team's
operation, it is time to bringtechnical help.
For example, centralizedmonitoring can save subject
matter experts hundreds of hoursa month.

ELIZABETH (09:20):
So you have a new division of labor.
Devops handles uptime securityplatform updates.
Business owners handle prompts,knowledge and user feedback.
Okay, luis, we have to wrap itup.
What is your?
One more thing.

LUIS (09:32):
In our research organizations hit the management
crisis around months four orsix of agent adoption.
So you can start with a simpleaudit today Map who's using
which agent, who owns what andwhat breaks if each agent goes
down.
Most organizations discoverthey're far more dependent than
they realized.

ELIZABETH (09:51):
That is a very practical next step, and that's
it for today's episode.
For more resources and ourresearch, visit us at AI4SPorg.
Stay curious and we'll see younext time.