Customer Success Playbook S3 E39 - Matt LeMay - AI Is Not a Strategy

Send us a text

In the finale of our three-part series, Matt LeMay takes on the tech world's shiniest object: artificial intelligence. From summarizing meeting notes to influencing million-dollar decisions, AI is here to stay. But Matt warns against using AI as a goal unto itself. Product teams must ground their experimentation in business logic—or risk becoming distractions.

Detailed Analysis: Episode 39 brings a timely and pragmatic take on AI in product management. Matt acknowledges AI’s strengths—particularly in summarizing information and idea generation—but urges caution. When AI tools lack source context, they can mislead as easily as they can inform. The key is discernment: understanding when AI adds value and when it distracts.

The team unpacks how AI is impacting product roadmaps, budgeting, and even job security. Matt shares war stories of teams told to "do something with AI," only to later realize the result consumed $200K to close a $50K deal. Oops.

But this isn't an anti-AI rant. Matt offers frameworks to leverage AI wisely while keeping business impact front and center. If you’re going to play with shiny tools, make sure they’re solving real problems.

Now you can interact with us directly by leaving a voice message at https://www.speakpipe.com/CustomerSuccessPlaybook

Please Like, Comment, Share and Subscribe.

You can also find the CS Playbook Podcast:
YouTube - @CustomerSuccessPlaybookPodcast
Twitter - @CS_Playbook

You can find Kevin at:
Metzgerbusiness.com - Kevin's person web site
Kevin Metzger on Linked In.

You can find Roman at:
Roman Trebon on Linked In.

Episode Transcript

Kevin Metzger (00:05):
Customer success.

Roman Trebon (00:11):
Welcome back to the Customer Success Playbook
podcast.
I'm your host Roman Tree Bond.
Joined by uh, my co-host KevinMetzker.
It's Friday.
We've made it to the end of theweek, and we're wrapping up our
three part series with authorMatt LeMay.
Kev, we've had an amazing weekwith Matt.
I'm excited for Fry ai, AAFriday, whatever we call it

(00:32):
nowadays.
I think we've changed the titlefour times.
Are you excited to get intothis?

Kevin Metzger (00:36):
AI Friday.
Always good to you sound.
You sound excited, Kevin.
No, I love AI Friday.
It's always interesting to talkand, and get our guest
perspective about, about AI andhow it's impacting.
They're part of the business andhow they see how they see ai
because it's such an evolvingand really dynamic topic at this

(00:58):
point.
Matt, like I said, ai, it'stransforming everything right
from road mapping and userinsights to potential layoffs.
If automation displaces certainroles.
How can product managersleverage AI to enhance the
impact without losing sight ofcore business goals when budgets
or leadership priorities areshifting?

Matt LeMay (01:19):
I'm, I'm of several minds about ai, and I think as
with many of us, my opinionskeep shifting.
As the landscape of AI shifts,I've found that AI can be really
good for sort of summary tasks,for taking a lot of information
and synthesizing it andproviding a.
Basic defensible synthesis,which is sometimes really

(01:39):
valuable.
Like there are times where youwanna throw a bunch of meeting
notes in there and say, allright, what are some themes that
are coming up?
Or, you know, in the process ofestimating impact, which is
something I talk about quite abit in the book, what I
recommend doing generally is, isGoogling and looking for sources
and figuring out.
Kind of, you know, if you'retrying to figure out, for
example, what's the impact thatimproving password reset would

(02:00):
have on an e-commerce business?
Well, you can Google how manypeople forget their passwords.
What's the average value of ane-commerce cart?
Now where this gets tricky is AIcan give you sort of a
synthesized baseline for thosekinds of things.
But where you have to be moredeliberate in the way you use AI
and the way you write yourprompts is making sure that you

(02:22):
also understand the sourceswhere these things are coming
from.
A lot of the white papers andthings published that can help
address some of these commonchallenges are published by
companies that have a vestedinterest in making these things
sound important.
So for example.
You know, if you're looking upwhat's the value of password
reset, there are a lot ofidentity solutions companies
that are gonna have a reallystrong opinion about why

(02:44):
password reset is super valuableand you should work with them.
Now, if you understand that,that can be really valuable.
'cause you can look at that andsay, all right, well, it's
really unlikely that they wouldunderstate the importance of
this, right?
So if we assume that this is amaximum, how valuable could it
be?
But again, that requires that.
That kind of element of humandiscernment of the motives and

(03:07):
context and tone of information,which can get lost easily in AI
unless you are really thoughtfulabout how you use it and
checking the sources and askingAI systems to provide citations
and things like that, which itwill also hallucinate sometimes.

(03:27):
So again, I think.
As with a lot of things, youknow, I went through this when
it was machine learning 10 yearsago.
The question of like, how wrongcan we afford to be in this case
is always a really importantquestion to ask.
And I think that with AI, it canbe a great way to get a pulse
check.
It can be a great way to take abunch of messy information and

(03:48):
synthesize it into a goodstarting point from which you
can then further refineinformation.
You know, I think for productpeople.
A blank page can be reallyterrifying.
And if you go into, you know, alarge language model and say, if
this is what my team is thinkingabout, can you give me what a
one page product spec would looklike, it's probably gonna be

(04:09):
easier for some folks to editand modify that than to just
start writing from scratch.
So I, I think as a thoughtstarter, as a tool for
synthesizing information, AI canbe really valuable.
But I think that that finallevel of discernment, that
ability to contextualizesources, to understand what
really matters, to, again, tiethings back to the particular

(04:30):
business model of your companyand communicate things in a way
that will resonate and makesense to the particular
personalities in your company,is something that will always be
valuable for product management.

Roman Trebon (04:41):
Matt, I'm curious from, from, uh, you, you know,
you're in the, you're, you're anexpert in, in the product space.
Uh, do you see or can you seethat AI can maybe be distracting
from impact that it's such a bigconcept that like we, it's over
indexing and it's taking awayfrom what real impact is?

Matt LeMay (04:59):
You know, I remember seven or eight years ago, eight
years ago, I think this was thepeak of Pokemon Go Mania.
I was working with a bank.
Somebody asked very earnestly,well, what's our augmented
reality strategy at a bank?
You're a bank.
Yeah.
Yeah.
You're make your app work andthen we'll talk about augmented

(05:20):
reality.
But I, I think, I think there'salways been some shiny new
thing.
And here's where I feel forproduct teams.
Again, you know, as, as wediscussed in, in our very first
conversation on Monday, thebusiness ultimately expects
something from you.
The business is ultimatelylooking to your team to deliver
success, and if the businesssays to you, go make something

(05:43):
having to do with ai, that's notgreat because there will come a
time.
Where the business turns aroundand says, all right, well, AI
has either been commoditized orit's not as exciting anymore.
Or, our CEO went to a conferenceand learned about something
else, which is what we careabout now, um, they're gonna
turn around to that team andsay, all right, well, what's the

(06:03):
impact of your work?
Why are you doing this?
And if the answer is, well, youtold us to do something with ai,
so we did.
Again, that's not a greatanswer.
I, I've talked to some productpeople who say that they're
being told to.
Build things in service of$50,000 contracts that will cost
$200,000 of just AI usage tobuild and.

(06:27):
You know, I, I, I don't, I don'tthink I'm a business genius, but
I think I could probably build abusiness that turns$200,000 into
$50,000 if I tried.
Yeah.
So again, I think part of thechallenge is, you know,
businesses are not alwaysrational.
It's, it's people I.
People get excited about things,and people want to jump on
trends and, and do things thatare cool and exciting and new.

(06:50):
And part of my challenge toproduct teams is to say, all
right, imagine a year from now,how are you going to explain the
benefit to the business of theAI product that you're working
on?
And if the answer is well.
You said, go do something ai,and we did something ai.
Again, I don't think thatactually puts you in a terribly
strong position.
And I think that come time forthe realities of a business to

(07:13):
kick in, whether that's, youknow, raising around of funding
or making a report toshareholders, if these go do an
AI features and products are notdelivering meaningful results to
the business, they are probablynot going to be.
Plentifully resourced.
Moving forward.

Kevin Metzger (07:33):
I had a couple of thoughts I wanted to actually
run by.
Yeah.
One.
I think your point's very good,right?
You can't just use AI to use ai.
That's from a productstandpoint, from I, I realize
that a lot of people are doingthat.
Realistically, you should belooking at, Hey, is there a way

(07:55):
AI can make an impact that canprovide value beyond what we're
doing in our product today?
And really.
Spending time doing evaluationson where, where that works,
where it doesn't work, whatvalue it can bring.
And just like any type ofproductization you should be
looking at the value of whatyou're trying to accomplish and

(08:18):
whether you're getting a returnon that value.
I guess that's more of astatement.
I think the other, the otherpiece is, is a question on have
you had the opportunity to useany of like the deep research
tools yet?
I know it's, they're relativelynew to the market, but like open
AI's got a deep research there.
There's, there's a few toolsthat have the deep research out

(08:41):
yet, and I'm just wondering ifyou've had an opportunity to see
results from them yet, becausethose are getting very
interesting when doing.
Some of the queries that youwere talking about where you're
previously gettinghallucinations and don't have
all the references, well now youcan get reports back with
detailed references and opinionsand it's, it's really some

(09:04):
interesting details that you canget back.
So just wondering if you had achance to look at them.
I

Matt LeMay (09:09):
haven't had a chance to use those yet, but that does
seem like a really good use casefor ai.
I mean, and in a funny way, I, Ithink.
With new technologies like this,the initial impulse is, oh,
let's make this seem like magic.
But I think when you look atthings like deep research, part
of what it's doing is showingits work a little bit.
And I, I think that ultimatelythere are contexts where

(09:31):
understanding the why behindsomething is, is really valuable
and, and seeing some of thoseintermediate steps before you
just get the answer.
Certainly makes that informationmore valuable for folks who know
how to read and contextualizethat information.
So I'm, I'm excited to spendsome time with some of those
tools, but I haven't had achance to yet.

Roman Trebon (09:49):
That'll be your, your next book, Matt, your next
book.
Let's not easy yet, right?
No, Matt, thanks for joining usthis week.
We really appreciate it.
Your book.
It's my pleasure.
Impact first product team isMatt.
Like I told you, I I, I read it.
I love the book.
I'm a huge fan.
So for our audience, even ifyou've, I've never been in
product and I loved it.
I, you know, again, it's, youcan replace product, insert your

(10:10):
team.
There's tons of insights, tonsof takeaways.
Matt, for our audience, wherecan they find more about the
book yourself, uh, et cetera.

Matt LeMay (10:18):
Uh, Matt lame.com is the place to be.
That's, I got links to the book.
I've got some talks I've given,got ways to reach me, so I am,
I'm pretty easy to find.

Roman Trebon (10:29):
Awesome.
Matt lamay.com.
Check him out there.
Uh, again, definitely pick upthe book.
It's, it's tremendous.
So that is the end.
We've come to the end of our,our three-part series with Matt.
If you enjoyed these discussionsto our audience, again, thanks
for listening.
Please subscribe and share themwith your team.
We'll be, we'll be back nextweek with more tips and

(10:50):
strategies for your customersuccess playbook.
Until then, Kevin, keep onplaying.