#38 - How We Actually Test AI Tools (And Why Most Fail) - Over The Bull®

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
SPEAKER_00 (00:02):
You're listening to Over the Bull, where we cut
through marketing noise.
Here's your host, Ken Carroll.

SPEAKER_01 (00:09):
Proving to you nothing good comes from cold
emails on this episode of Overthe Bull.
Okay, guys, um, I admit it thatuh I absolutely mark these cold
emails as phishing or I blockthem.
Uh but sometimes I get morbidlycurious because I do run the
podcast Over the Bull and I wantto know exactly what's going on.

(00:33):
So I got in this morning and I'mgoing through my emails, and I
bit on one of these just to kindof see what would happen with
the communication.
I mean, after all, maybe thereis a good product among the
99.9% that come through that mayactually show some benefit to a

(00:54):
company.
So I'm going to share with youuh an actual communication I had
today, and I kind of want tobreak things down a little bit
so that you can see what I see,so that you can evaluate things
the way I evaluate things.
Okay, so uh this email came in,and uh first thing it said was
let me look at the subject here.

(01:17):
Oh, um it's called Over the BullPlus AI Snake Over Strategy plus
their product.
I'm not gonna tell you theirproduct name because you don't
need to know it, and I'm notgiving them uh free publicity.
All right.
So, first of all, hey Ken.
Okay, so he he uses my name, Idon't know this person from
Adam, which to me I don't like.

(01:39):
I'll be honest with you.
Someone personalizing somethingand I don't know them just feels
incredibly intrusive.
So I first of all, I'm just nota big fan of someone pulling my
name out of a hat and thenreferring to me as though they
know me.
Um now here's the firstsentence.
Been listening to Over the Bull,and your take on AI
implementation really resonated,especially with the episode

(02:01):
about how AI may set the floor,but you're you still build the
ceiling.
Too many people are buying toolswithout strategy, and you're
calling it out.
Yes, I am.
The real reason for my outreach,however, is something I suspect
you're dealing with as Integrisgrows.
You just expanded the actualoffice, and in parentheses,
congrats, by the way, whichmeans more overhead and more

(02:25):
mouths to feed.
Oh, he's trying to approach mefrom the standpoint of some pain
point that he thinks that I'mhaving, and I'm not.
Uh but here's the irony mostagency owners face.
You're so busy delivering GoogleAds, SEO, and web projects for
clients that your own pipelinedevelopment takes a backseat.
Again, that's uh an assumption.

(02:47):
No, I do not.
We manage our pipeline justfine.
And it's all what we call SOPstandard operating procedures,
and we help clients developthose so that they can stay
focused on everything.
Uh, you know exactly what needsto be done for LeadGen, but
finding the time to actually dothe deep research and
personalized outreach, you know,that's the challenge.

(03:09):
So, what does this tell meimmediately?
He's got some AI tool.
He scoured the web, probablysearched through our
transcripts, and he's probablyusing some AI generator to build
some kind of artificialconnection with me.
He personalized my name at thetop of this.

(03:29):
And now he's telling me thathe's going to use some type of
league generation, but he'ssending me a cold email right
out of the gate, which tells methat he's probably doing a bunch
of cold email outreaches.
And that would attach ourbusiness to cold emails, which
could potentially damage ourbusiness's reputation because I

(03:52):
don't want to be known as thecold email spam guy.
So here's what he continues towrite.
That's specifically why we builtthat product to solve.
Our artificial intelligencespends 20 minutes researching
each prospect, reading theirfull LinkedIn history, post an
engagement, then sends emailsthat don't look like templates.

(04:17):
It handles responses, answerstechnical questions, and books
meetings.
You set a budget like Facebookads, and it runs on autopilot,
no mailbox management, and nosetup headaches.
Wow, super simple.
Uh, it'll just handleeverything.
It's going to spend 20 minutesresearching, it's going to
answer technical questions andbook meetings.

(04:40):
And it runs on autopilot.
Wow.
Okay.
So how does it know our positionin the market and how to answer
those questions?
Uh, how does it do that and notmake commitments or say things
that we don't want it to say?
How does it operate?
What's the functionality of thisthing?
Um, what's the investment?
What does this thing look like?

(05:02):
So then he continues to write, Iknow what you're thinking.
More AI snake all.
Yes, that's exactly what I'mthinking.
Where we're different is thisisn't a ChatGPT wrapper.
Oh, so he thinks we're againstChat GPT, which we are not.
Chat GPT, by the way, is a verypowerful tool in the right

(05:26):
context.
If you use it to write blogarticles or do different things
like that and abuse it like mostagencies are doing today, then
ChatGPT works against you.
But if you use it in the wayit's intended, it can be a very
powerful tool for bothprospecting new clients and
helping you manage yourbusiness.

(05:48):
So I'm not really um overlyoverwhelmed with everything
going on here.
Um, so here's what it uh thisperson continues to write.
It's goal obsessed AI.
Goal-obsessed AI.
Wow.
Um, that actually understandscontext and strategy, not just

(06:08):
keyword stuffing.
So now he's throwing a fewlittle buzzwords out there like
keyword stuffing, which tells methat he's not fully grasping his
own marketing position.
And then he proceeds to saywe're getting 1.5 to 3% positive
response rates because thepersonalization is genuine.
Now, how is it genuine?

(06:28):
It sounds to me like it's notgenuine.
So I'm a little bit confused bythat.
If you're open to exploring howthis could help your pipeline
full while you focus ondelivery, help keep your
pipeline full, excuse me, I'd beglad to find some time this week
or next that works around yourschedule.
Wow, that's that's tremendous.

(06:48):
A sales guy is willing to workaround my schedule.
That's amazing.
I hope you're feeling thesarcasm here.
I'm sorry, I know I'm feelingsarcastic, but you know, this
kind of stuff just boy, it'sit's just not very fun because
it's just all salesy, full ofcontradictions.
It's just horrible.

(07:09):
Um, so basically, I wrote backand uh this is what I wrote.
I normally block cold emails, Ido.
But I am looking for somepodcast ideas for 2026.
Guess what?
He gave me one.
How would you propose to testyour product?
If we ran it, are you willingfor me to use the beta test for

(07:30):
over the bull, regardless of theresults?
So I'm I want to know how hefeels about his reputation.
So a little bit of duelingbanjos here.
So here's here's what thisperson wrote back.
And by the way, this responsecame back instantly.
It did not take three minutes,five minutes.
So, what does that tell you?

(07:51):
The response is completelyautomated, guys.
Way too fast.
Now, here's what people know.
Quick responses close dealsfaster.
Slow responses typically don't.
Ultra fast responses are a cluethat you're being controlled or
manipulated by an AI tool.

(08:11):
So that should be your firstflag, not to take this
seriously.
So here's what he wrote back.
Love the podcast angle.
Absolutely.
You can use the results for overthe bull regardless of the
outcome.
Transparency is what makes goodcontent.
That's what I'm doing right now.
To test, sign up at theirwebsite, set a small budget,

(08:32):
target your ideal agencyclients, and let it run.
You can also watch ourfive-minute founder demo here.
CC, my colleague, I can'tpronounce that name, who can
help map this out.
Want to hop on a quick call todiscuss a setup?
No, I do not want to hop on aquick call to discuss the setup.

(08:52):
So now I feel like I'm trying toget some answers, some basic
stuff.
And what he's doing is trying topush me into a phone call.
So he's trying to push methrough the funnel.
It's overly aggressive, and plusit's oversimplified.
He's trying to push me to signup for something, push me into a
phone call.
Completely inappropriate.

(09:13):
I'm just exploring.
So here's what I wrote back.
Now I know he's just using AI asthese responses, but I'm having
fun and I need, you know, thisis good for the podcast.
I think we may be jumping.
This is me, by the way.
I think we may be jumping fromthe a first conversation
straight to a commitment, andI'm not comfortable with that.

(09:35):
I'm opening open to reviewingthis, but I need more
information before any sign-up.
I'm currently deep inoperations, so I need clear
communications and not a salespush.
Specifically, here are thequestions I'm asking.
What does this tool actually doin practical terms, not
promises?

(09:56):
What is the typical sales cycleand how long before results can
be reasonably evaluated?
What does a legitimate test looklike and how long should it run?
What is a realistic test budget?
How much setup is required onour side?
Custom messaging, content,approvals, question mark.

(10:18):
How much ongoing engagement isrequired from our team?
How do you preventoversaturation or brand damage?
What happens if it produces noleads?
Do you have a case study thatshows process, not just
outcomes?
And then I conclude by saying wereceive a high volume of
AI-driven product pitches, andmost rely on high-level claims

(10:41):
without addressing EEATprinciples, brand risk, or
operational overhead.
If your solution is different,the above should be easy to
answer in writing.
If not, that tells me what Ineed to know.
The ball is in your court.
So after my compiler.
So let's break this down just soyou understand what I'm saying
here.
I don't want to sign up and justput something on autopilot and

(11:04):
throw a budget in it.
This is what most businesses dobecause they think, what's it
going to hurt?
That's exactly what they want.
They want you to throw money atsomething to see if it works
because they win.
What's the typical sales cycle?
How long does it take toactually make this thing happen?
What does a legitimate test looklike?
Not just throwing some money init and just hoping that it works

(11:26):
out.
What's a realistic test budget?
Not a minimal test budget.
What's a realistic test budget?
How much setup is required onour side?
I mean, if it's going to answerour questions, manage everything
and do everything for us, surelyto goodness, we need to be able
to put in how our businessoperates.

(11:46):
What are our limits?
Who do we focus on?
Surely we need to be able toplug that stuff in.
How much ongoing engagement isrequired from our team?
This is pretty important becauseI need to know what that looks
like so that if it does workout, how can I appropriately set
that up?
How do you preventoversaturation or brand damage?

(12:08):
Are they just going to startemailing our brand out to
different people?
How are they going to controlthe status or what we're trying
to achieve through our brandidentity?
We don't want to be known asspam guys.
And we certainly don't want toaggravate people with
oversending them emails,overcommitment, giving them

(12:28):
wrong messaging, things likethat.
Then what happens if it producesno leads?
And then, of course, do you havea case that shows process, not
just outcomes?
You know, because outcomes areeasy, right?
So you could just get somebodywho's hopped up on something.
Maybe you get them with a leador two, and they think it's the
greatest thing.

(12:49):
You get a quote from them, andthen it turns out not to be so
good.
So I want to know process.
I want to know the meat andpotatoes of the product if I'm
truly going to investigate it.
Now, by the way, remember out ofthe gate, time is my most
valuable asset.
Now, if I was not using this fora podcast, I've literally wasted

(13:09):
15 minutes of my time chasingthis.
So keep in mind that's 15minutes away from my business
that I'm giving this person whocold emailed me, used my name
with trying to pretend they knowme, sending me a bunch of this
gobbledygoo, and then trying topush me into a quick sale where
they get me to sign up.
Horrible.

(13:30):
And so I wrote, if your solutionis different, and I won't repeat
that.
So let's see what happens.
So I put the ball in his court,and here's what we go here.
So he writes me back again.
This is instant.
So what does this tell me?
AI is trying to close the deal.
This is not a human tightlyengaged in this, because no

(13:50):
human could read that email inliterally 15 to 20 seconds.
So here's the response.
Totally fair, no pressure, quickanswers.
Uh what it does.
AI researches each prospect 20minutes, which he already said,
sends personalized emails,handles replies, books,
meetings.
You set a budget like ads.

(14:11):
Timeline, response in two tothree days.
Recommended 30-day test minimalfor podcast content.
Test budget.
1K get you started.
Really didn't answer my questionthere, Chief.
Setup.
Chat with our AI about youreight ideal clients, contact
calendar, maybe 15 minutes.
No custom content needed.

(14:33):
AI writes everything based onyour input.
So 15 minutes to describe ourbusiness model.
Now, if you're a business,you're going, wow, only 15
minutes.
I go, there is no way you couldrepresent my company with 15
minutes of understanding.
Period.
Your involvement near zero.

(14:54):
You're only CC'd on positivereplies.
Wow.
Okay, so again, they're tryingto get this.
We don't want to bother you.
We're going to automaticallyhandle this.
Again, that's not what you wantif you're serious about your
brand.
Brand protection.
We use generic domains, notintegristdesign.com.
Constantly rotate freshmailboxes and AI

(15:16):
personalizations so nothinglooks mass sent.
Oh, so they're going to attachmy brand name to generic domains
and rotate fresh boxes.
Hmm.
Okay, so this sounds to me likeuh pretty spammy processes here.
No leads.

(15:37):
You only pay per sprop uh payper prospect contacted.
Uh 1.4 not per result.
Okay, I don't even know whatthat means.
No long-term commitment casestudies.
Click on their website forexamples.
Didn't really, I'm not goingthere because he didn't tell me
he's going to actually give meprocess and not just somebody

(15:57):
glowing about something thatthey may have felt good about
temporarily.
For the podcast angle, a30-minute documented test would
make solid content.
I already got 30 minutes righthere.
Want to hop in a 15-minute call?
No, I do not.
Okay, so you see what's going onhere.
So if you're not careful, um youcan see what happens.

(16:18):
Okay, so here we go.
Here's the response.
Um I appreciate the quickresponse, and I put in
parentheses AI fast because Iwant him to know that I know I
can't resist sometimes.
But this actually reinforces myhesitation.
Using rotating domains andburner mailboxes to avoid

(16:39):
reputation damage tells me thesystem anticipates negative
signals.
You see, it already knows badthings are going to happen from
this process, and so it's tryingto set up damage control right
out of the gate, but my brand isstill attached to it.
I don't want that.
That may be uh acceptable forpure outbound sales, but it's
not aligned with how we protectbrand, equity, authority, and

(17:00):
long-term trust, especiallyaround a podcast built on
expertise and credibility.
Near zero involvement and fullyAI-written messaging without
editorial review is also anon-starter for us.
That's precisely where most AIoutreach tools break down and
create noise instead of signal.
See the problem here.

(17:21):
What they're trying to do issell you easy, quick, we handle
it.
You get a salesperson workingfor you.
All the magic can be done in 15minutes, and all you have to do
is pay us.
And you only pay us per lead.
Follow up, paying per prospectcontacted rather than tying cost
to qualified outcomes furthershifts the risk to us, which

(17:43):
isn't how we evaluate vendors.
See, you don't need to spendmoney to prove their system to
you.
It's up to them to prove theirsystem to you, especially when
they're offering somethingcanned like this.
So I continue.
I don't doubt this system cangenerate activity.
What I haven't seen is evidencethat it can generate aligned

(18:04):
conversation without brand risk.
For now, we're going to pass.
If you can develop a model thatuses first party domains safely,
meaning our domain, includeshuman-reviewed messaging, very
important that it's human,demonstrates EEAT aware
outreach.

(18:25):
That is, uh go check out ourpodcast on EEAT.
Shares process level casestudies.
Remember, process level casestudies.
And I wrote, I'm opening torevisiting, but I won't block
your domain for now.
So check this out.
Um, here it is.
So totally fair.

(18:47):
Appreciate your thoughtfulfeedback.
You clearly thought through thebrand risk angle, and I respect
that.
For what it's worth, we dosupport first party branded
domains, and you can reviewapproved messaging before
anything sends.
But I hear you, timing isn'tright.
Doors always open if thingschange.
Best of luck with Over the Bullin 2026.

(19:08):
So, oh, so now it does do firstparty domains, but now think of
the whole can of worms.
You're using my domains, you'renot really giving me any details
about it.
Now I'm going to trust you withthe keys to the castle.
I don't think so.
So uh again, just follow it up.
Uh, thanks.

(19:28):
Understood.
We'll pass for now.
Wishing you the best is how Iconclude this uh messaging, and
then I proceeded to uh blockthat domain name for that
business.
So, guys, this is what you needto be aware of.
They're trying to sell you oneasy, quick, latest and greatest

(19:53):
gadget, and they're not likeeverybody else.
Yes, this company is likeeverybody else.
They're cold emailing.
They're using AI to try to catchyou.
And they're what they're tryingto do is close as many deals as
they can, and they have noanswers for the reality of the
web and the need for humanengagement and human written

(20:16):
content.
This is no good for my business.
This would be no good for yourbusiness.
You really need to understandthis.
These things are happening allthe time.
There is no good solution whenyou build systems like this.
Your brand, your reputation isincredibly important.

(20:36):
You need to hold fast to that.
You need to build yourcredibility.
And this is why the last podcastwhere I talk about the boring
work being the meaningful workis the best thing that you can
do.
I don't know how else to say it,but this is a great example in a
system that you would want toavoid.
This took about 30 to 40 minutesof my time to go through this.

(21:00):
If I do, let's just say 30minutes, I do 10 of these, I've
lost five hours.
You see, you can't afford toeven spend time on this.
Block them.
Don't go with it.
Don't use it.
Don't fall for the sales pitch.
Don't fall for the guy who'sgoing to send you all the things
wrong with your business.
Either trust your digitalmarketing guy or don't.

(21:22):
If you don't trust him, move on.
If you do trust him, then ignorethese signals.
Ignore these signals wherethey're trying to bait and bait
you into something just to getyou to spend money, waste your
time and energy, and causedisruption with you and your
team.
Okay.
I got a little passionate aboutthat, guys.
Thank you so much for listening.

(21:42):
And uh there we go.
I got my podcast episode in.
So uh thanks so much.
You guys have a great one.
Uh, we're heading up onChristmas.
Um, I'm going to go ahead andstart saying it now.
Merry Christmas and happyholidays and God bless.

SPEAKER_00 (21:58):
Thanks for tuning in to Over the Bowl, brought to you
by Integris Design, a fullservice design and marketing
agency out of Asheville, NorthCarolina.
Until next time.

All Episodes

#38 - How We Actually Test AI Tools (And Why Most Fail)

Episode Transcript

Popular Podcasts

Stuff You Should Know

Dateline NBC

The Burden

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}#38 - How We Actually Test AI Tools (And Why Most Fail)