Data That Will Actually Help Your Business

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
As a business owner, one of the most difficult things
we have to deal with is data and how to
analyze that data and how to use that data to
help grow our business and serve our customers. So today
we have an expert to help us understand what data
to use and how to leverage that data to better

(00:21):
grow our business to better serve our customers. And with
us today is doctor Joe Sutherland of Emory University. Joe,
Welcome to the show. Thanks for having me and Joe,
if you would tell our wealth Ability audience a little
bit about your background and how you got into analytics.

Speaker 2 (00:42):
Of all places, A thank you for asking. It's a
great story. I mean, I've kind of lived three different lives,
you know. One as I worked at the White House
during the Barack Obama administration. I was on his personal
Advanced Staff, which is basically, like, you know, kind of
being a bag man for the present. You run around,
you put together his events. You you care for the

(01:05):
various very high level individuals that are involved and make
sure that they all feel very bubbly and nice when
they finally meet the President. And it was a great
impactful way to really see how policy influenced people's lives.
I mean I was elected back in like you know,
the college days to the student government presidency, and I

(01:28):
remember seeing how just community organizations could do so much
just for you know, a little teeny student body of
like three thousand people. But you know, when I went
to uh, for example, like Fort Stewart, Georgia with the
President to announce a new program that prevented you know,
fraudsters from stealing you know, GI bill benefits for for

(01:50):
veterans who had brain injuries. Actually literally they would basically
sign them up for these courses and they would start
taking the GI benefits and they wouldn't These people wouldn't
even know, you know, because their caretakers aren't nessarily monitoring everything,
and there was no enforcement mechanism to prevent that from happening.
So we went to Fort Stewart, the President signs, you know,
an executive order that that stopped that from happening. And

(02:11):
you know, just think about the millions of people's lives
who were affected by that. So that's kind of the
first life that I lived.

Speaker 1 (02:21):
Well, it must be must be have been amazing to
be with the ultimate community organizer, Barack Obama, Right if
when you're when you're talking about communities and organizing things.
I mean, he's he's the master of that and uh
and certainly the master, I think, one of the masters
of data and using data in his approach to politics.

(02:42):
So thanks for having with us. Your book is Analytics
the Right Way, which I love that title because I
think we all are looking at analytics. We're all looking
at data, but are we looking at the right data?
Are we looking at the right analytics? So let me
start with this, Joe, how do you determine what the
right data is to even collect before I even get

(03:05):
to how to collect it?

Speaker 2 (03:07):
Yeah, I think what a lot of people get wrong.
And you know I should mention also like I have
a PhD and data analysis essentially, and I also worked
as an executive at Amazon and Cisco, these big data companies,
so you know, Barack Obama absolutely a data driven guy,
you know, but there's also some business orientation here. I

(03:28):
want to make sure that your viewers have for context.
So I think what a lot of people start with
is this idea that if you collect all the data,
and you collect all the metrics, and if you put
them all into a giant database, then ultimately you will
be able to find the answer to any question that

(03:50):
you have. One of my colleagues, Matt Gershoff, calls.

Speaker 1 (03:53):
Isn't that the premise of AI, Well, well, so it isn't.

Speaker 2 (03:57):
I don't believe that it is, and I believe that
the people that are marked that way marketing it that way,
actually have some answering to do. You know, I do
think that one of the issues I'm seeing in particular
these days is, you know, you get these large cloud
providers you know, Google Cloud platform aws, you know, Azure
et cetera, that you know, data bricks that the you know,

(04:20):
the list goes on. I think a lot of the
promises that they make are hinged on this idea that
if you just consolidate the data, you'll get the answers.
And they have an incentive to talk about that, right,
because that's how they build. They build on your data utilization,
your compute utilization, and ultimately, by the time you finally
get around to trying to get get answers, you know,

(04:42):
some of those executives who sold you on that deal
to begin with have like you know, they they went
they left right, they went on too their next position,
or there's not really a lot of accountability. And the
point that I make in the book with my co
author Tim Wilson, is that you have to know what
questions you are trying to ask first before going and
collecting the data to be able to answer those questions.

Speaker 1 (05:04):
Okay, so let's let's go there. What questions should you
be asking if you're you know, you're you're a small
medium sized business, you're an entrepreneur, you're wanting to grow.
What questions are the ones you should be asking?

Speaker 2 (05:19):
The question should always be, especially as an entrepreneur, things
that are relevant to the growth of your business. Right, Like,
you know, if you're sitting there thinking about answering a
question that your data can answer, so you know, good examples,
A lot of people use like Google Analytics to track
what's happening on their website, and you know, they look
at how many visits am I getting, how many clicks

(05:40):
am I getting on my advertise, et cetera. Right, A
lot of these are we've refer to those as vanity metrics,
because who cares how many people come to your website?
What really matters is how many people end up converting, right,
or how many people go and sign up for a conversation.
And like the key is, uh, when you look at
Google Analytics, you're just getting a lot of data that

(06:01):
might have absolutely nothing to do with what actually causes
people to convert and actually schedule a message with you, right,
And what people do is they go searching for answers
on how to grow their business and data that might
actually have absolutely nothing to do with the answer that
they need.

Speaker 1 (06:18):
So how do you sift through that? How how do
you sift through all that that and say, okay, well
I've got all this data, but what's actually going to
help me?

Speaker 2 (06:27):
Yeah, So you have to start with the questions like
if you're interested in, for example, how do I how
do I get more customers or clients? You know, if
you're if you're you know, building an accounting firm, right,
which is actually a fantastic business to be in these days.
If you're building an accounting firm and you want to
grow your customer base, you have to figure out where
the customers are that aren't already coming to your website.

(06:51):
You know that that's a different question altogether from what
data do I have? That's a question that has to
do with, well, you know, how do I develop a
referral base from my high value customers, Right, how do
I start getting introductions to conference organizers and event planners
who are going to be able to help direct traffic
and interest my way? And those are data points that

(07:13):
you have to go and collect separately. Now, now you
know we're talking a little bit more technically about the
data you need to answer the question. But the key
is that if you have a hypothesis, which is, if
I go out and I start a new go to
market motion, it's going to be all based on referrals.
I'm really going to try just getting three referrals from
every high value customer that I have. Right, Like your

(07:35):
hypothesis might be if I do this for three months,
I'm going to have more net new customers than I
did before. Right, that might be the hypothesis. And what
you need to now do is track the information that's necessary,
such as the actual referred customers coming in, so that
you can answer that question. And when you really take
a step back, you know, it's almost so simple when

(07:57):
you think about it, Like you know, you're hearing from
from all these artificial intelligence providers, and I do direct
the Artificial Intelligence Center Emory University, So I know a
thing or two about AI. You know, what you hear
from these providers is all you got to do is
just get all this information and it'll answer the question
for you. But in reality, all you needed to do
is actually go out and start testing a hypothesis.

Speaker 1 (08:19):
Interesting. Okay, so you've got a hypothesis. Okay, if I
get referral, let me get let me give you a
real life one for me. Okay. I love using I
like being the guinea pig. I have. My hypothesis is
that if I get enough referral partners and these don't

(08:43):
have to be clients, but these are people who might
want to refer business, that I can replace myself and
I don't actually need to go out on stage. I
don't need to travel, I won't need to write more books.
I won't have to do that because I can do it.
I want to, but I won't have to do it
because all of these referral people will come in. So

(09:04):
where how would I go about? You know, give us
a few steps. How would go about collecting that data,
testing that hypothesis, and then and basically analyzing and valuing
you know, is this actually working? And how long do that?
How long do you have to test it before you know?

Speaker 2 (09:23):
Yeah, So there's there's different types of evidence. You know,
we we always think of being data driven as getting
to a certain answer, but the truth is that there's
varying levels of evidence that will come in from the
collection of data that help us inform the choices and
decisions that we make. Like, ultimately, you're trying to make
good decisions that propel your business forward. So what I

(09:46):
would start with is what is the level of information,
Like what is the informative piece of data that I
would need to see to believe that my referral strategy
is working? You know, that's simple things like well, am
I getting more customers in than before? You know when
I wasn't doing that strategy? That requires you to go

(10:08):
and establish a baseline that is not you know, necessarily
biased towards you know, a really good month or really
bad month. You have to go back and look at, Okay,
what would be the average expectation of where we would
be now and how would we be performing now versus
that expectation. That's that's kind of the key counterfactual that
you want to examine. I should mention too, counter Are

(10:30):
you familiar with counterfactuals.

Speaker 1 (10:31):
No, go ahead, explain that for us.

Speaker 2 (10:34):
So you know, another part of the book is all
about counterfactions, which is I think a lot of people
don't realize whenever we make choices in life, we basically
create a fork in the world. One one pathway is
the path we would have walked down if we hadn't
done the thing that we chose to do. And the
other pathway is the thing that is the pathway would

(10:55):
walk down for the thing that we did choose to do.
And that the key is is never to go back
and look at where you were before and compare to
where you are now. That's the wrong comparison. What you
want to do is compare yourself now after the choice
was made, to where you would have been had you

(11:16):
not made the choice. And a lot of analytics is
about going back and figuring out what that other counterfactual
outcome would have looked like had you not made the
choice that you did. And so, yeah, yeah, sorry, go ahead.

Speaker 1 (11:29):
So does that matter of just extrapolating from what you've
been doing, because obviously and saying, well, look what I've
been doing got me here, and so if I continue
doing it, it should get me there and comparing that
to what I actually did.

Speaker 2 (11:47):
To a certain extent. Yes, But let's talk about like
a good example is if you go and hire some
new like you know rvps to go and do sales
in like Southeast. Right, Let's say you want to go
and expand your firm and make an investment in the Southeast.
That could the success that you see could be entirely
dependent on the timing of that decision, right, So you

(12:08):
might go and see, for example, okay, like I'm doing
a million a year right now in sales, and when
I hired this new RVP for Southeast, you know, our
sales actually decreased to nine hundred thousand, right, Like, you
know a lot of people look at that and oh,
this clearly isn't working. You know, we need to part ways.
But you might have seen even lower sales if you

(12:30):
hadn't have made that decision, because, for example, the economy
could have been more volatile than you expected.

Speaker 1 (12:36):
So how would you know that? I mean, I mean
it's like to me that would be like magic. How
would I possibly know how? It's almost like proving a negative?
How do I do that?

Speaker 2 (12:47):
Yeah? So now you're getting into analytics and a lot
about the book, which is, you know, there are different
methodologies you can use to establish what those counterfactuals would
have been and what they would have looked like. You know,
you hear a lot about like, you know, a lot
of people talk about this tree of data. I've ever
seen this. It's like, there's descriptive statistics, there's predictive statistics,

(13:11):
there's prescriptives to like have you seen this kind of
like I think these charts go around LinkedIn, you know,
they get a lot of traffic, which is which is like,
which is funny to me because I guess I should
do a chart that's really complicated, right, I should go
and like, oh this goes to this that a lot
of people look at it, you know, it gets more impression.
You know, so I actually think that that model is
like entirely, I just think it's not useful. The real

(13:33):
ladder of evidence that you should be looking at is
how strong of a signal do I need to get
from the data analysis that I do to feel comfortable
that the decision that I should make or did make
was a good one or will be a good one.
And that comes back to being able to predict what
that kinter factual looks like. You can use descriptive statistical

(13:53):
methodologies to do that. That's simply looking back at the
month by month variation for eggsample in your level of sales.
That gives you a boundary of errors and other pathways
that you would have seen, so that finally, when you
go and do test your hypothesis, you can know what
the probability was that you actually would have seen the
answer that you did see. You know, So there are

(14:15):
statistical methodologies to do that descriptively, but there's also scientific
methodologies you can use to do what's called causal inference,
which is when I did A, did it really cause
B to happen? And you can use those methodologies if
you think about them prior to actually making the choice.

Speaker 1 (14:35):
So give us an example of that, because I think
that is the holy grail to marketing basically is causation. Right,
did this you know when I put out when I
spent this much money on Google AdWords, did it cause
this result? So how do you go about that? I mean,

(14:59):
walk us through it if you if you if you
don't mind, walk us through it a little bit.

Speaker 2 (15:03):
Yeah, so you know that's your you're kind of this
is an homage to the old like Wanna Maker quote,
which is, you know, if you don't know who Wannamaker was,
he started like the modern department store. You know. He
used to say, you know, I know some of my
marketing is working, I just don't know which half of
it is. And causal inferences of methodology set you can

(15:25):
use to try to actually figure out, you know, what
mix of investments should you be making, for example, in
different channels, using different messages to be able to see
the returns on your ad spend or investment that you
want to see. Uh, if you would take a good
example for me was I had a large pharmaceutical client,
and I think I think we actually talked about this
in the book, So feel free to pick up the

(15:46):
book and kind of read it and more. The book
is full of these examples and use cases. You know,
large pharmaceutical client that was interested in figuring out when
they showed an advertisement that went directly to patients, Uh,
did that actually improve the probability that they were going
to ask their physicians for a type of medication? Right?
So that's called direct to consumer marketing as opposed to

(16:07):
just marketing to the physicians. Themselves. A lot of pharmaceutical
companies do this, and they spend a lot of money
doing it, but it's never really super clear if it
was if it was a successful effort, or if they
you know, I guess in a different in a different frame,
if they had made as much money using that strategy,
you know, television for example, as they would have if
they had gone out and done it on Google ads.

(16:28):
And so what you do is you can design, for example,
of one methodology, a randomized controlled trial, So you know, RCTs,
as we call them in the business, are ways of
trying to establish what the counterfactual would have looked like
using the beauty of randomization. When you randomize, essentially, what

(16:50):
it does is it splits your sample of subjects into
two perfectly usually you know you could check it right,
but perfectly an expectation comparable groups of people. And when
you go on all elements, even elements that you couldn't
collect data on or observed, and you get one group
of people who we don't give the advertising to, and

(17:10):
we get another people who we do. And then we
look at the difference between the level of sales coming
out of those areas, for example, where those people got
those advertisements, and that will help you determine scientifically really
what the true effect is of the marketing efforts.

Speaker 1 (17:28):
And now internet marketing, we would call that a B testing, right.

Speaker 2 (17:33):
Exactly, it's called a B testing. What people don't sort
of realize though, is I think a B testing is
usually thought of as something that you would do on
just an individual level basis. So when you buy television advertising,
for example, you're buying it at the address block level,
or you're buying it you know a lot. I mean,

(17:53):
radio is a it's it's just a broadcast that has
a catchment area, right that may or may not hit you.
So in a lot of ways, like people think AB
testing is only possible or RCTs are only possible when
you have individual level data. Uh uh, but that but
that's actually not the case. What you can do is

(18:14):
you can look at running RCT designs that are georandomized
or block randomized. There's lots of different ways to do
it where you can then go and aggregate the level
of sales in the areas or blocks as opposed to
the individual levels that you would want to see to
go and estimate your treatment effect.

Speaker 1 (18:32):
So do you go through. Do you go through these
in your book?

Speaker 2 (18:35):
Yeah, we go through all these different methodologies and and
you know it's kind of like a handbook. It's you know,
and it starts it starts very small, right. You know
that the easiest type of evidence to generate, it's just
anecdotal evidence, which is like when you went out and
talk to somebody about your product with a new type
of message. Let's just say, like, you know, before you

(18:55):
were selling toothpaste and you said it's going to make
your teeth clean. Now you're going to go out to say, like, actually,
it makes your breath menty too. Just asking for feedback
from customers can be a form of useful evidence, right,
But if you're about to make that's great if you're
not putting any money on the line, right, But if
you're about to put a million or two million bucks

(19:16):
on the line to run a new advertisement or you know,
seven million bucks in a lot of cases these days,
because it requires so much more frequency, you know, you
want to be a lot more sure or a lot
more certain that it's going to actually return and the
return on investment's going to be good. And so when
you run an RCT, that's you want to use that
level of evidence creation when you're trying to really answer

(19:39):
a really heavy question. Anecdotal's fine if you don't have
any money on the line, and descriptive is kind of
somewhere in between.

Speaker 1 (19:46):
Interesting. So what are some steps that you would encourage
an entrepreneur to do to start first of all, collecting
the right data and asking the right questions and then
going through the the appropriate analysis.

Speaker 2 (20:02):
Yeah, no, I love this question. I think too often
people get paralyzed by you know, I don't know, there's
so many signals out there what should I be looking at?
The First step is really just sitting down and connecting
with the core business needs and articulating what the decisions
that you think you want to make are to be
able to achieve some goal. So, if you're interested in growth,

(20:24):
what are the decisions that you might make that would
try to propel growth. If you're interested in cost minimization
or efficiency, what are the choices you would make internally
to try to decrease the costs associated with your operating model.
If you're interested in just simply like getting greater awareness, right, like,

(20:44):
what are the choices you would make to get better awareness?
And once you know your goal is, you can start
thinking about the choices you would make to achieve that goal,
and then the designs of you know, data collection and
analysis that you would then approach to be able to
answer whether or not the choices were successful or not.
And in the book we talk about what's called the

(21:04):
hypothesis library. It basically is a step by step plan
to go and create an Excel spreadsheet of all the
different choices that you might make and linking them back
to the things that you would need to see or
analyze to be able to know if you made the
right choice. And then you know, putting a rough estimate
on what the actual ROI you anticipate would be. And

(21:24):
once you go and sort the list on investment and return,
you'll get the top five ideas you should try right now.
I really think what entrepreneurs suffer from is you know,
analysis paralysis and other sorts of like, you know, this
problem of I'm only seeing the data that I think
I need when in reality, I'm not collecting the right
data to answer my question. And I think it causes

(21:46):
people not to act. And you have to be nimble
and you have to make choices that are smart and
mitigate your risk and business to be able to succeed.
Otherwise you're not really generating any growth at all.

Speaker 1 (21:58):
Well, it can certainly be overwhelming if you're not a
If you're not I mean a lot of entrepreneurs, they're
not detailed people to begin with, right, I mean, these
are people who like doing deals, they like selling, they
like being out with the customer. And that's your more
typical entrepreneur, right. And so now all of a sudden
you're saying, oh, but we need all this data and
we need to get into the details. And is there

(22:23):
one way you can think of? I mean, you mentioned
your I bought the success britcheet, but is there there
any other way you can think of to kind of
help an entrepreneur get into the data and be comfortable
with It's just a step by step process. It doesn't
have to be that complex complicated.

Speaker 2 (22:42):
It doesn't have to be complicated. I what I start
with with my clients is usually, you know, I take
out a yellow pad and I start just writing down,
you know, what are the what are the goals that
you want to achieve? What are the objectives that you desire?
And from there we start to think, well, okay, are
these objectives linkable to sorts of you know, metrics or

(23:07):
measures that might indicate whether or not they were successful
at being achieved, and a lot of that what people
find is, oh, these these metrics have absolutely nothing to
do with the data that is already available to me,
you know. And then from there, thinking through, well, how
important would it be to go and collect that data

(23:28):
to be able to answer your hypothesis. If you're just
trying to you know, I don't know, test a new message,
then it doesn't make sense to go and invest three
hundred thousand dollars to build a system to collect the
data to test your hypothesis. If you're if there's not
really a lot of risk associated with it, right, Whereas
if you're about to put down a five million dollar investment,

(23:51):
and could I mean even for a manufacturer, could be
a huge piece of machinery that they want to take
a loan out on. There's lots of implications to these
large was to get made. You know, then you really
want to think about investing in the right data collection infrastructure.
And from there we start thinking like, well, actually, you know,
you don't need to go for the full mante you
just need this this is the one thing you need

(24:12):
to be able to decrease your uncertainty about this decision.
And usually what we end up is it's actually a
lot easier than people are thinking.

Speaker 1 (24:20):
I love that it goes back to what I said
multiple times is that if you want to get a
better answer, you need to ask a better question, and
the data can help you ask better questions, but sometimes
you need to actually figure out what the question is
before you ever get to the data in the first place.
And I think that's a terrific message, Joe. The book

(24:42):
is Analytics the Right Way, a business Leader's guide to
putting data to production use. Joe, thank you so much
for being with us. Tell us where we could get
more information about you, and I know you're a professor,
but you also have a consulting company where we get
go for more information about what you do.

Speaker 2 (24:59):
Absolutely, you can go and visit my website Jlsutherland dot
com for more information and if you're interested in the book,
we sell it to the best seller actually now worldwide
on Amazon dot com.

Speaker 1 (25:12):
Awesome. Thank you so much and thanks everybody for listening.
And you know, if you can see the value of
analytics and how to do this, the right way. Please
share this with your other friends that are entrepreneurs, your
other business leaders. Make sure you like it, send it
to other people. Because what we want to know is
we want to know what's the right question. We want

(25:35):
to know do we have the data behind it? And
can we actually reduce the amount we spend and spend
our money more wisely so that our expenses are always
producing income that we want and by doing so, we'll
always make way more money and pay wayless tax. We'll
see you next time on the Wealth of By Show.
This podcast is a presentation of which Dad Media Network

All Episodes

Episode Transcript

Popular Podcasts

My Favorite Murder with Karen Kilgariff and Georgia Hardstark

24/7 News: The Latest

Dateline NBC

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Data That Will Actually Help Your Business

Episode Transcript

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}My Favorite Murder with Karen Kilgariff and Georgia Hardstark

24/7 News: The Latest

Dateline NBC

All Episodes

Data That Will Actually Help Your Business

My Favorite Murder with Karen Kilgariff and Georgia Hardstark