February 15, 2023 • 12 mins
In this episode, Brick and Caleb discuss different causes of untrustworthy data, and how to expose and address it to avoid flawed decision-making and improve adoption of your BI efforts.
Click here to watch this episode on our YouTube channel.
Blue Margin helps private equity owned and mid-market companies organize their data into dashboards to execute on strategy and create a culture of accountability. We call it The Dashboard Effect, the title of our book and podcast.
Visit Blue Margin's library of additional BI resources here.
For a free, downloadable copy of our book, The Dashboard Effect, click here, or buy a hardcopy or Kindle version on Amazon.
#BI #businessintelligence #adoption #PowerBI
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Brick Thompson (00:05):
Welcome to The Dashboard Effect Podcast. I'm
Brick Thompson.
Caleb Ochs (00:08):
Caleb Ochs.
Brick Thompson (00:09):
Alright, Caleb.
Today, we're gonna be talkingabout untrustworthy data. What
is untrustworthy data?
Caleb Ochs (00:17):
It's data you can't rely on.
Brick Thompson (00:20):
Very good. Yeah.
So untrustworthy data is bad fora lot of reasons, we'll talk
about that. There's sort of fourmain types. I mean, everything
falls into these buckets. You'vegot stale data. So data that's
out of date, or it's not beingcollected anymore. You've got
unclear data. This is the typeof data where you may have
(00:40):
naming conventions that aren'tbeing followed. The classic
example is, you've got acustomer list or customer data
set of customer names, and youhave two names that are slightly
different for the same customer,and getting those to line up.
You've got inaccurate data,which speaks for itself. And
then of course, no data, whichdefinitely happens sometimes. So
(01:02):
the problems with untrustworthydata, what's the main problem
that you see?
Caleb Ochs (01:09):
there's a lot of different problems, right? I
mean, there's the obvious ones,like, you can just make the
wrong decision because you'rerelying on something that's not
reliable. That's a good one.
Obviously, if people can't trustthe data, they're going to
figure out what they can do sothey can trust the data. And
that usually involves pullingthe data themselves and
(01:30):
reporting on it and mashingthings up on their own. So lots
of different ways.
Brick Thompson (01:37):
Yeah, yeah. And, we talked last week about
becoming a data drivenorganization, and how user
adoption and trust in the dataso that people actually start to
use it to drive the organizationcan really be hampered by a lot
of things. But untrustworthydata is one of the biggest ones.
Caleb Ochs (01:58):
Yeah, right. And it's so easy to get
untrustworthy data. Just to haveit start permeating through your
systems. And now you realizethat it's causing a problem if
you don't have visibility intoit. So you may have a process
that's not quite being followedthe way you intended. Or maybe
you have a poorly designedprocess that is causing some bad
(02:22):
data into your systems. Youmight have no way for people to
see that the data is actuallybad so they can't go clean it
up, and they can't fix it. Theycan't stay on top of it. Lots of
different ways that it justhappens throughout the course of
business if you're not keepingan eye on it. And you usually
uncovered at the mostinconvenient times.
Brick Thompson (02:43):
Yeah, yeah.
Well, hopefully you uncover it.
I mean, one of the problems withbad data is, you may not even
know it's bad, so you're makingdecisions on data that's not
good. I think one of the pointsyou made, as we were talking
about this earlier, was, one ofthe best ways to figure out your
(03:04):
bad data is just to giveeverybody exposure to the data,
because people will notice it,and then can bring it to
attention. But you also have tohave processes for fixing it
then.
Caleb Ochs (03:13):
Right, right. I think what we see a lot is, and
this even happened at one of myearly career jobs. I was an
analyst, and one of the thingsthat I had to do, I was creating
these inventory reports that Iwas sending out to salespeople,
so they knew where theirinventory was and what they
could sell and that type ofthing. And there was some data
(03:35):
in there that every week I hadto move one business unit's
inventory into another isbecause they were just
categorized incorrectly. Andthat's just a small example of
what happens in most of thecompanies that we see anyway,
where they have some sort ofmanual reporting process. And
that's just where your data getsfixed. The problem with that is
(03:57):
that, that happens every time,and you just hope that it's the
same every time. And then ifthat person leaves you hope that
the next day there is some gooddocumentation and it can turn
into a mess. And you see thatall the time.
Brick Thompson (04:09):
Yeah, those processes can fall apart fast.
So what are the best ways to goafter it? What are some best
practices to try to avoidgetting bad data?
Caleb Ochs (04:18):
Yeah, so we already touched on one, get the data out
there and visible. But I thinkone of the first things that you
should do is define what yourmetrics are. This seems so
elementary, but it's soimportant. It's surprising. It's
shocking, really, how many timeswe see companies come in. And
(04:41):
when they start working with us,and we start doing projects with
them, it's like redefining allof their metrics. They didn't
know that, this group of people,group A defines differently than
Group B. And there's actuallyconfusion on what the metrics
actually are, and then you gottago back and forth and get an
alignment on it. So a lot oftimes, you may not actually have
(05:05):
bad data. But it may beuntrustworthy, because people
are viewing it differently.
Brick Thompson (05:10):
Right, yeah. And you'd be surprised at how often,
well you wouldn't be, but somepeople would be surprised at how
often something that seems like,"Alright, you can't define this
differently." Like net profit.
Sure everybody's defining thatthe same. And then you find out
sometimes they're not. Yeah, so,so critical to align on that,
because that's not evennecessarily an underlying data
(05:30):
problem. But it's still aproblem when it shows up on the
reports, and now you haveinconsistent data. And so people
view that as untrustworthy,they're less likely to use it.
And you hamper your ability tobecome data driven even more
that way.
Caleb Ochs (05:47):
Right. So another thing that you could do is you
can designate an owner of yourmetrics. Picture like your org
chart, and then each personwould have one or many metrics
that they are responsible for,and they know they're
responsible for it. Everybodyknows who's responsible for
what, and then not only is thatjust great for your business,
(06:09):
but it also, I'll note, allowsthose people to start taking
action that's going to keep thatdata clean and kind of spring,
along process change, or justdata cleanliness efforts, those
types of things. If they knowthat, that's kind of their
thing. They want that to beaccurate, right?
Brick Thompson (06:27):
Right. If you're responsible for a number, and
you're looking at it, and it'snot reflecting well on your
responsibility, and you suspectit's because of bad data
underneath, they're gonna goafter that and fix it.
Caleb Ochs (06:38):
Yeah, right.
Brick Thompson (06:39):
And it's just good practice from a management
standpoint, anyway. Everybodyshould have accountability. And
in the best case, everybody hasa number. Everybody wants to
know, the score. So you sort ofget a double benefit with that.
Caleb Ochs (06:51):
Yeah, exactly. That it's a great thing to do.
Brick Thompson (06:53):
Yeah. Okay. What else?
Caleb Ochs (06:55):
So we already talked about just exposing the data,
just get it out there, get itinto the front hand, or the
frontline workers hands, right.
My first job out of college, itwas a lot of very manual stuff
that I did. And I was so in theweeds on handling specific
(07:17):
details at a time, I neverreally had a chance to step back
and look at, "Alright, iseverything that I'm doing here,
how's it winding up in thedata?" It was a lot of data
entry stuff. And it would havebeen great to have that. Every
day to be able to see the globalview of just my data, just,
(07:39):
"Does this look right?" And ifit didn't, then I could go fix
it that day. But every day forme, it was just record by record
by record by record. I thinkthat happens a lot. And your
frontline people will cleanstuff up if they know where the
problems are.
Brick Thompson (07:55):
So you're just doing data entry, but you don't
know sort of the macro view.
You're not seeing the checksumof, "Is this adding up to
something that makes sense?"
Caleb Ochs (08:03):
Right, I may have made a mistake or knew I was
gonna go back and fix somethingand forgot, or whatever. But
having that bigger view allowsyou to do that. And there's no
problems with giving that typeof view to those people, and
they will fix it. It's amazing.
Brick Thompson (08:19):
Yeah. Okay.
Yeah, that's a good example.
Another thing that comes up issomething we talked about, just
having unclear data. So we had acustomer that had a bunch of
products in a product database.
And there were things like, thesame product would be called
Snickers in one place, andSnickers Bar and another place,
and 12 ounce Snickers somewhereelse. But they were all the same
(08:43):
thing, and in the sales report,they wanted to be able to report
on that one thing. And so youcan actually go after that and
do what's called data qualityservices or used data quality
services to clean that up andactually give people who know
the ability to go in and say,"This is actually the same as
this," So that the the endreport is what you were hoping
(09:03):
for. "It's not actually threeSnickers bars. There's one."
Caleb Ochs (09:08):
Right, right. Yeah, that can be a really powerful
tool. And I think you touched onit early, about the customer
issue that's very, very similar.
Like you have the same customer,but they're spelled slightly
differently or something. Thatcan be really important when
you're looking at lifetime valueof your customers and things
like that. We see it a lot inthe private equity space of the
buy and build, where you'rebringing on different companies
(09:31):
that just name thingsdifferently. But it turns out
now you're going to want to rollthose up under a single
customer. And doing somethinglike that helps you do that.
Brick Thompson (09:43):
Yeah, for sure.
I think one of the mostimportant things, well, you've
already said expose the data,but people shouldn't be
surprised that once they startdoing reporting, it's going to
expose untrustworthy data. Butthat's a good thing. because
then you can go back and fix it.
Caleb Ochs (09:59):
Yeah, I think in our projects, I think back to so
many of them, when we get intothem, if we find out like,
"Sorry that you've beenreporting on this for 10 years,
but it's wrong." You know, youreally start to do an audit like
that, and that can be a goodthing. It shouldn't be viewed
badly. It's one of those thingsthat you just have to do, and
(10:22):
going through that process iskind of a necessary evil, but
ultimately it ends up in hugebenefit.
Brick Thompson (10:31):
Yeah, yeah, for sure. Yeah, I think another
thing, it touches on a podcastwe did maybe three or four weeks
ago, about, data projectsshouldn't be viewed as one and
done. It's not a discreteproject once in time and that's
it. Data is really sort of afoundational resource. And you
(10:52):
need to be ready to go in andmake changes and be okay with
that. So, just get that sort ofinto the data culture of the
company, that it's a livingthing, there's going to be
changes that we make to thebusiness. And so we've got to
change maybe data we'recollecting, or we change the
instrumentation, or we augmentthe data with a new data source.
(11:13):
That's just sort of part of it.
And the sooner you realize thatand make it part of your whole
data culture, the more likelyyou'll be to not end up with
untrustworthy data. You're stillgoing to have cases of
untrustworthy data, but you cango in and correct it.
Caleb Ochs (11:30):
Yeah, that's exactly right. And then another thing
that we touched on our lastepisode was that community of
practice. If you've gotsomething like that set up,
people know where to go askquestions, rather than see
something that looks maybewrong, and then just think that
that reports wrong. Or thinkthat the data is just bad. They
can go ask the question, theycan get a response. And then
(11:51):
that helps build the trust inthe data and the reports that
you've got out there.
Brick Thompson (11:57):
Yeah. Okay. All right. Anything else you want to
say on the topic?
Caleb Ochs (12:03):
You know, untrustworthy data is just bad.
Brick Thompson (12:07):
That's a good summary.
Caleb Ochs (12:08):
There you go.
Brick Thompson (12:10):
All right.
Thanks, Caleb.
Welcome to The Dashboard Effect Podcast. I'm
Brick Thompson.
Caleb Ochs (00:08):
Caleb Ochs.
Brick Thompson (00:09):
Alright, Caleb.
Today, we're gonna be talkingabout untrustworthy data. What
is untrustworthy data?
Caleb Ochs (00:17):
It's data you can't rely on.
Brick Thompson (00:20):
Very good. Yeah.
So untrustworthy data is bad fora lot of reasons, we'll talk
about that. There's sort of fourmain types. I mean, everything
falls into these buckets. You'vegot stale data. So data that's
out of date, or it's not beingcollected anymore. You've got
unclear data. This is the typeof data where you may have
(00:40):
naming conventions that aren'tbeing followed. The classic
example is, you've got acustomer list or customer data
set of customer names, and youhave two names that are slightly
different for the same customer,and getting those to line up.
You've got inaccurate data,which speaks for itself. And
then of course, no data, whichdefinitely happens sometimes. So
(01:02):
the problems with untrustworthydata, what's the main problem
that you see?
Caleb Ochs (01:09):
there's a lot of different problems, right? I
mean, there's the obvious ones,like, you can just make the
wrong decision because you'rerelying on something that's not
reliable. That's a good one.
Obviously, if people can't trustthe data, they're going to
figure out what they can do sothey can trust the data. And
that usually involves pullingthe data themselves and
(01:30):
reporting on it and mashingthings up on their own. So lots
of different ways.
Brick Thompson (01:37):
Yeah, yeah. And, we talked last week about
becoming a data drivenorganization, and how user
adoption and trust in the dataso that people actually start to
use it to drive the organizationcan really be hampered by a lot
of things. But untrustworthydata is one of the biggest ones.
Caleb Ochs (01:58):
Yeah, right. And it's so easy to get
untrustworthy data. Just to haveit start permeating through your
systems. And now you realizethat it's causing a problem if
you don't have visibility intoit. So you may have a process
that's not quite being followedthe way you intended. Or maybe
you have a poorly designedprocess that is causing some bad
(02:22):
data into your systems. Youmight have no way for people to
see that the data is actuallybad so they can't go clean it
up, and they can't fix it. Theycan't stay on top of it. Lots of
different ways that it justhappens throughout the course of
business if you're not keepingan eye on it. And you usually
uncovered at the mostinconvenient times.
Brick Thompson (02:43):
Yeah, yeah.
Well, hopefully you uncover it.
I mean, one of the problems withbad data is, you may not even
know it's bad, so you're makingdecisions on data that's not
good. I think one of the pointsyou made, as we were talking
about this earlier, was, one ofthe best ways to figure out your
(03:04):
bad data is just to giveeverybody exposure to the data,
because people will notice it,and then can bring it to
attention. But you also have tohave processes for fixing it
then.
Caleb Ochs (03:13):
Right, right. I think what we see a lot is, and
this even happened at one of myearly career jobs. I was an
analyst, and one of the thingsthat I had to do, I was creating
these inventory reports that Iwas sending out to salespeople,
so they knew where theirinventory was and what they
could sell and that type ofthing. And there was some data
(03:35):
in there that every week I hadto move one business unit's
inventory into another isbecause they were just
categorized incorrectly. Andthat's just a small example of
what happens in most of thecompanies that we see anyway,
where they have some sort ofmanual reporting process. And
that's just where your data getsfixed. The problem with that is
(03:57):
that, that happens every time,and you just hope that it's the
same every time. And then ifthat person leaves you hope that
the next day there is some gooddocumentation and it can turn
into a mess. And you see thatall the time.
Brick Thompson (04:09):
Yeah, those processes can fall apart fast.
So what are the best ways to goafter it? What are some best
practices to try to avoidgetting bad data?
Caleb Ochs (04:18):
Yeah, so we already touched on one, get the data out
there and visible. But I thinkone of the first things that you
should do is define what yourmetrics are. This seems so
elementary, but it's soimportant. It's surprising. It's
shocking, really, how many timeswe see companies come in. And
(04:41):
when they start working with us,and we start doing projects with
them, it's like redefining allof their metrics. They didn't
know that, this group of people,group A defines differently than
Group B. And there's actuallyconfusion on what the metrics
actually are, and then you gottago back and forth and get an
alignment on it. So a lot oftimes, you may not actually have
(05:05):
bad data. But it may beuntrustworthy, because people
are viewing it differently.
Brick Thompson (05:10):
Right, yeah. And you'd be surprised at how often,
well you wouldn't be, but somepeople would be surprised at how
often something that seems like,"Alright, you can't define this
differently." Like net profit.
Sure everybody's defining thatthe same. And then you find out
sometimes they're not. Yeah, so,so critical to align on that,
because that's not evennecessarily an underlying data
(05:30):
problem. But it's still aproblem when it shows up on the
reports, and now you haveinconsistent data. And so people
view that as untrustworthy,they're less likely to use it.
And you hamper your ability tobecome data driven even more
that way.
Caleb Ochs (05:47):
Right. So another thing that you could do is you
can designate an owner of yourmetrics. Picture like your org
chart, and then each personwould have one or many metrics
that they are responsible for,and they know they're
responsible for it. Everybodyknows who's responsible for
what, and then not only is thatjust great for your business,
(06:09):
but it also, I'll note, allowsthose people to start taking
action that's going to keep thatdata clean and kind of spring,
along process change, or justdata cleanliness efforts, those
types of things. If they knowthat, that's kind of their
thing. They want that to beaccurate, right?
Brick Thompson (06:27):
Right. If you're responsible for a number, and
you're looking at it, and it'snot reflecting well on your
responsibility, and you suspectit's because of bad data
underneath, they're gonna goafter that and fix it.
Caleb Ochs (06:38):
Yeah, right.
Brick Thompson (06:39):
And it's just good practice from a management
standpoint, anyway. Everybodyshould have accountability. And
in the best case, everybody hasa number. Everybody wants to
know, the score. So you sort ofget a double benefit with that.
Caleb Ochs (06:51):
Yeah, exactly. That it's a great thing to do.
Brick Thompson (06:53):
Yeah. Okay. What else?
Caleb Ochs (06:55):
So we already talked about just exposing the data,
just get it out there, get itinto the front hand, or the
frontline workers hands, right.
My first job out of college, itwas a lot of very manual stuff
that I did. And I was so in theweeds on handling specific
(07:17):
details at a time, I neverreally had a chance to step back
and look at, "Alright, iseverything that I'm doing here,
how's it winding up in thedata?" It was a lot of data
entry stuff. And it would havebeen great to have that. Every
day to be able to see the globalview of just my data, just,
(07:39):
"Does this look right?" And ifit didn't, then I could go fix
it that day. But every day forme, it was just record by record
by record by record. I thinkthat happens a lot. And your
frontline people will cleanstuff up if they know where the
problems are.
Brick Thompson (07:55):
So you're just doing data entry, but you don't
know sort of the macro view.
You're not seeing the checksumof, "Is this adding up to
something that makes sense?"
Caleb Ochs (08:03):
Right, I may have made a mistake or knew I was
gonna go back and fix somethingand forgot, or whatever. But
having that bigger view allowsyou to do that. And there's no
problems with giving that typeof view to those people, and
they will fix it. It's amazing.
Brick Thompson (08:19):
Yeah. Okay.
Yeah, that's a good example.
Another thing that comes up issomething we talked about, just
having unclear data. So we had acustomer that had a bunch of
products in a product database.
And there were things like, thesame product would be called
Snickers in one place, andSnickers Bar and another place,
and 12 ounce Snickers somewhereelse. But they were all the same
(08:43):
thing, and in the sales report,they wanted to be able to report
on that one thing. And so youcan actually go after that and
do what's called data qualityservices or used data quality
services to clean that up andactually give people who know
the ability to go in and say,"This is actually the same as
this," So that the the endreport is what you were hoping
(09:03):
for. "It's not actually threeSnickers bars. There's one."
Caleb Ochs (09:08):
Right, right. Yeah, that can be a really powerful
tool. And I think you touched onit early, about the customer
issue that's very, very similar.
Like you have the same customer,but they're spelled slightly
differently or something. Thatcan be really important when
you're looking at lifetime valueof your customers and things
like that. We see it a lot inthe private equity space of the
buy and build, where you'rebringing on different companies
(09:31):
that just name thingsdifferently. But it turns out
now you're going to want to rollthose up under a single
customer. And doing somethinglike that helps you do that.
Brick Thompson (09:43):
Yeah, for sure.
I think one of the mostimportant things, well, you've
already said expose the data,but people shouldn't be
surprised that once they startdoing reporting, it's going to
expose untrustworthy data. Butthat's a good thing. because
then you can go back and fix it.
Caleb Ochs (09:59):
Yeah, I think in our projects, I think back to so
many of them, when we get intothem, if we find out like,
"Sorry that you've beenreporting on this for 10 years,
but it's wrong." You know, youreally start to do an audit like
that, and that can be a goodthing. It shouldn't be viewed
badly. It's one of those thingsthat you just have to do, and
(10:22):
going through that process iskind of a necessary evil, but
ultimately it ends up in hugebenefit.
Brick Thompson (10:31):
Yeah, yeah, for sure. Yeah, I think another
thing, it touches on a podcastwe did maybe three or four weeks
ago, about, data projectsshouldn't be viewed as one and
done. It's not a discreteproject once in time and that's
it. Data is really sort of afoundational resource. And you
(10:52):
need to be ready to go in andmake changes and be okay with
that. So, just get that sort ofinto the data culture of the
company, that it's a livingthing, there's going to be
changes that we make to thebusiness. And so we've got to
change maybe data we'recollecting, or we change the
instrumentation, or we augmentthe data with a new data source.
(11:13):
That's just sort of part of it.
And the sooner you realize thatand make it part of your whole
data culture, the more likelyyou'll be to not end up with
untrustworthy data. You're stillgoing to have cases of
untrustworthy data, but you cango in and correct it.
Caleb Ochs (11:30):
Yeah, that's exactly right. And then another thing
that we touched on our lastepisode was that community of
practice. If you've gotsomething like that set up,
people know where to go askquestions, rather than see
something that looks maybewrong, and then just think that
that reports wrong. Or thinkthat the data is just bad. They
can go ask the question, theycan get a response. And then
(11:51):
that helps build the trust inthe data and the reports that
you've got out there.
Brick Thompson (11:57):
Yeah. Okay. All right. Anything else you want to
say on the topic?
Caleb Ochs (12:03):
You know, untrustworthy data is just bad.
Brick Thompson (12:07):
That's a good summary.
Caleb Ochs (12:08):
There you go.
Brick Thompson (12:10):
All right.
Thanks, Caleb.
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Therapy Gecko
An unlicensed lizard psychologist travels the universe talking to strangers about absolutely nothing. TO CALL THE GECKO: follow me on https://www.twitch.tv/lyleforever to get a notification for when I am taking calls. I am usually live Mondays, Wednesdays, and Fridays but lately a lot of other times too. I am a gecko.