006: Why our brains like order and how John Hattie's rankings impacted education

Send us a text

In this episode Stephanie and Zac talk to Dr. Ji Sun from Cal State Los Angeles to get a better understanding of why it's so hard to add complexity to information once we've got our minds made up as well as some helpful tricks to make new information stickier. Then, they dive in to the work of John Hattie and begin to examine how his attempts to share what's "effective" in education spread like wildfire.

Episode Transcript

SPEAKER_01 (00:00):
Hello, it's good to have you back.
Welcome to this episode ofAcademic Distinctions.
Today we're going to be talkingwith Professor G.
Sun on why our brains likethings to be in order and what
makes those things sticky.
And we're going to be discussinga guy named John Hattie and the
impact he had on education whenhe published a book that sold a

(00:22):
bunch of copies.
So stick around.
I think you'll like it.

UNKNOWN (00:27):
Music

SPEAKER_01 (00:39):
Before we talk about education today, we wanted to
stop and talk about somethingwe've noticed as a trend as we
prepare these episodes for you.
Each time we see someone'smagnum opus of education
research full of citations andcomplexity, we also see that the
history shows the most basicpieces of that research that
make it into the field.

SPEAKER_03 (01:00):
See Bloom's Triangle and the fact that he said it was
the most often quoted book abouteducation that no one has ever
read.
So today we're going to start byhelping all of us understand our
brains a little bit better.
To do that, we're happy towelcome Dr.
Ji Sun from Cal State LA.
Hi, Ji.
Hi.

(01:20):
Great to be on the pod.
We're so excited to have you.

SPEAKER_01 (01:23):
So I think we're going to have a pretty
interesting conversation today.
We are struggling with a lot ofhierarchies and a lot of lists.
And as we read these books thatpeople...
love but haven't read, wenoticed that there's a lot
that's getting left out.
And so we were hoping you canhelp us think through and

(01:44):
understand maybe how brains areworking.
And so here are two importantthings we noticed.
In our episode about Bloom'staxonomy, that he and his
colleagues never organized theirhierarchy into, or their
taxonomy, into a hierarchy or atriangle.
But the way that our brains andthe way that it gets passed on

(02:06):
culturally is that that taxonomyis a hierarchy and that it's in
a triangle.
And then the other piece thatwe're noticing, and these are
just kind of the roots of uswanting to talk to you are that
when Anderson and Marzano andothers tried to improve upon or
add complexity to bloom at all'soriginal work, the thing that

(02:31):
stuck was the, hierarchy.
So the thing that is stickingaround is the idea that is not
the idea that was put forth inthe first place.
And when we try to coursecorrect, culture and brains
stick to the stuff that's there.
Why does that happen?

SPEAKER_00 (02:48):
Well, first, I want to say the fact that the
triangle was sticky, it tells ussomething about humans.
So it's not so much like, oh,why do humans love triangles?
It's really, why did we lovethis triangle, right?
But I love triangles, G.

SPEAKER_01 (03:05):
More of an octagon man myself, but please carry

SPEAKER_00 (03:07):
on.
There was something about itthat struck a nerve.
And I would argue, I mean, Idon't really know why the
triangle had a moment, right?
But I would argue that itprobably hit a nerve.
It probably mimicked some partof people's experience where
they felt like, hey, I look atthat and I recognize something
in it.

(03:28):
Now, the question is, what isthe experience that kind of got
triggered by this triangle suchthat it's the thing that made it
into the cultural lexicon?

SPEAKER_01 (03:41):
Is it?
So we also noticed, or from kindof our own anecdotal experience,
is that people know the bottomlevel and they know the top
level.
Yeah.
But if you asked them to fill inall of the different pieces in
between, their minds are kind ofa blank and they're like, well,
this one is somewhere in

SPEAKER_00 (03:58):
there.
It's like a rainbow in between,right?
Yeah.
And I would argue...
And maybe Jim Stigler and manyothers would argue, we probably
really only remember three.
So Bloom's triangle, if I couldcall it that, the problem with
it is that it has more thanthree things.
So it doesn't really stick.

(04:18):
So we remember it almost as likea continuum, right?
Like there's the worstmemorization and there's the
best creating, applying,Whatever.
But they don't really evenmemorize the last one.
They just remember it as acontinuum.
Like one is the worst one andone is like the place we're
trying to get to.
And then there's a lot ofvariation in between.

(04:41):
But they could kind of be alittle loosey goosey about the
stuff in the middle because theythink of it like a grayscale.
You're getting like better andbetter and better.

SPEAKER_01 (04:50):
Okay, but see, this is really frustrating.
Because we're talking abouteducating human beings and
helping them improve their livesand their understanding and
their knowledge.
And these are really complex andinteresting ideas.
And they're all getting lostbecause this is what our brains
are holding.

SPEAKER_03 (05:08):
Yeah.
So maybe the question is, youknow, how do we get folks to
come to understand, like a newcomplex idea?

SPEAKER_00 (05:17):
Well, understanding complex ideas are a whole
shebang, right?
Because it's like, are humansreally made for complexity?
Are

SPEAKER_01 (05:26):
they?

SPEAKER_00 (05:27):
And so here I want to introduce a metaphor that I
think will be helpful for usgoing forward.
So Jonathan Haidt talks abouthuman minds, and it's not just
Jonathan Haidt, but he has thiswonderful metaphor that I think
will be helpful for us.
Imagine a human mind as a man, alittle rider, riding on top of
of an elephant, right?

(05:50):
In cognitive science, we oftencall the little man system one
and the elephant system two, butI like Jonathan Haidt's imagery
of this little man riding on topof an elephant much better.
The little man is the logicalpart.
It's the part that learnscomplex ideas and nuance.
It learns wonderful, complex,interesting, structural,

(06:14):
relational things.
But it's small for a reason.
The elephant has a lot of power.
And the elephant is everythingelse in our minds that define
and determine what we actuallyend up doing.
So a small part of us is logicand reasoning and complex ideas.

(06:37):
But the big elephant part of usis all that other stuff.
perception, how we feel in themoment, all our stereotypes, how
we've done things before, what'seasiest to do in the moment,
what the world is set up to do.
That's the kind of stuff that wealso have to think about.
So I love talking to you guysbecause you're so thoughtful.

(06:59):
And I, as an academic, wouldlove to just spend all our time
thinking about how to get peopleto learn complex ideas.
And Honestly, we have a paperabout that.
It's called the PracticingConnections Framework.
And we really think a lot aboutthat.
But one of the things that youhave to realize is how little of
an effect complex learning hason actual behavior.

SPEAKER_01 (07:29):
But then...

SPEAKER_00 (07:30):
You're stunned.

SPEAKER_01 (07:32):
Well...

SPEAKER_00 (07:34):
Are you sad?
Doesn't that make you sad?

SPEAKER_01 (07:37):
Yes.
There's so much to learn andunderstand.
And not even from an achievementside of things.
But as we are more connected ina world, our need to understand
the complexity of other humansis so great that what you're

(07:58):
talking about makes me worriedabout about our ability just to
connect to other people, notjust to understand like
trigonometry.

SPEAKER_00 (08:05):
So here's, you know, I'm not all doom and gloom.
Thank goodness.
We've had enough of thoseepisodes.

SPEAKER_01 (08:14):
Did not know what we were getting into.
We

SPEAKER_00 (08:16):
still as humans act in very complex ways.
I'm just saying that It's notbecause we just think complex
thoughts that we're able to actin complex ways.
We act in complex ways for many,many reasons.
And a small part of that is dueto our complex thoughts.

(08:37):
Like, can I give you an examplethat's relevant to teaching?

SPEAKER_01 (08:40):
Yes.

SPEAKER_00 (08:42):
Like, you know, it's...
It's interesting.
Many teachers have this idea,hey, we should get more students
to talk in class.
I want to hear from a broaderdiversity of students.
That feels like a great goal.
Often, that's the rider saying,hey, I value the diversity of
voices.

(09:02):
I want to hear from manydifferent students.
And so that's a great goal.
But if you're a teacher who'sstrapped for time, who has to
move on, you might really belike, oh, my God, I'm just going
to call on the kid who I knowknows this thing, who could kind
of say it, and we could call onhim, and then we can move on,
right?
And that's the rider.
However, if you change theteaching practice, like, for

(09:26):
example– The thing that they doin this classroom is cold
calling.
Then the teacher doesn't have todo it because of the complex
thought of like, hey, I valuediversity, nor do they have to
default to, oh, I'm just goingto call in the kid who knows the
answer.
They just default to coldcalling because that's the

(09:48):
culture of their class thatthey've set up.
You see what I mean?

SPEAKER_01 (09:54):
So today we're going to talk about John Hattie's
work.

UNKNOWN (09:56):
Okay.

SPEAKER_01 (09:57):
It is big and it is detailed and it is complex and
it has been oft misunderstood.
But it's a big old list ofthings.
And that's the appendix.
In the original, 138 things.
And that's the appendix.
But that's what people remember.
Yeah.
So how do you communicate a bigcomplex...

(10:20):
If there is an idea that needscommunicating, that needs to be
complexly understood...
What are some ways that we can,I mean, what you're saying is
like, let's replace thispractice with this practice.
Kind of like training it out,making it very simple, saying
we're going to do this now.

SPEAKER_00 (10:40):
Not only.
I mean, the thing is, I'm justsaying that you need to get the
elephant and the rider to align.
If you're just planning onmaking the little rider do all
the work, that means you'regoing to go wherever the
elephant wants to go that day.

UNKNOWN (10:55):
Yeah.

SPEAKER_00 (10:55):
And so whatever your values are, you got to make sure
that that really is in theclassroom practice in a way that
when a teacher is tired, whenthe class is dragging, that they
could still do it.
Because you know what?
That's 90% of our experience,right?
And so what I'm saying with evenlike Hattie's work is that if

(11:18):
you have 180 things, people aregoing to remember everything.
A random three at best and zerofor the vast majority of

SPEAKER_03 (11:29):
time.
Sure, sure.
Too much is too much.

SPEAKER_00 (11:30):
Too much is too much.
And it's because think about thewriter.
If you're a writer, if you're aperson in the classroom and
you're tired, you haven't hadlunch yet, half your students
came late, you're trying to dragthem through this topic that
nobody's particularly interestedin, right?
You are an elephant teachingother elephants.

SPEAKER_02 (11:52):
That

SPEAKER_00 (11:53):
is the situation we are all stuck in.
And when you're doing that, youcan't think of the 180 things
and think, which of those 180things is relevant for this
moment at hand?
That's just not something thatyou're going to do.
But if you have some biggerthing, if you have a priority
that you're going to be tryingto really do, boom.

(12:15):
That's something you couldcommit to.
And I would argue that's why theBloom's thing took off too.
It gave people a prioritystructure.
It said, look, this is the leastpriority.
This is the worst priority.
It gave people a decision tree.
If you don't have time, if youdon't have resources, just spend
your energy doing this.
And I would argue that happenedfor the top of the pyramid and

(12:37):
the bottom of the pyramid.

SPEAKER_02 (12:38):
Because

SPEAKER_00 (12:39):
it's like, if you have one thing you want all your
students to do, maybe you wantthem just to do the best thing.
And if there's only one thingyour students could do, maybe
you just all want to get them atlevel on the bottom rung, right?
But it gave people a way to makedecisions.

SPEAKER_03 (12:58):
I really like that.
But okay, so yes, I like that.
And We know that, you know,Blooms, Marzano, Hattie, they
all put out revisions andrefinements that did not stick.
So when new ideas come down thepipeline, you know, those
refinements and the revisions,how do we get that to stick?

(13:21):
Like that never sticks.
We're only ever connecting tothat first impression, that
first read through.

SPEAKER_00 (13:28):
Well, and that's if that first impression was any
good.
For every Hattie, there's10,000- You think it wasn't
good.
That's what sticks.
Right.
There's 10,000 learningscientists that you don't know
their names, right?
So for whatever reason, thereare some people who stick.
And then those people, whatevertheir theory was, some crazy

(13:49):
part of it stuck.
And it tells us more about usthan we realize, right?
And- I would argue thatespecially in American culture,
it's very hard to reviseanything.
Americans love an innovation,right?
AI is going to change the world.
We love a silver bullet.

(14:09):
Everyone likes to say there's nosilver bullet, but secretly
we're really wishing for asilver bullet.
Oh, yeah.
Right?
No one's really that interestedin a system that incrementally
improves.
So you actually have to hide it,you know?
I'm old enough where I rememberwhen YouTube didn't have a lot
of videos on it.

(14:29):
But YouTube was reallydifferent.
But it didn't say, we'rerebranding to a new website
where we're going to do thisother thing.
They just made incrementalimprovements where we still
think the whole thing is stillYouTube, right?

SPEAKER_01 (14:44):
But every feature was tagged as like new.

SPEAKER_00 (14:47):
But a lot of times people didn't even know that it
was happening, you know?
Yeah.
There was a point where inYouTube, you would have to click
next to get another 15 or 20videos.
But they just changed toinfinite scroll where try
scrolling down YouTube.
You never get to the

SPEAKER_01 (15:06):
end.
Right.

SPEAKER_00 (15:06):
That's true.
And they just switched itwithout telling anybody about
it.

SPEAKER_01 (15:11):
So then if we want to add complexity to somebody's
understanding, rather thansaying we're adding complexity
to your understanding or toour...
discussion about this, we say,we've got this new thing we're
going to try.

SPEAKER_00 (15:23):
Yes.
I would say if you want toreally change the framework, you
need to brand it as new and sexyand good luck in getting that to
pop because 10,000 people havetried to get their thing to pop.
Well,

SPEAKER_01 (15:38):
initiative fatigue is a real thing, right?

SPEAKER_00 (15:42):
There's

SPEAKER_01 (15:43):
that line of like, oh, another new thing?
No, thanks.
I'm going to go ahead and closemy door.

SPEAKER_00 (15:47):
Exactly.
I mean, you know, I love CommonCore, but I was around the math
reform game for a while.
And when you read it, it isbeautiful, but is it that
different from the previousframeworks?
Which is why it's so funny howmuch backlash there was because
it's like, all right.
It's always the thing.
It's like, is it the branding oris it the real truth of the

(16:09):
thing?
The truth of the thing onlychanges by 10%, but they branded
it differently, right?

UNKNOWN (16:16):
Yeah.

SPEAKER_00 (16:17):
which has its pros and cons.
And that's always the thing.
You could brand it as a newthing, but it's gonna have a
different set of pros and cons.
Or you could do the YouTubeangle and just secretly build it
in.
There are innovations, but guesswhat?
You don't really have to learnanything new.
It just stretches you a littlebit from what you used to do.

SPEAKER_03 (16:37):
So I have a related but unrelated question.

SPEAKER_00 (16:40):
Of course.

SPEAKER_03 (16:42):
We love a segue.
Confirmation bias, right?
For our listeners at home, it'show people tend to look for,
interpret, recall informationthat confirms their existing
beliefs or opinions, right?
They also tend to discredit orignore contradictory evidence,
which I kind of feel like is inline to some extent of what

(17:07):
you're talking about a littlebit in terms of like why our
first bit of information sticksso firmly to, to what we work
with, right?
Like revisions and refinementsdon't really work.
So what, We know about this ideaof confirmation bias, but so
what can folks do to kind ofsubvert that in their own

(17:29):
thinking and in the thinking ofothers?
I don't feel like it's a uniquething to American education.
Maybe it is, but, you know, sowhat can we

SPEAKER_00 (17:38):
do?
Americans don't particularlylike the word innovation and
new, you know?
Yeah.
They love it more than most.
But you're right.
Confirmation bias is a humanthing.
Everybody has it.
It's not something that you'renecessarily going to overcome.
But here's where that elephantand rider thing comes in handy.
Like, often people think whenyou feel happy, you smile.

(18:05):
Right?

SPEAKER_03 (18:07):
For those of us who just exist with RBF pretty
consistently, you wouldn't knowthat.

SPEAKER_00 (18:16):
Except for those.
But...
There's a fair amount ofresearch that also shows the
other direction is true too.
When you smile, you feelhappier, right?
So confirmation bias goes inthis way.
Like you have these experiencesand that changes how you think,

(18:37):
right?
And we wish how you thinkchanged too.
what you experienced and viceversa, right?
We wish that was more true.
The way that I would say it isyou need to give people new
experiences that don't quite fitinto their framework.
And then those experiences havea chance of doing something

(18:58):
besides confirmation bias,because there's no theory that
those new experiences have yetto tag onto.
And that's what we've seen inour work with teachers, where
it's like they're doing datascience, they're doing these new
things, they're doing coding inclasses.
And because they're doing thesenew things in their classes,
it's almost like an opportunityfor them to kind of shake up

(19:21):
their concepts because theydon't have a concept for this
stuff yet, but we're seedingthem with experiences.
And then once they have thoseexperiences, we tell them the
concept and they're like, oh myGod, that really...
with my experiences.
We're almost using confirmationbias by seeding experiences and

(19:42):
later presenting them theconcept that's going to confirm
that they're going to confirmwith their experiences.
It's a very long game.
Yes.

SPEAKER_01 (19:53):
Christopher Nolan was trying to get us to
understand that with Inception.
And

SPEAKER_00 (19:57):
then we just watched the top

SPEAKER_01 (19:59):
spin and we never got it.

SPEAKER_00 (20:00):
But we decided to do it with teachers and education
and math classes.

UNKNOWN (20:04):
Yes.

SPEAKER_01 (20:04):
This has me thinking about the human library out of
Denmark.
I don't know.
Do you both know how nerdy arewe?
So the whole idea there is thatyou can check out a person.
Oh,

SPEAKER_03 (20:17):
that gives me feelings.

SPEAKER_01 (20:19):
No, in a really cool way, right?
So people sign up if you have a–if you're an older person or if
you're somebody who's queer ortrans or of a different
ethnicity, you can put yourselfinto the library.
And then people can come andcheck you out to have a
conversation with you.

SPEAKER_02 (20:37):
Wow.

SPEAKER_01 (20:38):
Oh,

SPEAKER_02 (20:39):
that's beautiful.

SPEAKER_01 (20:40):
Right.
And it is a really fascinatingidea.
But what you're saying and whattheir whole theory here is
around kind of contacthypothesis is that once I've
talked to somebody who is thatdifference that I do not own,

SPEAKER_02 (20:50):
that

SPEAKER_01 (20:52):
that's probably going to shift my thinking at a
foundational level about otherpeople who have that difference
that is not my own.

SPEAKER_00 (21:00):
Right then is the right time to give them a new
concept is what I'm saying.
So that's where it brings itbecause it's wonderful to go out
and have new experiences.
That's great.
But if you want people to glomon to a new complex concept,
what you want to do is pair itwith those new experiences.

SPEAKER_01 (21:20):
Okay.
So novelty is key and important.
And

SPEAKER_00 (21:27):
Cause it gives you a shot.

SPEAKER_01 (21:29):
Cause it gives you a shot.
It opens the door.
And once you're there, give likethe easiest form of the solution
that you would like the mostbasic.
We want you to do this at themost basic level.
We're going to give you thecomplexity, but

SPEAKER_00 (21:43):
I'm not that cynical.
That's

SPEAKER_01 (21:46):
what it sounds like.

SPEAKER_00 (21:49):
I still think right at that moment is when you want
to introduce the complex idea of

SPEAKER_01 (21:55):
Okay.
I guess I'm more thinking aboutyour elephant isn't like when
you're tired and cranky and allthose things, you're not going
to think through the complexityof the idea.
So you want to make sure thatfirst and foremost, that they
get the most basic or key pieceof information.
No.

SPEAKER_00 (22:13):
No?
I would not say that at all.
Here's the way we think aboutit.
Okay.
First, you give people realexperiences.
That's really important.
That connects to the elephant.
That gives them this likemeaningfulness basis, right?
But after that, you do want toconnect to complex concepts
because you don't want theelephant to do things that are

(22:34):
unmoored from the rider.
But the thing is there's anordering.
The elephant comes first andthen the rider comes second
because ideally the elephant andrider want to move in the same
direction together.
Okay.
The reason that I advocate forthat is there are going to be

(22:54):
times when the elephant wants tomove in one direction.
And what you want to do is beokay with that direction.
The rider wants to be like, hey,that's a great direction.
I love it for you.
I have no power in thesituation, but I'm happy to go
along for the ride.
You don't want the rider to beunhappy because ultimately

(23:14):
people are going to burn out.
If it's just elephant stuffwithout the higher order writer
stuff, giving them that sense ofinspiration and coherence that
people really love.
That's why we stay in hard jobslike teaching.
You need that.

SPEAKER_01 (23:32):
So this would be where like scripted curriculum
tried to do the thing that I wasdescribing.

SPEAKER_00 (23:37):
Correct.
It's just the elephant.

SPEAKER_01 (23:41):
You're all elephant.

SPEAKER_00 (23:43):
Stop thinking.
Leave the rider to do whateverhe wants.
We know if the elephant goes inthis direction,

SPEAKER_01 (23:49):
we'll be fine.

SPEAKER_00 (23:50):
Right.

SPEAKER_01 (23:50):
Yeah.

SPEAKER_00 (23:50):
And I don't think those are the right things
because you know what?
Even those scripted curricula,they always get something wrong
and you need a great teacher inthere for their writer to be
like, you know what?
I know I'm doing this, but Idon't think that's quite the
right way.
But I think if we tweak it, wecould get the elephant and
writer to go in the rightdirection.
And I really believe that thatkind of information from

(24:13):
researchers and teachers andcurriculum designers working
together is what's really goingto help both the elephant and
writer have coherence.

SPEAKER_01 (24:22):
One of the interesting pieces as we were
reading that book and thinking,I wish other people read this
book.
There's a part when they'retalking about the taxonomy that
says, of course, this should beused, and I'm paraphrasing here,
in concert with a school'stheory of learning and
philosophy of education.

(24:43):
To say that the ethos of yourschool is should also guide the
use of these tools as we aregiving them to you.
And I think about all the timeswhere we talk about the tools
and we never ask, how does thisalign with who we are or who we
want to be as an institution oran organization or a collective?

(25:05):
And I think what you're talkingabout right there is, again,
hitting on that like, tools aregreat.
Tools minus what do we believein, who are we, what are we
about, doesn't get us where wewant to go.

SPEAKER_00 (25:18):
Because you need them to be connected.
That's the thing.
And a lot of times, the toolsand the thinking are
disconnected.
The most relevant paper I havefor you guys is actually we have
this theory paper that we justsubmitted revisions on about how

(25:38):
embodied experiences in theclassroom, like hands-on sort of
activities– I think there's thisnice analogy where people are
like, do hands-on activities.
And I would argue that'selephant.
It's like, oh yeah, just dothis, play with these
manipulatives.
But a lot of times people doweird things with manipulatives
because it's unconnected to whatyou're supposed to do with these

(25:59):
manipulatives.
And one of the things we arguein this paper is there's a
reason why in the literature,there are lots of times when
manipulatives or whateverhands-on activities don't lead
to better learning.
And it's because you need tothink about that alignment
between hands-on activities andmore abstract, complex learning.

(26:22):
That's why my ultimate messageis don't give up hope.

SPEAKER_03 (26:27):
Oh, my goodness.
Okay.
So let me see if I can makesense of some of this.
To understand a new and complexidea, we got to– oh, dang.
Okay.
How do I want to say this?
Where

SPEAKER_00 (26:42):
do you start?
Do you start with the complexidea?
Nope.

SPEAKER_03 (26:45):
No, you piecemeal it.

SPEAKER_00 (26:47):
You do start with real concrete experiences.

SPEAKER_01 (26:52):
Okay.
And when their brains are open

SPEAKER_00 (26:53):
and when they're ready,

SPEAKER_01 (26:55):
you say, here's a complex idea to

SPEAKER_00 (26:57):
the complex abstraction.

SPEAKER_01 (27:01):
So that later when I want to use it, my brain's got
it all filed away nicely.
And it's

SPEAKER_00 (27:06):
interconnected.
Where when you think of theabstract thing, it reminds you
of the concrete stuff.
And when you do the concretestuff, it also activates the
abstract thing.

SPEAKER_03 (27:17):
Yeah.

SPEAKER_00 (27:18):
Isn't it beautiful?

SPEAKER_01 (27:20):
Yes.

SPEAKER_03 (27:21):
It is.
I love it.

SPEAKER_01 (27:23):
G, you're real fun.

SPEAKER_03 (27:24):
All right.
Thank you, G.

SPEAKER_01 (27:26):
Thank

SPEAKER_00 (27:26):
you.
It really has been an honor.
Thank you for letting me justpontificate.
It's

SPEAKER_01 (27:33):
our favorite kind of conversation.

SPEAKER_00 (27:34):
Yeah.

SPEAKER_03 (27:36):
When we come back, we'll dive into the work of John
Hattie and examine if what worksworks, even when the math might
not.

SPEAKER_01 (27:56):
Hey, Stephanie.

SPEAKER_03 (27:57):
Hey,

SPEAKER_01 (27:57):
Zach.
I shared that conversation withG, and it gave me an idea.
I want to tell you a story.

SPEAKER_03 (28:07):
Ooh, I love story time.
Okay, I'm ready.

SPEAKER_01 (28:10):
Once upon a time in the late 20th century, it's a
more specific time, I suppose, aman named John Hattie.
And one day, he stepped outsidehis metaphorical door, looked
across the world, or maybe theparts of the world that were
accessible to him throughjournal articles and databases.
And he did a meta analysis ofmore than 800 meta analyses in

(28:38):
education comprised of more than50,000 separate studies and
applied some statistical mumbojumbo that I sort of understand.
He wrote a book called VisibleLearning.
a synthesis of over 800meta-analyses relating to
achievement.
It rolls off the tongue.

SPEAKER_03 (29:00):
Zach?

SPEAKER_01 (29:01):
Yes, Stephanie?

SPEAKER_03 (29:02):
I feel like we're jargoning right now.
What?

SPEAKER_01 (29:05):
No, I was just telling the story.

SPEAKER_03 (29:07):
Yeah, I feel like we need to break down what a
meta-analysis actually is.
So meta-analysis is when youtake a bunch of studies about
the same thing and combine theirresults to try to find a general
pattern.
We can sort of think of it asfinding the average of a bunch
of different results andinterpreting those averages as

(29:29):
actions we can take.
Sort of.
It's a good enoughinterpretation for this episode
anyway.
So Hattie found more than 800meta-analyses that looked at
lots of smaller studies oneducation.
These meta-analyses had putsimilar studies together and

(29:52):
tried to figure out what usuallyhappens when you have a specific
thing in education or when youchange a specific thing in
education, like having studentsset goals for a class period or
tracking whether living inpoverty impacts a student's
learning.
And so what Hattie did is hetook those meta-analyses and did
another round of meta-analysison them.

(30:13):
He did a meta-meta-analysis,averaging the averages, if you
will.

SPEAKER_01 (30:19):
Thank you, Stephanie.

SPEAKER_03 (30:20):
Oh, you're so welcome.

SPEAKER_01 (30:22):
So, back to the story.
Yes.
Hattie took this book around theworld.
It was renowned.
In fact, this book and itssuccessors have sold more than 2
million copies.
I've never sold 2 million ofanything.
And they've been translated into29 languages.

(30:45):
Now, it is hard to undersell...
Why?
Why?

(31:17):
Because it gave a list.
And as G pointed out, it was newand sticky.
It satiated our need for asilver bullet, even when we know
there are no silver bullets.

SPEAKER_03 (31:32):
We know our brains love a list.

SPEAKER_01 (31:35):
Now, for those of you not in education, or who
maybe just have seen the effectsof Hattie's work, but never
actually looked at Hattie'swork, visible learning, deep in
Appendix B...
included a listing of 138influences on student
achievement covered by Hattie'smeta-analyses.

(31:55):
And there are hundreds of pages,about 300 pages almost, in this
original book.
But it was that list of 138different things that were
studied that made it sticky.
And that is what people noticed.
And that is what peopleremembered.

(32:15):
If you remember back to ourconversation with G not too long
ago, and if you remember ourconversation about Bloom's
taxonomy, you know that folkslove when that information is
presented in the hierarchy.
And so when they looked atHattie's information, it was
really easy to say these thingsat the top of the list look like
they're really going to improvestudent learning.

(32:37):
And these things at the bottomof the list, maybe I don't need
to pay as much attention to.
So schools...
Then took that and they said,you know what, we're going to do
these top 10 things or we'regoing to do these top 20 things.
That's going to be what we do toinfluence learning.
And they maybe let some otherthings go.
In fact, sometimes theydiscouraged teachers from using

(33:00):
practices lower on the list infavor of those at the top.

SPEAKER_03 (33:06):
Yeah, Hattie was all the rage when I started
teaching.
And let me tell you, as a firstyear teacher, I was very, very
ready to have someone tell mewhat was most likely to give me
the biggest bang for my buck.
I drank the Kool-Aid.
I did.
Yeah, I did the jigsaw method.
It was number seven on the list.
I brought up prior learningnumber nine on the list.

(33:28):
I told them they could do it.
It was number three on the list.
I was able to reference themoves I made from the list, and
my principal was so proud of me,little tiny first-year teacher
Stephanie.
But, you know, whether it wasbecause I was a newbie or
because the moves weren't rightfor my kids, I didn't actually
move the needle for my students.

(33:50):
But, you know, the moves madesense in my brain.
Like, logically and from apractitioner standpoint.
So I kept at it.
And as my teaching practiceimproved, the relationships I
built with my students improved.
And, you know, eventually theneedle did move.
But I don't know if it was fromthe moves from Hattie's list or
me figuring out my vibe andrhythm as an educator, you know?

(34:11):
It

SPEAKER_01 (34:12):
sounds like you need a study of your use of things
from studies about studies tohelp your students study.

SPEAKER_03 (34:21):
Yeah, maybe.

UNKNOWN (34:22):
Yeah.

SPEAKER_01 (34:23):
I have a story similar to yours.
In one district where I worked,there was an administrator who
frequently cited the importanceof reciprocal teaching and the
effect size that it had shown inHattie's work.
Now we'll get to effect sizelater, but it just is the number
that made it higher up on thelist.
But none of us knew anythingabout math or science.

(34:44):
Well, we knew about math, but wedidn't know about statistics.
We didn't know about effectsizes.
So we didn't challenge anything.
We didn't ask any questions.
And this was largely how thebook was received, is educators
who mostly don't have anybackground in understanding
research methodology orstatistics kind of took things

(35:05):
at face value.
Or they said, this isn't quitewhat I think happens in my
school or with my kids.

UNKNOWN (35:13):
Right.

SPEAKER_01 (35:13):
eventually, not too long after it was published,
statisticians started to look atthe math Hattie presented.
And they had some issues.
Not only was each of thesemeta-analyses not necessarily
comparing apples and oranges, itwas more like comparing apples
and orangutans.
And that, if you are astatistician, is not good.

SPEAKER_03 (35:37):
No.
I'd say even if you're amathematician, it's not good.

SPEAKER_01 (35:41):
If you...
are some sort of, you know,orange grower, also probably not
good because then you've got alot of primates and oranges.
And you don't want to juice bothof those.
Anyway, in our next episode,that's right, this is our very
first two-parter.
We're going to dive into whatwas wrong that kept the math

(36:05):
from mathing in Hattie's work.
And we're going to ask aquestion of our guest of whether
or not it matters.
Was the impact net good, netbad?
What was the effect size?
I know I'm using that completelywrong of Hattie's work on
education.

SPEAKER_03 (36:26):
Thanks for listening to Academic Distinctions,
friends.
Like, subscribe, and call yourparents every once in a while.
Follow us on Instagram atacademicdistinctionspod.
Find us on Blue Sky atfixingschools or find us on
Facebook at And as always, inthese unprecedented times,
remember to wear your sunscreenand stay hydrated.
If you've got a question or atopic you'd like us to tackle on

(36:48):
the show, send them to mail atacademicdistinctions.com.
Until next time, I'm StephanieMelville.

SPEAKER_01 (36:57):
And I'm Zach Chase.

SPEAKER_03 (36:59):
And you just got a little smarter.
Bye for now.
This podcast is underwritten bythe Federation of American
Scientists.
Find out more at fas.org.