September 21, 2023 • 27 mins
The IQ test is supposed to be a measure of human intelligence. But is it? Today, we're joined by UC Berkeley professor Dr. Steven Piantadosi to understand why the diversity of human intellect can't be captured by a single number. We'll explore the history of IQ, from its innocent beginnings in French schools to its dark role in the eugenics movement. We'll understand why IQ testing is fallible, thanks to rising scores, the effects of motivation, and cultural bias, and learn about a test in the 1970s that was designed to flip that bias upside down. Go full galaxy brain and tune in to hear why IQ isn't all it's cracked up to be.
See Dr. Piantadosi's website here: http://colala.berkeley.edu/people/piantadosi/
See citations and a transcript here: https://www.tabooscience.show/s3e11-iq-intelligence/
Need music for a project? Use my Epidemic Sound referral link: https://share.epidemicsound.com/kbva2h
Connect with the show:
Newsletter: https://www.tabooscience.show/newsletter
Instagram: https://www.instagram.com/tabooscience
YouTube: https://www.youtube.com/@tabooscience
Twitter: https://www.twitter.com/tabooscience
Website: https://www.tabooscience.show/
Loved this episode? Leave a review and rating here: https://podcasts.apple.com/us/podcast/taboo-science/id1533606473
Mentioned in this episode:
Indecent With Kiki Andersen
Listen to Indecent with Kiki Andersen here: https://pod.link/1687409567
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Ashley (00:08):
Something I've learned by doing this podcast is that when you
try to measure the vast diversity ofthe human experience with a single
number, you're gonna have a bad time.
I mean, take something assimple as your credit score.
It's a single number, taken fromhow much debt you have, how much
credit you have, and how oftenyou pay your credit card on time.
(00:31):
We use that single number to judgehow trustworthy and reliable you
are, not only for car loans andmortgages, but even for employment.
And yet it hardly captureswhat it's supposed to capture.
Someone who's so financially responsiblethat they've paid everything in cash
and never even used a credit cardwould have a lousy credit score.
(00:53):
And in the U.
S., so would someone who's paid everythingon time their whole life but got a
serious medical condition that saddledthem with medical bills that they
couldn't pay that went to collections.
That one number works most ofthe time, but when it doesn't
work, it really hurts people.
The same is true of BMI, the ratioof height to weight that purports
(01:17):
to measure how fat you are, andby extension, your overall health.
Race is another one.
It's not a number, but it is a singlelabel we give people, and with it
we make all sorts of extrapolationsabout their personality, abilities,
income, and criminal background.
So when you learn that there's a singlenumber that can measure all of human
(01:40):
intelligence, You should be skeptical.
There are 8 billion ways to be human,and a single measurement like IQ
is not going to capture all that.
Instead, it's going to cause problems.
And today, we're going to find out why.
I'm Ashley Hamer, and this is TabooScience, the podcast that answers the
(02:03):
questions you're not allowed to ask.
(02:24):
-- I've been interested in the shortcomings of IQ for a few years now, and when I set
out to find the right researcher for thisepisode, I did what any journalist does.
I turned to Twitter.
And boy, did I findwhat I was looking for.
Steven Piantadosi (02:37):
I wrote a pretty long Twitter thread kind of outlining some of
the problems that I saw with IQ research,and a lot of that came about because part
of my lab's work is doing work with anindigenous group in Bolivia, and we look
and see what happens with language andmath learning without formal schooling.
Ashley (02:55):
That author of a 62 tweet thread back in 2020 that starts
with, Here's why IQ is bullshit?
That's Dr.
Steven Piantadosi.
Steven Piantadosi (03:04):
I'm a professor at UC Berkeley in psychology and
neuroscience, um, and my lab studies,language and math learning in kids.
Ashley (03:12):
The indigenous group in Bolivia Dr.
Piantadosi works withis called the Chimane.
There are a group of foraging farmers,sometimes called hunter gatherers, who
are incredibly popular with scientists.
If you've ever seen a headline thatsays, Hunter Gatherer Societies Have
the Healthiest Hearts or Hunter GathererLifestyle May Be Key to Healthy Brain
Aging, that was about the Chimane.
(03:35):
The idea is that their way of life iscloser to that of early humans than that
of people in industrialized societies.
So by studying them, people inindustrialized societies can get
a better idea of where they camefrom, and what industrialized
society is doing to their health.
Likewise, studying children inthese societies can tell you what
(03:56):
kind of knowledge is fundamentalto being human, and what you need
school and written language to learn.
Steven Piantadosi (04:03):
One thing that's interesting is that is that if you work
with people who've never been to school,many of the kind of familiar school
tasks or things that are familiar tous are completely unfamiliar to them.
And that means that if you givethem, for example, an IQ test,
they won't score very well on it.
So, that made me start thinking aboutIQ and looking into some of the problems
(04:24):
that people have argued are presentfor IQ research, namely big cultural
differences and different kinds ofexpectations and motivations and things
that people bring into an IQ test.
And that's what led to the Twitter thread.
Ashley (04:37):
Like so many of these numbers we use to describe humanity,
IQ started innocently enough.
Around the turn of the 20th century,French psychologists Alfred Binet and
Theodore Simon came up with the BinetSimon test, which was designed to identify
(04:57):
schoolchildren whose mental abilitieswere developing more slowly than average.
This is where we get theconcept of someone's mental age.
If your score on the test said your mentalage was younger than your chronological
age, you needed extra help in school.
And if it was older, you were gifted.
Binet, for his part, did notbelieve that intelligence was
(05:19):
one thing that you got at birth.
He believed that there were many waysto be intelligent, and that children
could learn to become more intelligent.
But then America got a hold of the testand gave it the old red, white, and blue.
In 1916, Lewis M.
Terman published the first StanfordBinet test of intelligence, which was an
(05:40):
expanded version of the Binet Simon test.
It was the most popular test in the U.
S.
for decades, and it's where we get thetypical IQ numbers that too many of
my Tinder dates have thrown around.
You get it by dividing a person'smental age by their chronological
age, then multiplying by 100.
By that measure, 100 is average.
(06:01):
It's what you get if your mentalage matches your chronological age.
And anything higher or lower isless common, dropping off toward
the extremes in a bell curve.
In the original test manual, Terman laidout the benefits and uses of the new test.
Next to simple things like identifyinggifted students and determining
vocational fitness, there was the line,quote, "This will ultimately result in
(06:25):
curtailing the reproduction of feeblemindedness and in the elimination
of an enormous amount of crime,pauperism, and industrial inefficiency."
End quote.
If you were playing the eugenicsdrinking game, it's time to take a shot.
Another important figure in whatwe now know as IQ was British
psychologist Charles Spearman.
(06:47):
He observed that children's grades inunrelated subjects tend to be correlated.
Steven Piantadosi (06:52):
So for example, people who tended to get good grades in math
also tended to get good grades in English.
Okay.
And you could think about thatand think like, that's not
necessarily how it has to be, right?
Like maybe you have the intuitionthat it could be the other way around,
that if you get good grades in math,you're less good in English, right?
But that's not kind of empiricallyor quantitatively how it turns out.
People's objective scores on differentthings tend to correlate positively.
Ashley (07:17):
Speerman proposed that all mental traits were related to
a single common factor, which hecalled G for general intelligence.
Pretty much every intelligence testsince has been designed to correlate
as much as possible to this G factor.
Steven Piantadosi (07:32):
For example, if somebody takes the SATs, that number will
be pretty highly correlated with G as youwould have measured it by looking at their
grades across a bunch of different topics.
So, what kind of intelligence testinghas tried to do over the years is
find tests which are highly G loaded,meaning some test I can give you
hopefully something that's kind ofshort and simple, which when I get
(07:54):
your number, it's correlated with yourG, your, general intelligence factor.
And therefore also would do a goodjob at predicting, say, your grades
across a, a bunch of different topicsor other things that intelligence
tests are supposed to correlate with.
Ashley (08:08):
The problem is that nobody has ever definitively
identified what G even is.
Steven Piantadosi (08:14):
It's a real kind of magician's trick to call
it general intelligence, right?
When nobody actually knows if it'sgeneral intelligence, as opposed
to, for example, motivation orexperience or any of these other
kinds of cultural factors that matter.
One thing that happens in psychologyand in some of the kind of culture wars
around this is people sometimes say, well,intelligence is the most statistically
(08:36):
robust area of psychology, right?
It's the thing that is most replicable.
And, there's tons of studies and it's truethat there's tons of studies that robustly
find a general intelligence score, right?
And once you find it, it's truethat there's high replicability
in terms of what tasks arehighly g loaded, for example.
So on kind of a raw statisticallevel, it's true that intelligence
(08:58):
is very well justified.
Where it's very poorly justifiedis on the interpretation.
So, when I say that, you know,there's other things that could
determine G or there's other kindsof confounding factors, those things
have not been well examined byintelligence research and ruled out.
Um, in fact, kind of the opposite, right?
There have been these people whohave manipulated those factors
(09:19):
and shown that those really affecthow you do on intelligence tests.
And therefore, it's really not good tocall them intelligence tests, right?
That's kind of the trick is incalling them intelligence tests.
And once you say that, it sort of feelslike they're measuring intelligence.
Ashley (09:33):
If you don't know exactly what the test measures, and you don't even
have a scientific definition of generalintelligence — what it is, whether
it's inherited, whether you can changeit — that leaves a ton of wiggle room for
people to make their own interpretationsto further their own goals.
A score that a school administratormight say is reason to give a
student extra guidance could be thesame score that makes a dictator
(09:56):
call for their extermination.
I'm not exaggerating.
Nazi Germany systematically wiped outpeople with disabilities, and that
included intellectual disabilities.
They often used IQ testing todetermine who was unfit to live.
But the U.
S.
did some pretty horrificstuff around IQ, too.
(10:17):
During the 20th century, more than60, 000 people were sterilized in 32
states, based on the idea that preventingthe, quote, feeble minded and other,
quote, degenerate stock from havingkids would reduce crime, cut healthcare
costs, and generally improve society.
Some of this continued asrecently as 2010, by the way.
(10:40):
The eugenics drinking game peopleare out of whiskey at this point.
Throughout all of this, the typesof people who got low scores
on the IQ test fit a pattern.
Recent immigrants, peopleof color, people in poverty.
While some take that as evidencethat these groups are just less
intelligent, researchers throughoutthe decades have found that
(11:01):
there are other elements at play.
Steven Piantadosi (11:04):
This is a point actually that has been made by
a number of sociologists over theyears, that the kind of content of
IQ tests is inherently biased towardswhite and dominant cultural groups.
And they make this point by coming upwith other versions of IQ tests, right?
If you ask different kinds ofquestions, which emphasize, say,
black culture, then black kids will dobetter than white kids on those tests.
(11:27):
And so the fact that you canconstruct tests like that, I think
really emphasizes the fact that thetests are constructed and they're
not constructed in a vacuum, right?
They're, they're constructed in away which happens to preferentially
treat, white kids, for example, orrich kids or kids that can afford,
the kinds of tutoring and, andpractice and, schools, which lead
(11:48):
to high performance on the tests.
So, I think that the main mythologyaround IQ tests is that somehow because
it's a test, it's objective, right?
And that I think is the mainthing that's wrong, right?
It's true it's objective in the sensethat you can take it and get a number out,
but somebody had to construct that test.
And if you construct it in a differentway, you'll end up, biasing in
(12:09):
favor of some other racial orethnic or socioeconomic group.
Ashley (12:14):
One example of a test constructed to be biased in favor of another racial
group was developed in the 1970s by RobertWilliams, a psychologist and professor
at the Washington University in St.
Louis.
It was called the Black IntelligenceTest of Cultural Homogeneity.
Yes, that forms the acronym BITCH.
(12:34):
From what I understand, it lookslike it was also sometimes called
the Black Intelligence TestCounterbalanced for Honkies.
Which honestly makes a lot moresense than cultural homogeneity.
I mean, it was counterbalancedfor honkies, so I don't know.
Dr.
Williams had done a lot of researchabout the bias of standardized IQ tests.
(12:57):
One question, for example, hadthe child point to a squirrel that
is beginning to climb the tree.
His Black subjects didpoorly on this question.
But when he changed the question tosay, point to the squirrel that is
fixing to climb the tree, suddenly Blackchildren did better than White children.
So the Black Intelligence Test ofCultural Homogeneity flipped that
(13:20):
bias and took it to its extremes.
This test listed 100 words fromthe Dictionary of Afro American
Slang and asked test takersto identify their meaning.
Things like black draft and applealley, along with words that exist
in white culture but had differentmeanings in 1970s black culture,
like clean and Mother's Day.
(13:41):
Those mean well dressed and the day thewelfare checks come in, respectively.
Unsurprisingly, white test takers do muchworse on this test than black test takers.
Steven Piantadosi (13:52):
There's quite a few historical examples of racial differences
and people reading into those as beingtrue differences in intelligence as
opposed to cultural differences, right?
Differences in schooling, forexample, or opportunity, right?
These other kinds of things thatwe know influence intelligence
test performance, but which wereally shouldn't call intelligence.
This is true in the group I work with.
(14:14):
When people give them intelligencetests, they test close to the
threshold for intellectual disability.
So down in the seventiesor eighties, right?
And, I think anyone who works withthem, like, nobody who works with
them would think that they areintellectually disabled, right?
The issue is that they're notused to taking tests, right?
Because if you've never been to schooland somebody brings you in and starts
(14:35):
showing you, you know, geometricshapes and numbers and asking you
to find patterns and things, thoseare completely unfamiliar tasks
that they've never done before.
And so, of course, they don't score well.
And so you find those kinds of culturaldifferences without the difference
being actually reflective of adifference in intelligence, right?
It's a difference in culture orpractice or something like that.
Ashley (14:58):
One difference might be motivation.
Turns out you can actually improvepeople's scores by offering them money.
A 2011 meta analysis led byAngela Duckworth found that a
$10 reward can increase your IQscore by as much as 20 points.
You can imagine how thismight play out in a classroom.
(15:19):
A kid who knows their family can affordcollege and as long as they get good
test scores they can go to any schoolthey want is probably going to try a lot
harder than a kid who knows that no matterwhat score they get, the needs of their
family mean that college is off the table.
And if IQ tests are really measuringa person's general intelligence,
average scores over the decadesshouldn't be changing that much.
(15:42):
We haven't had enough time toevolve better brains, you know?
And yet.
Steven Piantadosi (15:48):
If you look at IQ tests over the years, like over decades of time,
in general, scores are going up on them.
And this has been a bit of amystery in intelligence testing.
You know, if you thought intelligencetesting was a measure of true
intelligence, that people aregetting, you know, smarter over
time, that I think goes against somedominant cultural narratives, right?
Ashley (16:09):
This phenomenon is known as the Flynn effect, discovered
by researcher James Flynn in 1984.
He found that over 46 years,representative samples of Americans scores
on IQ tests rose by about 14 points.
Which is weird, right?
I mean, name the greatest geniuses inhistory Newton, Galileo, Edison, Einstein.
(16:32):
They all lived a long time ago.
But if you believe that IQ measurestrue intelligence, you'd have to also
believe that your average Joe todayis way smarter than any one of them.
So what's the deal?
Steven Piantadosi (16:45):
But I think it's maybe interesting for people
who care about the mechanisms ofwhat's going on cognitively, right?
Because you want to know whatit is that's changing, right?
Is it something abouteducation that's changing?
Is it something about test prep, right?
Maybe we're testing kids more or somethingand they're, they're getting better at it.
I've heard theories that our educationalsystems are emphasizing abstraction more.
(17:08):
I think this is what, what Flynnhimself thought, was that we taught
abstraction from younger ages and triedto encourage abstraction and thinking
about problems abstractly and that wassomething that maybe wasn't as true in,
for example, our grandparents generation.
And many of the questions you get in anIQ test involve, you know, recognizing
abstract patterns among shapes ornumbers or something like that, right?
(17:31):
And so if we're reinforcing thatkind of abstraction early on and
more in school, then maybe you wouldtend to score better on those tests.
Ashley (17:38):
Yeah.
Is there any evidence that weactually are getting smarter?
Like maybe it's nutrition or maybeit's like something else that isn't
about learning how to take the tests?
Steven Piantadosi (17:48):
I mean, people take the Flynn effect as evidence of that.
I don't know, so, I think it's it's avery hard kind of question to answer
because, you need some objective wayof measuring how smart people are.
And I think that thatpretty much doesn't exist.
Ashley (18:11):
if you didn't call it intelligence, what would you call it?
Steven Piantadosi (18:14):
I don't know.
So that's actually somethingi've thought about a little bit.
Like what is the right kindof name for these things?
I think that often they have kind ofclinical names and and that's fine.
So for instance, there's atest called Raven's Test.
Named after Raven who was thepsychologist who developed it.
And I think it's perfectly fineto talk about what somebody's
Raven score would be, right?
(18:36):
That's often how we might talkabout it in the lab, right?
You know, you're looking at somethingand you say, what was their Raven score?
And that doesn't bring any kindof inherent judgment that Ravens
is the measure of intelligence.
Right?
And I think that that'sprobably good, right?
It'd be good to, to kind of removethose loaded terms from these tests.
Ashley (18:56):
But Dr.
Piantadosi doesn't believe thatintelligence testing is totally useless.
It does have its place.
Steven Piantadosi (19:03):
There's certain settings where I think this kind of
testing is actually useful, right?
So, so people use it in a clinicalsetting, for example, and oftentimes in a
clinical setting, you might be comparing,you know, within a patient, right?
So, you know, patient at one timepoint to another time point, or,
within patients maybe that, thatare very tightly normed in terms of
(19:24):
experience or age or other kinds ofthings like that, education level.
And there, like, you often needa quantitative cutoff, right?
So you might need a quantitativecutoff to decide what kind of treatment
you should do with this person or ifthey are okay and you can continue
watching them or, or whatever, right?
So, I think that there's lots ofareas where we need some quantitative
(19:44):
number, but the problem is when youconvince yourself that that number
is the objective thing, right?
Ashley (19:51):
I don't know about you, but I don't see people abandoning
the IQ test anytime soon.
It's too embedded in ourculture at this point.
So if people are using it, for entranceinto the military, or for school
placement, or a medical diagnosis,what should they be looking out for?
Steven Piantadosi (20:07):
If you use them for something, you have to be very
aware of these kinds of biases, inparticular, the racial biases in the U.
S.
That awareness, I think, meansnot taking the numbers you
know, very seriously, right?
So they might be kind ofuseful as a guide or something.
But if you're looking at a student, forexample, and trying to decide what level
to put them in, you can't just use thenumber because the number is, is biased.
(20:31):
Right?
And so you might look at other kindsof information, like, you know what
their background was or what theirfamily background was try to evaluate
their performance relative to theiropportunities or to their training
or to the motivation that they have.
All of those things arerelevant and it means that the
decision isn't a simple one.
So I think that the people incharge of these decisions probably
(20:51):
won't like that advice, right?
It's you can't just you know, drawa line on a spreadsheet and say
everybody above this is above the line.
Uh, It's really a complicatedkind of context dependent
decision that you have to make.
And, that really comes from just nottaking them very seriously because
they're not measuring what they'resupposed to or what they claim to.
Ashley (21:09):
Right.
Yeah.
So just, you're just makingthings more complicated, but that
is what things are in reality.
So people just need to deal with it.
Steven Piantadosi (21:15):
Yeah, exactly.
Yeah.
I think it's almost an effort againstthe complicated nature of reality.
And so if you're, if you'redenying that, then you're gonna
end up making worse decisions.
Ashley (21:25):
That's it.
If you're fighting against thecomplicated nature of reality,
you're going to make worse decisions.
That's race.
That's BMI.
That's all sorts of medical decisions.
I wanted to hear what Dr.
Piantadosi thought of thisbig picture view I was taking.
(21:46):
It seems like every time scientiststry to come up with a way to simplify
human diversity, um, It causes problems.
Like, even though it's like, we're,it, it, it's understandable they're
trying to do that because things arecomplex and you, you wanna be able
to, to understand the world better.
And, and, and organizingthings, uh, helps you do that.
(22:09):
But it, it seems like that'salways, it always seems to go wrong.
Could, could you talkabout that a little bit?
Steven Piantadosi (22:14):
That's a, a nice observation.
You know, I, I think that there'ssomething maybe even inherent to
doing science, which is tryingto find generalizations, right?
And those generalizations are reallynecessarily simplifications of
what we see happening in the world.
And you can think about even like,textbook kind of scientific laws or,
(22:34):
or results, you know, think about likeNewton's Law of Gravity or something,
or if you, you know, remember inphysics class computing, you know,
the trajectory of a ball that youthrow or something like that, right?
Like all of those are simplificationsbecause you, you ignore things like
friction or you ignore wind resistanceor, or whatever, and those physical
laws are, are useful, even thoughthey're simplifications, right?
(22:56):
Because those simplifications allow youto solve the problem and get you kind
of a good enough answer or a good enoughapproximation in, in those situations.
And I agree with your observation.
I mean, it seems like manyof our simplifications of
human nature aren't like that.
And it could be that human natureis just really complicated, right?
And I think there's other systemswhich are kind of intrinsically or
(23:17):
inherently complicated, like thestock market, for example, right?
That, you know, there may not be a simplekind of law that you could write down
for the stock market that describesits behavior in the same way that
you have for, uh, for Newton's Laws.
And I'm not sure why this happens withtheories of human psychology or people.
Part of it might be that it'sa little bit hard to appreciate
(23:38):
all of the complexity thatthere is within people, right?
So, where we grow up in some communityand we're used to people who have
similar experiences to us, and that mightmake us think that our experience is
universally shared among other people.
But when you go to a completely differentculture, right, a completely, even
industrialized culture, but, but also,you know, a non industrialized group that
(24:00):
lives in the Amazon, for example, right?
Like, you start to see that thingsare just really, really different, and
they approach problems and have a verydifferent set of kind of cognitive skills
and tools that they need for their lives.
And it's different thanwhat you need for your life.
But it makes it clear thatpeople are just very flexible and
very adaptable across differentenvironments and different settings.
(24:27):
there's all kinds of sophisticatedand really beautiful ways that people
have of thinking about the world andthinking about different problems.
And those vary among different humangroups and among cultural groups and
among different languages and Noneof that kind of interestingness is
really captured by detecting shapepatterns or, or something, right?
So, for people like me who are, who areinterested in the real kind of mental
(24:50):
mechanisms of how our brains work, right?
This one number is just, akind of hopeless attempt,
right, at characterizing that.
And is almost completely irrelevant toall of the interesting kinds of processes
that actually happen right inside people'sbrains as they're thinking about things.
Ashley (25:13):
Thanks for listening.
Big, big thanks to Steve Piantadosi.
You can find more of hisresearch at his website, which
is linked in the show notes.
Taboo Science is written andproduced by me, Ashley Hamer.
The theme was by DannyLopatka of DLC Music.
Episode music is from Epidemic Sound.
There's a referral link in the show notesif you want to use it for your own stuff.
(25:35):
Hey, did you know September isInternational Podcast Month?
I think you should celebrate by leavinga nice review for a podcast you love.
The link to leave a review on ApplePodcasts for this podcast, is at
the bottom of the show description.
Just saying.
Anyway, we've just got a few episodesleft in the season, and I have
big, big plans for the next one.
(25:57):
Hope you tune in!
I won't tell anyone.
Something I've learned by doing this podcast is that when you
try to measure the vast diversity ofthe human experience with a single
number, you're gonna have a bad time.
I mean, take something assimple as your credit score.
It's a single number, taken fromhow much debt you have, how much
credit you have, and how oftenyou pay your credit card on time.
(00:31):
We use that single number to judgehow trustworthy and reliable you
are, not only for car loans andmortgages, but even for employment.
And yet it hardly captureswhat it's supposed to capture.
Someone who's so financially responsiblethat they've paid everything in cash
and never even used a credit cardwould have a lousy credit score.
(00:53):
And in the U.
S., so would someone who's paid everythingon time their whole life but got a
serious medical condition that saddledthem with medical bills that they
couldn't pay that went to collections.
That one number works most ofthe time, but when it doesn't
work, it really hurts people.
The same is true of BMI, the ratioof height to weight that purports
(01:17):
to measure how fat you are, andby extension, your overall health.
Race is another one.
It's not a number, but it is a singlelabel we give people, and with it
we make all sorts of extrapolationsabout their personality, abilities,
income, and criminal background.
So when you learn that there's a singlenumber that can measure all of human
(01:40):
intelligence, You should be skeptical.
There are 8 billion ways to be human,and a single measurement like IQ
is not going to capture all that.
Instead, it's going to cause problems.
And today, we're going to find out why.
I'm Ashley Hamer, and this is TabooScience, the podcast that answers the
(02:03):
questions you're not allowed to ask.
(02:24):
-- I've been interested in the shortcomings of IQ for a few years now, and when I set
out to find the right researcher for thisepisode, I did what any journalist does.
I turned to Twitter.
And boy, did I findwhat I was looking for.
Steven Piantadosi (02:37):
I wrote a pretty long Twitter thread kind of outlining some of
the problems that I saw with IQ research,and a lot of that came about because part
of my lab's work is doing work with anindigenous group in Bolivia, and we look
and see what happens with language andmath learning without formal schooling.
Ashley (02:55):
That author of a 62 tweet thread back in 2020 that starts
with, Here's why IQ is bullshit?
That's Dr.
Steven Piantadosi.
Steven Piantadosi (03:04):
I'm a professor at UC Berkeley in psychology and
neuroscience, um, and my lab studies,language and math learning in kids.
Ashley (03:12):
The indigenous group in Bolivia Dr.
Piantadosi works withis called the Chimane.
There are a group of foraging farmers,sometimes called hunter gatherers, who
are incredibly popular with scientists.
If you've ever seen a headline thatsays, Hunter Gatherer Societies Have
the Healthiest Hearts or Hunter GathererLifestyle May Be Key to Healthy Brain
Aging, that was about the Chimane.
(03:35):
The idea is that their way of life iscloser to that of early humans than that
of people in industrialized societies.
So by studying them, people inindustrialized societies can get
a better idea of where they camefrom, and what industrialized
society is doing to their health.
Likewise, studying children inthese societies can tell you what
(03:56):
kind of knowledge is fundamentalto being human, and what you need
school and written language to learn.
Steven Piantadosi (04:03):
One thing that's interesting is that is that if you work
with people who've never been to school,many of the kind of familiar school
tasks or things that are familiar tous are completely unfamiliar to them.
And that means that if you givethem, for example, an IQ test,
they won't score very well on it.
So, that made me start thinking aboutIQ and looking into some of the problems
(04:24):
that people have argued are presentfor IQ research, namely big cultural
differences and different kinds ofexpectations and motivations and things
that people bring into an IQ test.
And that's what led to the Twitter thread.
Ashley (04:37):
Like so many of these numbers we use to describe humanity,
IQ started innocently enough.
Around the turn of the 20th century,French psychologists Alfred Binet and
Theodore Simon came up with the BinetSimon test, which was designed to identify
(04:57):
schoolchildren whose mental abilitieswere developing more slowly than average.
This is where we get theconcept of someone's mental age.
If your score on the test said your mentalage was younger than your chronological
age, you needed extra help in school.
And if it was older, you were gifted.
Binet, for his part, did notbelieve that intelligence was
(05:19):
one thing that you got at birth.
He believed that there were many waysto be intelligent, and that children
could learn to become more intelligent.
But then America got a hold of the testand gave it the old red, white, and blue.
In 1916, Lewis M.
Terman published the first StanfordBinet test of intelligence, which was an
(05:40):
expanded version of the Binet Simon test.
It was the most popular test in the U.
S.
for decades, and it's where we get thetypical IQ numbers that too many of
my Tinder dates have thrown around.
You get it by dividing a person'smental age by their chronological
age, then multiplying by 100.
By that measure, 100 is average.
(06:01):
It's what you get if your mentalage matches your chronological age.
And anything higher or lower isless common, dropping off toward
the extremes in a bell curve.
In the original test manual, Terman laidout the benefits and uses of the new test.
Next to simple things like identifyinggifted students and determining
vocational fitness, there was the line,quote, "This will ultimately result in
(06:25):
curtailing the reproduction of feeblemindedness and in the elimination
of an enormous amount of crime,pauperism, and industrial inefficiency."
End quote.
If you were playing the eugenicsdrinking game, it's time to take a shot.
Another important figure in whatwe now know as IQ was British
psychologist Charles Spearman.
(06:47):
He observed that children's grades inunrelated subjects tend to be correlated.
Steven Piantadosi (06:52):
So for example, people who tended to get good grades in math
also tended to get good grades in English.
Okay.
And you could think about thatand think like, that's not
necessarily how it has to be, right?
Like maybe you have the intuitionthat it could be the other way around,
that if you get good grades in math,you're less good in English, right?
But that's not kind of empiricallyor quantitatively how it turns out.
People's objective scores on differentthings tend to correlate positively.
Ashley (07:17):
Speerman proposed that all mental traits were related to
a single common factor, which hecalled G for general intelligence.
Pretty much every intelligence testsince has been designed to correlate
as much as possible to this G factor.
Steven Piantadosi (07:32):
For example, if somebody takes the SATs, that number will
be pretty highly correlated with G as youwould have measured it by looking at their
grades across a bunch of different topics.
So, what kind of intelligence testinghas tried to do over the years is
find tests which are highly G loaded,meaning some test I can give you
hopefully something that's kind ofshort and simple, which when I get
(07:54):
your number, it's correlated with yourG, your, general intelligence factor.
And therefore also would do a goodjob at predicting, say, your grades
across a, a bunch of different topicsor other things that intelligence
tests are supposed to correlate with.
Ashley (08:08):
The problem is that nobody has ever definitively
identified what G even is.
Steven Piantadosi (08:14):
It's a real kind of magician's trick to call
it general intelligence, right?
When nobody actually knows if it'sgeneral intelligence, as opposed
to, for example, motivation orexperience or any of these other
kinds of cultural factors that matter.
One thing that happens in psychologyand in some of the kind of culture wars
around this is people sometimes say, well,intelligence is the most statistically
(08:36):
robust area of psychology, right?
It's the thing that is most replicable.
And, there's tons of studies and it's truethat there's tons of studies that robustly
find a general intelligence score, right?
And once you find it, it's truethat there's high replicability
in terms of what tasks arehighly g loaded, for example.
So on kind of a raw statisticallevel, it's true that intelligence
(08:58):
is very well justified.
Where it's very poorly justifiedis on the interpretation.
So, when I say that, you know,there's other things that could
determine G or there's other kindsof confounding factors, those things
have not been well examined byintelligence research and ruled out.
Um, in fact, kind of the opposite, right?
There have been these people whohave manipulated those factors
(09:19):
and shown that those really affecthow you do on intelligence tests.
And therefore, it's really not good tocall them intelligence tests, right?
That's kind of the trick is incalling them intelligence tests.
And once you say that, it sort of feelslike they're measuring intelligence.
Ashley (09:33):
If you don't know exactly what the test measures, and you don't even
have a scientific definition of generalintelligence — what it is, whether
it's inherited, whether you can changeit — that leaves a ton of wiggle room for
people to make their own interpretationsto further their own goals.
A score that a school administratormight say is reason to give a
student extra guidance could be thesame score that makes a dictator
(09:56):
call for their extermination.
I'm not exaggerating.
Nazi Germany systematically wiped outpeople with disabilities, and that
included intellectual disabilities.
They often used IQ testing todetermine who was unfit to live.
But the U.
S.
did some pretty horrificstuff around IQ, too.
(10:17):
During the 20th century, more than60, 000 people were sterilized in 32
states, based on the idea that preventingthe, quote, feeble minded and other,
quote, degenerate stock from havingkids would reduce crime, cut healthcare
costs, and generally improve society.
Some of this continued asrecently as 2010, by the way.
(10:40):
The eugenics drinking game peopleare out of whiskey at this point.
Throughout all of this, the typesof people who got low scores
on the IQ test fit a pattern.
Recent immigrants, peopleof color, people in poverty.
While some take that as evidencethat these groups are just less
intelligent, researchers throughoutthe decades have found that
(11:01):
there are other elements at play.
Steven Piantadosi (11:04):
This is a point actually that has been made by
a number of sociologists over theyears, that the kind of content of
IQ tests is inherently biased towardswhite and dominant cultural groups.
And they make this point by coming upwith other versions of IQ tests, right?
If you ask different kinds ofquestions, which emphasize, say,
black culture, then black kids will dobetter than white kids on those tests.
(11:27):
And so the fact that you canconstruct tests like that, I think
really emphasizes the fact that thetests are constructed and they're
not constructed in a vacuum, right?
They're, they're constructed in away which happens to preferentially
treat, white kids, for example, orrich kids or kids that can afford,
the kinds of tutoring and, andpractice and, schools, which lead
(11:48):
to high performance on the tests.
So, I think that the main mythologyaround IQ tests is that somehow because
it's a test, it's objective, right?
And that I think is the mainthing that's wrong, right?
It's true it's objective in the sensethat you can take it and get a number out,
but somebody had to construct that test.
And if you construct it in a differentway, you'll end up, biasing in
(12:09):
favor of some other racial orethnic or socioeconomic group.
Ashley (12:14):
One example of a test constructed to be biased in favor of another racial
group was developed in the 1970s by RobertWilliams, a psychologist and professor
at the Washington University in St.
Louis.
It was called the Black IntelligenceTest of Cultural Homogeneity.
Yes, that forms the acronym BITCH.
(12:34):
From what I understand, it lookslike it was also sometimes called
the Black Intelligence TestCounterbalanced for Honkies.
Which honestly makes a lot moresense than cultural homogeneity.
I mean, it was counterbalancedfor honkies, so I don't know.
Dr.
Williams had done a lot of researchabout the bias of standardized IQ tests.
(12:57):
One question, for example, hadthe child point to a squirrel that
is beginning to climb the tree.
His Black subjects didpoorly on this question.
But when he changed the question tosay, point to the squirrel that is
fixing to climb the tree, suddenly Blackchildren did better than White children.
So the Black Intelligence Test ofCultural Homogeneity flipped that
(13:20):
bias and took it to its extremes.
This test listed 100 words fromthe Dictionary of Afro American
Slang and asked test takersto identify their meaning.
Things like black draft and applealley, along with words that exist
in white culture but had differentmeanings in 1970s black culture,
like clean and Mother's Day.
(13:41):
Those mean well dressed and the day thewelfare checks come in, respectively.
Unsurprisingly, white test takers do muchworse on this test than black test takers.
Steven Piantadosi (13:52):
There's quite a few historical examples of racial differences
and people reading into those as beingtrue differences in intelligence as
opposed to cultural differences, right?
Differences in schooling, forexample, or opportunity, right?
These other kinds of things thatwe know influence intelligence
test performance, but which wereally shouldn't call intelligence.
This is true in the group I work with.
(14:14):
When people give them intelligencetests, they test close to the
threshold for intellectual disability.
So down in the seventiesor eighties, right?
And, I think anyone who works withthem, like, nobody who works with
them would think that they areintellectually disabled, right?
The issue is that they're notused to taking tests, right?
Because if you've never been to schooland somebody brings you in and starts
(14:35):
showing you, you know, geometricshapes and numbers and asking you
to find patterns and things, thoseare completely unfamiliar tasks
that they've never done before.
And so, of course, they don't score well.
And so you find those kinds of culturaldifferences without the difference
being actually reflective of adifference in intelligence, right?
It's a difference in culture orpractice or something like that.
Ashley (14:58):
One difference might be motivation.
Turns out you can actually improvepeople's scores by offering them money.
A 2011 meta analysis led byAngela Duckworth found that a
$10 reward can increase your IQscore by as much as 20 points.
You can imagine how thismight play out in a classroom.
(15:19):
A kid who knows their family can affordcollege and as long as they get good
test scores they can go to any schoolthey want is probably going to try a lot
harder than a kid who knows that no matterwhat score they get, the needs of their
family mean that college is off the table.
And if IQ tests are really measuringa person's general intelligence,
average scores over the decadesshouldn't be changing that much.
(15:42):
We haven't had enough time toevolve better brains, you know?
And yet.
Steven Piantadosi (15:48):
If you look at IQ tests over the years, like over decades of time,
in general, scores are going up on them.
And this has been a bit of amystery in intelligence testing.
You know, if you thought intelligencetesting was a measure of true
intelligence, that people aregetting, you know, smarter over
time, that I think goes against somedominant cultural narratives, right?
Ashley (16:09):
This phenomenon is known as the Flynn effect, discovered
by researcher James Flynn in 1984.
He found that over 46 years,representative samples of Americans scores
on IQ tests rose by about 14 points.
Which is weird, right?
I mean, name the greatest geniuses inhistory Newton, Galileo, Edison, Einstein.
(16:32):
They all lived a long time ago.
But if you believe that IQ measurestrue intelligence, you'd have to also
believe that your average Joe todayis way smarter than any one of them.
So what's the deal?
Steven Piantadosi (16:45):
But I think it's maybe interesting for people
who care about the mechanisms ofwhat's going on cognitively, right?
Because you want to know whatit is that's changing, right?
Is it something abouteducation that's changing?
Is it something about test prep, right?
Maybe we're testing kids more or somethingand they're, they're getting better at it.
I've heard theories that our educationalsystems are emphasizing abstraction more.
(17:08):
I think this is what, what Flynnhimself thought, was that we taught
abstraction from younger ages and triedto encourage abstraction and thinking
about problems abstractly and that wassomething that maybe wasn't as true in,
for example, our grandparents generation.
And many of the questions you get in anIQ test involve, you know, recognizing
abstract patterns among shapes ornumbers or something like that, right?
(17:31):
And so if we're reinforcing thatkind of abstraction early on and
more in school, then maybe you wouldtend to score better on those tests.
Ashley (17:38):
Yeah.
Is there any evidence that weactually are getting smarter?
Like maybe it's nutrition or maybeit's like something else that isn't
about learning how to take the tests?
Steven Piantadosi (17:48):
I mean, people take the Flynn effect as evidence of that.
I don't know, so, I think it's it's avery hard kind of question to answer
because, you need some objective wayof measuring how smart people are.
And I think that thatpretty much doesn't exist.
Ashley (18:11):
if you didn't call it intelligence, what would you call it?
Steven Piantadosi (18:14):
I don't know.
So that's actually somethingi've thought about a little bit.
Like what is the right kindof name for these things?
I think that often they have kind ofclinical names and and that's fine.
So for instance, there's atest called Raven's Test.
Named after Raven who was thepsychologist who developed it.
And I think it's perfectly fineto talk about what somebody's
Raven score would be, right?
(18:36):
That's often how we might talkabout it in the lab, right?
You know, you're looking at somethingand you say, what was their Raven score?
And that doesn't bring any kindof inherent judgment that Ravens
is the measure of intelligence.
Right?
And I think that that'sprobably good, right?
It'd be good to, to kind of removethose loaded terms from these tests.
Ashley (18:56):
But Dr.
Piantadosi doesn't believe thatintelligence testing is totally useless.
It does have its place.
Steven Piantadosi (19:03):
There's certain settings where I think this kind of
testing is actually useful, right?
So, so people use it in a clinicalsetting, for example, and oftentimes in a
clinical setting, you might be comparing,you know, within a patient, right?
So, you know, patient at one timepoint to another time point, or,
within patients maybe that, thatare very tightly normed in terms of
(19:24):
experience or age or other kinds ofthings like that, education level.
And there, like, you often needa quantitative cutoff, right?
So you might need a quantitativecutoff to decide what kind of treatment
you should do with this person or ifthey are okay and you can continue
watching them or, or whatever, right?
So, I think that there's lots ofareas where we need some quantitative
(19:44):
number, but the problem is when youconvince yourself that that number
is the objective thing, right?
Ashley (19:51):
I don't know about you, but I don't see people abandoning
the IQ test anytime soon.
It's too embedded in ourculture at this point.
So if people are using it, for entranceinto the military, or for school
placement, or a medical diagnosis,what should they be looking out for?
Steven Piantadosi (20:07):
If you use them for something, you have to be very
aware of these kinds of biases, inparticular, the racial biases in the U.
S.
That awareness, I think, meansnot taking the numbers you
know, very seriously, right?
So they might be kind ofuseful as a guide or something.
But if you're looking at a student, forexample, and trying to decide what level
to put them in, you can't just use thenumber because the number is, is biased.
(20:31):
Right?
And so you might look at other kindsof information, like, you know what
their background was or what theirfamily background was try to evaluate
their performance relative to theiropportunities or to their training
or to the motivation that they have.
All of those things arerelevant and it means that the
decision isn't a simple one.
So I think that the people incharge of these decisions probably
(20:51):
won't like that advice, right?
It's you can't just you know, drawa line on a spreadsheet and say
everybody above this is above the line.
Uh, It's really a complicatedkind of context dependent
decision that you have to make.
And, that really comes from just nottaking them very seriously because
they're not measuring what they'resupposed to or what they claim to.
Ashley (21:09):
Right.
Yeah.
So just, you're just makingthings more complicated, but that
is what things are in reality.
So people just need to deal with it.
Steven Piantadosi (21:15):
Yeah, exactly.
Yeah.
I think it's almost an effort againstthe complicated nature of reality.
And so if you're, if you'redenying that, then you're gonna
end up making worse decisions.
Ashley (21:25):
That's it.
If you're fighting against thecomplicated nature of reality,
you're going to make worse decisions.
That's race.
That's BMI.
That's all sorts of medical decisions.
I wanted to hear what Dr.
Piantadosi thought of thisbig picture view I was taking.
(21:46):
It seems like every time scientiststry to come up with a way to simplify
human diversity, um, It causes problems.
Like, even though it's like, we're,it, it, it's understandable they're
trying to do that because things arecomplex and you, you wanna be able
to, to understand the world better.
And, and, and organizingthings, uh, helps you do that.
(22:09):
But it, it seems like that'salways, it always seems to go wrong.
Could, could you talkabout that a little bit?
Steven Piantadosi (22:14):
That's a, a nice observation.
You know, I, I think that there'ssomething maybe even inherent to
doing science, which is tryingto find generalizations, right?
And those generalizations are reallynecessarily simplifications of
what we see happening in the world.
And you can think about even like,textbook kind of scientific laws or,
(22:34):
or results, you know, think about likeNewton's Law of Gravity or something,
or if you, you know, remember inphysics class computing, you know,
the trajectory of a ball that youthrow or something like that, right?
Like all of those are simplificationsbecause you, you ignore things like
friction or you ignore wind resistanceor, or whatever, and those physical
laws are, are useful, even thoughthey're simplifications, right?
(22:56):
Because those simplifications allow youto solve the problem and get you kind
of a good enough answer or a good enoughapproximation in, in those situations.
And I agree with your observation.
I mean, it seems like manyof our simplifications of
human nature aren't like that.
And it could be that human natureis just really complicated, right?
And I think there's other systemswhich are kind of intrinsically or
(23:17):
inherently complicated, like thestock market, for example, right?
That, you know, there may not be a simplekind of law that you could write down
for the stock market that describesits behavior in the same way that
you have for, uh, for Newton's Laws.
And I'm not sure why this happens withtheories of human psychology or people.
Part of it might be that it'sa little bit hard to appreciate
(23:38):
all of the complexity thatthere is within people, right?
So, where we grow up in some communityand we're used to people who have
similar experiences to us, and that mightmake us think that our experience is
universally shared among other people.
But when you go to a completely differentculture, right, a completely, even
industrialized culture, but, but also,you know, a non industrialized group that
(24:00):
lives in the Amazon, for example, right?
Like, you start to see that thingsare just really, really different, and
they approach problems and have a verydifferent set of kind of cognitive skills
and tools that they need for their lives.
And it's different thanwhat you need for your life.
But it makes it clear thatpeople are just very flexible and
very adaptable across differentenvironments and different settings.
(24:27):
there's all kinds of sophisticatedand really beautiful ways that people
have of thinking about the world andthinking about different problems.
And those vary among different humangroups and among cultural groups and
among different languages and Noneof that kind of interestingness is
really captured by detecting shapepatterns or, or something, right?
So, for people like me who are, who areinterested in the real kind of mental
(24:50):
mechanisms of how our brains work, right?
This one number is just, akind of hopeless attempt,
right, at characterizing that.
And is almost completely irrelevant toall of the interesting kinds of processes
that actually happen right inside people'sbrains as they're thinking about things.
Ashley (25:13):
Thanks for listening.
Big, big thanks to Steve Piantadosi.
You can find more of hisresearch at his website, which
is linked in the show notes.
Taboo Science is written andproduced by me, Ashley Hamer.
The theme was by DannyLopatka of DLC Music.
Episode music is from Epidemic Sound.
There's a referral link in the show notesif you want to use it for your own stuff.
(25:35):
Hey, did you know September isInternational Podcast Month?
I think you should celebrate by leavinga nice review for a podcast you love.
The link to leave a review on ApplePodcasts for this podcast, is at
the bottom of the show description.
Just saying.
Anyway, we've just got a few episodesleft in the season, and I have
big, big plans for the next one.
(25:57):
Hope you tune in!
I won't tell anyone.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.