CSI On Trial: Ep 1 – The Flaws of Forensic Science

Popular TV shows and high-profile televised trials have made forensic science famous…but what’s the science and history behind it? Host and filmmaker Molly Hermann traces the roots of these tools and the questions surrounding their accuracy.

CSI On Trial is a co-production of iHeart Podcasts and School of Humans. It is a Curiosity Podcast based on the Curiosity Stream series CSI On Trial.  You can watch all six episodes of the video series here if you sign up for Curiosity Stream.

See omnystudio.com/listener for privacy information.

Episode Transcript

Speaker 1 (00:05):
Oh can you name this TV show? So the width
of the burn and the pinpoint detail in the fuel,
I've narrowed down what our suspects using. Burned the tags
off a blowtworks. Yeah, burns a small precise, just like

(00:29):
what you get with compact torches used in dental labs.
It's CSI Crime Scene Investigation, a CBS crime drama featuring
a team of investigators solving crimes by analyzing the physical evidence.
See once a dried, the pattern revealed itself. You see
this discoloration right here. Looks at the tread from a
boot well I initially thought was a V is actually

(00:52):
a WU And that's the logo for Woodridge Hunting boots.
They look at things like bloodstain patterns, footprints, and bullet casings.
The original show launched in two thousand, then came the
spin and offs, inspiring still more shows that dig into
forensic science, Bones and CIS Dexter. The list is long.

(01:13):
These crime scene procedurals. They're fun, they make investigations digestible,
engaging millions watch them. But these shows also propel the
idea that forensic science is precise in proving guilt or innocence.
It's called the CSI effect. The CSI effect, from a
defense attorney's perspective is is that and shows like that

(01:34):
they depict forensic evidence as infallible, and that it's quick,
it's fast, and they do it in a half an hour,
and they wrap it all up right, and that jurors
come into lajers come into a criminal case with the
expectation that forensic evidence can conceivably do that. That's Chris
fabricand I'm the director of Strategic Litigation at the Innocence

(01:55):
Project in New York, and i am the author of
Junk Science and the American Criminal Justice System. CSI and
shows like it give jurors a false sent of security
that forensic science methods are totally accurate, that expert testimony
from CSI analysts is infallible. Studies show that juries put
an outsized amount of trust in the testimony of experts.

(02:19):
But as we'll discuss in this podcast, forensic science methods
and the expert testimony that shares that evidence with juries
can both be flawed. It's an issue that has serious consequences.
CSI this all crap. That's Keith Harward, one of several
people you'll meet in this series, who was wrongly convicted

(02:40):
based on forensic evidence that turned out to be wrong.
I'm Molly Herman. I'm a filmmaker and producer of a
six part documentary series for Curiosity Stream that digs into
forensic crime science, one discipline at a time. I've interviewed
nearly seventy practitioners, attorneys, scientists, journalists, judges, and innocent people

(03:04):
who spend decades in prison, some who are still there.
When I started this project, I believed that old adage
that the US justice system isn't perfect, but it's still
the best system out there. But what I discovered is
that many methods of forensic science used to build criminal
cases aren't just imperfect, but irresponsible and unscientific with profound

(03:29):
impacts on human lives. And that's the mission of this
podcast to investigate how and why forensic methods are flawed
and how the system can be fixed. From School of Humans,
Curiosity Stream and iHeart Podcasts, this is CSI on Trial.

(03:51):
Two or three stays are really not enough to call
something an impacts patter from gunshot that's going to put
someone in prison the rest of their life. Sto making
up a lie was gonna get you home center. What
is it about a bite mark that would make a
dentist an expert in this area who shot at you?

(04:12):
He said, I will sit in this jail and I
will rot before I take the problem with forensic science
in the criminal legal system today is that it's an
awful lot of forensic and not an awful lot of science.
Episode one, the Flaws of Forensic Science. Popular television has

(04:35):
made forensic science famous. True crime has been a long
time fascination for the American public, from the Salem witch
trials to the Missing Lindbergh Baby, to the prosecution of
gangsters like al Capone. Al Capone goes to jail, the
federal government strikes its first telling law at the underworld
by convicting public Enemy number one on income tax brauds.

(04:58):
Modern forensic science launches because of a massive influx of
federal funding in the late nineteen sixties, after PubL unrest
during the Civil Rights Movement and protests against the Vietnam
War spur President Lyndon Johnson to declare a war on
crime to attack and to overcome growing crime and lawlessness.

(05:18):
I think we must have a stepped up program to
help modernize and strengthen our local police forces. Our people
have a right to feel secure in their homes and
on their streets, and that right just must be secured.
It marks the first moment in American history where the

(05:42):
federal government becomes directly involved in local law enforcement. Menka
Sinha spent a decade as a public defender in Washington,
d c. She's now a law professor and teaches the
Criminal Defense Clinic at the University of Maryland School of Law.
She's published on how forensic science can contribute to wrongful convictions.

(06:03):
And they believe that this crime problem originates in these
black urban communities, and without really consulting with the residents
of these communities themselves, they do this investigation and they
do the study, and they produced this really sweeping report
that was issued in nineteen sixty seven. President Johnson put

(06:25):
together a crime Commission with a mix of backgrounds in
law education, social work, and law enforcement. The report they
issued was called The Challenge of Crime in a Free Society.
It had over two hundred recommendations, one of them the
creation of a national emergency number, you know it as
nine one. One. One of the things the commission points

(06:46):
to is it says we need to improve our law
enforcement technique by improving our capability of analyzing evidence left
at scenes. So we should be improving our ability to
do fingerprint nails, we should be improving our ability to

(07:07):
do shoeprint analysis, and it mentioned the whole bunch of them,
and it says we should be developing a centralized fingerprint database.
And this recommendation becomes the backbone of the development of
crime labs. Using the Commission's report and energized by the
political climate of the time, Congress creates a major federal

(07:28):
funding pipeline for state and local police forces, including funding
for forensic science development and training. In order for police
to be eligible for many of the grants, they have
to show an increase in arrest rates. In other words,
the development of modern forensic science was tied to and
incentivized by arrests and prosecutions. Notice what I didn't say

(07:52):
fueled modern forensic science, and that's science, independent, rigorous science.
Here's Manca again. We tend to think of forensics as
really reliable, really accurate, really scientific, And what most people
don't realize is that most forensic methods they weren't developed

(08:14):
in scientific labs. They weren't developed through scientific method They
weren't developed through scientific research. They were developed by and
for law enforcement to help law enforcement pursue law enforcement functions,
meeting surveillance, investigation, prosecution, securing convictions, and sometimes even at

(08:37):
sentencing to secure a higher punishment. The problem with forensic
science in the criminal legal system today is that it's
an awful lot of forensic and not an awful lot
of science. That's Kate Judson. I'm the executive director of
the Center for Integrity and Forensic Sciences. Much of it
was developed for law enforcement for use in courtrooms rather

(09:00):
than at a lab bench in academia, for example, and
for that reason attend to be quite skewed. It tends
to be in pursuit of a results or an agenda,
rather than simply going where the science leads. In the
late nineteen eighties, a proverbial bomb dropped on forensic sciences.
DNA analysis developed by the scientific community far away from

(09:23):
the world of arrests and prosecutions. DNA analysis held up
a mirror to forensic science methods. My name's Julia Layton.
I'm the former General Counsel for the DC Public Defender Service,
but I've been retired for several years now. When I
went to law school, most of the forensic disciplines that
had been admitted for decades. So in the nineteen eighties

(09:46):
when I started practicing, we weren't challenging the evidence. We
weren't even looking very closely at the evidence. That sort
of changed with the introduction of DNA forensic evidence, because
suddenly this was something new and our responsibility as lawyers
was to investigate it, learn about it, and so slowly
but surely we started to learn about the technology behind it.

(10:08):
We started to learn about the scientific method, and I
think once we had started to master those topics, we
naturally started to turn to look at the other disciplines.
In contrast to DNA analysis, common forensic science disciplines firearms analysis,
footprint comparison, byte mark analysis were revealed as unscientific. But

(10:30):
most troubling and most revealing, DNA analysis was directly contradicting
the validity of forensic evidence that had locked people in
prison or even put them to death. In the early nineties,
DNA technology becomes ripe enough for us to use in
criminal cases. We can go back and test samples that

(10:51):
have never been tested what they learned was that actually,
we have convicted a lot of innocent people, and one
of the major contributing factors they found was some sort
of shoddy forensic evidence that was used in the first place.
So everyone saw what was happening with DNA and exonerations
and there was a huge change in the way we

(11:13):
developed and used forensic science. Right. Well, no, it turns
out there were more powerful forces than science and truth
at play. It was a very impressive set of expertise
in this committee and their report was really damning, and
so for us it was like finally somebody's opened the door,

(11:35):
opened Pandora's box about how little research, how little foundational, empirical,
scientifically valid research has been done, and we thought we
were going to go forth and have a sea change,
and we were wrong. When DNA analysis began proving the

(12:13):
innocence of people originally convicted on the weight of forensics science,
Congress did act in the late two thousands. It funded
a commission to study the issue and tapped the National
Academy of Sciences to run it. The commission released its
report in two thousand and nine. It was titled Strengthening
Forensic Science in the United States a path forward. Manka

(12:35):
Sinha says, the reports authors made sure it had a
tone of optimism rather than blame. So the report's goal
was really to improve the forensic system. So it makes
all these recommendations for how to do that. We're gonna
come up with a research strategy and guide how research
should be funded. We're going to work on standards and
regulations to regulate the use of forensics, and we're going

(12:57):
to all do that centrally and through this national governing body.
David Fegman is the Chancellor and dean at UC Hastings
College of Law in San Francisco. So the report was
published in two thousand and nine, and one of the
things that the report advocated was for a federal agency
to oversee the development of forensic science that would be

(13:19):
able to accredit and certify labs, would oversee research and
fund research, not unlike the National Institutes of Health might do.
I'll just say it, this report came out in two
thousand and nine. It's many many years after that, and
there's no national governing forensic body. It never happened, It
never got off the ground. Forensic analysts and their industry

(13:42):
associations made some cosmetic changes to the language they used.
The Department of Justice continued to provide research money, and
some of the money did go to actual scientists. Some
forensic associations and labs required more training hours, assessments, or accreditations,
but there were no fundamental changes to what techniques were used.

(14:05):
To be fair, the report was released during a tumultuous time.
The research was done under the Bush administration, but released
a few months after President Obama took office. The global
economic crash sucked up all of the oxygen in US
public policy, But years later, the Obama administration decided to
pick up where the Congressional study left off. The President's

(14:28):
Council of Advisors on Science and Technology, also known as PEACAST,
pulled together the nation's top scientific talent to dig deeper
into specific forensic science areas. University of Maryland physics professor
Jim Gates was a principal author. Science is all about measurement,
So the PEACASH report said, we are going to look

(14:51):
at parts of forensic science that are based on measurement
and ask the question, when we look at processes and
practices in forensic science, how closely do those adhere to
measurements made outside of forensic science, and unfortunately not much
improvement had occurred. Professor Thomas Albright from the Salk Institute

(15:11):
also contributed to the report, which came out in twenty sixteen.
The Peacast report broadly condemned a particular category of forensic
practices known as the feature comparison techniques. These are kenner fingerprints,
tire tracks, bulletmarks, and so forth, and the major concern

(15:33):
was that they hadn't been in most cases validated properly validated.
Validation simply means that you've empirically tested the technique so
that you're certain that a large fraction of the time
it gives you the correct answer. But the Department of Justice,
which has been entwined with forensic science since lbj's War
on Crime agenda, rejected the findings on the day of

(15:57):
its release. Attorney General Lauretta Lynch, who was the Attorney
General at the time, she says, no, thank you, Peacast.
We appreciate your efforts, but we are not going to
adopt your recommendations. Attorney General Lynch said in a statement
reported at the time by The Wall Street Journal that quote,

(16:19):
we remain confident that when used properly, forensic science evidence
helps juries identify the guilty and clear the innocent, and
the Department believes that the current legal standards regarding the
admissibility of forensic evidence are based on sound science and
sound legal reasoning. The National District Attorney's Association they denounced

(16:41):
the report immediately. The FBI criticizes the report. A whole
bunch of bodies criticize the report and decline to adopt
the recommendations. William Fitzpatrick from the National District Attorney's Association
made it pretty clear how local prosecutors like himself felt
if I were to have taken the Peacast reports seriously,

(17:04):
which I did, or the two thousand and nine NAS report,
which I did not, as well, and to embrace some
recommendations suggested, I could actually picture myself in my office
telling a grieving father of the domestic violence victim that
even though his daughter's ex boyfriend was ideed fleeing the scene,

(17:25):
even though he left a fingerprint that matched his left
thumb in the victim's blood at the scene, and even
though a friends at godentologists matched a bite mark on
the victim's chest to the defendant and the thirty two
caliber revolver recovered from his jacket pocket was ballistically matched
to the projectile and his daughter's head. It's just right now,

(17:45):
the case is too thin to proceed and I won't
be able to prosecute please. They were worried that what
it would mean for these closed cases, would they open
back up? Would we have to relitigate all of this?
And they were also worried about what it meant for
open cases, even though the authors of the Peacast report
were careful to say they had no view on the

(18:06):
legal question of whether any past cases were erroneously decided.
People convicted of a crime where questionable forensic science was
used might be able to appeal their case and even
walk free. Professor Jim Gates, again one of the report's authors,
there's a large part of the pushback that comes from attorneys,
quite frankly, particularly prosecuting attorneys. And one can understand why

(18:31):
are there motivations for this, because in the current state,
a forensic science is a tool that basically they can
use to get convictions. The only problem, as I said,
as an outside scientist, many of us outside the field say,
this isn't science. You're using the power science. But you're
not doing science. And this is a deep moral as

(18:52):
well as practical problem. I have no criticism at all
of the people who actually do the forensic science. It's
a systemic problem that exists, and that systemic problem impact
some communities more than others. Black people and brown people
and other people for marginalized communities, but especially black people,

(19:15):
suffer the punitive effects of our criminal legal system more
and more harshly at every single stage of the process.
A recent report by the National Registry of Exonerations found
that Black Americans are seven times more likely to be
wrongly convicted of serious crimes than white Americans. So, if

(19:38):
that's the snapshot, and then you have a prosecutor or
a prosecutorial entity saying, well, I won't be able to
get a conviction if I don't have this tool. Well,
if you particularly want to take somebody's liberty away, who
is more likely to be black or brown? In this context,
we should be making sure it's reliable and producing reliable results.

(20:02):
At the least, if you should be able to do
any of this at all, you should be doing it
with tools that are reliable. Both prosecutors and defense attorneys
call on expert witnesses to enter forensic science evidence into
court and to explain that evidence to the jury. Judges
are supposed to be the gatekeepers of those expert witnesses

(20:22):
to make sure junk science or junk scientists aren't allowed
in court. But that's not happening. Sometimes you have experts
who come in and lab codes and it feels really scientific,
even when it's not. It can have this effect of
making other aspects of a case that are questionable sort

(20:44):
of wash them away. We've talked about how law enforcement

(21:10):
and prosecutors rely on forensic science, but what about judges.
How do they fit in a critical Supreme Court ruling
in nineteen ninety three set as standard on evidence judges
are supposed to follow. Here's David Fegman again. He specializes
in the use of scientific research and legal decision making.

(21:31):
And then you had in nineteen ninety three, the United
States Supreme Court decided what is now a landmark case,
Daubert versus Merldal Pharmaceuticals. And what the US Supreme Court
said was that judges under the federal rules of evidence
had to be gatekeepers and they had to look at
the underlying methods and principles for scientific evidence that's being

(21:54):
offered in court. It's called the Daubert standard after the
court case that established it. So what happened in the
early nineties is courts started look at what the data
actually were, what the methods that were being used, and
they realized that there wasn't much there. And a good
example of that is handwriting in the handwriting area. Quote

(22:17):
started asking well, where the data, and the forensic scientists
basically said, well, we don't have data, we don't have studies,
but we know we can do it because we've been
doing it for twenty five years. So trust us. And
judges said, well, under the Daubert rule, you have to
tell us the statistics. You have to give us the
research studies. They have to be pure viewed in mainstream

(22:39):
scientific journals. And these experts started saying, we don't have
any of that, and judges, I think we're surprised. Through
pre trial evidentiary hearings, judges are supposed to evaluate the
forensic science and rule on whether or not a jury
should hear it. What people forget is that judges are
human being subject to the same kind of cognitive pressures

(23:01):
that other people are This is Nancy Gertner. I was
a criminal defense lawyer and a civil rights lawyer for
twenty four years, and I was on the bench federal
bench for seventeen and now I teach, I write, I
do some litigation. I'm sort of a gadfly in lots
of areas. When judges are asked to decide on these issues,

(23:24):
they often fall back on established precedent. If forensic evidence
has been admitted over the years have in an unchallenged way,
then there's precedent for it, and a judge, a trial judge,
knows that allowing it in is consistent with the precedent.
It enforces this feedback loop that keeps us from critically

(23:45):
examining forensic evidence. Overruling that precedent is rare because it's
a big deal. A ruling like that is like a
rock thrown in a pond. It can send ripples system wide.
So judges will often let the evidence and the expert
witness in and then the chances are that if you
let in evidence like this, you'll be affirmed. You will

(24:10):
be affirmed on appeal, and you'll be affirmed on appeal.
Because the review of evidentiary decisions that a judge makes
is actually quite cursory, is really not very substantial. So
if a judge says, I'm going to allow in this
arson evidence based on the precedent, even if it's unexamined
precedent over years and years and years and years, the

(24:31):
odds are you'll be affirmed. And I think that that
kept lawyers from raising the issues. It's good for the
system to have standards of some threshold reliability and the
irony or the tragedy is on the civil side, the
courts are pretty good at restricting what comes in. And

(24:51):
it's really been on the criminal side and the forensic
side that it's been a much steeper held a quie.
So that means on the criminal side juries not judges
are often weighing forensic evidence. The judge, you'll say, well,
I'll err on the side of let the jury decide
that way. It won't be error that way. I won't
be making a mistake that way. Know, the acquittal in

(25:14):
this case of the hung jury won't be on me.
So if you're a juror and you're sitting in a courtroom,
you are likely to weigh evidence that you perceive to
be scientific or from an expert more highly than other
pieces of evidence. In that case. This is Manka Sinha again,
and that's a fundamental problem because the jurors don't know

(25:37):
about the carcerel origins of these disciplines. Jurors don't know
about the criticism and the scrutiny that these disciplines have
received for failing to really shore up the scientific underpinnings.
Jurors don't know that stuff. Scientific studies have revealed that
forensic science testimony has outsized influence with juries. Jay Kohler

(25:59):
teaches at Northwestern's Pritzker School of Law and studies behavioral
decision theory. Studies show that jurors generally do give a
lot of weight to expert testimony, and because this testimony
is often technical in nature, jurors tend to rely on
the experts credentials and experience when deciding how much weight
to give to this testimony. But some studies show that

(26:20):
jurors don't particularly care about whether the methods used by
the experts have been scientifically tested or not. Instead, they're
generally content to trust the credential and experienced expert. This
is a problem in some forensic sciences, where most experts
are trained and experienced, but where the methods used haven't
been subjected to rigorous scientific testing. When these experts testify

(26:43):
in trials, they tend to use one phrase over and over,
reasonable scientific certainty. But what does that even mean? To
a reasonable degree of scientific certainty? Is kind of a
bogus phrase. It doesn't really have any meaning. It's just
a phrase that has caught on in the courtrooms over
the last ten or twenty years. Unfortunately, now people are

(27:04):
onto it and realizing that's just a misleading wave of saying.
In my opinion, here's what I think has happened. The
phrase came about in nineteen thirty five as a ploy
from a lawyer. The case was about a death resulting
from a capsized boat. The plaintiff's lawyer asked a witness,
a man who had experienced building this kind of motor boat,

(27:26):
if he could quote determine with reasonable scientific certainty the
cause of the capsizing of the boat. So a guy
who had experience in building boats, not a scientist, was
providing scientific certainty. David Fegman, most of the folks that
testify in court for prosecutors are not scientists themselves. They

(27:48):
are technicians. They are simply applying a technology that has
been applied for decades, if not a century. And if
you keep applying the same protocols and the protocols are
not producing valid, good information, then you just keep on
doing it over and over again, and there's no real
testing of it. So let's recap the state of forensic science.

(28:12):
Many of the methods were all familiar with from popular
TV shows like CSI are used in crime labs and
presented in courtrooms all across the country without ever having
been subject to scientific validation. That means there is no
proof the methods work. And then DNA comes along and
starts proving that some people who you'll hear from in

(28:34):
this podcast whose convictions were based on CSI evidence are
actually innocent. The wider consequences of those exonerations is, of course,
that the CSI evidence therefore had to be wrong. There
have been at least two major national studies calling for
independent oversight of forensic science outside the criminal legal system.

(28:57):
That oversight hasn't happened. Meanwhile, the Supreme Court says judges
are supposed to play a key role in keeping bad
science out of the courtroom, but many don't want to
make waves. So instead, juries are often left to decide,
and juries find scientific experts to be very convincing and influential. Sometimes,

(29:21):
when my kids were little, we'd resolve a conflict in
family court. Each kid could make their case and then
the judge me would rule, and everyone had to agree
to abide by the ruling. I think I mostly got
it right, but even if someone felt wronged, it was
better than fighting it out in the backyard, and hopefully
they knew I was trying. That's sort of how I

(29:42):
thought the justice system worked. Sure, a few eggs might
get broken, but generally it's better than the alternative. But
no amount of fair mindedness really matters. If the basic
tools a decision is based on our flawed. So if
the s in CSI isn't verified science, where does that

(30:02):
leave things? How should we view forensic science. Here's what
former public defender Julia Layton thinks. Forensics is a high
risk industry. It's no different than medicine and aviation. People's
lives are at stake, public safety is at stake. Getting
it right matters, and we should be treating it as

(30:25):
a high risk industry. We should be regulating it. We
should be investing in it and we should be overseeing it.
So think about that. Think about sitting on a plane
with your family about to take off for a long flight.
You might be afraid of flying, but logic tells you
it's safe because the FAA, the air traffic controllers, the airline,

(30:46):
the pilot, and the crew have all checked the data.
The plane couldn't even be in service if it wasn't safe.
Think about picking up your medication from the pharmacist. Do
you pause before swallowing a pill to consider whether or
not the pharmaceutical company was on the ball when they
developed it, or do you assume it's safe because the
industry is regulated. Now, what's being done to protect you

(31:09):
or a loved one from junk forensic science? We now
have four hundred public crime labs operating across the US,
along with countless private contractors. In the coming episodes of
this podcast, we will dive into specific methods like bite marks,
bloodstained pattern analysis, and footprints. We'll explain the history behind

(31:31):
the method, how it's practiced, and how reliable it really is.
Kate Judson wants us to keep in mind that most
people that walk into a courtroom are doing it with
the best of intentions to find justice for a victim.
I think that the majority of people who contribute to
bad science coming into the courtroom are doing so because

(31:53):
they are really trying to be right. I mean, they
want the right result. They want to make sure that
yield to people go to prison and in US and
people go free. And they just are working within a
system that doesn't have the quality assurances it should have.
On the next episode of CSI on Trial, what is

(32:16):
it about a bite mark that would make a dentist
an expert in this area? Why use bite mark evidence
when just this year long to other people besides myself
have gotten out and bite mark evidence was used in
the trial. I mean, explain to me what does it
take to realize this stuff's all crap. CSI on Trial

(32:42):
is a coproduction of iHeart Podcasts and School of Humans
based on the Curiosity Stream series CSI on Trial, created
by Eleanor Grant and produced by The Biscuit Factory. You
can watch all six episodes of the video series right
now at curiosity stream dot com. This episode is hosted

(33:03):
and written by me Molly Herman and researched by Katie
Dunn and myself. Our producer is Miranda Hawkins. Jessica Metzger
is the senior producer. Virginia Prescott, Jason English, Brandon Barr
and L. C. Crowley are the executive producers. Sound design
in mix by Miranda Hawkins. Special thanks to John Higgins,

(33:28):
Rob Burke, Rob Lyle and Brandon Craigie. If you're enjoying
the show, leave a review in your favorite podcast app.
Check out the Curiosity Audio Network for podcasts covering history,
pop culture, true crime, and more. School of Humans