December 3, 2025 • 30 mins
To understand what a concept is, we must also understand what it is not. In this episode, Bill Goffe joins us to discuss his study of the effect of the use of negative examples on student learning outcomes.
Bill is a Teaching Professor in the Economics Department at Penn State. He had previously been one of our colleagues here at SUNY Oswego. Bill is very well known in the profession for his Resources for Economists on the Internet, which was one of the very first internet guides available for economists, and it’s now hosted and sponsored by the American Economic Association. He has served as a member of the American Economic Association’s Committee on Economic Education, the Secretary-Treasurer for the Society of Computational Economics, an Associate Editor for Computational Economics and the online section of the Journal of Economic Education. Bill is currently co-chair of the Liberal Arts Teaching Group, a faculty learning community at Penn State.
A transcript of this episode and show notes may be found at http://teaforteaching.com.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
To understand what a concept is, we must also understand what it is not. In this episode,
we discuss a study of the effect of the use of negative
examples on student learning outcomes. Thanks for joining us for Tea for Teaching,
(00:20):
an informal discussion of innovative and effective practices in teaching and learning.
This podcast series is hosted by John Kane, an economist...
...and Rebecca Mushtare, a graphic designer...
...and features guests doing important research and advocacy work to make higher education more
inclusive and supportive of all learners.Our guest today is Bill Goffe. Bill is a
(00:51):
Teaching Professor in the Economics Department at Penn State. He had previously been one of
our colleagues here at SUNY Oswego. Bill is very well known in the profession for
his Resources for Economists on the Internet, which was one of the very first internet guides
available for economists, and it’s now hosted and sponsored by the American Economic Association.
He has served as a member of the American Economic Association’s Committee on Economic Education,
(01:15):
the Secretary-Treasurer for the Society of Computational Economics,
an Associate Editor for Computational Economics and the online section of the Journal of Economic
Education. Bill is currently co-chair of the Liberal Arts Teaching Group, a faculty learning
community at Penn State. You can also find Bill on many listservs and forums devoted to teaching
(01:37):
and learning. Welcome back, Bill. Thanks, John. Appreciate that.
Today's teas are:... Bill, are you drinking any tea with us today?
I am. I'm drinking Twinings green tea, decaf.Nice, good choice. How about you, John?
I am drinking a peppermint tea from the same vendor.
Nice. I have hot cinnamon spice today because my daughter requested a pot of it. So we're sharing
(02:03):
a pot of hot cinnamon spice this afternoon.Very nice as we approach the holiday season
here when we're recording. So we've invited you here today to discuss your recent working paper
with Austin Boyle entitled “Negative Examples in Lecture Improve Student Learning. First, can you
tell us what you mean by negative examples? Sure. So you're describing a concept to someone,
(02:26):
let's say a young child, and describing what a dog is, and you talk about different types of dogs:
Dalmatians, Pomeranians, whatever. And unless you describe how a cat is not a dog, they're
probably not going to have a great idea of what a dog actually is. And this might be the case,
say, for a second language learner, where you might have different terms for different types
of mammals, also related to something called contrasting cases, where experts are very good
(02:52):
at delineating different types of phenomena or different types of facts, if you will,
say a physician or variation theory used in math. So these are all ways of getting students
a rich understanding of something. Can you talk a little bit about what
prompted your interest in negative examples? Sure. So I've used clickers to teach for a long,
long time. You currently use iClicker Cloud. Love it. And I tried to write a paper on just
(03:17):
in general, using clickers to teach economics, and the paper didn't really work very well.
There's reasons it's never been written, and the reason was I use clickers in many different ways,
so a little bit more focused approach to having a paper. And a few years ago, I came across the
book How to Teach Economics by Yousuf Hamid. He was a high school teacher in Great Britain,
(03:40):
and I had to read that. Well, to be honest, I read it during graduation. You have to go to graduation
every year, but, you know, it always kind of work. And he talked a lot about negative examples. And I
thought, “Oh, that's pretty interesting.” And it also followed work by Kirschner, Sweller,
and Clark on the idea of direct instruction lowercase “d” and “i”, and I thought, “Well,
(04:02):
that'd be an interesting to try, and it's fairly focused,” and that's the genesis of the paper,
and pulled Austin into it as well.We'll include citations to those resources
in the show notes. So, many people, when they're teaching, tend to use primarily
positive examples. What's the problem with just using positive examples of a concept? I think
you've already addressed that a little bit.One example, the original work on this, at least
(04:27):
formal original work… the idea goes back a long, long time… it was by Engelmann and Carnine, a book
on direct instruction. And they give an example of a glem, a G- L-E,-M. Could you define that just
with one example? Well, you could not. And they give an example of a pencil. Well, is it wood, or
is the pencil horizontal, whatever? But it helps you focus in better on an idea when you understand
(04:50):
what it's not. For example, in economics, we talk about capital quite a bit. Unless you tell
students, it's not stock, they probably won't pick up on that. Or you give examples of money,
how money is very specific to economics. To economists, they're probably going to pick up
on it's not some general sense of wealth.And I can affirm that the concept of capital
(05:11):
is one tht students have always had a lot of trouble about because the way we use it in
economics differs from how it's used in everyday use. And just for our listeners, could you explain
the difference between the everyday use of capital and the way we use it in economics?
I'm an everyday listener.Great. We'll see if this works. So,
to an economist, capital is something owned by business, manufactured and used to make things,
(05:37):
so, a commercial mowing machine, a commercial oven, a jet liner,
construction equipment. In everyday usage, it might also include financial capital, stocks,
for example, might be considered capital. I would add one thing economists really haven't done,
the Bureau of Economic Analysis actually redefined capital recently, includes intellectual capital,
(05:58):
and I just don't teach that. It would be confusing, but there's actually about as
much intellectual capital these days as physical capital in the United States. But maybe in an
intermediate course. I would do that.Seems like an easy thing to get confused.
Can you talk a little bit about how you pick which concepts to use negative examples for?
(06:19):
I can imagine that some concepts are more ripe for this strategy than others.
Yes, we wanted something fairly limited, not a long concept of how to manipulate a
graph or something, that could be a further work, but just a definitional sort of thing,
but the students have misconceptions about this, as we described there with capital. And so we
picked three different ones, money, capital, and technology. Technology in everyday use,
(06:44):
our phones, for economists, it's basically efficiency in the production process. You know,
it is rather unfortunate, there's what, 50,000 words in the English language, and we use words
that are used in other ways just to make it more complicated. And I joke to students,
I wish I could use different words, but that would hinder their further education, I fear..
And this has been a problem for economists for well over a century in those very same issues,
(07:08):
and it would be nice if we didn't have to do that, because it introduces that confusion
right when students are just starting to learn the vocabulary of economics, and the way I
talk to them about I say, “Hey, part of this is you're just learning the language of economics,
which may look similar to how we use these words in everyday use, but it's a bit different,
and you need to focus on these things because of the ways in which we're using the term.”
(07:32):
I would add that I've seen some STEM education research papers. They teach concepts first and
jargon second, and they find that actually improves learning because you're trying to
remember the terms and they're using the terms, it overloads your working memory. And there you teach
jargon later, they find improvements. We do that sometimes in design, where we
separate the technical skill from the concept, because learning them all at the same time can
(07:54):
be complicated. I was thinking of a term that I run into all the time where there's confusion,
where this strategy might work well, which is digital accessibility,
where people confuse the idea of meeting these standards to help people with disabilities gain
access to content and material with the idea of “just available,” like it's on the internet.
(08:15):
I learned that distinction just recently myself. I would add on choosing these terms, we found even
after instruction, students got it right between 60 and 71% of the time. So we seem to have picked
a nice set of terms and definitions to use.Which is actually much better than they would
tend to if you had only used positive examples. And we'll talk about those
(08:36):
results a bit in just a minute.Can you tell us a little bit about
your study? How you had it set up? How many students were involved?
Yes, we had about 1200 students. I teach large sections, about three sections every semester,
about 340 students that did this over two semesters, basically just repeated things,
get more power for our tests, and did a so-called crossover study. Crossover study,
(08:58):
have your participants that are both controls and the intervention side of things. We did a 3x3 one,
which is a little bit hard to describe on a podcast, where you can't visualize things,
but there's a fairly famous one for ultra processed diets. You might have heard of by
Hall and company in 2019. They put people into hospitals, they control directly what they ate,
(09:18):
and one group got ultra processed foods for two weeks, the other group got more wholesome foods,
I guess you could say, and then they switched who got the wholesome foods, who got the ultra
processed foods. So thus you could deal with the fact that people might have responded differently,
that different foods have different effects on people. And we did the same thing here in
our study, except we had three concepts and three sections and three teaching methods.
(09:43):
So what were the three teaching methods that you used in the study?
So we just use a positive example, where I would list a half dozen examples of capital or money,
or not quite of money, because there are quite six examples of that. Then with lecture,
we had both positive and negative examples, where these things are money, this is not.
This is capital. This is not. And a third type of teaching method was with a cliicker question,
(10:09):
where I gave a student's list of things. Some are positive, some are negative examples,
and they just choosed which was which.So what did you find?
We found that including negative examples improved learning in the following class by
about 21 percentage points. Let's say the class got 50% correct just with positive examples,
(10:31):
the class that got positive and negative examples got a 71% correct. So it is very
statistically significant, it was robust with lot different statistical methods we used. So
we're pretty pleased with that, it was a clean result. Sample size helps, and I think probably
doing it just with precise definitions, not long drawn out concepts or something like that.
(10:52):
Based on these results, would you suggest that perhaps we should all focus a bit more
on using negative examples as well as positive examples in our teaching?
Yes, most certainly it's an easy intervention. For example, you might know Jim Lang's book Small
Teaching, where he talks about easy interventions you can do, and this one's easy, as long as you
remember to do it. But you're making up your notes, you think of things that aren't whatever
(11:14):
you're talking about. You've probably seen those on exams, overall. Interesting result we had too
that the clicker questions, the active learning, was no different than the lecture. So we'd like
to do further work on that at some point, most active learning studies have been semester long,
active learning not very well defined, and we didn't find any difference here,
(11:35):
so presumably, at some point, active learning make a difference be interesting to find out what that
difference might actually be. That's an interesting result,
and that's probably worth some follow up.I think partly on if you do see similar results
with the flipped class and active learning and such, people don't really have a framework on
what they actually ought to be doing before class and what active learning should look like. I've
(11:56):
got another paper on this, I mentioned, that uses so-called deliberate practice,
That's the 10,000 hour rule. It's really not 10,000 hours, but there's some key parts of
active learning people don't do that much. One is to focus on sub skills, for us in economics,
things like change in demand versus change in quantity demanded. We generally don't teach that
specifically, is my thinking. And then also, you want to make connections between things. How are
(12:19):
supply curves and demand curves separate? How are they similar? Because we remember things in terms
of frameworks or schemas. The negative examples help develop schemas. You understand what things
are not, kind of triangulating, if you will. And good active learning does the same thing,
but if they're just filling in a worksheet with numbers, they plug into the formula, that's
active learning, I suppose, but not much learning there. So we ought to focus more on exactly what
(12:44):
active learning is doing for people.Following your study, what are some
different ways that you've incorporated this practice into your own teaching practice?
So I definitely try to do the negative examples, capturing, oftentimes what I see on these exams
and when I’m thinking of a question, I always try to have it connected to something else.
Oftentimes one concept’s related to another one, and we make that connection explicitly
(13:07):
in clicker questions so that, “Oh, if you remember this, this is a variation of that.” For example,
how is consumer price index similar and different than the GDP deflator would be one thing we do,
and why are they fairly similar? To understand better how those things work. Oftentimes,
I explain things based upon what they've done before. Another thing I do is we have very nice
(13:29):
technology at Penn State, I have a main screen in the classroom. I have a side screen. Side screens,
I put up an outline includes both the outline for the day and our bigger topics. So I'm constantly
referring to that outline saying “what we're doing here is a variation of this, is different
than that,” and so forth. And if I do surveys as students, they find that very useful. A better
(13:49):
study would be to not have the outline there someday and see how learning goes. I just now
thought of that as an idea to do a study on.We know that students come in with a weaker set
of connections than faculty, and sometimes we assume that they're able to make these
connections and see the relationships, but they haven't yet developed that skill,
and sometimes it's generally believed that that type of scaffolding, by putting up an outline of
(14:14):
the main concepts you're trying to get across, can help. But I don't know if I've ever seen any
tests of that, certainly not in economics, but I'm not sure if I've seen it in other areas either.
I have not. I can't say I’ve looked very hard. I would add, to do my literature
searches. Now I oftentimes start with ChatGPT, but that's another whole podcast, I suspect.
(14:34):
Yes, and there's some other good tools out there, too. There's research rabbit and other things that
will actually, even in some cases, will provide more accurate results. But ChatGPT is getting a
lot better all the time for these things.Other than in the test results,
have students responded? Or have you asked students about using the negative
questions and how that impacts their learning?I have not actually, that's a good point. That'd
(14:58):
be another thing to add. It'd be interesting see if they actually can pick up on that. Oftentimes,
they're not that great on understanding their own learning. Something we’ve all seen as instructors,
someone thought they did much better on the exam than they actually did,
which is something you want to avoid. That's a tough place to be in and you have to be
sympathetic as instructor when that occurs.Now, one thing to keep Rebecca happy, we should
(15:18):
also ask a little bit about the methodology that you used. In the study, you used both
pooled data as well as panel data, and you also used ordinary least squares and binary dependent
variable models. So could you just talk a little bit about why you used multiple approaches and how
consistent were the results across those. Yeah, the results are very consistent. So
(15:41):
sometimes you don't quite know what econometric approach you take, you can argue this is better,
that's better. And another approach, and it's pretty common these days, I think it's by far
the most common, is you take multiple approaches and see if the results are robust, to see if you
get similar results. Indeed, we did. So it's a very happy occurrence when that occurs.
So were you surprised at the magnitude and strength of these results? I know when I
(16:04):
read the paper, I was surprised at just how large that effect was, because it's
very rare to see effect sizes that large in a study of alternative teaching approaches.
I don't think I had many preconceptions here. That's odd. I certainly was pleased, but I can’t
say I was quite surprised. It certainly makes me want to take this approach again with where we
do the crossover studies, you can do many things with that. For example, we're told we should teach
(16:28):
about things students find interesting. Well, is that really true? I did an example of that
this semester where Penn State students, they do a large fundraising project called Dance Marathon,
or THON, and last year it raised $17 million, about half the student body is involved. And so I
do a question about real prices, and the question was, did THON actually raise more money when you
(16:48):
correct for inflation, you know, something they're invested in a lot, and informally looking at the
data, that didn't seem to have any impact on that segment, on calculating real prices,
I need to do more work on that, but I hope more people take this methodological approach,
the crossover studies, where you could study a lot of things, the outlines,
is it interesting or not, and so forth?Because with the crossover study, you're
(17:11):
ruling out the possibility that there was just some systematic difference between the sections
or between the other conditions, right?Yes, exactly. We actually found there were
differences between the sections. Some did much better on the questions than others,
but by effectively pooling these different interventions and controls together,
we can adjust for that. We can correct for that. I don't see it that much in economics. John, have
(17:31):
you seen it very much, the crossover studies?I haven't seen very many, but one that stands out,
and I think it was done by economists, was back when the initial studies came out that suggested
that when people took notes by hand, they remembered more than when they used a computer.
I do remember a study, and I'll include a citation to this in the show notes. What they did is they
(17:53):
had all the students use mobile devices for note taking for a portion of the semester,
they broke up the group into two halves, and half of them, for the first half of the semester,
took all the notes using mobile devices, and the other half took notes by hand. And in the second
half of the semester, they switched, and what they found was there was no difference between
(18:16):
how students did when they took notes by hand and how those same students did when they took
notes on the computer. However, those students who regularly chose to take notes on a computer,
on average, did less well regardless of whether they were taking notes by hand or on a computer,
which suggests that the students who choose to take notes by hand may be perhaps a little
(18:40):
bit more focused on the material no matter how they're taking the notes, and those students who
may bring in mobile devices to use, perhaps may do it because they prefer having the availability of
distractions. So it's not quite the same thing, but it's similar in that you're controlling for
the individual effects and separating it out so that you are getting a better comparison,
(19:02):
correcting for any self selection in the people who are involved, which is really, I think, what
crossover studies are trying to remove, the effect of any systematic selection between the groups.
I would add, I've surveyed my students, and generally my students aren't performing quite
as wel do choose to use laptops, and higher performing students do not, so consistent
(19:23):
with your more serious study, for sure.And another thing I've been noticing a lot is that
for quite a while, I've been walking around quite a bit, but I've also noted that the students in
the front third or so of the class tend to be very actively engaged in answering questions, and they
tend to do really well, and they tend to ask more questions. But when I'm in the back of the room,
I see a lot of laptops where there perhaps may be a video playing, there may be some online
(19:48):
shopping going on, or there may be some sort of video game that they're working on on the mobile
device and their laptop or one or the other. And I think we often do see that sort of sorting in
terms of the way in which students choose to work. I'm not sure how I could get rid of that effect,
other than moving them all up to the front.it does seem like at least the multi taskers,
(20:09):
in quotes, are at least removing themselves, and because they're probably distracting to
other people, they're just moving themselves to the back so they're not a distraction
to the people who are trying to focus.Except they’re surrounded by other people,
and there may be a bit of a contagion effect going on back there.
I've seen papers on the contagion effect, yes. There was a Carl Wieman paper. They did look
(20:30):
at seating in large classrooms and found a very definite effect. They even switched seats halfway
through the semester to bring the people in the back up to the front, and they still did worse.
Some of our students are just much more interested in the material and much more engaged than others,
and there's just so much we can do with that.It's tough, but I would be a little bit
sympathetic to students. It's been a long time since I've had to sit through a typical
(20:53):
undergraduate today, and you know, you're three, maybe four courses a day, a lot of sitting. That
would be tough, a challenge. There's a book called My Freshman Year, and there’s a field
anthropologist who went to college for a year to understand her students better,
and she found it was harder than she thought.And she found some dramatic differences, too,
in the way in which students connected with each other from when she had been a student,
(21:16):
that they had much tighter networks with the people they knew before they came to college,
because it's less costly to connect with them over social media and so forth.
Yeah, friendships have changed with the rise of social media.
It used to be costly. I remember having to bring all these quarters to put in pay
phones back when I was in school.You mean, in those 1900s phones
that my daughter refers to, John?Yes, I think that would be the case.
(21:40):
What's a quarter?Yeah. Well, I think we
still know what quarters are. We just don't know what pennies are anymore.
Well, that could be, yes. Are there any other things
you'd like to share about this study?Well, I think that's pretty much it. I think
I would emphasize this little bit more about, again, developing student schemas. But we talked
about that earlier today, this broad network, and it’s through connections, and this helps them make
(22:01):
connections more focused on whatever the topic might be at hand. So I think negative examples
could be used for teaching most anything.Including our colleagues when we're doing
professional development and all kinds of other contexts.
Yes. And it's
easy for us to lose focus, because we already can distinguish these things more easily, and students
(22:22):
have not yet developed that capability.The curse of knowledge really reigns here,
where you know it well. It's hard to understand how other people don't. I was doing a think
aloud exercise the end of last semester where I interviewed students half the semester in how
they'd interpret different things. And I spent a lot of time on aggregate supply, aggregate demand
model in my course, standard staple things course, asked about certain phenomena, no one actually
(22:46):
used the model. I was only interviewing four students, so, I mean, it's not much power there,
but it was kind of crushing, but it gave me insights on how they think about things.
In an earlier podcast with Doug McKee, he talked about having students come in and
just work through the problems verbally, so he could see what sort of process they were using,
(23:08):
and by seeing that, it was easier to help guide them in a more productive direction.
Because if we don't really understand how students are processing things, it's hard
to help them understand things more deeply.Yeah, we use that talk-aloud method and design
a lot too around troubleshooting, because troubleshooting strategies require, especially
(23:30):
around technical skills, you need to think about problems in particular kinds of ways to be able
to solve them correctly, or to be able to solve them at all, and doing that kind of talk aloud
strategy works really well for figuring out where something went awry in the troubleshooting.
And experts think a lot differently than students, I would suspect. You
would never think someone thought that way.And fMRI studies confirm that different areas
(23:52):
of the brain are activated for experts and for novices when trying to solve a problem. So it's
easier when you become an expert, but we have to help students develop the skills to get there.
While we've got you here, could you also tell us a little bit about how you teach classes so
large with so many students, every semester.I use Perusall as a way to have them read ahead.
(24:14):
They come into class, they have some sense of things and then a lot of clicker questions,
typically, as long as it's not something new. For example, I was doing long-run growth. I do that a
fair amount, and the book doesn't really talk much about the institutions and the role of markets
in that. So that was a day or two of me talking. I'm going to be doing the business cycle starting
tomorrow, and that's a fair amount of me talking. But other than that, it's them pretty much asking
(24:37):
questions, or me asking questions. Probably on Thursday, we'll get to calculate employment
rates like read about that in Perusall. I’ll ask them, “Who in this group of people is unemployed?”
Then I'll show them some data, what we compute from that data. So it's a lot of using ideas. I
do a lot of peer instruction where they first do votes on their own, and then I say, “Okay,
there's a lot of uncertainty here. Please convince your neighbors.” And I use an app by iClicker, so
(25:03):
I can see their votes on my phone, and then they do a revote, and they usually improve 20, maybe
30 points, then I provide further instruction on the topic at hand. And there's a couple papers
I've seen. Let's see Michelle Smith at Cornell, a computer science paper. They found that if you
have students revote, that helps them; if you have a further explanation from the instructor
(25:23):
that helps them; if you both, it helps them even more, and it's probably enriching their schemas,
they're seeing several explanations that reinforce the connections between things. Also I do a series
of quizzes, rather than midterms. I do seven quizzes. We're very fortunate at Penn State,
we have a testing center, so I send all my students there for testing. So I run the quizzes
(25:44):
for two days, after the quiz they do a second stage of the quiz. In class they start, then they
answer at home using any resource, and finally I release the answers, and then they reflect on and
describe what they found most difficult on that quiz. So we do a two-week cycle. There's a lot
of feedback, even though the course is quite large, with about 1000 students a semester.
(26:05):
So having that testing center gives you a bit more security in the assessments than many of
us have when we give quizzes online. Many of the assessments I use are done online,
and I used to feel a whole lot better about them before we had ChatGPT and other AI tools appear,
especially with some of the browser extensions, which will directly answer things, so you're able
(26:27):
to avoid some of that, which is really nice.I really feel very fortunate, yes. I would add,
it's a good thing that the AI tools came after Covid, rather than before.
So one of the things that you mentioned when you're describing your class was
Perusall. Can you describe what that is for people that don't know?
Oh, yes. An e-text goes inside perusal, and then students mark it up with comments, questions,
(26:48):
and responses to other people's questions. So it has them thinking bit about the text. It grades
them automatically. They need to start early so there's a little bit of spacing there. They need
to make sure they document across the assignment, and they're also rewarded for opening up multiple
times, not just the evening before the exam. And it's fairly straightforward. Most students get
full credit on each assignment. So it's a bit of encouragement there to read ahead. I used to use
(27:13):
so called JITs for just-in-time teaching assignments where I'd ask some questions,
including what they found confusing, but I like Perusall better. It's less grading work,
and also, I think it covers a little bit more topics or not just looking at the
answers to my JIT questions.Now, do you break them up into
groups when they're using Perusall?Yes, Perusall actually does that. When I
look at the marked up text from 1000 students, the page does not load basically, or has difficulty
(27:38):
loading. And you can also download, I've not done this, you could download all their comments,
and you can run through a chatbot or whatever to get a sense of what they have trouble with,
or what they find confusing. I think textbook authors would find that really,
really useful. You could say your textbook was designed with the input from 1000s of students,
and so your students can read this more carefully. I've not found an author willing to do that, but I
(28:02):
think that would be a big win for someone.It could be, and from a marketing standpoint,
that could be really helpful. When iClicker announced that they had AI tools being built in,
I was really hoping that they would integrate that into the text responses, because right now,
it's just generating questions, which is not bad, and it doesn't do a really bad job of that,
(28:24):
but it would be so nice when you got maybe 1000 or more responses coming in, or a few 1000 responses,
if it could very quickly generate a summary of the responses to free-response questions. That
seems like something that wouldn't be all that hard to do, and it would be a nice extension.
I agree. I think there's some polling technologies that do that. For some
(28:45):
reason iClicker doesn't. I wonder if there's a legal issue or some other constraint. I would
love to see that as well so I'm not just asking five different possibilities.
So we always wrap up by asking (28:54):
what's next? So I want to do more work on active learning,
the ideas from deliberate practice. Does it make a difference if students have good sub skills. And
does it make a difference if you teach by making connections between different topics? There's many
ways active learning are implemented. And it would be cool to do a study where you look at active
(29:17):
learning in many different implementations, but some get those interventions, some do
not, in a crossover study, of course. So that's what I would like to work on next a bit.
That sounds like a very productive area of research that could really
be helpful for all disciplines.Yes, well thank you. That's the idea, I think,
a bit about marketing of papers, you know, who would find it interesting, and so forth.
(29:38):
Well, thank you so much for joining us, and we look forward to your future research.
Thank you so much. It was great fun. It's great talking to you, Bill,
always. I've learned a lot from you over the years, and it's great to still be able to see
your work out there in various places.Great. Okay. Thanks so much, John. It's
great to connect up again. If you've enjoyed this podcast,
(30:02):
please subscribe and leave a review on Apple Podcasts or your favorite podcast service.
To continue the conversation, join us on our Tea for Teaching Facebook page.
You can find show notes, transcripts and other materials on teaforteaching.com.
Music by Michael Gary Brewer.Editing assistance provided by Ryan Metzler.
To understand what a concept is, we must also understand what it is not. In this episode,
we discuss a study of the effect of the use of negative
examples on student learning outcomes. Thanks for joining us for Tea for Teaching,
(00:20):
an informal discussion of innovative and effective practices in teaching and learning.
This podcast series is hosted by John Kane, an economist...
...and Rebecca Mushtare, a graphic designer...
...and features guests doing important research and advocacy work to make higher education more
inclusive and supportive of all learners.Our guest today is Bill Goffe. Bill is a
(00:51):
Teaching Professor in the Economics Department at Penn State. He had previously been one of
our colleagues here at SUNY Oswego. Bill is very well known in the profession for
his Resources for Economists on the Internet, which was one of the very first internet guides
available for economists, and it’s now hosted and sponsored by the American Economic Association.
He has served as a member of the American Economic Association’s Committee on Economic Education,
(01:15):
the Secretary-Treasurer for the Society of Computational Economics,
an Associate Editor for Computational Economics and the online section of the Journal of Economic
Education. Bill is currently co-chair of the Liberal Arts Teaching Group, a faculty learning
community at Penn State. You can also find Bill on many listservs and forums devoted to teaching
(01:37):
and learning. Welcome back, Bill. Thanks, John. Appreciate that.
Today's teas are:... Bill, are you drinking any tea with us today?
I am. I'm drinking Twinings green tea, decaf.Nice, good choice. How about you, John?
I am drinking a peppermint tea from the same vendor.
Nice. I have hot cinnamon spice today because my daughter requested a pot of it. So we're sharing
(02:03):
a pot of hot cinnamon spice this afternoon.Very nice as we approach the holiday season
here when we're recording. So we've invited you here today to discuss your recent working paper
with Austin Boyle entitled “Negative Examples in Lecture Improve Student Learning. First, can you
tell us what you mean by negative examples? Sure. So you're describing a concept to someone,
(02:26):
let's say a young child, and describing what a dog is, and you talk about different types of dogs:
Dalmatians, Pomeranians, whatever. And unless you describe how a cat is not a dog, they're
probably not going to have a great idea of what a dog actually is. And this might be the case,
say, for a second language learner, where you might have different terms for different types
of mammals, also related to something called contrasting cases, where experts are very good
(02:52):
at delineating different types of phenomena or different types of facts, if you will,
say a physician or variation theory used in math. So these are all ways of getting students
a rich understanding of something. Can you talk a little bit about what
prompted your interest in negative examples? Sure. So I've used clickers to teach for a long,
long time. You currently use iClicker Cloud. Love it. And I tried to write a paper on just
(03:17):
in general, using clickers to teach economics, and the paper didn't really work very well.
There's reasons it's never been written, and the reason was I use clickers in many different ways,
so a little bit more focused approach to having a paper. And a few years ago, I came across the
book How to Teach Economics by Yousuf Hamid. He was a high school teacher in Great Britain,
(03:40):
and I had to read that. Well, to be honest, I read it during graduation. You have to go to graduation
every year, but, you know, it always kind of work. And he talked a lot about negative examples. And I
thought, “Oh, that's pretty interesting.” And it also followed work by Kirschner, Sweller,
and Clark on the idea of direct instruction lowercase “d” and “i”, and I thought, “Well,
(04:02):
that'd be an interesting to try, and it's fairly focused,” and that's the genesis of the paper,
and pulled Austin into it as well.We'll include citations to those resources
in the show notes. So, many people, when they're teaching, tend to use primarily
positive examples. What's the problem with just using positive examples of a concept? I think
you've already addressed that a little bit.One example, the original work on this, at least
(04:27):
formal original work… the idea goes back a long, long time… it was by Engelmann and Carnine, a book
on direct instruction. And they give an example of a glem, a G- L-E,-M. Could you define that just
with one example? Well, you could not. And they give an example of a pencil. Well, is it wood, or
is the pencil horizontal, whatever? But it helps you focus in better on an idea when you understand
(04:50):
what it's not. For example, in economics, we talk about capital quite a bit. Unless you tell
students, it's not stock, they probably won't pick up on that. Or you give examples of money,
how money is very specific to economics. To economists, they're probably going to pick up
on it's not some general sense of wealth.And I can affirm that the concept of capital
(05:11):
is one tht students have always had a lot of trouble about because the way we use it in
economics differs from how it's used in everyday use. And just for our listeners, could you explain
the difference between the everyday use of capital and the way we use it in economics?
I'm an everyday listener.Great. We'll see if this works. So,
to an economist, capital is something owned by business, manufactured and used to make things,
(05:37):
so, a commercial mowing machine, a commercial oven, a jet liner,
construction equipment. In everyday usage, it might also include financial capital, stocks,
for example, might be considered capital. I would add one thing economists really haven't done,
the Bureau of Economic Analysis actually redefined capital recently, includes intellectual capital,
(05:58):
and I just don't teach that. It would be confusing, but there's actually about as
much intellectual capital these days as physical capital in the United States. But maybe in an
intermediate course. I would do that.Seems like an easy thing to get confused.
Can you talk a little bit about how you pick which concepts to use negative examples for?
(06:19):
I can imagine that some concepts are more ripe for this strategy than others.
Yes, we wanted something fairly limited, not a long concept of how to manipulate a
graph or something, that could be a further work, but just a definitional sort of thing,
but the students have misconceptions about this, as we described there with capital. And so we
picked three different ones, money, capital, and technology. Technology in everyday use,
(06:44):
our phones, for economists, it's basically efficiency in the production process. You know,
it is rather unfortunate, there's what, 50,000 words in the English language, and we use words
that are used in other ways just to make it more complicated. And I joke to students,
I wish I could use different words, but that would hinder their further education, I fear..
And this has been a problem for economists for well over a century in those very same issues,
(07:08):
and it would be nice if we didn't have to do that, because it introduces that confusion
right when students are just starting to learn the vocabulary of economics, and the way I
talk to them about I say, “Hey, part of this is you're just learning the language of economics,
which may look similar to how we use these words in everyday use, but it's a bit different,
and you need to focus on these things because of the ways in which we're using the term.”
(07:32):
I would add that I've seen some STEM education research papers. They teach concepts first and
jargon second, and they find that actually improves learning because you're trying to
remember the terms and they're using the terms, it overloads your working memory. And there you teach
jargon later, they find improvements. We do that sometimes in design, where we
separate the technical skill from the concept, because learning them all at the same time can
(07:54):
be complicated. I was thinking of a term that I run into all the time where there's confusion,
where this strategy might work well, which is digital accessibility,
where people confuse the idea of meeting these standards to help people with disabilities gain
access to content and material with the idea of “just available,” like it's on the internet.
(08:15):
I learned that distinction just recently myself. I would add on choosing these terms, we found even
after instruction, students got it right between 60 and 71% of the time. So we seem to have picked
a nice set of terms and definitions to use.Which is actually much better than they would
tend to if you had only used positive examples. And we'll talk about those
(08:36):
results a bit in just a minute.Can you tell us a little bit about
your study? How you had it set up? How many students were involved?
Yes, we had about 1200 students. I teach large sections, about three sections every semester,
about 340 students that did this over two semesters, basically just repeated things,
get more power for our tests, and did a so-called crossover study. Crossover study,
(08:58):
have your participants that are both controls and the intervention side of things. We did a 3x3 one,
which is a little bit hard to describe on a podcast, where you can't visualize things,
but there's a fairly famous one for ultra processed diets. You might have heard of by
Hall and company in 2019. They put people into hospitals, they control directly what they ate,
(09:18):
and one group got ultra processed foods for two weeks, the other group got more wholesome foods,
I guess you could say, and then they switched who got the wholesome foods, who got the ultra
processed foods. So thus you could deal with the fact that people might have responded differently,
that different foods have different effects on people. And we did the same thing here in
our study, except we had three concepts and three sections and three teaching methods.
(09:43):
So what were the three teaching methods that you used in the study?
So we just use a positive example, where I would list a half dozen examples of capital or money,
or not quite of money, because there are quite six examples of that. Then with lecture,
we had both positive and negative examples, where these things are money, this is not.
This is capital. This is not. And a third type of teaching method was with a cliicker question,
(10:09):
where I gave a student's list of things. Some are positive, some are negative examples,
and they just choosed which was which.So what did you find?
We found that including negative examples improved learning in the following class by
about 21 percentage points. Let's say the class got 50% correct just with positive examples,
(10:31):
the class that got positive and negative examples got a 71% correct. So it is very
statistically significant, it was robust with lot different statistical methods we used. So
we're pretty pleased with that, it was a clean result. Sample size helps, and I think probably
doing it just with precise definitions, not long drawn out concepts or something like that.
(10:52):
Based on these results, would you suggest that perhaps we should all focus a bit more
on using negative examples as well as positive examples in our teaching?
Yes, most certainly it's an easy intervention. For example, you might know Jim Lang's book Small
Teaching, where he talks about easy interventions you can do, and this one's easy, as long as you
remember to do it. But you're making up your notes, you think of things that aren't whatever
(11:14):
you're talking about. You've probably seen those on exams, overall. Interesting result we had too
that the clicker questions, the active learning, was no different than the lecture. So we'd like
to do further work on that at some point, most active learning studies have been semester long,
active learning not very well defined, and we didn't find any difference here,
(11:35):
so presumably, at some point, active learning make a difference be interesting to find out what that
difference might actually be. That's an interesting result,
and that's probably worth some follow up.I think partly on if you do see similar results
with the flipped class and active learning and such, people don't really have a framework on
what they actually ought to be doing before class and what active learning should look like. I've
(11:56):
got another paper on this, I mentioned, that uses so-called deliberate practice,
That's the 10,000 hour rule. It's really not 10,000 hours, but there's some key parts of
active learning people don't do that much. One is to focus on sub skills, for us in economics,
things like change in demand versus change in quantity demanded. We generally don't teach that
specifically, is my thinking. And then also, you want to make connections between things. How are
(12:19):
supply curves and demand curves separate? How are they similar? Because we remember things in terms
of frameworks or schemas. The negative examples help develop schemas. You understand what things
are not, kind of triangulating, if you will. And good active learning does the same thing,
but if they're just filling in a worksheet with numbers, they plug into the formula, that's
active learning, I suppose, but not much learning there. So we ought to focus more on exactly what
(12:44):
active learning is doing for people.Following your study, what are some
different ways that you've incorporated this practice into your own teaching practice?
So I definitely try to do the negative examples, capturing, oftentimes what I see on these exams
and when I’m thinking of a question, I always try to have it connected to something else.
Oftentimes one concept’s related to another one, and we make that connection explicitly
(13:07):
in clicker questions so that, “Oh, if you remember this, this is a variation of that.” For example,
how is consumer price index similar and different than the GDP deflator would be one thing we do,
and why are they fairly similar? To understand better how those things work. Oftentimes,
I explain things based upon what they've done before. Another thing I do is we have very nice
(13:29):
technology at Penn State, I have a main screen in the classroom. I have a side screen. Side screens,
I put up an outline includes both the outline for the day and our bigger topics. So I'm constantly
referring to that outline saying “what we're doing here is a variation of this, is different
than that,” and so forth. And if I do surveys as students, they find that very useful. A better
(13:49):
study would be to not have the outline there someday and see how learning goes. I just now
thought of that as an idea to do a study on.We know that students come in with a weaker set
of connections than faculty, and sometimes we assume that they're able to make these
connections and see the relationships, but they haven't yet developed that skill,
and sometimes it's generally believed that that type of scaffolding, by putting up an outline of
(14:14):
the main concepts you're trying to get across, can help. But I don't know if I've ever seen any
tests of that, certainly not in economics, but I'm not sure if I've seen it in other areas either.
I have not. I can't say I’ve looked very hard. I would add, to do my literature
searches. Now I oftentimes start with ChatGPT, but that's another whole podcast, I suspect.
(14:34):
Yes, and there's some other good tools out there, too. There's research rabbit and other things that
will actually, even in some cases, will provide more accurate results. But ChatGPT is getting a
lot better all the time for these things.Other than in the test results,
have students responded? Or have you asked students about using the negative
questions and how that impacts their learning?I have not actually, that's a good point. That'd
(14:58):
be another thing to add. It'd be interesting see if they actually can pick up on that. Oftentimes,
they're not that great on understanding their own learning. Something we’ve all seen as instructors,
someone thought they did much better on the exam than they actually did,
which is something you want to avoid. That's a tough place to be in and you have to be
sympathetic as instructor when that occurs.Now, one thing to keep Rebecca happy, we should
(15:18):
also ask a little bit about the methodology that you used. In the study, you used both
pooled data as well as panel data, and you also used ordinary least squares and binary dependent
variable models. So could you just talk a little bit about why you used multiple approaches and how
consistent were the results across those. Yeah, the results are very consistent. So
(15:41):
sometimes you don't quite know what econometric approach you take, you can argue this is better,
that's better. And another approach, and it's pretty common these days, I think it's by far
the most common, is you take multiple approaches and see if the results are robust, to see if you
get similar results. Indeed, we did. So it's a very happy occurrence when that occurs.
So were you surprised at the magnitude and strength of these results? I know when I
(16:04):
read the paper, I was surprised at just how large that effect was, because it's
very rare to see effect sizes that large in a study of alternative teaching approaches.
I don't think I had many preconceptions here. That's odd. I certainly was pleased, but I can’t
say I was quite surprised. It certainly makes me want to take this approach again with where we
do the crossover studies, you can do many things with that. For example, we're told we should teach
(16:28):
about things students find interesting. Well, is that really true? I did an example of that
this semester where Penn State students, they do a large fundraising project called Dance Marathon,
or THON, and last year it raised $17 million, about half the student body is involved. And so I
do a question about real prices, and the question was, did THON actually raise more money when you
(16:48):
correct for inflation, you know, something they're invested in a lot, and informally looking at the
data, that didn't seem to have any impact on that segment, on calculating real prices,
I need to do more work on that, but I hope more people take this methodological approach,
the crossover studies, where you could study a lot of things, the outlines,
is it interesting or not, and so forth?Because with the crossover study, you're
(17:11):
ruling out the possibility that there was just some systematic difference between the sections
or between the other conditions, right?Yes, exactly. We actually found there were
differences between the sections. Some did much better on the questions than others,
but by effectively pooling these different interventions and controls together,
we can adjust for that. We can correct for that. I don't see it that much in economics. John, have
(17:31):
you seen it very much, the crossover studies?I haven't seen very many, but one that stands out,
and I think it was done by economists, was back when the initial studies came out that suggested
that when people took notes by hand, they remembered more than when they used a computer.
I do remember a study, and I'll include a citation to this in the show notes. What they did is they
(17:53):
had all the students use mobile devices for note taking for a portion of the semester,
they broke up the group into two halves, and half of them, for the first half of the semester,
took all the notes using mobile devices, and the other half took notes by hand. And in the second
half of the semester, they switched, and what they found was there was no difference between
(18:16):
how students did when they took notes by hand and how those same students did when they took
notes on the computer. However, those students who regularly chose to take notes on a computer,
on average, did less well regardless of whether they were taking notes by hand or on a computer,
which suggests that the students who choose to take notes by hand may be perhaps a little
(18:40):
bit more focused on the material no matter how they're taking the notes, and those students who
may bring in mobile devices to use, perhaps may do it because they prefer having the availability of
distractions. So it's not quite the same thing, but it's similar in that you're controlling for
the individual effects and separating it out so that you are getting a better comparison,
(19:02):
correcting for any self selection in the people who are involved, which is really, I think, what
crossover studies are trying to remove, the effect of any systematic selection between the groups.
I would add, I've surveyed my students, and generally my students aren't performing quite
as wel do choose to use laptops, and higher performing students do not, so consistent
(19:23):
with your more serious study, for sure.And another thing I've been noticing a lot is that
for quite a while, I've been walking around quite a bit, but I've also noted that the students in
the front third or so of the class tend to be very actively engaged in answering questions, and they
tend to do really well, and they tend to ask more questions. But when I'm in the back of the room,
I see a lot of laptops where there perhaps may be a video playing, there may be some online
(19:48):
shopping going on, or there may be some sort of video game that they're working on on the mobile
device and their laptop or one or the other. And I think we often do see that sort of sorting in
terms of the way in which students choose to work. I'm not sure how I could get rid of that effect,
other than moving them all up to the front.it does seem like at least the multi taskers,
(20:09):
in quotes, are at least removing themselves, and because they're probably distracting to
other people, they're just moving themselves to the back so they're not a distraction
to the people who are trying to focus.Except they’re surrounded by other people,
and there may be a bit of a contagion effect going on back there.
I've seen papers on the contagion effect, yes. There was a Carl Wieman paper. They did look
(20:30):
at seating in large classrooms and found a very definite effect. They even switched seats halfway
through the semester to bring the people in the back up to the front, and they still did worse.
Some of our students are just much more interested in the material and much more engaged than others,
and there's just so much we can do with that.It's tough, but I would be a little bit
sympathetic to students. It's been a long time since I've had to sit through a typical
(20:53):
undergraduate today, and you know, you're three, maybe four courses a day, a lot of sitting. That
would be tough, a challenge. There's a book called My Freshman Year, and there’s a field
anthropologist who went to college for a year to understand her students better,
and she found it was harder than she thought.And she found some dramatic differences, too,
in the way in which students connected with each other from when she had been a student,
(21:16):
that they had much tighter networks with the people they knew before they came to college,
because it's less costly to connect with them over social media and so forth.
Yeah, friendships have changed with the rise of social media.
It used to be costly. I remember having to bring all these quarters to put in pay
phones back when I was in school.You mean, in those 1900s phones
that my daughter refers to, John?Yes, I think that would be the case.
(21:40):
What's a quarter?Yeah. Well, I think we
still know what quarters are. We just don't know what pennies are anymore.
Well, that could be, yes. Are there any other things
you'd like to share about this study?Well, I think that's pretty much it. I think
I would emphasize this little bit more about, again, developing student schemas. But we talked
about that earlier today, this broad network, and it’s through connections, and this helps them make
(22:01):
connections more focused on whatever the topic might be at hand. So I think negative examples
could be used for teaching most anything.Including our colleagues when we're doing
professional development and all kinds of other contexts.
Yes. And it's
easy for us to lose focus, because we already can distinguish these things more easily, and students
(22:22):
have not yet developed that capability.The curse of knowledge really reigns here,
where you know it well. It's hard to understand how other people don't. I was doing a think
aloud exercise the end of last semester where I interviewed students half the semester in how
they'd interpret different things. And I spent a lot of time on aggregate supply, aggregate demand
model in my course, standard staple things course, asked about certain phenomena, no one actually
(22:46):
used the model. I was only interviewing four students, so, I mean, it's not much power there,
but it was kind of crushing, but it gave me insights on how they think about things.
In an earlier podcast with Doug McKee, he talked about having students come in and
just work through the problems verbally, so he could see what sort of process they were using,
(23:08):
and by seeing that, it was easier to help guide them in a more productive direction.
Because if we don't really understand how students are processing things, it's hard
to help them understand things more deeply.Yeah, we use that talk-aloud method and design
a lot too around troubleshooting, because troubleshooting strategies require, especially
(23:30):
around technical skills, you need to think about problems in particular kinds of ways to be able
to solve them correctly, or to be able to solve them at all, and doing that kind of talk aloud
strategy works really well for figuring out where something went awry in the troubleshooting.
And experts think a lot differently than students, I would suspect. You
would never think someone thought that way.And fMRI studies confirm that different areas
(23:52):
of the brain are activated for experts and for novices when trying to solve a problem. So it's
easier when you become an expert, but we have to help students develop the skills to get there.
While we've got you here, could you also tell us a little bit about how you teach classes so
large with so many students, every semester.I use Perusall as a way to have them read ahead.
(24:14):
They come into class, they have some sense of things and then a lot of clicker questions,
typically, as long as it's not something new. For example, I was doing long-run growth. I do that a
fair amount, and the book doesn't really talk much about the institutions and the role of markets
in that. So that was a day or two of me talking. I'm going to be doing the business cycle starting
tomorrow, and that's a fair amount of me talking. But other than that, it's them pretty much asking
(24:37):
questions, or me asking questions. Probably on Thursday, we'll get to calculate employment
rates like read about that in Perusall. I’ll ask them, “Who in this group of people is unemployed?”
Then I'll show them some data, what we compute from that data. So it's a lot of using ideas. I
do a lot of peer instruction where they first do votes on their own, and then I say, “Okay,
there's a lot of uncertainty here. Please convince your neighbors.” And I use an app by iClicker, so
(25:03):
I can see their votes on my phone, and then they do a revote, and they usually improve 20, maybe
30 points, then I provide further instruction on the topic at hand. And there's a couple papers
I've seen. Let's see Michelle Smith at Cornell, a computer science paper. They found that if you
have students revote, that helps them; if you have a further explanation from the instructor
(25:23):
that helps them; if you both, it helps them even more, and it's probably enriching their schemas,
they're seeing several explanations that reinforce the connections between things. Also I do a series
of quizzes, rather than midterms. I do seven quizzes. We're very fortunate at Penn State,
we have a testing center, so I send all my students there for testing. So I run the quizzes
(25:44):
for two days, after the quiz they do a second stage of the quiz. In class they start, then they
answer at home using any resource, and finally I release the answers, and then they reflect on and
describe what they found most difficult on that quiz. So we do a two-week cycle. There's a lot
of feedback, even though the course is quite large, with about 1000 students a semester.
(26:05):
So having that testing center gives you a bit more security in the assessments than many of
us have when we give quizzes online. Many of the assessments I use are done online,
and I used to feel a whole lot better about them before we had ChatGPT and other AI tools appear,
especially with some of the browser extensions, which will directly answer things, so you're able
(26:27):
to avoid some of that, which is really nice.I really feel very fortunate, yes. I would add,
it's a good thing that the AI tools came after Covid, rather than before.
So one of the things that you mentioned when you're describing your class was
Perusall. Can you describe what that is for people that don't know?
Oh, yes. An e-text goes inside perusal, and then students mark it up with comments, questions,
(26:48):
and responses to other people's questions. So it has them thinking bit about the text. It grades
them automatically. They need to start early so there's a little bit of spacing there. They need
to make sure they document across the assignment, and they're also rewarded for opening up multiple
times, not just the evening before the exam. And it's fairly straightforward. Most students get
full credit on each assignment. So it's a bit of encouragement there to read ahead. I used to use
(27:13):
so called JITs for just-in-time teaching assignments where I'd ask some questions,
including what they found confusing, but I like Perusall better. It's less grading work,
and also, I think it covers a little bit more topics or not just looking at the
answers to my JIT questions.Now, do you break them up into
groups when they're using Perusall?Yes, Perusall actually does that. When I
look at the marked up text from 1000 students, the page does not load basically, or has difficulty
(27:38):
loading. And you can also download, I've not done this, you could download all their comments,
and you can run through a chatbot or whatever to get a sense of what they have trouble with,
or what they find confusing. I think textbook authors would find that really,
really useful. You could say your textbook was designed with the input from 1000s of students,
and so your students can read this more carefully. I've not found an author willing to do that, but I
(28:02):
think that would be a big win for someone.It could be, and from a marketing standpoint,
that could be really helpful. When iClicker announced that they had AI tools being built in,
I was really hoping that they would integrate that into the text responses, because right now,
it's just generating questions, which is not bad, and it doesn't do a really bad job of that,
(28:24):
but it would be so nice when you got maybe 1000 or more responses coming in, or a few 1000 responses,
if it could very quickly generate a summary of the responses to free-response questions. That
seems like something that wouldn't be all that hard to do, and it would be a nice extension.
I agree. I think there's some polling technologies that do that. For some
(28:45):
reason iClicker doesn't. I wonder if there's a legal issue or some other constraint. I would
love to see that as well so I'm not just asking five different possibilities.
So we always wrap up by asking (28:54):
what's next? So I want to do more work on active learning,
the ideas from deliberate practice. Does it make a difference if students have good sub skills. And
does it make a difference if you teach by making connections between different topics? There's many
ways active learning are implemented. And it would be cool to do a study where you look at active
(29:17):
learning in many different implementations, but some get those interventions, some do
not, in a crossover study, of course. So that's what I would like to work on next a bit.
That sounds like a very productive area of research that could really
be helpful for all disciplines.Yes, well thank you. That's the idea, I think,
a bit about marketing of papers, you know, who would find it interesting, and so forth.
(29:38):
Well, thank you so much for joining us, and we look forward to your future research.
Thank you so much. It was great fun. It's great talking to you, Bill,
always. I've learned a lot from you over the years, and it's great to still be able to see
your work out there in various places.Great. Okay. Thanks so much, John. It's
great to connect up again. If you've enjoyed this podcast,
(30:02):
please subscribe and leave a review on Apple Podcasts or your favorite podcast service.
To continue the conversation, join us on our Tea for Teaching Facebook page.
You can find show notes, transcripts and other materials on teaforteaching.com.
Music by Michael Gary Brewer.Editing assistance provided by Ryan Metzler.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!