December 16, 2024 • 31 mins
My guests for this episode are Kane Murdoch and Shaun Lehmann. They work in the area of academic integrity at one of Australia's major universities and have been thinking seriously about the problems facing us in academia of ensuring that students behave with integrity in their assessments. The issue is typically referred to as "contract cheating" which includes students outsourcing completion of their work, or *contracts it out, to a third party. This practice risks devaluing university degrees and destroying trust in the quality of degrees.
Kane and Shaun argue that the way to combat this practice in the age of AI is to change the way that we assess students in higher education. Have a listen to their arguments and let us know what you think.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
Welcome to another episode of the Data Revolution podcast.
(00:18):
This time we're headed back to education and a couple of old friends, Kane Murdoch and
Sean Layman who are now at Macquarie, who are very interested in how we assess students
and it'll be really interesting conversation because you might not have thought how data
plays into that but welcome Kane, welcome Sean.
Thanks Kate, lovely to see you again.
(00:38):
Yep, lovely to be here.
It's something that, you know, with COVID, the way we assess people at universities
was kind of thrown into a bit of disarray for a couple of years and since then I think
everybody's trying to work out now with AI how do we change the way we assess people
(01:02):
and I know you've both been thinking a lot about this so let's have a chat about that.
It's yeah, there've been kind of big waves come through over the last decade really.
It's like for those of a certain age they might remember the My Master scandal in 2014-15
(01:25):
where students were paying a mob just down the road in Sydney to do essays for them and
it kind of opened a window into what was happening and that kind of progressed and then COVID
hit which was like another big wave and now we have generative AI and it starts to make
(01:47):
us think about whether our structure is working.
So and that's where we kind of come in because sometimes the main problem is changing the
culture rather than changing any particular part of it and so I think that some of the
cases that we ran from about 2018 onwards kind of opened out different data sets in
(02:12):
ways just frankly higher quality information about what is and isn't happening among students
and as Sean mentions fairly regularly it really kind of starts to give us much better information
when we form our decision making rather than chasing students per se.
(02:33):
We don't have any vested interest in having more cases.
You're really talking about identifying cases of academic misconduct which are typically
a breach of student rules and kind of prosecuting those cases, aren't you?
Yeah, that's pretty much what we're talking about but for a number of years now for like
(02:59):
six or seven years finding them isn't really the problem.
It's perhaps the scale that becomes more problematic and the resources required to be deployed
to respond to that and in a kind of relatively resource light environment like this isn't
(03:22):
something that uni's really want to spend a lot of money on using us as information.
Once you've found out about misconduct it's already happened, the damage is already done
so the thing that you really want to be able to do is divert them and this is why I'm
assuming you've moved to looking at assessment rather than just wait for the damage to be
(03:44):
done because one of the things that I remember Cain telling me many years ago was how not
only did they pay the essay mills to write the essays but then they'd get blackmailed.
There's a whole bunch of risks that students are unaware of and uni's are, uni's do have
(04:06):
an awareness of them but when students are engaging in this kind of behaviour it's constantly
opening out new risks like for example when a student hands off their login credentials
that opens out risks for the uni and their uncontrolled risks so it's risk for the student
(04:26):
but the uni doesn't know what someone might be able to do with that login and MFA isn't
very helpful when someone wants to share that information.
Can you share some of the kinds of activity that you used to see?
I remember having this conversation with you Cain pre-COVID where somebody was logging
(04:47):
in from the same IP address multiple times for different students and the kind of risks
that opens up to the university.
I'll pass over to Sean at this point because the reason that I originally employed Sean
is because he had skills that I didn't particularly around coding and stuff like this and it's
really opened out our ability to see what's happening where I could see parts of it and
(05:15):
I could prove cases but I think the work that Sean's done has really opened the door.
Yeah, so I think for anyone working in the education space a term that people have probably
heard used is learning analytics which is the idea of looking at the data around student
(05:35):
interactions with the learning systems to understand how students learn, where they're
learning, what times they're doing their learning and various other things.
Essentially where our work ends up fitting into that idea is that there is actually a
faulty underlying assumption in learning analytics which is that all of the data associated
(05:57):
with a particular student and what that student is doing in the learning management system
must necessarily be because that student was doing things.
The reason why that's faulty is as Cain's mentioned, commercial contract cheating is
a thing where a student will hand off their username and password to a cheating provider
who logs in and does things for them.
(06:18):
So essentially what I'm trying to argue and what Cain and I are doing with our work is
inverting learning analytics in a way, the idea that we call non-learning analytics which
is where essentially with learning analytics that underlying assumption is false and interestingly
the data itself can tell you where that underlying assumption has been breached because you can
(06:39):
identify where the data might be pointing to a student doing different things which
we've been able to do at a much larger scale through in background wise I was a bioinformatician
in the past I'm pretty handy with R and doing things efficiently with big data sets which
(06:59):
is really what you're doing here.
So it allows us to do things like identify the things you were just talking about Kate
where we might have a single unusual VPN IP address turning up and doing a weekly quiz
for 10 students in a row or that sort of thing.
But as you mentioned before, you know if you're responding to that as an integrity case in
(07:19):
a way the kind of the horse is already bolted and you're in damage control.
If you have access to that kind of data and you can see where your assumption of learning
is being breached what kinds of assessments what kinds of degree programs and so on and
so forth, you can then use that integrity data landscape to make decisions about future
(07:40):
assessment.
You can allow it to inform your views about the validity of particular assessment types
and that's really what we're more interested in.
I tend not to be that interested in the moral failure of an individual student when it comes
to cheating it's not a useful thing in my view.
If they've outsourced an assessment it just means that there is an evidence of learning
(08:04):
and the assessment outcome should be a zero.
It's not a punishment.
It's not a punishment. It's something we need to know to maintain the integrity of the entire
system, isn't it?
Yes.
So it all comes back to assessment and not necessarily moralising and punishing which
is a thing Cain really paved a path with the courageous conversations approach to doing
(08:25):
things which is a much more restorative way of talking to students about these issues.
I think that the approach, the non-judgmental approach is good but how are you working in
with the educators?
How are you able to influence the fact that their assessments might not be very good?
(08:50):
One of the things, again, like taking that kind of non-judgmental approach, like when
we're able to provide visualisations of what's happening in a particular subject and pointing
out that this was all kind of invisible to them to say electorate, that takes away that
(09:13):
sense of, oh, I'm being critiqued or I'm being criticised.
So I think when you can present it that way, people can take on board that information
much more easily rather than say, for example, if they had their boss or they had a colleague
(09:34):
come and say to them, that assessment is crap, stop doing that.
We find it yet.
Oh, no, they love that.
Oh, they're famous for being able to take on criticism.
Yes.
So we're not presenting it as criticism at all.
What we're saying is that there's objectively a problem here and it's related to these choices.
(09:58):
And so, as I said, like driving the culture change piece is that's a part of that, to
be able to have a conversation and just going, well, this is kind of objectively a little
bit bent or broken.
And therefore, here's what we might suggest.
But it still leaves all the agency in their hands in effect.
(10:21):
You know, because after COVID and with the advent of Gen A, my husband, who's a high
school mathematics teacher, he was just like, well, we'll lock all the students in a room
with pen and paper and that's how we'll assess.
And it's fine for a high school because they can do that.
But the scale that we're dealing with education in Australian higher education, it's actually
really challenging to do those old pen and paper and vigilated exams, isn't it?
(10:44):
It makes me think that assessing on a kind of subject basis is like, I'm sure someone
can come up with a model where it works.
But to my mind, for most universities on the planet, it's not really tenable anymore.
Like for the reasons that you say like trying to schedule tens of thousands of exam sittings
(11:07):
when you don't have enough space to do it becomes increasingly problematic.
And it was a problematic at UNSW when we had the entire race course across the road.
So we at UNSW have a race course across the road with lots of space.
You think if you could do it anywhere, you'd be able to do it somewhere where you've got
that kind of facility across the road.
But it was really challenging even for us.
(11:29):
Yeah.
And so when you're thinking about how do you possibly even run this many exams?
I think it had come a cropper pretty quickly.
And so it makes me start to think about summative assessments at a higher level.
So like more like stage gates after a year.
(11:50):
Just explain the concept of summative assessments.
Sure.
So if we think about, okay, Sean engages in martial arts.
And so if we think about someone doing practice and they are working towards a belt like a
(12:11):
black belt, black belt, the test to do it or the demonstration of it is the summative
assessment where the process of getting to that test point is formative assessment.
So they're getting feedback based on what they're doing and what they can improve on.
But that point at which it's decided, yes or no, you reach that level is summative.
(12:37):
And so I tend to think that when we can bring the summative points up a level, that makes
it more manageable, frankly.
And I think you can actually lock those down a bit easier.
It's not necessary that you have to run exams for everything, but rather when you're conducting
(12:58):
less total assessment, managing the assessment, you do run in a fairly secure way or using
different data points like we have to inform the decision making rather than an overworked
academic marking exams at 10 o'clock on a Sunday night, which is really one potentially
(13:21):
fallible data point.
Well, you know, even worse, you've got a huge pile of exam papers and you start
to open a bottle of red.
Well, I didn't want to mention the bottle of red.
But yeah.
The people who started the bottle, maybe, oh, maybe, maybe good luck to those who are
at the bottom of the bottle.
But you know, it's a fallible human process.
(13:43):
And does this really indicate that we actually need to go back and start to think about what
are the nuggets of knowledge that people need to be able to demonstrate?
And that that martial arts example is a really good one because it's fairly well prescribed.
And you know, there are some disciplines where there is a really strong body of knowledge,
(14:06):
like engineering.
You know, you know, you need to understand solid mechanics if you're going to build things.
So there is a solid group of knowledge, things that you need to know and you need to be able
to demonstrate your confidence at.
And Engineers Australia thankfully prescribes those things so that you can be certified
as an engineer.
Not every discipline has those kind of things.
(14:28):
Do we need to start to think about developing those kind of things?
I think so.
And it's just that where they sit might look a bit different.
So I mean, I think going back slightly to talking about data and the integrity data landscape,
I think it's important to not throw the baby out with the bathwater when it comes to certain
(14:50):
kinds of assessment.
When you start looking at the landscape of integrity data, you realise quite quickly
that things like commercial contract cheating, for example, are not distributed evenly across
your degree programmes.
So there are some degree programmes where it's more common than others.
And if you actually have the data to tell you which ones, then it means that things that
(15:14):
are efficient online forms of assessment that you may actually be safer to use in some areas
than others.
So you can keep efficiencies where it's safe to do so.
And if you have the data to tell you where that is.
The other thing there, going back to what that could look like, that that higher level
(15:34):
summative stage gated assessment in disciplines that don't have that core bolus of knowledge
like engineering or in my case, my first degree was in medical science.
So it was learning a lot of anatomy and physiology and things which doesn't change dramatically.
If I think about the other parts of what I've studied, I also have an anthropology degree.
(15:55):
The way you could do things there would just look a bit different.
So what if you had a large, important, sort of not quite a black belt exam, let's say
it's at the end of the first year, so it's more of an orange belt exam.
But what you could be doing there is giving students a quite complex problem, a multifaceted,
(16:19):
complicated, real-worldy sort of issue, and ask them, have a think about everything you've
learned over the course of this year that might be across different things.
You might have learned some sociology, you might have learned some anthropology, you
might have learned some politics, you might have done some first year philosophy or critical
reasoning or propositional logic or whatever.
(16:42):
And then from there, they can look at this complex real-world problem and say, well,
this piece here and this piece here and this piece here, I can see how these things I've
learned over the year.
I can abstract out the concepts from here and apply them to this problem.
Because I think really what people want out of graduates is that ability to take a solution
(17:05):
to an issue, identify its principles and then apply them to a different set of issues.
That's an assessment that you've got, Chris, to do that.
Sorry.
Will you just ask AI to do that for us now?
It depends how you're doing it.
So I'm a pretty big advocate for things like interactive oral assessment, for example.
(17:28):
And as soon as you mention that, you inevitably get a reply from people saying, I'll bet it's
too costly to do it.
If you're doing it in a more programmatic way, where you're doing it, say, at the end
of first year, you can then actually pull the marking labor across multiple subjects
because it's not necessarily being assessed in the same way at a granular level at every
(17:48):
single subject.
The other thing, too, is that some of the reasoning underlying that whole concept of
interactive oral assessment being expensive is itself faulty.
A, because where people have done it, it actually wasn't that expensive.
So most of the time when people are objecting to it, they haven't tried.
And second, you actually need to factor in the cost of responding to breaches of integrity
(18:13):
in poorly secured assessment, too.
So if you do a sort of traditional take home online exam, on the best research that's published,
you should probably expect 10% to 15% of students to not actually do it, to cheat it
in some way.
You've then got a factor.
So let's say that's a class of 400 students.
(18:34):
So you're talking about 40 to 50 students there.
What is the actual cost to the institution of carrying out integrity cases against those
40 to 50 students?
You find that that stacks up to be tens of thousands of dollars very quickly.
And when you're thinking about that, it actually would have been cheaper to go ahead and leave
a vote, say all of those students to begin with.
(18:55):
I think the real challenge here is that the teachers are used to doing things a certain
way.
I can remember a time when I was at UNSW and a very senior academic walked into my office
and said, I want to wait for you to electronically force the students to come to my class.
I was like, sit down and have a chat, mate.
(19:18):
Let's find out what's underneath this.
And it was because his students weren't coming to class.
And I was like, what are you doing in the class?
He's like, I stand in front of my PowerPoints and I talk about them.
I was like, maybe you need to do something different.
And he said, I don't want to.
So that was the fundamental problem that really, really what he was doing was not meeting their
needs and he didn't want to do anything different.
(19:41):
Yeah.
Kind of swings back to that.
It's something you mentioned about going back to some base principles and taking a look
at what we're doing and rather than trying to, the analogy that I use is like a rickety
house where we've kind of kept adding on and slapping paint and adding wings and taking
(20:06):
out a window here.
And I think we need to kind of reset the foundations a bit.
And that's pretty difficult in a kind of current moving dynamic system.
But I think that's what we really ought to be thinking about as a sector about going,
okay, so how do we touch academic integrity issues?
(20:29):
How do we touch assessment redesign?
How do we touch academic culture?
And so like that fellow, there's plenty of them out there.
They've learned it the way they've learned it and it was never anything but correct.
And you're also talking to people who are very used to being expert in something and
(20:49):
then pointing out that the world's changed and they are no longer expert in that thing.
So I mean lecturing.
I can think of really interesting ways that you could use AI to help with that kind of
assessment that Sean was just outlining.
So you don't have to have a human being sit there and actually assess that.
(21:11):
You could actually use AI.
With AI, I tend to think performative stuff, it's good.
I think we want human decision makers to be making those decisions about whether a student
has learned or not.
Oh, no, no, no.
But just assessing, you know, what Sean was saying of like having people do performative
(21:35):
things that they can demonstrate that they're understanding.
But you can get AI to do that.
I did a proof of concept with mainly Min Kim Eng at UNSW along those lines and it was quite
interesting results.
Yeah, it'd be well worth looking at.
Yeah, it'd be interesting.
I mean, I think the key, I mean, ultimately, I think the fundamental question that universities
(21:59):
need to be able to answer at this point in time is that they need to be assuring that
there is a difference between a university graduate and somebody who just has access
to generative AI.
If there is no difference between those two things, then we should just take the shingle
down and stop wasting everyone's time.
So I think one of the ways to assure that is conversations.
(22:26):
So conversations with students where you're talking to them, it's not just a presentation,
it's an interactive conversation.
So you can say to them, oh, you raised idea X, that's a really interesting idea.
How did that come into your mind?
How does this relate to so-and-so?
How does this relate to something else?
So it's a live conversation.
Where that gets really interesting, I think, particularly when you're thinking about traditional
(22:49):
written tasks that are sort of the domain of generative AI, people say it's cheated.
I'm kind of not that interested in calling the use of generative AI cheating at this
point.
It seems a bit sysophian.
But let's say, for example, instead of you giving an assignment to a student, which is
(23:11):
ostensibly writing a report, but instead of marking that report by sitting in a room and
reading the report and marking what's written on the paper with no idea whether any of those
ideas actually came from the student, instead, you sit down with the student and have a conversation
with them about the report.
That's the marking.
And that is, in effect, the assessment.
(23:32):
Because ultimately, if the student didn't produce that document and they use generative
AI to do it, but they can speak competently and interactively to every single concept
that's in that document and explain...
Yeah, they know it.
So...
Yeah, they know it.
Then who cares whether they wrote it with their fingers or generative AI?
And I think that's where the learning conversations sort of need to go.
(23:57):
There are a lot of academics...
That's a very big shift.
It is a very big shift, yeah.
But I think it's a necessary one.
So how are these conversations...
I know you guys are cooking to a national and global network of people who are working
on the same stuff.
How are they thinking about this internationally?
(24:20):
Is this the kind of things that people are talking about?
I think everyone's facing very, very similar issues.
And I think part of the context for all this happening now is the financial issues that
Unis are having.
So it's kind of trying to sail the boat while the boat is halfway underwater.
(24:45):
So there's definitely interesting conversations happening, but without any patriotism, Australia
seems to lead the world in these discussions.
All right.
I think it's genuinely because we looked at the problems we were facing with integrity
(25:06):
quite a bit earlier than anyone else has.
And so we just got to a level of advancement or sophistication in the conversation before
anyone else did.
But from the perspective that other people like Ireland is an example for me, they've
really come on board and they're starting to lead in their own ways, which I'm enormously
(25:30):
impressed by.
But yeah, like I think anyone who thinks we can keep doing it exactly the same way without
any blowback or consequence is misinformed.
Do you think there's anything that can be done at the end of educating kids about this before
(25:53):
they hit uni, university, you know, like can the parents talk to the kids?
What should the parents be saying to kids?
There's a lot of parents that listen to the show.
I think that schools train students to the test.
And it's not that they're doing anything untoward, but it's rather it's talking about
incentives and so like the incentive is to get good marks or the incentive is to pass.
(26:19):
The incentive is to get to uni rather than the incentive being to learn.
And when students...
Netplanes the cause of all of our problems.
I wouldn't go that far, but yeah, like it's a problematic feature of the education landscape.
And I have this belief that young people are just natural learners.
(26:44):
Like a lot of them just want to learn whether it's learning about, you know, riding a skateboard
or doing karate or whatever.
They just kind of information sponges.
And I think we kind of squeeze it out of them a bit.
So for parents, what I would say is, talk to your kids about what they've learned.
(27:04):
You know, you should be concerned if they can't speak with any kind of lucidity about
the things that they've learned.
But more, I would say that talking to their teachers and getting a sense of them as people
rather than just students, we're producing like a factory.
Because I think they come out of high school and not enough changes at uni.
(27:30):
So they're still so mark incentivised.
And I think it's driving a whole bunch of anxiety, like literal anxiety was in our space.
And I'm also a chair of the appeals panel for the university.
We see a heck of a lot of clinical anxiety and depression.
And it seems to me this is not a very happy or healthy generation of young people.
(27:52):
And part of that is to do with the stresses around marks and achievement and lacking confidence
in those things.
Yeah.
Yeah.
And Sean, just a question about what higher institutions can do to start to build this
kind of data landscape that can help facilitate these kind of conversations?
Yeah, sure.
(28:14):
So I think there's a couple of things that can be done.
I mean, I think that the first thing every institution should really do is appoint as
an actual position in a reasonably well-paid position, an integrity data scientist, someone
like that.
If that should be somebody whose job it is to look at the learning data, that can be
(28:38):
document metadata, that can be learning management system interaction data.
It can be MFA data.
It can be all kinds of things to understand the patterns of student interaction and be
able to identify where the data is telling you that your presumption of learning is being
undermined and then use that data.
(28:58):
So if it's very apparent to you that weekly online quizzes are very frequently outsourced
one way or another, then that's a good signal that they're probably not a good thing to
use as a summative that is graded form of assessment.
That sounds like a really obvious thing, doesn't it?
It does.
I think the other thing that this data helps you understand too that's very useful is there
(29:23):
has been a bit of a panic with generative AI and a lot of people thinking that there
is a new kind of assessment validity crisis.
If you've been working in the area as long as Kane and I have and looking at this data
as long as we have, the assessment validity crisis predates generative AI.
(29:44):
There was already a validity crisis through commercial outsourcing of assessment.
Generative AI as far as I see is not actually presenting any new questions.
It's just further highlighting an existing validity issue with assessment.
So the data can tell you a lot about where the validity issues are and help you identify
(30:05):
what you can do in response to them and help you identify as well how that can be done
cost effectively.
And a key part of that again is if you're actually good at identifying where assessments
have been outsourced, for example, your institution then has to respond to when that has happened
which gives you some real numbers about what it costs you financially as an institution
(30:30):
to have to be chasing bolted horses when it comes to integrity breaches.
It should become apparent quite quickly that it is literally cheaper to engage in quite
radical assessment change that just involves an awful lot more conversations with students.
It's not nearly as expensive as most people think especially when you have a proper picture
(30:52):
of the cost of what we're doing now.
That sounds like a very good place to wrap up.
Thank you both for joining me.
It was really good to chat with you and nice to see you.
Lovely to see you again, Kate.
Thank you for having us.
And that is it for another episode of the Data Revolution podcast.
I'm Kate Crothers.
Thank you so much for listening.
(31:13):
Please don't forget to give the show a nice review and a like on your podcast app of choice.
See you next time.
Welcome to another episode of the Data Revolution podcast.
(00:18):
This time we're headed back to education and a couple of old friends, Kane Murdoch and
Sean Layman who are now at Macquarie, who are very interested in how we assess students
and it'll be really interesting conversation because you might not have thought how data
plays into that but welcome Kane, welcome Sean.
Thanks Kate, lovely to see you again.
(00:38):
Yep, lovely to be here.
It's something that, you know, with COVID, the way we assess people at universities
was kind of thrown into a bit of disarray for a couple of years and since then I think
everybody's trying to work out now with AI how do we change the way we assess people
(01:02):
and I know you've both been thinking a lot about this so let's have a chat about that.
It's yeah, there've been kind of big waves come through over the last decade really.
It's like for those of a certain age they might remember the My Master scandal in 2014-15
(01:25):
where students were paying a mob just down the road in Sydney to do essays for them and
it kind of opened a window into what was happening and that kind of progressed and then COVID
hit which was like another big wave and now we have generative AI and it starts to make
(01:47):
us think about whether our structure is working.
So and that's where we kind of come in because sometimes the main problem is changing the
culture rather than changing any particular part of it and so I think that some of the
cases that we ran from about 2018 onwards kind of opened out different data sets in
(02:12):
ways just frankly higher quality information about what is and isn't happening among students
and as Sean mentions fairly regularly it really kind of starts to give us much better information
when we form our decision making rather than chasing students per se.
(02:33):
We don't have any vested interest in having more cases.
You're really talking about identifying cases of academic misconduct which are typically
a breach of student rules and kind of prosecuting those cases, aren't you?
Yeah, that's pretty much what we're talking about but for a number of years now for like
(02:59):
six or seven years finding them isn't really the problem.
It's perhaps the scale that becomes more problematic and the resources required to be deployed
to respond to that and in a kind of relatively resource light environment like this isn't
(03:22):
something that uni's really want to spend a lot of money on using us as information.
Once you've found out about misconduct it's already happened, the damage is already done
so the thing that you really want to be able to do is divert them and this is why I'm
assuming you've moved to looking at assessment rather than just wait for the damage to be
(03:44):
done because one of the things that I remember Cain telling me many years ago was how not
only did they pay the essay mills to write the essays but then they'd get blackmailed.
There's a whole bunch of risks that students are unaware of and uni's are, uni's do have
(04:06):
an awareness of them but when students are engaging in this kind of behaviour it's constantly
opening out new risks like for example when a student hands off their login credentials
that opens out risks for the uni and their uncontrolled risks so it's risk for the student
(04:26):
but the uni doesn't know what someone might be able to do with that login and MFA isn't
very helpful when someone wants to share that information.
Can you share some of the kinds of activity that you used to see?
I remember having this conversation with you Cain pre-COVID where somebody was logging
(04:47):
in from the same IP address multiple times for different students and the kind of risks
that opens up to the university.
I'll pass over to Sean at this point because the reason that I originally employed Sean
is because he had skills that I didn't particularly around coding and stuff like this and it's
really opened out our ability to see what's happening where I could see parts of it and
(05:15):
I could prove cases but I think the work that Sean's done has really opened the door.
Yeah, so I think for anyone working in the education space a term that people have probably
heard used is learning analytics which is the idea of looking at the data around student
(05:35):
interactions with the learning systems to understand how students learn, where they're
learning, what times they're doing their learning and various other things.
Essentially where our work ends up fitting into that idea is that there is actually a
faulty underlying assumption in learning analytics which is that all of the data associated
(05:57):
with a particular student and what that student is doing in the learning management system
must necessarily be because that student was doing things.
The reason why that's faulty is as Cain's mentioned, commercial contract cheating is
a thing where a student will hand off their username and password to a cheating provider
who logs in and does things for them.
(06:18):
So essentially what I'm trying to argue and what Cain and I are doing with our work is
inverting learning analytics in a way, the idea that we call non-learning analytics which
is where essentially with learning analytics that underlying assumption is false and interestingly
the data itself can tell you where that underlying assumption has been breached because you can
(06:39):
identify where the data might be pointing to a student doing different things which
we've been able to do at a much larger scale through in background wise I was a bioinformatician
in the past I'm pretty handy with R and doing things efficiently with big data sets which
(06:59):
is really what you're doing here.
So it allows us to do things like identify the things you were just talking about Kate
where we might have a single unusual VPN IP address turning up and doing a weekly quiz
for 10 students in a row or that sort of thing.
But as you mentioned before, you know if you're responding to that as an integrity case in
(07:19):
a way the kind of the horse is already bolted and you're in damage control.
If you have access to that kind of data and you can see where your assumption of learning
is being breached what kinds of assessments what kinds of degree programs and so on and
so forth, you can then use that integrity data landscape to make decisions about future
(07:40):
assessment.
You can allow it to inform your views about the validity of particular assessment types
and that's really what we're more interested in.
I tend not to be that interested in the moral failure of an individual student when it comes
to cheating it's not a useful thing in my view.
If they've outsourced an assessment it just means that there is an evidence of learning
(08:04):
and the assessment outcome should be a zero.
It's not a punishment.
It's not a punishment. It's something we need to know to maintain the integrity of the entire
system, isn't it?
Yes.
So it all comes back to assessment and not necessarily moralising and punishing which
is a thing Cain really paved a path with the courageous conversations approach to doing
(08:25):
things which is a much more restorative way of talking to students about these issues.
I think that the approach, the non-judgmental approach is good but how are you working in
with the educators?
How are you able to influence the fact that their assessments might not be very good?
(08:50):
One of the things, again, like taking that kind of non-judgmental approach, like when
we're able to provide visualisations of what's happening in a particular subject and pointing
out that this was all kind of invisible to them to say electorate, that takes away that
(09:13):
sense of, oh, I'm being critiqued or I'm being criticised.
So I think when you can present it that way, people can take on board that information
much more easily rather than say, for example, if they had their boss or they had a colleague
(09:34):
come and say to them, that assessment is crap, stop doing that.
We find it yet.
Oh, no, they love that.
Oh, they're famous for being able to take on criticism.
Yes.
So we're not presenting it as criticism at all.
What we're saying is that there's objectively a problem here and it's related to these choices.
(09:58):
And so, as I said, like driving the culture change piece is that's a part of that, to
be able to have a conversation and just going, well, this is kind of objectively a little
bit bent or broken.
And therefore, here's what we might suggest.
But it still leaves all the agency in their hands in effect.
(10:21):
You know, because after COVID and with the advent of Gen A, my husband, who's a high
school mathematics teacher, he was just like, well, we'll lock all the students in a room
with pen and paper and that's how we'll assess.
And it's fine for a high school because they can do that.
But the scale that we're dealing with education in Australian higher education, it's actually
really challenging to do those old pen and paper and vigilated exams, isn't it?
(10:44):
It makes me think that assessing on a kind of subject basis is like, I'm sure someone
can come up with a model where it works.
But to my mind, for most universities on the planet, it's not really tenable anymore.
Like for the reasons that you say like trying to schedule tens of thousands of exam sittings
(11:07):
when you don't have enough space to do it becomes increasingly problematic.
And it was a problematic at UNSW when we had the entire race course across the road.
So we at UNSW have a race course across the road with lots of space.
You think if you could do it anywhere, you'd be able to do it somewhere where you've got
that kind of facility across the road.
But it was really challenging even for us.
(11:29):
Yeah.
And so when you're thinking about how do you possibly even run this many exams?
I think it had come a cropper pretty quickly.
And so it makes me start to think about summative assessments at a higher level.
So like more like stage gates after a year.
(11:50):
Just explain the concept of summative assessments.
Sure.
So if we think about, okay, Sean engages in martial arts.
And so if we think about someone doing practice and they are working towards a belt like a
(12:11):
black belt, black belt, the test to do it or the demonstration of it is the summative
assessment where the process of getting to that test point is formative assessment.
So they're getting feedback based on what they're doing and what they can improve on.
But that point at which it's decided, yes or no, you reach that level is summative.
(12:37):
And so I tend to think that when we can bring the summative points up a level, that makes
it more manageable, frankly.
And I think you can actually lock those down a bit easier.
It's not necessary that you have to run exams for everything, but rather when you're conducting
(12:58):
less total assessment, managing the assessment, you do run in a fairly secure way or using
different data points like we have to inform the decision making rather than an overworked
academic marking exams at 10 o'clock on a Sunday night, which is really one potentially
(13:21):
fallible data point.
Well, you know, even worse, you've got a huge pile of exam papers and you start
to open a bottle of red.
Well, I didn't want to mention the bottle of red.
But yeah.
The people who started the bottle, maybe, oh, maybe, maybe good luck to those who are
at the bottom of the bottle.
But you know, it's a fallible human process.
(13:43):
And does this really indicate that we actually need to go back and start to think about what
are the nuggets of knowledge that people need to be able to demonstrate?
And that that martial arts example is a really good one because it's fairly well prescribed.
And you know, there are some disciplines where there is a really strong body of knowledge,
(14:06):
like engineering.
You know, you know, you need to understand solid mechanics if you're going to build things.
So there is a solid group of knowledge, things that you need to know and you need to be able
to demonstrate your confidence at.
And Engineers Australia thankfully prescribes those things so that you can be certified
as an engineer.
Not every discipline has those kind of things.
(14:28):
Do we need to start to think about developing those kind of things?
I think so.
And it's just that where they sit might look a bit different.
So I mean, I think going back slightly to talking about data and the integrity data landscape,
I think it's important to not throw the baby out with the bathwater when it comes to certain
(14:50):
kinds of assessment.
When you start looking at the landscape of integrity data, you realise quite quickly
that things like commercial contract cheating, for example, are not distributed evenly across
your degree programmes.
So there are some degree programmes where it's more common than others.
And if you actually have the data to tell you which ones, then it means that things that
(15:14):
are efficient online forms of assessment that you may actually be safer to use in some areas
than others.
So you can keep efficiencies where it's safe to do so.
And if you have the data to tell you where that is.
The other thing there, going back to what that could look like, that that higher level
(15:34):
summative stage gated assessment in disciplines that don't have that core bolus of knowledge
like engineering or in my case, my first degree was in medical science.
So it was learning a lot of anatomy and physiology and things which doesn't change dramatically.
If I think about the other parts of what I've studied, I also have an anthropology degree.
(15:55):
The way you could do things there would just look a bit different.
So what if you had a large, important, sort of not quite a black belt exam, let's say
it's at the end of the first year, so it's more of an orange belt exam.
But what you could be doing there is giving students a quite complex problem, a multifaceted,
(16:19):
complicated, real-worldy sort of issue, and ask them, have a think about everything you've
learned over the course of this year that might be across different things.
You might have learned some sociology, you might have learned some anthropology, you
might have learned some politics, you might have done some first year philosophy or critical
reasoning or propositional logic or whatever.
(16:42):
And then from there, they can look at this complex real-world problem and say, well,
this piece here and this piece here and this piece here, I can see how these things I've
learned over the year.
I can abstract out the concepts from here and apply them to this problem.
Because I think really what people want out of graduates is that ability to take a solution
(17:05):
to an issue, identify its principles and then apply them to a different set of issues.
That's an assessment that you've got, Chris, to do that.
Sorry.
Will you just ask AI to do that for us now?
It depends how you're doing it.
So I'm a pretty big advocate for things like interactive oral assessment, for example.
(17:28):
And as soon as you mention that, you inevitably get a reply from people saying, I'll bet it's
too costly to do it.
If you're doing it in a more programmatic way, where you're doing it, say, at the end
of first year, you can then actually pull the marking labor across multiple subjects
because it's not necessarily being assessed in the same way at a granular level at every
(17:48):
single subject.
The other thing, too, is that some of the reasoning underlying that whole concept of
interactive oral assessment being expensive is itself faulty.
A, because where people have done it, it actually wasn't that expensive.
So most of the time when people are objecting to it, they haven't tried.
And second, you actually need to factor in the cost of responding to breaches of integrity
(18:13):
in poorly secured assessment, too.
So if you do a sort of traditional take home online exam, on the best research that's published,
you should probably expect 10% to 15% of students to not actually do it, to cheat it
in some way.
You've then got a factor.
So let's say that's a class of 400 students.
(18:34):
So you're talking about 40 to 50 students there.
What is the actual cost to the institution of carrying out integrity cases against those
40 to 50 students?
You find that that stacks up to be tens of thousands of dollars very quickly.
And when you're thinking about that, it actually would have been cheaper to go ahead and leave
a vote, say all of those students to begin with.
(18:55):
I think the real challenge here is that the teachers are used to doing things a certain
way.
I can remember a time when I was at UNSW and a very senior academic walked into my office
and said, I want to wait for you to electronically force the students to come to my class.
I was like, sit down and have a chat, mate.
(19:18):
Let's find out what's underneath this.
And it was because his students weren't coming to class.
And I was like, what are you doing in the class?
He's like, I stand in front of my PowerPoints and I talk about them.
I was like, maybe you need to do something different.
And he said, I don't want to.
So that was the fundamental problem that really, really what he was doing was not meeting their
needs and he didn't want to do anything different.
(19:41):
Yeah.
Kind of swings back to that.
It's something you mentioned about going back to some base principles and taking a look
at what we're doing and rather than trying to, the analogy that I use is like a rickety
house where we've kind of kept adding on and slapping paint and adding wings and taking
(20:06):
out a window here.
And I think we need to kind of reset the foundations a bit.
And that's pretty difficult in a kind of current moving dynamic system.
But I think that's what we really ought to be thinking about as a sector about going,
okay, so how do we touch academic integrity issues?
(20:29):
How do we touch assessment redesign?
How do we touch academic culture?
And so like that fellow, there's plenty of them out there.
They've learned it the way they've learned it and it was never anything but correct.
And you're also talking to people who are very used to being expert in something and
(20:49):
then pointing out that the world's changed and they are no longer expert in that thing.
So I mean lecturing.
I can think of really interesting ways that you could use AI to help with that kind of
assessment that Sean was just outlining.
So you don't have to have a human being sit there and actually assess that.
(21:11):
You could actually use AI.
With AI, I tend to think performative stuff, it's good.
I think we want human decision makers to be making those decisions about whether a student
has learned or not.
Oh, no, no, no.
But just assessing, you know, what Sean was saying of like having people do performative
(21:35):
things that they can demonstrate that they're understanding.
But you can get AI to do that.
I did a proof of concept with mainly Min Kim Eng at UNSW along those lines and it was quite
interesting results.
Yeah, it'd be well worth looking at.
Yeah, it'd be interesting.
I mean, I think the key, I mean, ultimately, I think the fundamental question that universities
(21:59):
need to be able to answer at this point in time is that they need to be assuring that
there is a difference between a university graduate and somebody who just has access
to generative AI.
If there is no difference between those two things, then we should just take the shingle
down and stop wasting everyone's time.
So I think one of the ways to assure that is conversations.
(22:26):
So conversations with students where you're talking to them, it's not just a presentation,
it's an interactive conversation.
So you can say to them, oh, you raised idea X, that's a really interesting idea.
How did that come into your mind?
How does this relate to so-and-so?
How does this relate to something else?
So it's a live conversation.
Where that gets really interesting, I think, particularly when you're thinking about traditional
(22:49):
written tasks that are sort of the domain of generative AI, people say it's cheated.
I'm kind of not that interested in calling the use of generative AI cheating at this
point.
It seems a bit sysophian.
But let's say, for example, instead of you giving an assignment to a student, which is
(23:11):
ostensibly writing a report, but instead of marking that report by sitting in a room and
reading the report and marking what's written on the paper with no idea whether any of those
ideas actually came from the student, instead, you sit down with the student and have a conversation
with them about the report.
That's the marking.
And that is, in effect, the assessment.
(23:32):
Because ultimately, if the student didn't produce that document and they use generative
AI to do it, but they can speak competently and interactively to every single concept
that's in that document and explain...
Yeah, they know it.
So...
Yeah, they know it.
Then who cares whether they wrote it with their fingers or generative AI?
And I think that's where the learning conversations sort of need to go.
(23:57):
There are a lot of academics...
That's a very big shift.
It is a very big shift, yeah.
But I think it's a necessary one.
So how are these conversations...
I know you guys are cooking to a national and global network of people who are working
on the same stuff.
How are they thinking about this internationally?
(24:20):
Is this the kind of things that people are talking about?
I think everyone's facing very, very similar issues.
And I think part of the context for all this happening now is the financial issues that
Unis are having.
So it's kind of trying to sail the boat while the boat is halfway underwater.
(24:45):
So there's definitely interesting conversations happening, but without any patriotism, Australia
seems to lead the world in these discussions.
All right.
I think it's genuinely because we looked at the problems we were facing with integrity
(25:06):
quite a bit earlier than anyone else has.
And so we just got to a level of advancement or sophistication in the conversation before
anyone else did.
But from the perspective that other people like Ireland is an example for me, they've
really come on board and they're starting to lead in their own ways, which I'm enormously
(25:30):
impressed by.
But yeah, like I think anyone who thinks we can keep doing it exactly the same way without
any blowback or consequence is misinformed.
Do you think there's anything that can be done at the end of educating kids about this before
(25:53):
they hit uni, university, you know, like can the parents talk to the kids?
What should the parents be saying to kids?
There's a lot of parents that listen to the show.
I think that schools train students to the test.
And it's not that they're doing anything untoward, but it's rather it's talking about
incentives and so like the incentive is to get good marks or the incentive is to pass.
(26:19):
The incentive is to get to uni rather than the incentive being to learn.
And when students...
Netplanes the cause of all of our problems.
I wouldn't go that far, but yeah, like it's a problematic feature of the education landscape.
And I have this belief that young people are just natural learners.
(26:44):
Like a lot of them just want to learn whether it's learning about, you know, riding a skateboard
or doing karate or whatever.
They just kind of information sponges.
And I think we kind of squeeze it out of them a bit.
So for parents, what I would say is, talk to your kids about what they've learned.
(27:04):
You know, you should be concerned if they can't speak with any kind of lucidity about
the things that they've learned.
But more, I would say that talking to their teachers and getting a sense of them as people
rather than just students, we're producing like a factory.
Because I think they come out of high school and not enough changes at uni.
(27:30):
So they're still so mark incentivised.
And I think it's driving a whole bunch of anxiety, like literal anxiety was in our space.
And I'm also a chair of the appeals panel for the university.
We see a heck of a lot of clinical anxiety and depression.
And it seems to me this is not a very happy or healthy generation of young people.
(27:52):
And part of that is to do with the stresses around marks and achievement and lacking confidence
in those things.
Yeah.
Yeah.
And Sean, just a question about what higher institutions can do to start to build this
kind of data landscape that can help facilitate these kind of conversations?
Yeah, sure.
(28:14):
So I think there's a couple of things that can be done.
I mean, I think that the first thing every institution should really do is appoint as
an actual position in a reasonably well-paid position, an integrity data scientist, someone
like that.
If that should be somebody whose job it is to look at the learning data, that can be
(28:38):
document metadata, that can be learning management system interaction data.
It can be MFA data.
It can be all kinds of things to understand the patterns of student interaction and be
able to identify where the data is telling you that your presumption of learning is being
undermined and then use that data.
(28:58):
So if it's very apparent to you that weekly online quizzes are very frequently outsourced
one way or another, then that's a good signal that they're probably not a good thing to
use as a summative that is graded form of assessment.
That sounds like a really obvious thing, doesn't it?
It does.
I think the other thing that this data helps you understand too that's very useful is there
(29:23):
has been a bit of a panic with generative AI and a lot of people thinking that there
is a new kind of assessment validity crisis.
If you've been working in the area as long as Kane and I have and looking at this data
as long as we have, the assessment validity crisis predates generative AI.
(29:44):
There was already a validity crisis through commercial outsourcing of assessment.
Generative AI as far as I see is not actually presenting any new questions.
It's just further highlighting an existing validity issue with assessment.
So the data can tell you a lot about where the validity issues are and help you identify
(30:05):
what you can do in response to them and help you identify as well how that can be done
cost effectively.
And a key part of that again is if you're actually good at identifying where assessments
have been outsourced, for example, your institution then has to respond to when that has happened
which gives you some real numbers about what it costs you financially as an institution
(30:30):
to have to be chasing bolted horses when it comes to integrity breaches.
It should become apparent quite quickly that it is literally cheaper to engage in quite
radical assessment change that just involves an awful lot more conversations with students.
It's not nearly as expensive as most people think especially when you have a proper picture
(30:52):
of the cost of what we're doing now.
That sounds like a very good place to wrap up.
Thank you both for joining me.
It was really good to chat with you and nice to see you.
Lovely to see you again, Kate.
Thank you for having us.
And that is it for another episode of the Data Revolution podcast.
I'm Kate Crothers.
Thank you so much for listening.
(31:13):
Please don't forget to give the show a nice review and a like on your podcast app of choice.
See you next time.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.