Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Brian (00:06):
Hello and welcome to the
gaming with science podcast
where we talk about the sciencebehind some of your favorite
games. In this bonus episode,we're going to discuss publish
or perish by Dr Max Bai
Jason (00:20):
All right, everyone,
welcome back. We're well, Brian
says that this one is actuallyscience with gaming instead of
gaming with science as we'redoing a game about the
scientific process itselfinstead of about a science
topic. This is Jason. This isBrian. And welcome technically,
we're in our between seasonbreak right now, but we like
giving y'all bonus episodes, andhonestly, this was something we
couldn't pass up. So we're justgoing to dive right into it.
(00:41):
Publish or Perish, is a gamethat is just out by Dr Max Bai.
So he is an independent socialpsychologist. So he got his PhD
in social psychology, did apostdoc at Stanford, and now
runs an independent researchlab, which I don't know exactly
how that works. He says on hisKickstarter page he started a
few companies, so I assume theyprovide him enough income he can
just do his own research what hewants. And maybe social research
(01:02):
is less expensive than biologyresearch. I don't know. I do not
have enough money to be anindependent researcher and run
my own lab.
Brian (01:07):
Well I mean, research can
vary widely in how expensive or
inexpensive it is, depending onhow you're doing it.
Jason (01:12):
but whatever the case is,
he's running an independent
Yeah it's kind of one of theunfortunate realities that, like
research lab, which means hedoesn't have any of the
administrative overhead and allthe deans and stuff that we
spend all our time complainingabout, and that's probably a
preview for how this episode isgoing to go, because this
episode is about the scientificprocess itself, not really about
any specific scientificdiscipline. And so you're going
to see maybe a peek behind thecurtains, if you don't know it
(01:35):
already, if you're already inthe sciences, then hopefully
science can be thought of asthis very pure thing, the
this is not too traumatizing, aswe bring up maybe some of the
less fun parts of being aresearcher and a scientist. So
anyway, what is this game? Idon't know if he made this as a
graduate student or as apostdoc. It is a light party
game meant to kind of poke funat the scientific publishing
enterprise. So we've mentionedthis a few times on the episode.
(01:55):
We scientists don't have verymuch. We're generally not in it
for the money. Most of us don'tget very famous like the one
thing we have is our reputationwith other scientists, and we
establish that by publishingacademic research papers. And
pursuit of knowledge, thegeneration, the advancement of
people look at those papers, andthat's how things like promotion
and tenure, which is basicallyjob security and being hired by
human knowledge and humankind.It's also a job that's kind of
another university or going offinto industry is important, like
(02:17):
if you're going on the jobmarket either as a freshly
minted Master's or PhD student,or as a professor who's been in
it for 20 years, people aregoing to look at your
publication record to see, areyou actually a good scientist?
Are you actually putting out alot of work, and hopefully good
quality work. But as whathappens with any field, anytime
you reward people for something,then there's ways to sort of
(02:37):
abuse the system, and things gowrong, and the publish or perish
of the title is talking abouthow, as a researcher, especially
a university researcher, youhave to publish or you'll
perish, like if you don't getthe publications out, you're
going to lose your job.
(03:00):
difficult. And the unit ofscientific information that your
typical scientist cares about ispublishing a paper. We did a
study. This is what we found.You take that paper, you write
it all up, you submit it to ajournal. It'll go through a
process called peer review,which is where three anonymous
colleagues typically will lookat it and say, like, is this
good science or not? And thatprocess kind of works. It's, I
(03:21):
think it's like democracy. It'sthe worst political system
except for all the other ones.
Churchill's famous quote,
Brian (03:26):
yeah.
Jason (03:26):
So this system has
evolved over a century or two of
time as the scientificenterprise kind of started
building as we recognize it now.And no, it's not perfect by any
means, and people are looking atways of changing it. So there's
like, open peer review, whereyou publish and then people just
comment on it, which has all thepros and cons of everything else
on the internet that's just openfor people to comment on,
Brian (03:47):
just like Wikipedia,
right?
Jason (03:49):
Yeah, peer review, as
currently, as it currently
stands, also has the same prosand cons, because peer reviewers
are usually anonymous. And onething the Internet has taught us
is that sometimes when you haveanonymity, you get permission to
be nasty, and some people dothat. And that is actually the
infamous reviewer two that thegame talks about a few times is
that it's usually it's reviewernumber two is that nasty person
(04:09):
who just hates your work anddidn't read it, and is just
saying how much horrible stuffis in there, and you have to
change all this stuff and so onand so forth.
Brian (04:16):
Being on the other side
of this, you know, being someone
who does publish papers inalmost everything that we do as
scientists, you have to justifywhat you're saying. You either
did the experiment or you cancite someone else who did the
experiment. You can citesomething that's passed peer
review, that's out in commonknowledge. When you are an
anonymous reviewer, people willjust say random stuff that's not
justified. They don't providetheir sources. They don't have
(04:38):
to explain anything. And that'sone of the only times that
scientists get away with thiscrap.
Jason (04:43):
And I will say this is
not like this isn't every time,
Brian (04:46):
no, no, no,
Jason (04:47):
but it's not rare enough
that it's a weird thing. Like
most people have had anexperience with THAT reviewer
who just didn't get their thing,
Brian (04:54):
and then you usually have
an editor who's kind of acting
like a referee. They will readthe paper, they will. Also look
at the reviewers comments andsay, like, hey, is this person
out of line? They're the, prettymuch the only people that get to
say that.
Jason (05:06):
But we don't have any of
that in the game. In the game,
all you have to do is you getenough cards to get to publish.
So let's actually talk aboutthis game before we dig into the
nitty gritty of scientificpublishing. The game has a few
different types of cards. You'vegot your action cards, or what
you draw into your hand and youused to play the game. There are
publication cards, manuscriptcards, which are what you
collect to try to win the game.Each one's worth a certain
(05:27):
number of citation points,because that's how we grade
papers in science. Is how manytimes do other people cite the
paper? And then there's sometrivia cards, actually, that get
used in a few cases, where theyhave random trivia from all
sorts of different fields ofscience. And the experience
there is, if you are in thatfield, then they're easy, and if
you're not, they're super hard.
Brian (05:46):
We experienced this. It's
like, oh, these biology
questions are way too easy. Idon't even know what these terms
are in the economics question,
Jason (05:53):
yeah, which is probably
exactly what it's meant to be.
They're meant to be high levelstuff, so that if you're not in
the field, it's basically justrandom guessing. And the idea is
that you go through, you'redrawing your hand, you're
collecting cards. And if you getcertain numbers of the right
resources that you can then geta manuscript. And there are five
resources which are basicallythere just to have five
different things you have tocollect. There's writing,
theory, references, data andideas, and so different papers
(06:15):
require different numbers ofeach of those. Usually they just
require, like, two or three ofsome of them. Honestly, the
manuscript cards, I think arethe best part of this game,
because they're all fullyfleshed out with a journal and
an article and a title and anabstract, which is like a brief
summary of it, and they're justridiculous. So I'm just going to
flip through here. We've got theeconomics of Santa Claus, an
analysis of infinite resourcemanagement procrastination
(06:36):
patterns among academics, a casestudy of myself unpacking the
aerodynamics of flying pigs. Andthen there's a whole paragraph
describing what this article isabout, and he does say he used a
little bit of generative AI tohelp with that, because I know
there's like, 50 or 60 of thesecards. I don't know how someone
would come up with all that forall of them. So they're actually
quite fun to read, becausethey're just totally ridiculous.
(06:57):
They've all got punny authornames as well.
So I'm just picking the top ofthe stack here. Got a myth
busting microwave minutes isdefrost, just a placebo, and the
author is luke warm. SCD fromthe University of convenient
conclusions.
Brian (07:12):
You can also hit your
opponent with maladies, budget
cuts, which will affect things,or a citation error, which will
remove some of the citationvalues. And those all have funny
little quips on them too.
Jason (07:22):
Yes, they actually the
the flavor text on the card. So
not the mechanics. The mechanicsare pretty standard across a lot
of the card types, but theflavor text of what exactly went
wrong is actually quiteentertaining. So there are
mishap cards which you play onother people to reduce their
citation counts. Just grab onehere. As it turns out, a
critical citation for your paperwas hallucinated by chat GTP,
(07:43):
that's your thing, and then youhave to spend resource cards to
fix that error so you can getyour citations back.
Brian (07:47):
Have you ever experienced
this? Because I know that this
was the thing in the news wherechat GPT will fully manufacture
scientific articles, theauthors, the title, the date,
everything. It'll even give youfake Digital Object Identifier
URLs for papers that do notexist.
Jason (08:02):
Yes, I tried that early
on. It's like, okay, let's see
how this is. Write me a shortscientific literature with
citations for whatever topic Iwas looking at the time. And it
did it. It wrote a very niceparagraph, and it had citations.
And I looked them up, and theywere all completely non
existent.
Brian (08:17):
They sounded real.
Jason (08:18):
I've noticed they've
started adjusting that now where
they'll actually have, like,footnotes, maybe not on chat
GPT, just different AIs I'veused, they'll put footnotes as
where this information camefrom, so you can actually look
Brian (08:26):
anyway. This is a
different conversation.
up your sources. And there's afew where they very specifically
were making it for scientificresearch, where they would have,
Jason (08:34):
Yeah, different one. So
game, back to the game. So, and
like, this sentence came fromthis paper that you can then
look up. So they're trying tofix that problem
your goal is just to get thefirst one to five citations ends
the game and you finish out theround. But they don't
necessarily win, because youhave to get your citation count.
And then we didn't do this partbecause it was late and we were
(08:54):
tired.
Brian (08:55):
Well, it's also only two
of us. This game says it's got a
player count of three, becauseit's a party game. We, I mean,
we just did it because we've gotthe game for a week and we
wanted to play. Wanted to playit right.
Jason (09:03):
Yeah, so we had to review
an early review copy that we
have to send back in a few days.But whenever someone publishes,
everyone else is supposed toclap and then congratulate them
or give snarky comments andreviews phrased in the form of a
question, like you were areviewer trying to tear apart
their publication, and at theend of the game, there's
actually a vote for whoever wasthe snarkiest reviewer, and they
get extra points. You also takeall the publications you managed
(09:26):
to collect and put togetherbasically a fake dissertation
defense, where you're defendingyour line of research with these
random papers you've cobbledtogether. And that goes on for
supposedly one minute, althoughusually longer, and then you get
a vote, and whoever did that thebest also gets additional
points. And then there's the"almost there" award, which is
for the person who tried to getthose previous two and failed.
(09:47):
And so they get, instead ofthree extra citations, they get
2.9 extra citations. So it's,it's, this is definitely a party
game. It's a light fluff game.This is not deep gameplay.
There's not deep strategies. Toexplore here. It's just you
gather some people around thetable, you sit down, you play,
you have fun. Obviously, hitstrongest if you're in academia,
(10:07):
so like, if you've been througha graduate program or stuff, but
you don't have to do that toplay. Like, you can kind of get
it, and the cards are funnyenough that you can kind of get
what's going on without that.But I think its original target
audience was definitelyacademics.
Brian (10:19):
Didn't Didn't you say
that this kind of started making
the rounds because it got someattention in Nature magazine.
Jason (10:23):
I think it got some
attention on social media, and
then it got picked up by Nature,which is one of the big
scientific journals, and so lotsof people see that. And so,
yeah, it got within at least thescientific community. It got a
lot of traction.
Brian (10:36):
I mean, I had two people
send it to me. They're like,
Hey, you should talk about this.Did you have that as well?
Jason (10:40):
I did not. Maybe I short
circuited, because as soon as I
found it, I sent it out to otherpeople, so maybe
Brian (10:46):
they didn't have the
opportunity.
Jason (10:47):
Yeah,
Brian (10:48):
well, by the time they
were sending it to me, you had
already done that as well. Soactually, I think I got it from
three or four different people,you included. I think the best
player count for this game.Party games are always better
with more people. What's the maxsuggested player count?
Jason (10:59):
Let's see. So the game
says it is for somewhere. I
think it's three to six, yeah,three to six players. Says it
lasts anywhere from half an hourto two hours.
Brian (11:09):
Two hours? That seems
crazy.
Jason (11:11):
If it's really cutthroat,
maybe, I don't know.
Brian (11:13):
I mean, we played two
rounds in an hour. There were
only two of us, though,
Jason (11:16):
yeah, and it was fun,
like it was a nice, quick little
fluff game. I'm gonna be playingit with my lab over lunch today,
so we'll see how that goes, andI'm going to try to get them to
do the improv and the sillierparts that you were not on board
for
Brian (11:26):
improv is more fun when
it's not just one person across
the table from you. So
Jason (11:30):
that is fair.
Brian (11:31):
Yeah, I think five to six
is going to be a lot more fun
than three.
Jason (11:34):
Yeah, Isuspect because
you have more people to riff off
of, there's more chance to besnarky because you're not like
you're not always snarky. Ican't come up with a snarky
comment for every paper, but themore people there are, the more
likely that is to happen. I'vealready pre ordered a copy of
this that I'm gonna have in thelab just to blow off some steam
or do over lunch or take todepartment retreats or
something. And on theKickstarter, not only do they
(11:54):
have the core game, typical,they already have the first
three expansions made. One justadds extra action cards. One
sets extra trivia cards, and oneadds extra manuscript cards,
including predatory journalsthat will basically publish
anything you give them. And infact, one of them, the abstract
is lorem ipsum, which is thisLatin text that is basically a
Unknown (12:08):
which believe it or
not, that has been an actual
fill in for editors. They justuse to fill space. I think that
the title of that one wasscientific evidence that
predatory journals will publishanything you give them.
thing, like, there have been acouple studies on that very
topic for real in real life.Maybe I'll find a link for one
of those.
Jason (12:25):
Yeah, so that's really
the game. I mean, it's a light
game. It's a fun game. Lookslike when it goes live, probably
by the time we get this edited,Kickstarter is going to be done.
Sorry about that. There weresome delays in getting us our
review copy, but it is going tobe available commercially after
that. Looks like the retailprice is going to be about $40
for the base game, and somewherearound 20 for each expansion you
look out for that either online.I'm sure he'll have it available
(12:46):
online somewhere, maybe managedto get into some big box stores
or local game stores orsomething.
Brian (12:51):
What do you wanna do for
this one? So this is a is this
getting a science grade or justa fun grade?
Jason (12:55):
I think mostly it's
getting a fun grade because, I
mean, if we were trying to gradeon what is this like according
to actual scientific publishing,actual scientific publishing is
a grind and is generally notfun.
Brian (13:07):
Yeah, stretch this out,
where each round you submit, and
then you wait for about fourmonths to hear something back.
Jason (13:13):
And the thing is, like,
we're making this sound bad, and
part of it's because most of usdo not go into science because
we love writing papers. We gointo science because we love
doing the science, and we lovedoing experiments and solving
problems and writing the paperhas to happen, but it's kind of
like doing your taxes orotherwise filing forms. It's
like, it's not necessarily thefun part of what we do,
Brian (13:33):
but super critical. It's
the whole thing, right? I mean,
you bring in money to producescience, and this is how we do
it. You gotta publish it. Yougotta get it out there. Or what
was the point of doing it?
Jason (13:42):
Yeah, unpublished
sciences. I mean, technically,
it is science, but doesn't it'snot part,
Brian (13:46):
it's not useful.
Jason (13:47):
Science that doesn't make
it into the public
consciousness, that doesn't makeit out there for other people to
use, is, I wouldn't say useless,because that's companies do that
all the time. It's proprietary,but we're in the public sector.
Our job is to create knowledgefor the public good, which is
why people in industry publishless, not zero. So industry
researchers can actually publishpapers on what they are doing.
They usually have to go throughsome hoops to lock down
(14:08):
intellectual property first,because a company's goal is to
get a competitive advantage, butif they can do that, then they
can put it out, and it doesactually boost their reputation
some too, especially if there'slike a tool they've developed
that lots of people would bereally interested in using they
want people to know about it sothat they can license to them
and make money off of it thatway. So he even has a role in
the private sector, just not asmuch a role in the public
(14:29):
sector.
Brian (14:29):
OK, so we're probably
gonna skip giving it a science
accuracy grade, maybe just toprotect my own mental health.
But in terms of fun, I probablythink for the right audience,
this would be an A, but ifyou're thinking about a general
public thing where their livesare not touched by the
scientific publishing endeavor,I don't really know if this is
gonna hit. So am I allowed togive a split grade?
Jason (14:51):
This is a bonus episode.
You give whatever you want,
Brian (14:53):
all right, if you have
had science publication, or know
someone who has in your life, Ithink this will probably be an
A. And if you don't, I thinkit's probably a B-. I don't
think the jokes are going to hitall that well.
Jason (15:03):
yeah, hopefully they'd
enjoy the manuscript pages,
though, the abstracts and thetitles are actually quite fun.
The names are funny. Being onewho's in academia, I'd probably
give it like A-, B+. But I likecrunchier games. So there's very
few party games that I justenjoy sitting around and playing
with people, because I prefermore ones where there's tactics
and there's some rules I can tryto figure out how to master, and
my advantage comes from beingable to master those rules
(15:24):
better than other people, ratherthan just doing goofy stuff with
my friends, which there's aplace for that. My family loves
Cards Against Humanity, which isdefinitely just a light, fluffy
party game. But most of the timeI prefer something with more
meat to it, but that's mypersonal preference.
Brian (15:37):
all right. Well, I mean,
we're gonna skip the science
again, just because it hurts myheart too much to have to get
into the details here.
Jason (15:42):
It almost feels like we
did this episode backwards. We
talked about the actual stuffbefore we talked about the game.
Brian (15:47):
Well, I mean, but it is a
backwards episode. It's the
science with gaming.
Jason (15:52):
Here's something else I
want to talk about, though, and
that's actually the use ofgenerative AI to make these
things and the thing is, there'sa big conversation now, my wife
and I have argued back and forthabout what's acceptable use of
AI and not we have verydifferent opinions on that.
We're not going to get into thatmore, just the fact that the
genie is out of the bottle itsgoing to happen. So you
mentioned that generative AI cancreate an entire fake scientific
paper, and the game actuallygets a little bit into like the
(16:14):
predatory Journal, the ones thatbasically make money off of
people paying to publish inthem, because they need
publication counts to letanything through. And my concern
is like, where are we gonna gowith generative AI, for this in
the future? Because it used tobe that writing a paper was
really hard, even if you're abad actor and you lied about
your results, it took a lot ofwork to put a paper together and
get it through the publicationprocess. But now generative AI,
makes that easy, and you combinethat with predatory journals,
(16:37):
and I just worry that thescientific literature is gonna
get pollutedwith a bunch of crappapers, that's a big problem,
especially when people try toreproduce what you do, which is
a key part of science, onepublication means nothing.
Brian (16:47):
It's a body of work.
Jason (16:49):
Yes, like one publication
is like, okay, that's an
indication, but until otherpeople also repeat it and get
the same result, then it's just,it's a data point. It's not
actually considered real unlessit can be replicated.
Brian (17:00):
Although I would say that
there's a problem with that
idea, though, because you'reright. It is supposed to be part
of the process, but replicationstudies often don't happen
because there's this unfortunatehuman rationale of like, well,
that's already been done. I'mnot going to waste my time doing
it. So the problems come whensomeone tries to build on
previous work and then itdoesn't work. Like, oh, wait,
something's gone wrong here.That also faces the publication
filter. People are less likelyto publish the results that
(17:22):
don't, quote, unquote, work. Soit should be a self-correcting
process, but the sort of messyhuman sociology, this is my
jobness of it very much, can getin the way of that process.
Jason (17:32):
That's true, and that's
kind of where I'm concerned
here, just like with everythingelse, with generative AI, it's
possible that the noise willstart swamping out the signal,
because there are definitelypeople who are motivated just
get a publication count outthere, because that's what will
be used to boost up theircitation count. It might get
them a bonus at their job. Thereare some countries that very
(17:52):
specifically tie publications toyour salary and to your
promotion. And anytime you getthat sort of perverse incentive,
then you start encouragingpeople to cheat. Hopefully most
people won't, but there willalways be some number of people
who will. Scientists are stillhuman, and that means there's a
whole bunch of different typesof scientists, and some of them
are going to game the systembecause it helps them get ahead,
unfortunately.
Brian (18:13):
So predatory journals.
Why do those exist?
They're 100% are
predatory conferences. I get
Jason (18:15):
Ah, so we have to go into
a little bit of background of
publication for this. So it usedto be that you would send your
manuscript to a big publishingarea, they'd go review it,
assuming it passed all that andgot published. They then made
their money off of sellingsubscriptions to universities
and companies and such. Well,starting probably 20 years ago,
maybe more is this open accessmovement, because lots of people
can't afford to pay for that.And the idea is, if we're making
(18:37):
this the common domain ofknowledge for humanity, then
humanity as a whole should beable to access it. So people
started making results openaccess, where you could just you
could access them no matter whoyou are. You didn't have to pay
for them or anything like that,and that's good. But the fact
is, it still costs money to tohave a copy editor to make
things look nice, and so themoney has to come from
somewhere. And so now, when youhave an open access publication,
(18:59):
the person publishing it ispaying the cost of that. They're
paying essentially the cost ofmaking it look nice, of running
the peer review process and allthat sort of thing. And once
that happened, there are somemalicious people figured, hey,
we can turn this into a businessof having people pay us to
publish their work. And goodopen access publications still
keep a high quality standard,and they still make sure it gets
filtered. And bad ones who areout just to make a buck don't.
(19:21):
They're just using it as a wayof getting money from
researchers, many of them whodon't know better, because they
don't realize the publication ispredatory, because, like with
all scams and phishing attemptsand such, they're very good at
hiding and looking likesomething legit, and so we have
this whole issue going on. Maybethis is a different topic, but
the same thing happens withconferences. The other way we
get our work out is not justpapers, but we talk about them
(19:43):
at conferences. And there arepredatory conferences out there.
invitations for "conferences"all over the world all the time,
and so this is, maybe, I
hesitate to call it, the seedy
underbelly of science. It's morejust the unfortunate reality of,
as Brian said, science is. Ajob. Science is a human endeavor
that costs time and energy andmoney, and there are rewards
(20:04):
attached to it. And anytime youhave that, there can be some
people gaming the system. Andthe reason for talking about
this is not to like doom andgloom and Oh, science is
horrible. It's like, no, no.It's like, most people involved
are good actors most of thetime. It's okay, but it's
imperfect. And some people havepointed out that there's an idea
of scientism nowadays. Of like,oh, science is great. We must
follow everything. It's like,recognize it's still a human
(20:25):
endeavor. It's imperfect.
Brian (20:27):
Look at the history of
science, it will show you how
much humans can put theirfingerprint on data.
Jason (20:33):
Yeah. And the thing is, I
think on the whole, it still is
one of the better systems wehave for knowledge discovery,
and betterment like you comparehow we are now to 500 years ago?
A lot of that is due toscientific advancements. Just
before this, we talked aboutPandemic and our lifespan,
infectious disease control. Allof that is due to scientific
stuff, where people have takenthings, some of it from
indigenous knowledge that gottested in more extensive ways.
(20:54):
Some of it from pure Westernscientific research from all
over the place. But it gottested, it got validated. It
went through the system. And soyes, the system makes mistakes.
The system is imperfect, but itdoes more good than it does harm
by and large. And so we don'twant to under cut it, but we do
want to make it more realistic.The game pokes fun at science,
but it does in a way that isbased in truth. I mean, all the
best satire is based in truth.And so while we may be talking a
(21:15):
bit of a downer in terms oflike, Oh, these are some of the
flaws in the system, this isalso a good safety valve. Yeah,
we know things are imperfect, sothis is a good way of like being
able to laugh at theimperfections and then maybe
move on and try to make thembetter.
Yeah, best not to ignore thatthere's a problem that was a lot
of deep philosophy for a gamethat is primarily based on puns,
puns in a fluff party game. Allright? Well, this is just a
(21:36):
quick bonus episode, so I thinkwe're gonna call it then. So
look up publish or perish by Drmax by hope you can have some
fun if you decide to go for it,and until then, have a good
break. Happy gaming, and we'llsee you next time.
Brian (21:48):
Have fun playing dice
with the universe. See ya.
Jason (21:52):
This has been the gaming
with Science Podcast copyright
2024 listeners are free to reusethis recording for any non
commercial purpose, as long ascredit is given to gaming with
science. This podcast isproduced with the support from
the University of Georgia. Allopinions are those of the hosts,
and do not imply endorsement bythe sponsors. If you wish to
purchase any of the games wetalked about, we encourage you
to do so through your friendlylocal game store. Thank you and
(22:14):
have fun playing dice with theuniverse. You.