December 27, 2019 • 33 mins
Learn what happens during study section and the role of a scientific review officer from SROs Drs. Ernie Lyons (NINDS Review Branch Chief), Shanta Rajaram, and Bill Benzing.
Building Up the Nerve is a podcast from the National Institute of Neurological Disorders and Stroke for neuroscience trainees that takes you through the life cycle of a grant from idea to award at NINDS with the people who make it happen. We know that applying for NIH funding can be daunting, but we’re here to help—it’s our job!
This episode has been edited since its original publication.
Transcript: http://ninds.buzzsprout.com/558574/2309078-episode-6-scientific-review
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Lauren (00:02):
Welcome to the National Institute of Neurological Disorders
and Stroke's Building Up the Nerve,a podcast for neuroscience trainees
that takes you through the life cycleof a grant from idea to award at NINDS
with the people who make it happen.
We know that applying for NIH fundingcan be daunting, but we're here to help.
It's our job.
Marguerite Matthews (00:22):
Hi, I'm Marguerite Matthews, a health
program specialist at NINDS.
Lauren (00:26):
and I'm Lauren Ullrich, a scientific program manager at NINDS
and we're your hosts for this episode.
So last time we discussed submittingyour application and receipt and
referral for assignment to an Institute.
Today we're going to talk aboutwhat happens during review.
Marguerite Matthe (00:41):
And as always we want to provide a disclaimer that
everything we talk about in thispodcast may only be relevant to NINDS.
And if you're applying to a differentNIH Institute or center, it's best to
check with them about their policies.
Lauren Ullrich (00:54):
So I was really lucky in my graduate work--I went to
Georgetown, so we are very close toNIH--and we had mock study sections
at our first year with actual SROs.
It really was central to opening myeyes to how the process works because I
think there's nothing like experiencingit to really understand what goes
through the mind of a reviewer.
(01:15):
And Marguerite, did you have anyexperiences like that in grad school?
Unfortunately not as a trainee.
The NIH review process was completelyforeign to me and since I didn't submit
an F31, I never got a chance to getany summary statement or reviews back.
So I didn't learn about the NIH reviewprocess until I started working at NIH.
(01:35):
Yeah.
And I think unfortunately that's alot of trainees' experience, so...
Marguerite Matthews (01:39):
So that's why we're talking about it today!
Lauren Ullrich (01:41):
Exactly.
Marguerite (01:41):
Have a lot in store.
Lauren Ullrich (01:47):
Joining us today are Dr Ernie Lyons, chief of NINDS's
scientific review branch, Dr ShantaRajaram, scientific review officer who
runs the NINDS clinical trials NSD-Kstudy section and Dr Bill Benzing,
scientific review officer who runsthe NST-1 study section and other
(02:08):
training-related special emphasis panels.
So let's start with introductions.
Ernie Lyons (02:13):
As you mentioned, I'm the chief of review here.
So I manage the review branch and I manageall of the assignment of all the incoming
applications to the study sections andother receipt and referral processes.
I help manage developing our FOAsand setting up receipt dates.
Just, you know, make sure thatall our reviews follow all
the rules and regulations andare conducted appropriately.
(02:36):
And I've worked here since2001 so quite a while.
I was an SRO for 10 yearsbefore I became chief.
And before I started here I was aresearch associate at a university
and you know, academic research.
I had small grants and private grants.
I did write some NIHgrants; I never got funded.
My hobbies, I have several hobbies.
(02:56):
I like to garden and I like to play music.
Shanta Ra (02:59):
I, as you said, I'm a scientific review officer here at NINDS.
I have been in some waysa jack-of-all-trades.
I've done all sorts of reviews through my15 years at NINDS, but I primarily manage
the reviews of our clinical programs.
Uh, in addition to that, I'm involvedin initiatives such as the BRAIN
(03:20):
and HEAL and I've done a few oddtraining reviews here and there.
What I did before I started here,before I started at NINDS, I was in the
intramural research program at NIMH.
That allowed me an opportunity to do adetail in, in a branch that I might be
interested in, um, for as a future career.
(03:41):
And that's how I ended up, uh, as adetail yet at NINDS and ended up getting
a job as an SRO here 15 years ago.
Passions outside of workis actually rescuing dogs.
Um, I have two of them at home rightnow, but I'm very much involved
in the volunteer organizationthat rescues those animals.
Lauren Ullrich (04:01):
Nice.
I want to see pictures.
Marguerite Matthews (04:03):
Yeah, same.
Shanta Rajaram (04:06):
Absolutely
Bill Benzing (04:06):
I'm a scientific review officer at the NINDS as well.
And, uh, I've been at the NIH as longas Ernie actually almost 20 years.
And I was at CSR, center for scientificreview, first for a while, and then
I came to NINDS about 13 years ago.
Uh, what I did before here, I workedin academia and then in biotech.
(04:28):
My research focus was age-relatedneurodegenerative disorders.
As far as hobbies or passions, obviouslymy kids and gardening and music and
I've rescued a few dogs for my own sake.
Dogs are a large part of my life as well.
Marguerite Matthews (04:51):
All right.
So can one of you tell us aboutthe relationship between the center
for scientific review and reviewsthat are done at the IC level?
Er (04:59):
So the center for scientific review, as you mentioned, is one of
the 27 institutes or centers at NIH.
And they handle the majority ofreviews at NIH, about 75% of the
applications are reviewed there.
And just in general you could say theyreview the more standard applications
that are typically submitted inresponse to parent funding opportunity
(05:21):
announcements, like the standard R01,just regarded as the gold standard.
I think it's the most common NIHapplication and it's, as I said, sort of
a gold standard at academic institutionsto have your lab funded through an R01.
They also review R21s, R15s, smallbusiness, so many other mechanisms.
But institutes as well have theirown review branch, which tend to,
(05:45):
and again this is general, reviewapplications that are submitted in
response to more specialized fundingannouncements and often have um,
institute-specific review criteria oreligibility requirements or definitions.
For example, translationalresearch may be defined differently
at different institutes.
So often, uh, we review that.
(06:06):
We have a translational reviewprogram that reviews our translational
research, clinical research,training, as you'll hear a lot about.
And as well as many RFAs and otherprograms like our centers awards,
centers grants, core grants, andthose kinds of complex mechanisms.
Lauren Ullrich (06:23):
That's a lot of different kinds of mechanisms.
I don't think when I was a trainee, anyof those things were really on my radar.
And so specifically for trainingat NINDS, which training mechanisms
are reviewed at CSR and whichones are reviewed by us at NIDS?
Bill Be (06:38):
Um, the ones that we do in NINDS review branch, uh, specifically
for training, while we do review othertypes of grant mechanisms, are the F32s
the various K01 mechanisms, clinicianscientists, independent junior faculty
K02, and the clinician scientists mentoredtraining programs, K08, K22, and K23.
(07:01):
Uh, we also review, um, themore departmental type training
programs like T32s and summerresearch programs like R25s.
Marguerite Matthews (07:10):
Once I've submitted my application, um,
where can I check exactly wheremy application is being reviewed?
Shanta Raj (07:16):
So you can check the locus of review for your application
in your commons account--in NIH ERAcommons account once you've submitted it.
An alternate place to look, althoughspecific review panel may not be listed
at that spot, a good place to check isalso to look under the agency contacts
listed under the funding opportunityannouncement for your review locus.
Ernie Ly (07:40):
And in section five of the FOA, section five number 2 lists
the locus of review, so you can knowahead of time where it's going to go.
Lauren Ullrich (07:48):
Right.
It's kind of buried in there a little bit.
Hard to find if you don'tknow what you're looking for.
And specifically if I want to know whatstudy section my application will be
reviewed in, where do I go to check that?
And also who is going tobe on the review panel?
Ernie (08:04):
I believe the easiest way to know your study section assignment is:
it's posted in your ERA commons accountonce it's assigned to a study section and
I think there may be even a link thereto the roster to see who's on the roster.
Rosters typically become available 30days before the study section meets.
That would be the meeting roster.
(08:25):
You can also preview the committeemembers for standing committees,
you know, to see who the regularmembers are on the NIH website.
There's a helpful tool in, in the NIHreporter called the matchmaker tab and
you can paste in a keywords or an abstractand it will match up a potential study
sections and you can actually suggesta study section when you submit your
(08:48):
application either in the cover letteror on the assignment request form that's
available during the submission process.
Marguerite Matthews (08:55):
And how are reviewers chosen to review applications?
Bill (08:59):
Well, a number of factors go into how we choose reviewers.
At the level of, I'm looking forreviewers for an individual application
or a group of applications or fora whole study section, we typically
look for individuals, definitelywith a doctoral degree or equivalent.
Most importantly, we look fordemonstrated scientific expertise in the
relevant area of either an individualapplication or a group of applications.
(09:20):
One of the criteria that welook at is a track record of
independent research support anddemonstrated breadth of perspective.
When we get to know reviewers, we reallywant reviewers that can provide an
overall broad objective perspective,free of bias and we want reviewers that
can exercise sound and mature judgment.
And most importantly individualreviewers should also be free of personal
(09:42):
professional or financial conflicts ofinterest to review the set applications.
At the level of entire review panel,additional factors go into creating
the makeup of an of a whole panel andthese include geographic distribution
of the reviewers, and some other factorsthat get factored into those things.
Marguerite Matthew (09:57):
So you don't purposely pick investigators who are
going to be the third reviewer tobe out to get a given application?
That's not how it works?
[Laughter]You're not looking for people to
be extremely harsh in their review?
Cause I think as trainees especially, wesee what our mentors and other faculty
are going through and we just assumethat reviewers are purposely picked
(10:18):
to just be mean and give bad reviews.
Shanta Rajaram (10:21):
And actually we come to some of this in one of
your questions in the sense thatapplicants are almost always wrong
when they try to guess which reviewerson the panel was out to get them.
The support for your application maycome from the direction you least expect.
So I always advise applicantsdon't guess and certainly don't
(10:43):
assume the worst based on a roster.
Erni (10:45):
And I would add just in my long experience
really aren't out to get anyone.
I think the important thing to rememberis that study sections are made up of
scientists, they're all humans and webasically ask them for their opinions.
Maybe, I think overall the processworks very well and it's probably the
best we have, but that doesn't meanthey're always perfect or they always
(11:09):
give us the right answer or the rightopinion or you might not agree with
some of the things that they say, but...
Marguerite Matthews (11:14):
Right.
Lauren Ullrich (11:15):
Earlier we mentioned standing study section.
And so what is the difference betweena standing study section and a
special emphasis panel and how, um,how is it decided what applications
go to which kind of review panel?
Shanta Raja (11:29):
So a standing study section has appointed members.
Usually it's anywhere between 16 to 24and they are appointed for terms that are
typically between four or six years long.
The standing study sections usually meetthree times a year and as additional
expertise not provided by the standingmembers is required based on applications
(11:54):
that get assigned to the study section, adhoc expertise gets recruited each round.
A special emphasis panel or aSEP as NIH tends to call it--one
of its millions of acronyms--
Laur (12:05):
Yeah we love our acronyms.
Shanta Rajaram (12:07):
Is assembled de novo each time.
It is usually used for reviewingapplications in response to say to request
for applications commonly known as RFAs.
These are typicallyone time solicitations.
So these are the kinds of initiativesthat get reviewed in special emphasis
panels or they may be used for specialprograms that might require unique
(12:30):
review criteria, for example, or verycommon other use of special emphasis
panels is to review applications frommembers of standing study sections.
The so called member conflict SEPs.
In terms of deciding whichapplications, I think I talked a
little bit about how applicationsor which applications typically get
assigned to special emphasis panels.
(12:52):
In study sections they could bedivided by a number of different ways.
In NINDS we have different studysections for clinical, translational,
and training applications.
And each of these, even between thetranslational applications, the two study
sections have divided based on areasof science that NINDS funds, that are
(13:14):
divided between the two study sections.
Marguerite Matth (13:17):
And ultimately how does this influence the
actual review of applications?
Ernie L (13:22):
I think in general, the process between the two types of study
sections, standing committees or specialemphasis panels, is basically the same.
As Shanta mentioned, SEPs areusually created in response to
special, you know, typically one timeannouncements and they're tailored
specifically for that review.
(13:43):
Standing committees tend to review thingsthat come in on a regular basis like
many of our training awards that we have.
They come in three times a year,and so we have two standing study
sections that review our training.
I think that the difference thereis that a standing committee
provides some continuity andconsistency from round to round.
Those study sections develop a culture,they become familiar with the training
(14:05):
mechanisms and hopefully providesome consistency in their scoring
behaviors across different rounds.
Laur (14:12):
So shifting gears a little bit, in terms of the application, are
applicants allowed to send any updates ontheir application after it's submitted?
Bill Benzing (14:23):
Yes.
Up to 30 days prior to the review meeting.
You're only allowed to submit itemsthat are allowed in the NIH policy
for post submission materials, unlessthe funding opportunity announcement
that you're applying to, the FOA,states otherwise that no post-mission
material can be submitted orotherwise allows specific additional
materials than in the standard policy.
(14:45):
The NIH policy on post submissionmaterials can be found at
the NIH notice NOT-OD-19-083.
Lauren Ullrich (14:56):
And what kinds of materials generally
are allowed by the policy?
Bi (15:00):
There's a handful of things.
But the most common thing are acceptedpublications or recently published
manuscript that have become publishedsince application has been submitted.
Shanta Rajaram (15:09):
And what is most commonly not allowed and which we
get the maximum questions for, isanything that is a fixing of an error
that you made during submission.
So check the final form of theapplication once the PDF is in.
Ernie Lyons (15:23):
And some of our announcements do allow late
submission of um, of recent data.
So if, if you have more preliminarydata that would support your
application, some of our FOAs allowlate breaking data to be included.
Bill Benzing (15:38):
That would be listed specifically in the
funding opportunity announcement.
Lauren Ullrich (15:40):
So read the FOA.
Carefully.
Shanta Rajaram (15:43):
Absolutely.
Marguerite Ma (15:44):
So now that we've established that selecting reviewers
is not quite the hunger games, whatactually does go on during the review?
Shanta Raj (15:52):
The process actually starts a lot earlier than the
actual reviews; about four to sixweeks prior to the actual meeting.
Each application is assigned to a numberof reviewers, um, typically three, but
the more complex the application, the morethe number of reviewers assigned to it.
The reviewers get access, as I said, tonot just their assignments but also to
(16:14):
review instructions, relevant guidelinesthrough a secure web interface that
NIH uses for this for this purpose.
About a few days before the actualreview the reviewers are expected to
upload preliminary scores and critiques.
Now if it's a special program orspecial initiative with special review
(16:35):
criteria, as we said, there mighteven be a reviewer orientation call to
make NIH ensures with involvement ofboth program and review staff on those
calls that all reviewers are on thesame page in terms of understanding
the goals of that specific program.
Once the reviewers have postedtheir own critiques and scores,
(16:55):
they are also allowed to read thecomments posted by others assigned
to review the same application.
So there is a little bit ofan understanding of where the
others are coming at when thediscussions actually start.
At the meeting, there is a chairpersonwho facilitates discussions.
Discussions very typically focuson the strongest half of the
(17:17):
applications that have been assignedto that study section or that panel.
Reviewers take turns stating thepreliminary scores and going over
their main score driving issues.
We try to get them to focuson both the strengths and the
weaknesses in the application.
All non-conflicted members are theninvited to join the discussion once the
(17:40):
assigned reviewers have had their say.
Once it appears that the scoredriving issues have been hashed
out completely in a thorough discussion,the chair will then specifically invite
comments on other score driving criteriasuch as human subjects, protections,
vertebrate animals, inclusion policies,biohazards, so on and so forth.
(18:03):
Um, once this is done, the reviewchair person asks for final scores.
The assigned reviewer state their finalscores, which sets the score range,
and reviewers are asked to vote theirconscience based on that score range.
Now our reviewers in the room may chooseto vote outside the score range, but any
intent to do so must be declared publiclyat the meeting when the chair asks.
(18:25):
Lastly, the chair will ask forcomments or concerns on any
non score influencing elements.
These are administrative elementssuch as budget, if the organization
requesting funding is a foreign entity,issues such as resource sharing plans,
authentication plans for biologicalchemical resources, et cetera.
(18:47):
And that typically ends the reviewdiscussion for an application.
Lauren U (18:52):
One of the things that we get questions about a lot is this
difference between the different criterionscores that you might see on your
summary statement and the overall score.
So how do reviewers comeup with that overall score?
Do they just average the fivecriterion scores or how does that work?
Bill Benzing (19:10):
Well, um, let me give a little bit of background
on just the scoring first.
The NIH uses a nine point scoringsystem with one being the best
score and nine being the worst.
And most research grant applicationsreviewed at the NIH are evaluated
on five core review criteriaof which you're alluding to is
the these individual criterion.
(19:31):
Um, these are typically significance,investigator, innovation,,
approach and environment.
For training and careerdevelopment applications.
These individually scoring criteria aredifferent and may include applicant,
sponsors, which is also mentors,collaborators, consultants, research plan,
training, plan, training, potential, andinstitutional environment and commitment.
(19:54):
Each of these get an individual score,which is called a criterion score.
The overall impact score, which is anoverall opinion of of the uh, application
by a reviewer is an assessment of thelikelihood of the project to exert a
sustained powerful influence on theresearch fields involved or the training
of the individual applicant if it'sa training grant, and its determined
(20:16):
by reviewers by considering the fiveindividually scored criteria, but it's
weighted on the overall reviewer judgment.
And the overall score includes humansubjects issues, animal welfare
concerns, and things like that as well.
Laure (20:30):
And those are things that they're not individually scored, but they
can count towards the final overall score
Bill Benzing (20:35):
Right.
If there's issues with humansubjects, issues with the scientific
justification for using the humanssubjects or the animals, um, those
can affect the overall score.
So the overall score is an impact scoreof the whole application and individual
criteria and scores are a judgment eachof those different individual criteria.
Marguerite (20:54):
So the final number that applicants see is the impact score.
How is that calculated?
Bill (20:59):
The overall impact scores provided by all reviewers in the study
section are averaged and then roundedmathematically to one decimal place.
And then this is multiplied by 10 togive you the two digit impact score.
So if something had an average of sayhalf the panel or roughly half the
panel gave 1s to an application andthe other half gave 2s that yielded
(21:20):
an average of say 1.5, it wouldyield an overall impact score of 15.
Lauren Ullrich (21:25):
And in addition to the impact score, many summary
statements also provide a percentile.
And what is that and how is it calculated?
Ernie Lyons (21:33):
So a percentile is essentially just a way
of ranking the applications.
It's a ranking mechanism andit's expressed as a percentile.
And it's really the rank of yourapplication's score divided by
the total number of applications.
So for example, if there are hundredapplications and your score ranked
(21:54):
10th on the list, it was the 10th best,you would receive a 10th percentile.
And that's a little bit ofan oversimplification, but
that's essentially what it is.
But importantly, how we use it andwhat it allows us to do is to compare
scoring or ranking across differentstudy sections so we can fund the same
percentage of applications, the top 15%for example, from each study section.
(22:17):
So where study sections may haveslightly different scoring behaviors,
we're still funding the samepercentage from each study section.
And percentiles are also calculated acrossthree council rounds, which helps to
account for round to round variability.
So if you get a particularly strongset of applications, one round and a
weaker set the other round, that tendsto normalize over the three rounds.
Shan (22:40):
I would like to add though that NINDS does not percentile any of
the applications, either in our studysections or special emphasis panels.
So really reading the summary statementand looking at your score and what
those numbers mean in terms ofdescriptors is usually very helpful.
Laur (22:59):
Yes, that is a very common question that we get because most of our
training grants are reviewed in house andso most of them don't get a percentile.
And I think a lot of mentors, they'reused to just zeroing in on that percentile
and disregarding the impact score.
And so when a trainee gets theirreview back and it only has the impact
score, they're a little bit at a lossof how to, um, how to interpret that.
Marguerite Ma (23:22):
How do applicants know the outcome of their review?
Shanta Rajaram (23:26):
So the score and the summary statement are
both posted in the ERA commons.
And again, the applicants can access itby signing into their commons accounts.
Typically applicants receive an automatedemail notification as soon as their
scores are released, but we alwaysrecommend that you check periodically.
(23:46):
The scores are released within acouple of days of the study section,
of this review panel meeting.
The summary statements can takelonger, anywhere from four weeks from
the review date to a month beforethat particular council review round.
So we always ask that applicants checktheir commons accounts periodically
(24:07):
because there's always technical glitchesdue to which that automated email
may never actually make it to them.
Lauren Ullrich (24:14):
And if you have questions about the review,
who is the best contact person?
Bill Benzing (24:20):
This is a question that often gets misunderstood.
If its prior to the review andprior to the score coming out, the
person to contact is the scientificreview officer, which can be found
in the review section related to theapplication in your commons account.
Prior to the review, anything that'srelated to submission of additional
(24:41):
materials or other questions should bedirected to the scientific review officer.
If it's post-meeting and the scoreshave been out and/or the summary
statement is out, they should becontacting their program director
or their assigned program officer.
Marguerite Matthews (24:55):
Are there opportunities for early career
scientists to participate in review?
Shanta Rajaram (24:59):
Yes, absolutely.
In fact, most SROs welcome theperspective that junior investigators
often bring to these panels.
The center for scientific review, whichyou've heard of--us mention several
times now, it actually has an earlycareer reviewer program, so called the
ECR program, that is designed to helpemerging researchers advance their careers
(25:23):
by exposing them to review early on.
In fact, I always tell applicants thatone of the best ways to improve your grant
writing skills is to serve as a reviewer.
So I encourage applicants to both review,uh, both register for the ECR program
or if they are interested in an ICreview panel, to just send their CVS,
(25:43):
reach out to the chief of the scientificreview branch and send us your bios.
Lauren Ullrich (25:48):
And is there anything, common misconceptions,
things that you wish that applicantsunderstood about this process?
Ernie (25:56):
So what I was planned to say here, I, I've already mentioned
a little bit (25:59):
to remember that the study sections are comprised
of scientists who are humans.
They have a lot of grants to read.
They're not going to geteverything right all the time.
But for the most part,their intentions are good.
They're trying to do their best toidentify the best science and discriminate
between large numbers of applications.
It's very stressful for reviewers;they have to present critiques of
(26:22):
multiple applications in front of theircolleagues, in front of their peers.
And for the most part, again, they'rejust doing their best to try to review
these as accurately as they can.
But it's, you know, it'scertainly not a perfect process.
Marguerite Matthews (26:36):
And I think especially for training
opportunities, reviewers reallywant the best for the trainee.
So they really look for opportunitieswhere the trainee has, has highlighted
some great things and often wantto highlight the weaknesses more
as an opportunity for growth forboth the trainee and the mentor.
Shanta Raja (26:53):
I absolutely agree.
I sometimes think that, especially forsome of our training study sections,
if the applicant could be a fly onthe wall when the study sections
actually go on, they would really geta flavor for how much the reviewers
actually look out for these--for thetraining and for the applicant and for
what is important for their career.
(27:14):
And the reason why sometimes thingsdon't score as well is because they just
don't think that that meets what theapplicant needs to further their career.
Lauren Ullrich (27:25):
Right.
And resubmission is a chance to get thoseweaknesses addressed and commitments
to you and your training in writing.
And once they're in writing, in agrant, then we at NIH through the
progress reports have some authorityto make sure that those things happen.
But if they're not in writingthen we can't help you as well.
(27:53):
So thank you all so much forcoming on the podcast today.
And can I ask each of you for one lastpiece of parting advice for applicants?
Shanta Rajaram (28:03):
Sure.
Um, if I were to pick the top threethat significantly influenced review
outcome, my top three woul (28:08):
make sure that the mentor and the trainee
have read each others' portions of theapplications and the application clearly
states, mentor really states what thementee can take away with him or her.
Another issue is the descriptionof the actual training plan.
It must fill specific gaps in theapplicant's research experience and
(28:33):
the goals should be specific of thetraining plan should be specifically
linked to the proposed research plan.
The last point amongst my top threewould be the mentoring team should
be well suited to the proposed plan.
If your primary mentor doesn't haveexperience mentoring candidates who have
moved on to independent positions, makesure there is a co-mentor who's senior
(28:57):
and much more experienced in this area.
If you are a clinician moving towardsa clinician-scientist career, make
sure that if your primary mentoris an academic doctor, make sure
that you have a clinician or aprofessional doctor as a co-mentor.
Essentially, if you don't get fundedthe first time, always try again.
(29:20):
You know, you miss 100% ofthe shots that you don't take.
So carefully look through the critiques,seek advice from others who have both
been successful grantees and as well asexperienced recent reviewers on NIH panels
and think carefully, revise and resubmit.
Sometimes it's simply a matterof knowing what just needs to be
(29:42):
explained better versus somethingthat actually you have to go back
to the drawing board and readdress.
Ernie Lyons (29:49):
And I agree with all of those.
I would add, as Shanta said,it's very helpful to show your
application to colleagues.
Maybe even besides your mentor,someone who's had experience on
review panels, start early and readthe funding opportunity announcement
carefully that you're applying.
Too often we see applications comein where it appears that they've left
(30:09):
things out or just maybe haven't readthe opportunity announcement or don't
understand it and have submitted tothe sort of to the wrong thing and
if you aren't clear about this oreven in general, talk to the NIH,
talk to your program directors,folks like Lauren and Marguerite.
They're here to help you andguide you through the process.
Lauren Ullrich (30:27):
That's our tagline of this podcast.
Marguerite Matthews (30:29):
Yes.
Ernie Lyons (30:30):
Make use of them.
Bill Be (30:32):
and going last, they've already stolen some of my thunder but,
Ernie I know just indicated to have otherpeople read your application before you
submit and it's really important to getpeople that do not know what you're doing
or not know enough about what you'redoing because they can fill in the gaps
where things are not written clearly.
(30:52):
And for training applications, I wantedto add: specifically, very clearly
didactically state how the researchplan addresses the gaps in your career
development or your training plan.
Be very, I mean don't, don't assume thatthe reviewers will link those together
because they want the research planto support the gaps in your training
that you're trying to fill and howthis research will accomplish that.
Lauren Ullrich (31:16):
And Marguerite, do you have advice?
Marguerite Matthews (31:18):
Yeah, I'd like to build on something
Shanta mentioned earlier.
Um, think about serving as a peerreviewer for your colleagues and other
folks that you're with so you get tobe in the reviewer seat and you get
to see their application and reallybe thoughtful about what's in the
criteria for the funding announcementand have they addressed all of that.
(31:38):
And you can give critical feedback thatmay help them think about have they
reached all the points that reviewersshould know about their application?
And again, you serving as a reviewerwill help you be a better grant writer.
Lauren Ullrich (31:50):
That's exactly what I was going to say.
Marguerite Matthews (31:52):
See now we're sharing a brain.
Lauren (31:53):
Great minds think alike.
But just to build on that, I think there,there are opportunities to serve as a
reviewer even as like, a graduate student.
So, for example, at Georgetown we starteda very small scientific grant program that
was completely run by graduate students.
So you would submit, you know, a smallproposal and graduate students would
(32:15):
review, have a discussion--we modeledit exactly after NIH--and then you could
get, you know, a couple thousand dollars.
And so there's opportunities like that.
Oftentimes like travel grants or throughlike post-doctoral offices and things
like that, they're looking for reviewers.
So even if you're not at the stage whereyou can review at NIH, you can still
get the perspective of what reviewersare looking for and other opportunities.
(32:44):
So that's all that we have timefor today on Building Up the Nerve.
Thank you again to our guests thisweek for sharing their expertise and
thank you to program director BobRiddle for our theme song and music and
we'll see you next time when we tackleprogram review and recommendation.
You can find past episodes of thispodcast and many more grant application
(33:05):
resources on the web at ninds.nih.gov.
Marguerite Matthe (33:09):
Email us with questions at nindsnervepod@nih.gov and
be sure to subscribe to the podcaston Apple Podcasts or your favorite
podcast app of choice so that you don'tmiss an episode of this great content.
We'll see you next time.
Welcome to the National Institute of Neurological Disorders
and Stroke's Building Up the Nerve,a podcast for neuroscience trainees
that takes you through the life cycleof a grant from idea to award at NINDS
with the people who make it happen.
We know that applying for NIH fundingcan be daunting, but we're here to help.
It's our job.
Marguerite Matthews (00:22):
Hi, I'm Marguerite Matthews, a health
program specialist at NINDS.
Lauren (00:26):
and I'm Lauren Ullrich, a scientific program manager at NINDS
and we're your hosts for this episode.
So last time we discussed submittingyour application and receipt and
referral for assignment to an Institute.
Today we're going to talk aboutwhat happens during review.
Marguerite Matthe (00:41):
And as always we want to provide a disclaimer that
everything we talk about in thispodcast may only be relevant to NINDS.
And if you're applying to a differentNIH Institute or center, it's best to
check with them about their policies.
Lauren Ullrich (00:54):
So I was really lucky in my graduate work--I went to
Georgetown, so we are very close toNIH--and we had mock study sections
at our first year with actual SROs.
It really was central to opening myeyes to how the process works because I
think there's nothing like experiencingit to really understand what goes
through the mind of a reviewer.
(01:15):
And Marguerite, did you have anyexperiences like that in grad school?
Unfortunately not as a trainee.
The NIH review process was completelyforeign to me and since I didn't submit
an F31, I never got a chance to getany summary statement or reviews back.
So I didn't learn about the NIH reviewprocess until I started working at NIH.
(01:35):
Yeah.
And I think unfortunately that's alot of trainees' experience, so...
Marguerite Matthews (01:39):
So that's why we're talking about it today!
Lauren Ullrich (01:41):
Exactly.
Marguerite (01:41):
Have a lot in store.
Lauren Ullrich (01:47):
Joining us today are Dr Ernie Lyons, chief of NINDS's
scientific review branch, Dr ShantaRajaram, scientific review officer who
runs the NINDS clinical trials NSD-Kstudy section and Dr Bill Benzing,
scientific review officer who runsthe NST-1 study section and other
(02:08):
training-related special emphasis panels.
So let's start with introductions.
Ernie Lyons (02:13):
As you mentioned, I'm the chief of review here.
So I manage the review branch and I manageall of the assignment of all the incoming
applications to the study sections andother receipt and referral processes.
I help manage developing our FOAsand setting up receipt dates.
Just, you know, make sure thatall our reviews follow all
the rules and regulations andare conducted appropriately.
(02:36):
And I've worked here since2001 so quite a while.
I was an SRO for 10 yearsbefore I became chief.
And before I started here I was aresearch associate at a university
and you know, academic research.
I had small grants and private grants.
I did write some NIHgrants; I never got funded.
My hobbies, I have several hobbies.
(02:56):
I like to garden and I like to play music.
Shanta Ra (02:59):
I, as you said, I'm a scientific review officer here at NINDS.
I have been in some waysa jack-of-all-trades.
I've done all sorts of reviews through my15 years at NINDS, but I primarily manage
the reviews of our clinical programs.
Uh, in addition to that, I'm involvedin initiatives such as the BRAIN
(03:20):
and HEAL and I've done a few oddtraining reviews here and there.
What I did before I started here,before I started at NINDS, I was in the
intramural research program at NIMH.
That allowed me an opportunity to do adetail in, in a branch that I might be
interested in, um, for as a future career.
(03:41):
And that's how I ended up, uh, as adetail yet at NINDS and ended up getting
a job as an SRO here 15 years ago.
Passions outside of workis actually rescuing dogs.
Um, I have two of them at home rightnow, but I'm very much involved
in the volunteer organizationthat rescues those animals.
Lauren Ullrich (04:01):
Nice.
I want to see pictures.
Marguerite Matthews (04:03):
Yeah, same.
Shanta Rajaram (04:06):
Absolutely
Bill Benzing (04:06):
I'm a scientific review officer at the NINDS as well.
And, uh, I've been at the NIH as longas Ernie actually almost 20 years.
And I was at CSR, center for scientificreview, first for a while, and then
I came to NINDS about 13 years ago.
Uh, what I did before here, I workedin academia and then in biotech.
(04:28):
My research focus was age-relatedneurodegenerative disorders.
As far as hobbies or passions, obviouslymy kids and gardening and music and
I've rescued a few dogs for my own sake.
Dogs are a large part of my life as well.
Marguerite Matthews (04:51):
All right.
So can one of you tell us aboutthe relationship between the center
for scientific review and reviewsthat are done at the IC level?
Er (04:59):
So the center for scientific review, as you mentioned, is one of
the 27 institutes or centers at NIH.
And they handle the majority ofreviews at NIH, about 75% of the
applications are reviewed there.
And just in general you could say theyreview the more standard applications
that are typically submitted inresponse to parent funding opportunity
(05:21):
announcements, like the standard R01,just regarded as the gold standard.
I think it's the most common NIHapplication and it's, as I said, sort of
a gold standard at academic institutionsto have your lab funded through an R01.
They also review R21s, R15s, smallbusiness, so many other mechanisms.
But institutes as well have theirown review branch, which tend to,
(05:45):
and again this is general, reviewapplications that are submitted in
response to more specialized fundingannouncements and often have um,
institute-specific review criteria oreligibility requirements or definitions.
For example, translationalresearch may be defined differently
at different institutes.
So often, uh, we review that.
(06:06):
We have a translational reviewprogram that reviews our translational
research, clinical research,training, as you'll hear a lot about.
And as well as many RFAs and otherprograms like our centers awards,
centers grants, core grants, andthose kinds of complex mechanisms.
Lauren Ullrich (06:23):
That's a lot of different kinds of mechanisms.
I don't think when I was a trainee, anyof those things were really on my radar.
And so specifically for trainingat NINDS, which training mechanisms
are reviewed at CSR and whichones are reviewed by us at NIDS?
Bill Be (06:38):
Um, the ones that we do in NINDS review branch, uh, specifically
for training, while we do review othertypes of grant mechanisms, are the F32s
the various K01 mechanisms, clinicianscientists, independent junior faculty
K02, and the clinician scientists mentoredtraining programs, K08, K22, and K23.
(07:01):
Uh, we also review, um, themore departmental type training
programs like T32s and summerresearch programs like R25s.
Marguerite Matthews (07:10):
Once I've submitted my application, um,
where can I check exactly wheremy application is being reviewed?
Shanta Raj (07:16):
So you can check the locus of review for your application
in your commons account--in NIH ERAcommons account once you've submitted it.
An alternate place to look, althoughspecific review panel may not be listed
at that spot, a good place to check isalso to look under the agency contacts
listed under the funding opportunityannouncement for your review locus.
Ernie Ly (07:40):
And in section five of the FOA, section five number 2 lists
the locus of review, so you can knowahead of time where it's going to go.
Lauren Ullrich (07:48):
Right.
It's kind of buried in there a little bit.
Hard to find if you don'tknow what you're looking for.
And specifically if I want to know whatstudy section my application will be
reviewed in, where do I go to check that?
And also who is going tobe on the review panel?
Ernie (08:04):
I believe the easiest way to know your study section assignment is:
it's posted in your ERA commons accountonce it's assigned to a study section and
I think there may be even a link thereto the roster to see who's on the roster.
Rosters typically become available 30days before the study section meets.
That would be the meeting roster.
(08:25):
You can also preview the committeemembers for standing committees,
you know, to see who the regularmembers are on the NIH website.
There's a helpful tool in, in the NIHreporter called the matchmaker tab and
you can paste in a keywords or an abstractand it will match up a potential study
sections and you can actually suggesta study section when you submit your
(08:48):
application either in the cover letteror on the assignment request form that's
available during the submission process.
Marguerite Matthews (08:55):
And how are reviewers chosen to review applications?
Bill (08:59):
Well, a number of factors go into how we choose reviewers.
At the level of, I'm looking forreviewers for an individual application
or a group of applications or fora whole study section, we typically
look for individuals, definitelywith a doctoral degree or equivalent.
Most importantly, we look fordemonstrated scientific expertise in the
relevant area of either an individualapplication or a group of applications.
(09:20):
One of the criteria that welook at is a track record of
independent research support anddemonstrated breadth of perspective.
When we get to know reviewers, we reallywant reviewers that can provide an
overall broad objective perspective,free of bias and we want reviewers that
can exercise sound and mature judgment.
And most importantly individualreviewers should also be free of personal
(09:42):
professional or financial conflicts ofinterest to review the set applications.
At the level of entire review panel,additional factors go into creating
the makeup of an of a whole panel andthese include geographic distribution
of the reviewers, and some other factorsthat get factored into those things.
Marguerite Matthew (09:57):
So you don't purposely pick investigators who are
going to be the third reviewer tobe out to get a given application?
That's not how it works?
[Laughter]You're not looking for people to
be extremely harsh in their review?
Cause I think as trainees especially, wesee what our mentors and other faculty
are going through and we just assumethat reviewers are purposely picked
(10:18):
to just be mean and give bad reviews.
Shanta Rajaram (10:21):
And actually we come to some of this in one of
your questions in the sense thatapplicants are almost always wrong
when they try to guess which reviewerson the panel was out to get them.
The support for your application maycome from the direction you least expect.
So I always advise applicantsdon't guess and certainly don't
(10:43):
assume the worst based on a roster.
Erni (10:45):
And I would add just in my long experience
really aren't out to get anyone.
I think the important thing to rememberis that study sections are made up of
scientists, they're all humans and webasically ask them for their opinions.
Maybe, I think overall the processworks very well and it's probably the
best we have, but that doesn't meanthey're always perfect or they always
(11:09):
give us the right answer or the rightopinion or you might not agree with
some of the things that they say, but...
Marguerite Matthews (11:14):
Right.
Lauren Ullrich (11:15):
Earlier we mentioned standing study section.
And so what is the difference betweena standing study section and a
special emphasis panel and how, um,how is it decided what applications
go to which kind of review panel?
Shanta Raja (11:29):
So a standing study section has appointed members.
Usually it's anywhere between 16 to 24and they are appointed for terms that are
typically between four or six years long.
The standing study sections usually meetthree times a year and as additional
expertise not provided by the standingmembers is required based on applications
(11:54):
that get assigned to the study section, adhoc expertise gets recruited each round.
A special emphasis panel or aSEP as NIH tends to call it--one
of its millions of acronyms--
Laur (12:05):
Yeah we love our acronyms.
Shanta Rajaram (12:07):
Is assembled de novo each time.
It is usually used for reviewingapplications in response to say to request
for applications commonly known as RFAs.
These are typicallyone time solicitations.
So these are the kinds of initiativesthat get reviewed in special emphasis
panels or they may be used for specialprograms that might require unique
(12:30):
review criteria, for example, or verycommon other use of special emphasis
panels is to review applications frommembers of standing study sections.
The so called member conflict SEPs.
In terms of deciding whichapplications, I think I talked a
little bit about how applicationsor which applications typically get
assigned to special emphasis panels.
(12:52):
In study sections they could bedivided by a number of different ways.
In NINDS we have different studysections for clinical, translational,
and training applications.
And each of these, even between thetranslational applications, the two study
sections have divided based on areasof science that NINDS funds, that are
(13:14):
divided between the two study sections.
Marguerite Matth (13:17):
And ultimately how does this influence the
actual review of applications?
Ernie L (13:22):
I think in general, the process between the two types of study
sections, standing committees or specialemphasis panels, is basically the same.
As Shanta mentioned, SEPs areusually created in response to
special, you know, typically one timeannouncements and they're tailored
specifically for that review.
(13:43):
Standing committees tend to review thingsthat come in on a regular basis like
many of our training awards that we have.
They come in three times a year,and so we have two standing study
sections that review our training.
I think that the difference thereis that a standing committee
provides some continuity andconsistency from round to round.
Those study sections develop a culture,they become familiar with the training
(14:05):
mechanisms and hopefully providesome consistency in their scoring
behaviors across different rounds.
Laur (14:12):
So shifting gears a little bit, in terms of the application, are
applicants allowed to send any updates ontheir application after it's submitted?
Bill Benzing (14:23):
Yes.
Up to 30 days prior to the review meeting.
You're only allowed to submit itemsthat are allowed in the NIH policy
for post submission materials, unlessthe funding opportunity announcement
that you're applying to, the FOA,states otherwise that no post-mission
material can be submitted orotherwise allows specific additional
materials than in the standard policy.
(14:45):
The NIH policy on post submissionmaterials can be found at
the NIH notice NOT-OD-19-083.
Lauren Ullrich (14:56):
And what kinds of materials generally
are allowed by the policy?
Bi (15:00):
There's a handful of things.
But the most common thing are acceptedpublications or recently published
manuscript that have become publishedsince application has been submitted.
Shanta Rajaram (15:09):
And what is most commonly not allowed and which we
get the maximum questions for, isanything that is a fixing of an error
that you made during submission.
So check the final form of theapplication once the PDF is in.
Ernie Lyons (15:23):
And some of our announcements do allow late
submission of um, of recent data.
So if, if you have more preliminarydata that would support your
application, some of our FOAs allowlate breaking data to be included.
Bill Benzing (15:38):
That would be listed specifically in the
funding opportunity announcement.
Lauren Ullrich (15:40):
So read the FOA.
Carefully.
Shanta Rajaram (15:43):
Absolutely.
Marguerite Ma (15:44):
So now that we've established that selecting reviewers
is not quite the hunger games, whatactually does go on during the review?
Shanta Raj (15:52):
The process actually starts a lot earlier than the
actual reviews; about four to sixweeks prior to the actual meeting.
Each application is assigned to a numberof reviewers, um, typically three, but
the more complex the application, the morethe number of reviewers assigned to it.
The reviewers get access, as I said, tonot just their assignments but also to
(16:14):
review instructions, relevant guidelinesthrough a secure web interface that
NIH uses for this for this purpose.
About a few days before the actualreview the reviewers are expected to
upload preliminary scores and critiques.
Now if it's a special program orspecial initiative with special review
(16:35):
criteria, as we said, there mighteven be a reviewer orientation call to
make NIH ensures with involvement ofboth program and review staff on those
calls that all reviewers are on thesame page in terms of understanding
the goals of that specific program.
Once the reviewers have postedtheir own critiques and scores,
(16:55):
they are also allowed to read thecomments posted by others assigned
to review the same application.
So there is a little bit ofan understanding of where the
others are coming at when thediscussions actually start.
At the meeting, there is a chairpersonwho facilitates discussions.
Discussions very typically focuson the strongest half of the
(17:17):
applications that have been assignedto that study section or that panel.
Reviewers take turns stating thepreliminary scores and going over
their main score driving issues.
We try to get them to focuson both the strengths and the
weaknesses in the application.
All non-conflicted members are theninvited to join the discussion once the
(17:40):
assigned reviewers have had their say.
Once it appears that the scoredriving issues have been hashed
out completely in a thorough discussion,the chair will then specifically invite
comments on other score driving criteriasuch as human subjects, protections,
vertebrate animals, inclusion policies,biohazards, so on and so forth.
(18:03):
Um, once this is done, the reviewchair person asks for final scores.
The assigned reviewer state their finalscores, which sets the score range,
and reviewers are asked to vote theirconscience based on that score range.
Now our reviewers in the room may chooseto vote outside the score range, but any
intent to do so must be declared publiclyat the meeting when the chair asks.
(18:25):
Lastly, the chair will ask forcomments or concerns on any
non score influencing elements.
These are administrative elementssuch as budget, if the organization
requesting funding is a foreign entity,issues such as resource sharing plans,
authentication plans for biologicalchemical resources, et cetera.
(18:47):
And that typically ends the reviewdiscussion for an application.
Lauren U (18:52):
One of the things that we get questions about a lot is this
difference between the different criterionscores that you might see on your
summary statement and the overall score.
So how do reviewers comeup with that overall score?
Do they just average the fivecriterion scores or how does that work?
Bill Benzing (19:10):
Well, um, let me give a little bit of background
on just the scoring first.
The NIH uses a nine point scoringsystem with one being the best
score and nine being the worst.
And most research grant applicationsreviewed at the NIH are evaluated
on five core review criteriaof which you're alluding to is
the these individual criterion.
(19:31):
Um, these are typically significance,investigator, innovation,,
approach and environment.
For training and careerdevelopment applications.
These individually scoring criteria aredifferent and may include applicant,
sponsors, which is also mentors,collaborators, consultants, research plan,
training, plan, training, potential, andinstitutional environment and commitment.
(19:54):
Each of these get an individual score,which is called a criterion score.
The overall impact score, which is anoverall opinion of of the uh, application
by a reviewer is an assessment of thelikelihood of the project to exert a
sustained powerful influence on theresearch fields involved or the training
of the individual applicant if it'sa training grant, and its determined
(20:16):
by reviewers by considering the fiveindividually scored criteria, but it's
weighted on the overall reviewer judgment.
And the overall score includes humansubjects issues, animal welfare
concerns, and things like that as well.
Laure (20:30):
And those are things that they're not individually scored, but they
can count towards the final overall score
Bill Benzing (20:35):
Right.
If there's issues with humansubjects, issues with the scientific
justification for using the humanssubjects or the animals, um, those
can affect the overall score.
So the overall score is an impact scoreof the whole application and individual
criteria and scores are a judgment eachof those different individual criteria.
Marguerite (20:54):
So the final number that applicants see is the impact score.
How is that calculated?
Bill (20:59):
The overall impact scores provided by all reviewers in the study
section are averaged and then roundedmathematically to one decimal place.
And then this is multiplied by 10 togive you the two digit impact score.
So if something had an average of sayhalf the panel or roughly half the
panel gave 1s to an application andthe other half gave 2s that yielded
(21:20):
an average of say 1.5, it wouldyield an overall impact score of 15.
Lauren Ullrich (21:25):
And in addition to the impact score, many summary
statements also provide a percentile.
And what is that and how is it calculated?
Ernie Lyons (21:33):
So a percentile is essentially just a way
of ranking the applications.
It's a ranking mechanism andit's expressed as a percentile.
And it's really the rank of yourapplication's score divided by
the total number of applications.
So for example, if there are hundredapplications and your score ranked
(21:54):
10th on the list, it was the 10th best,you would receive a 10th percentile.
And that's a little bit ofan oversimplification, but
that's essentially what it is.
But importantly, how we use it andwhat it allows us to do is to compare
scoring or ranking across differentstudy sections so we can fund the same
percentage of applications, the top 15%for example, from each study section.
(22:17):
So where study sections may haveslightly different scoring behaviors,
we're still funding the samepercentage from each study section.
And percentiles are also calculated acrossthree council rounds, which helps to
account for round to round variability.
So if you get a particularly strongset of applications, one round and a
weaker set the other round, that tendsto normalize over the three rounds.
Shan (22:40):
I would like to add though that NINDS does not percentile any of
the applications, either in our studysections or special emphasis panels.
So really reading the summary statementand looking at your score and what
those numbers mean in terms ofdescriptors is usually very helpful.
Laur (22:59):
Yes, that is a very common question that we get because most of our
training grants are reviewed in house andso most of them don't get a percentile.
And I think a lot of mentors, they'reused to just zeroing in on that percentile
and disregarding the impact score.
And so when a trainee gets theirreview back and it only has the impact
score, they're a little bit at a lossof how to, um, how to interpret that.
Marguerite Ma (23:22):
How do applicants know the outcome of their review?
Shanta Rajaram (23:26):
So the score and the summary statement are
both posted in the ERA commons.
And again, the applicants can access itby signing into their commons accounts.
Typically applicants receive an automatedemail notification as soon as their
scores are released, but we alwaysrecommend that you check periodically.
(23:46):
The scores are released within acouple of days of the study section,
of this review panel meeting.
The summary statements can takelonger, anywhere from four weeks from
the review date to a month beforethat particular council review round.
So we always ask that applicants checktheir commons accounts periodically
(24:07):
because there's always technical glitchesdue to which that automated email
may never actually make it to them.
Lauren Ullrich (24:14):
And if you have questions about the review,
who is the best contact person?
Bill Benzing (24:20):
This is a question that often gets misunderstood.
If its prior to the review andprior to the score coming out, the
person to contact is the scientificreview officer, which can be found
in the review section related to theapplication in your commons account.
Prior to the review, anything that'srelated to submission of additional
(24:41):
materials or other questions should bedirected to the scientific review officer.
If it's post-meeting and the scoreshave been out and/or the summary
statement is out, they should becontacting their program director
or their assigned program officer.
Marguerite Matthews (24:55):
Are there opportunities for early career
scientists to participate in review?
Shanta Rajaram (24:59):
Yes, absolutely.
In fact, most SROs welcome theperspective that junior investigators
often bring to these panels.
The center for scientific review, whichyou've heard of--us mention several
times now, it actually has an earlycareer reviewer program, so called the
ECR program, that is designed to helpemerging researchers advance their careers
(25:23):
by exposing them to review early on.
In fact, I always tell applicants thatone of the best ways to improve your grant
writing skills is to serve as a reviewer.
So I encourage applicants to both review,uh, both register for the ECR program
or if they are interested in an ICreview panel, to just send their CVS,
(25:43):
reach out to the chief of the scientificreview branch and send us your bios.
Lauren Ullrich (25:48):
And is there anything, common misconceptions,
things that you wish that applicantsunderstood about this process?
Ernie (25:56):
So what I was planned to say here, I, I've already mentioned
a little bit (25:59):
to remember that the study sections are comprised
of scientists who are humans.
They have a lot of grants to read.
They're not going to geteverything right all the time.
But for the most part,their intentions are good.
They're trying to do their best toidentify the best science and discriminate
between large numbers of applications.
It's very stressful for reviewers;they have to present critiques of
(26:22):
multiple applications in front of theircolleagues, in front of their peers.
And for the most part, again, they'rejust doing their best to try to review
these as accurately as they can.
But it's, you know, it'scertainly not a perfect process.
Marguerite Matthews (26:36):
And I think especially for training
opportunities, reviewers reallywant the best for the trainee.
So they really look for opportunitieswhere the trainee has, has highlighted
some great things and often wantto highlight the weaknesses more
as an opportunity for growth forboth the trainee and the mentor.
Shanta Raja (26:53):
I absolutely agree.
I sometimes think that, especially forsome of our training study sections,
if the applicant could be a fly onthe wall when the study sections
actually go on, they would really geta flavor for how much the reviewers
actually look out for these--for thetraining and for the applicant and for
what is important for their career.
(27:14):
And the reason why sometimes thingsdon't score as well is because they just
don't think that that meets what theapplicant needs to further their career.
Lauren Ullrich (27:25):
Right.
And resubmission is a chance to get thoseweaknesses addressed and commitments
to you and your training in writing.
And once they're in writing, in agrant, then we at NIH through the
progress reports have some authorityto make sure that those things happen.
But if they're not in writingthen we can't help you as well.
(27:53):
So thank you all so much forcoming on the podcast today.
And can I ask each of you for one lastpiece of parting advice for applicants?
Shanta Rajaram (28:03):
Sure.
Um, if I were to pick the top threethat significantly influenced review
outcome, my top three woul (28:08):
make sure that the mentor and the trainee
have read each others' portions of theapplications and the application clearly
states, mentor really states what thementee can take away with him or her.
Another issue is the descriptionof the actual training plan.
It must fill specific gaps in theapplicant's research experience and
(28:33):
the goals should be specific of thetraining plan should be specifically
linked to the proposed research plan.
The last point amongst my top threewould be the mentoring team should
be well suited to the proposed plan.
If your primary mentor doesn't haveexperience mentoring candidates who have
moved on to independent positions, makesure there is a co-mentor who's senior
(28:57):
and much more experienced in this area.
If you are a clinician moving towardsa clinician-scientist career, make
sure that if your primary mentoris an academic doctor, make sure
that you have a clinician or aprofessional doctor as a co-mentor.
Essentially, if you don't get fundedthe first time, always try again.
(29:20):
You know, you miss 100% ofthe shots that you don't take.
So carefully look through the critiques,seek advice from others who have both
been successful grantees and as well asexperienced recent reviewers on NIH panels
and think carefully, revise and resubmit.
Sometimes it's simply a matterof knowing what just needs to be
(29:42):
explained better versus somethingthat actually you have to go back
to the drawing board and readdress.
Ernie Lyons (29:49):
And I agree with all of those.
I would add, as Shanta said,it's very helpful to show your
application to colleagues.
Maybe even besides your mentor,someone who's had experience on
review panels, start early and readthe funding opportunity announcement
carefully that you're applying.
Too often we see applications comein where it appears that they've left
(30:09):
things out or just maybe haven't readthe opportunity announcement or don't
understand it and have submitted tothe sort of to the wrong thing and
if you aren't clear about this oreven in general, talk to the NIH,
talk to your program directors,folks like Lauren and Marguerite.
They're here to help you andguide you through the process.
Lauren Ullrich (30:27):
That's our tagline of this podcast.
Marguerite Matthews (30:29):
Yes.
Ernie Lyons (30:30):
Make use of them.
Bill Be (30:32):
and going last, they've already stolen some of my thunder but,
Ernie I know just indicated to have otherpeople read your application before you
submit and it's really important to getpeople that do not know what you're doing
or not know enough about what you'redoing because they can fill in the gaps
where things are not written clearly.
(30:52):
And for training applications, I wantedto add: specifically, very clearly
didactically state how the researchplan addresses the gaps in your career
development or your training plan.
Be very, I mean don't, don't assume thatthe reviewers will link those together
because they want the research planto support the gaps in your training
that you're trying to fill and howthis research will accomplish that.
Lauren Ullrich (31:16):
And Marguerite, do you have advice?
Marguerite Matthews (31:18):
Yeah, I'd like to build on something
Shanta mentioned earlier.
Um, think about serving as a peerreviewer for your colleagues and other
folks that you're with so you get tobe in the reviewer seat and you get
to see their application and reallybe thoughtful about what's in the
criteria for the funding announcementand have they addressed all of that.
(31:38):
And you can give critical feedback thatmay help them think about have they
reached all the points that reviewersshould know about their application?
And again, you serving as a reviewerwill help you be a better grant writer.
Lauren Ullrich (31:50):
That's exactly what I was going to say.
Marguerite Matthews (31:52):
See now we're sharing a brain.
Lauren (31:53):
Great minds think alike.
But just to build on that, I think there,there are opportunities to serve as a
reviewer even as like, a graduate student.
So, for example, at Georgetown we starteda very small scientific grant program that
was completely run by graduate students.
So you would submit, you know, a smallproposal and graduate students would
(32:15):
review, have a discussion--we modeledit exactly after NIH--and then you could
get, you know, a couple thousand dollars.
And so there's opportunities like that.
Oftentimes like travel grants or throughlike post-doctoral offices and things
like that, they're looking for reviewers.
So even if you're not at the stage whereyou can review at NIH, you can still
get the perspective of what reviewersare looking for and other opportunities.
(32:44):
So that's all that we have timefor today on Building Up the Nerve.
Thank you again to our guests thisweek for sharing their expertise and
thank you to program director BobRiddle for our theme song and music and
we'll see you next time when we tackleprogram review and recommendation.
You can find past episodes of thispodcast and many more grant application
(33:05):
resources on the web at ninds.nih.gov.
Marguerite Matthe (33:09):
Email us with questions at nindsnervepod@nih.gov and
be sure to subscribe to the podcaston Apple Podcasts or your favorite
podcast app of choice so that you don'tmiss an episode of this great content.
We'll see you next time.
Popular Podcasts
Joy 101 with Hoda Kotb
Joy is essential. And it's also elusive. You can't order it, borrow it, or simply hope it into life. But now, there's a new and exciting way to start your journey toward a more joyful existence: The Joy 101 Podcast with Hoda! Best known for her Emmy-winning work and co-anchoring Today, Hoda Kotb infuses her authenticity, curiosity, and warmth into conversations with the world’s most fascinating people. Entertainment legends, sport icons, wellness experts, and everyday folks will share how they find, allow, and experience joy. Hoda will offer her own tips and takes on seeking a more balanced, harmonious life. If you're craving inspiration, support, and useful tools to maximize your joy, tune in to these candid, uplifting, and moving on-air chats. Joy after a breakup, joy as an empty-nester, joy after loss, joy as a caretaker — Hoda's new podcast will speak to you. Joy 101 with Hoda Kotb, an iHeartPodcast.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com