Elements of a Successful NIH Grant Application: Peer Review Perspectives

1
Elements of a Successful NIH Grant Application
Peer Review Perspectives

• Society for Personality and Social Psychology
• 22 February 2005

2
1. Principal Components of Peer Review, Michael
Micklin, Center for Scientific Review,
NIH 2. The Social Psychological Imagination
Elements of a Compelling Argument for Research
Funding, Joel Cooper, Princeton
University 3. Similarities and Differences in
Writing Successful Articles and Grant
Applications, Harry Reis, University of
Rochester 4. Grant Application Preparation
Skills and the Graduate Training Program, Jenny
Crocker, University of Michigan. 5. Making Use
of Reviewer Critiques The Successful Amended
Application, Donal Carlston, Purdue
University 6. Assigning Reviewers to Grant
Applications An SRA Perspective. Anna Riley,
Center for Scientific Review, NIH
3
Center for Scientific Review

• Responsible for review of 70 of the research
  grant applications submitted to NIH
• 4 Divisions, each with own Director
• 22 Integrated Review Groups (IRGs), each with own
  Chief
• 5-10 study sections/IRG, each with own SRA
  (Scientific Review Administrator)

4
Study Section Organization

• Study section vs. Special Emphasis Panel
• Study sections meet 3 times/year
• SRA manages study section
• 16-24 study section members (4-year terms one
  member serves as Chair)
• Temporary members vary each meeting
• Typical study section reviews 60-100 applications
  per meeting

5
Pre-Meeting Review Process

• SRA assigns appropriate reviewers to each
  application
• Conflicts of interest identified
• Applications sent 6 weeks prior to meeting
• Reviewers post critiques and provisional scores
  on NIH web site 3-4 days prior to meeting
• Reviewers read colleagues critiques (excluding
  conflicts) prior to meeting

6
Meeting Review Process I

• Approximate lower half applications identified,
  not discussed (streamlined)
• Remaining applications discussed in order
• Typically 3 reviewers
• Conflicts absent from room
• Reviewers assess adequacy of inclusion of women,
  minorities, children protection of human
  subjects
• Assigned reviewers establish range of priority
  scores, but all present vote
• Confidentiality of review a high priority

7
Meeting Review Process II

• Standard review criteria significance approach
  innovation investigator environment
• Human subjects inclusion and protection can
  affect the score
• Priority scores can range from 1.0 to 5.0
• CSR median is 2.5
• Scores in regular study sections percentiled
  against 2 prior study section meeting
  distributions
• Special consideration for new investigators
  (no prior R01 or R29 as PI) R01 submissions
• Unpaid applications, including those streamlined,
  can be resubmitted twice

8
Post Meeting Review Process

• Reviewers have opportunity to revise written
  critiques based on discussion reading of
  colleagues critiques
• SRA writes Resume Summary of Discussion for
  scored applications summary statements
• Unscored applications receive critiques only
• Summary statements available within 4-6 weeks
• For post meeting information
• Contact IC program official, NOT SRA

9
The Social Psychological Imagination Elements of
a Compelling Argument for Research Funding

• Joel Cooper
• Princeton University
• NIH (and NIMH) review panel retired

10
Reviewers review a proposal

• A proposal for grant funding is a matter of
  persuasion.
• The proposal needs to be sold to a group of
  mostly smart, mostly sympathetic, and mostly
  overworked reviewers.
• There is very little room for peripheral route
  persuasion in a grant proposal

11
(No Transcript)
12

• This is central route stuff. Make the words
  matter!

13
How we write reviewsPersuasion and the priority
score

• What are the criteria that the committee
  considers to assign priority scores?
• The approach. Reviewers will carefully analyze
  the methods and techniques you use. Do the
  methods adequately address the issues you raised?
  Are the plans for quantitative analysis
  sufficient?
• But There are two other criteria that are
  crucial for a good score that emphasize the
  imagination and creativity you bring to the
  request
• Significance and Innovation

14
What sells? What convinces the readers that the
proposal is significant and innovative?

• Excite the readers about the importance of the
  topic
• Excite the readers about the innovative new
  methods you have devised for studying a topic
  better
• Speak to the Big Picture
• Convince the reader that there is something new,
  broad and interesting in the research you will
  propose.

15
Where is the big picture communicated?

• In an NIH grant application, you will write an
  abstract
• You are also asked to write a bizarre-sounding
  section on Specific Aims
• This is the primacy effect in persuasion. As
  readers, our interest is piqued by these early
  sections.
• You can get us back later
• You can lose us later
• But this is the primary opportunity to convince
  the reader that the proposal is unique,
  imaginative and significant.

16
You tell the reviewers

• Why you think your total body of work is new,
  important, integrative.
• Do it early and do it often.
• Dont ask the reviewers to make your case for
  you. Its your research area. You know why its
  important, groundbreaking or innovative. You
  should tell the reviewers (while avoiding the
  impression of arrogance.)

17
(No Transcript)
18
The Seven Deadly Assumptions of Grant
Applications How Grant Writing is, and is not,
the Same as Writing for Journals
Harry T. Reis, Ph.D. University of Rochester
Fresh meat, Center for Scientific Review Old hat,
Journal Reviewing
19
Same!
1. Both require a compelling idea with a strong
scientific foundation
? A topic that engages the readers interest and
imagination
? A strong theoretical argument that is squarely
grounded in the existing literature but that
builds upon this literature to advance knowledge
? Methods that are reasonably free of
methodological artifacts and alternative
explanations
? Analyses that are appropriate to the questions
at hand -- i.e., that neither trivialize nor
baffle with bull
20
Different
2. Grants are promissory notes journal articles
are notes paid off
? Grants sell feasibility journal articles sell
credibility
? A grant application is a delicate balancing
act. It must be ambitious and realistic at the
same time one must convince the reader that the
research can be done and that the methods will
work -- trust my track record and were
confident that these methods will work leaves
you dead in the water. But a grant application
must also be inventive enough to do more than
replicate ones pilot study. It must move the
field forward.
? A journal article involves telling a credible
story about work already completed the more
original, the better.
21
May be Different
3. Journal articles may offer small increments
grants should take large leaps
? Journal articles may fill niches in the
literature or improve on existing technology,
samples, and methodological gaps for an archival
science, this is important.
? Grant applications must offer an original
theoretical contribution, must fill a significant
(pknowledge or application to the next step.
? Journal articles can be narrow in focus grant
applications should be broad in conceptual scope
(but not too broad!).
22
Same Different!
4. Grant applications emphasize the big picture
journal articles dot every i and cross every t
? Writing a journal article involves anticipating
every question a reviewer might think of, and
covering nearly every detail that would be needed
to replicate the research.
? Grant applications must do the above, but must
also have a vision about a coherent set of
principles and a diverse program of studies to
examine them. Pilot studies should establish
feasibility, but the program of work must venture
into new territory. The need for the research
and its importance to science and society is
paramount. Both internal and external validity
matter.
23
Same!
5. Write well and follow guidelines
? Write clearly, succinctly, and within
page/space/font guidelines.
? Be sure your excitement shows. If you arent
excited about the work, the reviewers wont be,
either.
? Dont beat around the bush. Get to the point,
quickly. Dont expect the reviewers to figure out
for you why the work might be important.
? Dont write by reviewer -- the draft you
submit should be the best you can make it.
24
Different
6. Itll never get funded
? Many grant applications are never funded, but
most papers find a home in print. This, combined
with the long odds, leads many to not bother or
to be overly discouraged about rejection.
? Grants have a more flexible revise-and-resubmit
process, and therefore leave more room for
improvement. If reviewers criticize a study or
method, a better one can be substituted. With
journal articles, ultimately, what you write is
what you have.
? There are many, often obscure, untapped funding
resources. Look long and hard at websites and
funding directories. Pay attention to feedback
from colleagues and reviewers.
? The best predictor of funding success is the
number of applications submitted. Beware the
self-fulfilling prophecy.
25
Different
7. Journal articles activate avoidance
motivation grants activate approach motivation.
? Journal reviewers tend to look for whats wrong
about an article. Avoid major flaws and
weaknesses and youll be published.
? Grant reviewers and funding panels tend to look
for whats right about an application Why is
the proposed work needed, important, and
exciting? How does your application give the
funders an opportunity to move our science
forward? Where is the originality and challenge?
Give the reviewers and funders a reason to
believe that 20 years from now, theyll be
basking in the reflected glory of having funded
your work.
26
Special Resources
http//www.csr.nih.gov/
http//www.csr.nih.gov/resources.htm
http//www.med.uwo.ca/physiology/courses/survivalw
ebv3/ ArtofGrantsmanship.html
http//www.sfu.ca/ors/forms/pdf/SSHRC.pdf
http//cpmcnet.columbia.edu/research/writing.htm
27
Preparing Students to Write Successful Grant
Applications

• Jennifer Crocker
• University of Michigan

28
Why is that Important?

• Once in academic or research positions, students
  may be expected to get grants as part of their
  job
• To do their work
• It is evidence that their work meets high
  standards
• This can weigh heavily in tenure decisions
• Its good for their department and university
• Indirect costs, increasing number of students in
  the program, etc.
• Many additional benefits

29
Benefits of Applying for Grant Support

• It teaches programmatic thinking
• It demonstrates you value this activity
• Getting feedback from smart, dedicated people
  about your work
• Becoming known to reviewers

30
Benefits of Receiving Grant Support

• Being able to do your work!
• Support for your students
• Travel funds
• Summer support
• Upgrade your lab space, or even access to lab
  space
• Relief from teaching responsibilities
• Credibility

31
How Can We Teach Students this Skill?

• Write applications for external fellowships
• NSF, Ford, Javits, Fulbright, NIMH Predocs
• Involve them in writing applications with you
• Allow extra time to plan, discuss, negotiate
  jointly
• Dont let them take the lead YOU are the one who
  knows best what it takes to get funded
• Seek their input, and be the boss
• Encourage (or require) them to write postdoc
  proposals

32
Thinking Programmatically Start with the Big
Picture

• The compass What do I want to contribute?
• Why is that important to me, to the discipline,
  to the nation or world?
• What gaps in existing knowledge would this fill?
• What is the best way to do that work?
• Without constraints on resources, what would be
  the best possible research to do?
• If the studies work, what do I want to have
  learned?

33
Who Might Fund this Work?

• What agencies, foundations are interested in this
  topic?
• What are their priorities, considerations,
  resources, requirements?
• Call them up! Tell them what you want to do.
• The usual suspects NSF, NIMH (R03, R01, K award)
• Foundations Spencer, Russell Sage, Templeton,
  etc. Ask for help finding sponsors
• Find out what you need to do. Read the review
  criteria.

34
Writing the Proposal

• Write the proposal with agency/foundation
  requirements in mind
• Deadlines
• Page limits
• Budget requirements
• Budget limitations and norms
• How many years of support do you need?
• Its hard to translate from one set of
  requirements to another

35
Deciding on the Studies

• Each study should make a distinct contribution to
  the overall goals
• Conceptual replications alone will seem redundant

36
Writing the Method

• For each study, provide
• Brief rationale Whats the question or
  hypothesis?
• Whats the design? Number of participants? Who
  are they? Why?
• How will you do it? Measures, manipulations
• Justify your choices
• How will you analyze it?
• Search for economy of presentation

37
Obstacles Things that Derail

• Writing to appear smart, well-read
• Writing for your critics
• Writing for specific reviewers (they change!)
• Assuming its obvious
• Assuming they know that you know how to do it
• Being overly ambitious, or underambitious
• Overbudgeting (you look bad, it hurts the field)
• Underbudgeting (you cant do the work for that)

38
Goals that will Help

• Remember what you want to contribute Explain
  that!
• Write for the reader help them understand the
  importance of the work
• Answer their questions, acknowledge limitations
• Keep it clear and simple
• In conclusion, explain the contribution
• Stay in learning mode seek out feedback and use
  it to improve revise and resubmit, or start over

39
Review Proposals

• You will really see how it is done well, and
  badly!
• Its a crucial contribution to the field

40
Good Luck!
41
Making Use of Reviewer Critiques The Successful
Amended Application
Donal E. Carlston Purdue University
Center for Scientific Review Panelist for 9
years, Panel Chair for 3 years Journal of Social
Cognition Associate Editor for 5 years, Editor
for 11 years
42
Investigators get back reviews from the Center
for Scientific Review that they should use in
amending their application.
The revisions process for grant proposals is in
some ways the same as, and in some ways different
from, the revisions process for journal
manuscripts.
Same!
Different
43
The reviewer is always right.
? Every reviewer comment, including those that
are wrong-headed, reflects a possible problem
with the submitted work, or a lack of clarify in
describing it.
Same!
? Authors who ignore reviewer comments miss out
on a valuable opportunity to improve their work.
? And they open themselves to one of the
deadliest criticisms that reviewers can make
that the author has been unresponsive to previous
feedback.
44
Unresponsiveness is potentially fatal.
Same!
? Reviewers want to be appreciated.
? And they dont want to read and criticize the
same problems in round 2 that were identified in
round 1.
? Failure to respond to reviews suggests a narrow
perspective, insularity, arrogance, and/or
incompetence and suggests that a writer is more
interested in beating the system than in
improving the quality of his or her work.
45
Different
But there are differences between resubmissions
to journals and to grant panels.
? Resubmissions to journals must generally be
invited by the editor, whereas investigators
ordinarily get 2 resubmissions.
? Resubmissions to journals almost always are
reviewed by the same people as the initial
submission, whereas resubmissions to grant panels
may be reviewed by an entirely new set of
reviewers .
? Journal editors may give credit for
improvement. Grant panels generally do not.
46
These pose a unique problem for investigators
resubmitting to a grant panel How to handle
revisions
? that may go to different individuals than those
who suggested the revisions in the first place.
? when the investigator is likely to be penalized
for failing to be responsive, but unlikely to be
credited simply for taking advice.
? and where the opportunity for additional
revision is strictly limited, even if each
submission is improving.
47
Somewhat similar
The solution is twofold.
First, improve your proposed research.
? Stay focused on bettering the proposed work.
? But keep in mind that every reviewer
suggestion, including those you dont much care
for, DOES hint at a possible problem.
Second, highlight your responsiveness.
? Describe changes in your proposal, and why you
made them (NOT because you were told to.) Also
address suggestions that you do not adopt, and
explain why.
? Mark those parts of your proposal where you
have made major changes.
48
Different
Investigators have more flexibility in making
revisions than do journal authors (Reis, 2005).
? Investigators can change their instructions,
materials, procedures, and subject samples,
whereas authors cannot.
? Investigators can remove, revise, or add
proposed studies more readily than can authors.
49
Some final suggested Dos and Donts
? DO Have faith in the sincerity and wisdom of
the grant panel.
? DONT Mindlessly adopt all of their
suggestions.
? DO Start running your initial studies while
you wait for feedback on your first submission.
? DONT Lose your focus on the overall
objectives of your work by adding numerous
trivial studies or measures.
? DO Revise and Resubmit. Very few proposals
are funded on first submission. And youve
already done the hardest part.
50
Assigning Reviewers to Grant Applications An SRA
Perspective

• Anna L. Riley, Ph.D.
• Scientific Review Administrator
• Social Psychology, Personality, and Interpersonal
  Processes Study Section
• Center for Scientific Review

51
Finding the Expertise Needed

• Abstract provides the basic or general
  information
• Specific Aims- purpose of the research
• Research Design and Methods hypotheses, sample,
  special analyses/procedures, measures/scales
• Citations- who is cited ?

52
The Reviewers

• Regular Members quite often the regular study
  section members will have the expertise needed.
• Adhoc Members In addition to regular study
  section members, adhoc reviewers are recruited to
  provide additional or special expertise on
  applications.
• Telephone Reviewer- Reviewers that cannot attend
  the meeting in person--- but have needed
  expertise for applications

53
Multidisciplinary Applications

• Multidisciplinary applications can be a
  challenge. It is sometimes difficult to cover
  all the application---- identify the principle
  parts.

54
Finding Reviewers

• IMPAC II or CRISP- NIH databases
• Library Database Ovid, PsycINFO, etc.
• Colleges and Universities Websites
• Recommendation from other reviewers