How to Be More Competitive in the NIH Peer Review Process for Grants

1
How to Be More Competitive in the NIH Peer Review
Process for Grants

• Olivia Bartlett, Ph.D.
• Chief, Research Programs Review Branch
• National Cancer Institute
• (301-496-7929, op2t_at_nih.gov)

2
Trends in NIH Peer Review of Clinical Research
Project Grants

• MDs have higher success rate than PhDs
• MDs avg 25.9 of applicants, 28.1 of awards
• BUT overall success rate of clinical research
  projects lower than for basic research projects
• 28 for basic research projects
• 22 for mechanism of disease clinical studies
• 20 for clinical trials projects
• Not due to
• Higher budgets for clinical research
• Review panel assignment
• Number or of clinical applications in review
  meeting
• Number or proportion of clinical reviewers on
  panel

3
There is no amount of grantsmanship that will
turn a bad idea into a good one..But there
are many ways to disguise a good idea.

• Dr. William Raub
• Past Deputy Director, NIH

4
Topics for Today

• Overview of NIH Peer Review Process
• Review criteria for research project grants
• Career Development award mechanisms
• Changes coming in the NIH grants process
• Hints for preparing a strong application

5
U. S. Department of Health and Human Services
The Secretary Deputy Secretary
Administration for Children and Families (ACF)
Administration on Aging (AoA)
Food and Drug Administration (FDA)
Health Resources and Services Administration (HRSA
)
Centers for Medicare and Medicaid Services
(CMS)
Indian Health Services (IHS)
National Institutes of Health (NIH)
Agency for Health Care Research and Quality
(AHRQ)
Centers for Disease Control and Prevention (CDC)
Program Support Center (PSC)
Agency for Toxic Substances and Disease
Registry (ATSDR)
Substance Abuse and Mental Health
Services Administration (SAMHSA)
6
National Institutes of Health
Office of the Director
National Institute on Alcohol Abuse and Alcoholism
National Institute of Arthritis
and Musculoskeletal and Skin Diseases
National Institute on Aging
National Institute of Allergy and Infectious
Diseases
National Cancer Institute
National Institute of Child Health and
Human Development
National Institute of Diabetes and Digestive
and Kidney Diseases
National Institute of Dental and Craniofacial Rese
arch
National Institute on Deafness and
Other Communication Disorders
National Institute on Drug Abuse
National Institute of Environmental Health
Sciences
National Eye Institute
National Heart, Lung, and Blood Institute
National Institute of Mental Health
National Institute of Neurological Disorders
and Stroke
National Institute of General Medical Sciences
National Institute of Nursing Research
National Human Genome Research Institute
National Library of Medicine
National Center for Complementary and
Alternative Medicine
Fogarty International Center
National Center for Research Resources
National Institute on Biomedical Imaging and
Bioengineering
National Center on Minority Health Health
Diagnosis

Center for Information Technology
Center for Scientific Review
Clinical Center
7
FY 2005 NIH Budget
8
FY 2005 NIH Extramural Budget
9
NIH Extramural Award Mechanisms
10
Responsibilities of NIH Extramural Staff

• Scientific Review Administrator (SRA)
• In Center for Scientific Review and in each NIH
  Institutes Scientific Review Office
• Organizes, manages, conducts, reports scientific
  peer review of grant applications and/or contract
  proposals
• Liaison between applicants and reviewers
• Program Officer/Director
• In NIH Institutes
• Manages a portfolio of awarded grants/contracts
• Monitors scientific progress made on
  grants/contracts
• Grants/Contracts Management Officer
• Fiscal stewardship of portfolio of awarded
  grants/contracts
• Negotiates fiscal aspects of awards
• Monitors financial progress made on
  grants/contracts

11
Dual Review System for Grant Applications

• First Level of Review
• Scientific Review Group (SRG)
• Provides initial Scientific Merit
• Review of grant applications
• Rates applications and makes
  recommendations for appropriate level of support
  and duration of award

• Second Level of Review
• Advisory Council/Board
• Assesses quality of SRG
• Review of grant applications
• Makes recommendation to
• Institute staff on funding
• Evaluates program priorities
• and relevance
• Advises on policy

12
Review Process for a Grant Application
National Institutes of Health
Institution
Principal Investigator
Center for Scientific Review
Assign to IC IRG/Study Section
Initiates Research Idea
Submits Application

Study Section
Review for Scientific Merit
Institute
Evaluate for Relevance
Allocates Funds
Advisory Councils and Boards
Conducts Research
Recommend Action
Institute Director
Takes final action
13
NIH Solicitations for Applications

• Announcements in the NIH Guide for Grants and
  Contracts indicate new or ongoing interest of one
  or more NIH I/Cs in supporting research,
  training, resources in a field
• Program Announcement (PA)
• Addresses a relatively broad field/category of
  research
• Usually no set-aside I/C budget
• Usually submit on regular receipt dates
• Regular review criteria for type of applications
• Request for Applications (RFA)
• Addresses a well defined area of research
• Set-aside I/C budget for RFA applications
• Submit on special, one time only receipt dates
• Often special eligibility and/or review criteria
• Often special application format and/or
  submission instructions

14
NIH Grant Receipt, Review, and Award Schedule

• Jan-May
• May-Sept Receipt Dates
• Sept-Jan
• June-July
• Oct-Nov Review Meetings
• Feb-Mar
• Sept-Oct
• Jan-Feb National Advisory Council Board Dates
• May-June
• Dec 1
• Apr 1 Earliest Possible Beginning Date
• July 1

15
NIH Center for Scientific Review (CSR)

• Central receipt point for grant applications for
  NIH and other DHHS components
• Assigns applications to NIH Institute(s) as
  potential funding component(s)
• Assigns applications to CSR Study Sections or
  to Institute Scientific Review Groups
• Manages more than 200 standing Study Sections and
  continuing Special Emphasis Panels for the
  initial scientific merit review of most research
  project grant (R01) applications submitted to the
  NIH

16
How is Your Application Assigned within NIH?

• Based on specific written guidelines
• To Study Sections based on
• Topics addressed in the application
• Areas of expertise in the Study Section
• To awarding Institute/Center based on
• Overall mission of the Institute/Center
• Specific programmatic mandates and interests of
  the Institute

17
Who will Handle Review of Your Application within
NIH?

• Center for Scientific Review Unsolicited
  Applications
• Research Project grants (R01)
• Fellowships (F32/33)
• Pilot Studies (R21)
• Small Grants (R03)
• SBIR (R43/44)
• Program Projects for some I/Cs
• Applications for simple PAs

• Institute Review Offices Solicited and I/C
  Mission-targeted Applications
• Training Grants (T32)
• Career Awards (Ks)
• Program Projects (P01)
• Centers (P20/30/50)
• Cooperative Agreements
• Multi-institutional clinical trials
• Applications for RFAs and complex PAs
• Contract Proposals for RFPs

18
Make Sure Your Application is Complete and
Correct as Submitted

• More than 80,000 grant applications were
  submitted to NIH in FY 2005
• All processed by CSR
• NIH cannot change pages after submission
• Errors, poor grammar, missing information will be
  very apparent to reviewers
• Contact the SRA if you need to send corrected
  information

19
Review Criteria for Research Project Grants

• Significance Does this study address an
  important problem? If the aims of the application
  are achieved, how will scientific knowledge or
  clinical practice be advanced? What will be the
  effect of these studies on the concepts, methods,
  technologies, treatments, services or preventive
  interventions that drive this field?
• Approach Are the conceptual or clinical
  framework, design, methods, and analyses
  adequately developed, well-integrated,
  well-reasoned and appropriate to the aims of the
  project? Does the applicant acknowledge
  potential problem areas and consider alternative
  tactics?
• Innovation Is the project original and
  innovative? For example Does the project
  challenge existing paradigms or clinical
  practice, address an innovative hypothesis or
  critical barrier to progress in the field? Does
  the project develop or employ novel concepts,
  approaches, methodologies, tools or technologies
  for this area?

20
Updated Review Criteria for Research Project
Grants Now in Effect

• Investigator Is the investigator appropriately
  trained and well suited to carry out this work?
  Is the work proposed appropriate to the
  experience level of the principal investigator
  and other researchers? Does the investigative
  team bring complementary and integrated expertise
  to the project (if applicable)?
• Environment Does the scientific environment in
  which the work will be done contribute to the
  probability of success? Do the proposed studies
  benefit from unique features of the scientific
  environment or subject populations, or employ
  useful collaborative arrangements? Is there
  evidence of institutional support?

21
Review CriteriaOther Considerations

• Human Subjects Protection
• Data and Safety Monitoring Plan
• Required for ALL clinical trials
• Plans for Inclusion of Women, Minorities and
  Children in Clinical Research
• Animal Welfare Protection
• Any RFA-specific criteria, if applicable
• Appropriateness of the Budget

22
Initial Review Group Options

• Not Scored (UN)
• Application not in top half of all applications
• Not Recommended for Further Consideration (NRFC)
• Lacks significant and substantial merit or
  serious ethical problems in Human Subject or
  Animal use
• Deferred
• Review Committee needs more information to decide
  on the scientific merit of the application
• Scientific Merit Rating (Priority Score) Assigned
• 1.0 (best) to 5.0 (worst)
• Target a mean score of 3.0 for all applications

23
Priority Score

• A single global score is assigned by each review
  committee member not in conflict for each scored
  application.
• The score is to reflect the overall impact that
  the project could have on the field.
• The emphasis on each review criterion may vary
  from one application to another, depending on the
  nature of the application and its relative
  strengths. An application does not need to be
  strong in all criteria to receive a high priority
  score.

24
What Determines Which Applications are Awarded?

• Scientific merit, as indicated by priority score
  and/or percentile ranking
• Each NIH Institute/Center sets its own paylines
• Paylines vary for different types of grants
• Usually more liberal payline for applications
  from new investigators
• Programmatic considerations of the awarding NIH
  Institute/Center
• Balance of models, geographic sites, approaches,
  etc in portfolio
• Availability of funds
• Funds for competing grant awards limited most
  of IC budget already committed to continuing
  grants and programs
• Doubling of NIH budget 1998 2003
• Essentially flat budget in FY 2005 and 2006
  means tighter paylines for all ICs

25
Career Development Awards(See NIH K Kiosk at
http//grants.nih.gov/training/careerdevelopmentaw
ards.htm)

• K01- Mentored Research Scientist Development
  Award
• Usually for Ph.D.s, for basic research
• K02 - Independent Scientist Award
• Additional time/effort support for researcher
  with R01
• K05 - Senior Scientist Award
• K07 - Academic Career Award
• K08 - Mentored Clinical Scientist Development
  Award
• For clinicians to get basic/laboratory research
  training
• K12 - Mentored Clinical Scientist Program Award
• K22 - Patient-Oriented Research (POR) Transition
  Awards
• K23 - Mentored Clinical Scientist Development -
  POR
• K24 - Mid-Career Patient-Oriented Research Award
• K99/R00 - Pathway to Independence (PI) Award
• Announced January 27, 2006
• For postdocs with no more than 5 yr of training
• 1 - 2 yr mentored phase followed by 3 yr
  independent phase

26
Review Criteria for Career Development Awards

• Qualifications of candidate
• Qualifications of mentor (if applicable)
• Appropriateness of career development plan for
  candidates career stage
• Quality of the career development plan
• Quality of the research plan
• Quality of training/institutional environment

27
NIH Loan Repayment Program

• Designed to attract health professionals into
  research in the following areas
• Clinical
• Pediatric
• Health disparities
• Contraception and fertility
• Also program for researchers from disadvantaged
  backgrounds
• Repays up to 35,000 per year of qualified
  educational debt (student loans) in exchange for
  2 3 yr commitment to research
• Must be US citizen
• One receipt date per year, special application
  form
• See http//lrp.nih.gov/about/extramural/index.htm

28
Trends to Watch..

• Increase in biodefense funding/initiatives
• Funds from DHS, managed by NIAID
• NIH Roadmap Initiatives
• Purpose To identify major opportunities and
  gaps in biomedical research that no single
  Institute at NIH could tackle alone but that the
  NIH as a whole must address, to make the biggest
  impact on the progress of medical research
• NIH Roadmap Website http//nihroadmap.nih.gov/

29
NIH Roadmap Initiatives

• New Pathways To Discovery
• Building Blocks, Biological Pathways, and
  Networks
• Molecular Libraries Molecular Imaging
• Structural Biology
• Bioinformatics and Computational Biology
• Nanomedicine
• Research Teams Of The Future
• High-Risk Research
• Interdisciplinary Research
• Public-Private Partnerships
• Re-engineering The Clinical Research Enterprise
• Clinical Research Networks/NECTAR
• Clinical Research Policy Analysis and
  Coordination
• Clinical Research Workforce Training
• Dynamic Assessment of Patient-Reported Chronic
  Disease Outcomes
• Translational Research

30
Pilots Underway.

• Pilot of shortened review cycle for revised R01
  applications from new investigators
• New July 20, 2006 receipt date (vs traditional
  Nov 1, 2006 receipt date)
• PI must decide if weaknesses amenable to quick
  fix
• 40 Study Sections
• See http//grants2.nih.gov/grants/guide/notice-fi
  les/NOT-OD-06-013.html
• Multiple Principal Investigators
• May designate more than 1 PI at one or more
  institutions
• Must include a Leadership Plan (evaluated under
  Approach)
• Linked awards anticipated
• Pilots during 2006, full implementation 2007?

31
Transition to Electronic Applications

• December 2005 through fall 2007
• R01 applications will transition October 1, 2006!
• K awards will transition in May 2007
• Submission through Grants.gov web portal
• Institutions must register with Grants.gov and
  NIH eRA Commons
• Institutions must register investigators as
  Principal Investigators
• NIH will issue Funding Opportunity Announcements
  (FOAs) for each award mechanism
• Grants.gov Find and NIH Guide for Grants and
  Contracts
• New application form SF 424 Research and
  Related
• Download specific application package for each
  FOA
• Start early to prepare application!
• See http//era.nih.gov/ElectronicReceipt/ for
  timeline, FAQs, training materials, tips, contacts

32
Choosing Your Research Project

• What Makes a Research Project Outstanding,
  Singling it Out From All Others Under
  Consideration?
• Has the potential to lead to seminal observations
• Leads to new ways of thinking
• Lays the foundation for further research in the
  field
• Clearly defines the importance of the research
  problem
• Has only two to three interrelated specific aims
  and links all parts of the application
• Addresses a difficult problem in a way that seems
  simple in retrospect, leaving the reviewers to
  wonder why they didn't think of the idea first
• Is written at a level understandable by all
  reviewers

33
Preparing to Write a Grant Application

• Critically Assess Yourself
• Do you have the necessary expertise, resources,
  personnel, and preliminary data to be
  competitive?
• Assess the Competition
• Who are the important contributors to the field?
  (remember, they might end up being your
  reviewers)
• What have competitors accomplished? What have you
  accomplished? How are you going to take what's
  been done a step further?
• Assess the Potential for Your Idea
• What's already been done/reported/funded in your
  area? What are the gaps?
• Search the literature and the database of funded
  grants in the field, e.g., NIH CRISP (Computer
  Retrieval of Information on Scientific Progress)
  system

34
Psychology of the Review Process

• Reviewers are
• Over committed, over worked and tired
• Inherently skeptical and critical
• Informed strangers
• A happy reviewer is likely to be a more positive
  one, so make their job easier
• Flow diagrams, charts, figures
• Well organized, clearly written application
• Avoid things that irritate reviewers
• Not following instructions ie, exceeding the
  page limits, font
  too small, putting information in the wrong
  section, omitting or mislabeling
  references/figures
• Spelling, grammar, and math errors, etc.

35
Preparing the Application

• START EARLY!
• Read instructions thoroughly and follow them
• Never assume the reviewers will know what you
  mean
• Refer to literature thoroughly and thoughtfully
• Explicitly state the rationale of the proposed
  studies
• Include well-designed, easy to follow tables and
  figures
• Include flow diagrams for overview, and for
  complex experiments and protocols
• Address priorities if patients, reagents or
  resources will be limited
• Include data analysis/interpretation plans and
  methods
• Involve the statistician EARLY in project design

36
Key Features of Successful Applications

• Hypothesis
• A meaningful hypothesis AND a means of testing it
• A sound rationale for the hypothesis
• Preliminary Data
• Shows proper training for the research proposed
  and the ability to interpret results
• Documents feasibility of the proposed project
• Include alternative interpretations of results
  and address limitations of methods
• Well Organized Research Plan
• Aims focused - and related to each other and the
  hypothesis
• Rationale for methods proposed, with alternatives
  addressed
• Research flow and priorities clearly indicated
• Sufficient experimental detail to show you
  understand methods
• Emphasize MECHANISM - avoid descriptive data
  gathering

37
In God We Trust.All Others Must Bring Data.
38
Key Features of Successful Applications, cont

• Biosketches
• Indicate your qualifications to carry out the
  work proposed
• Dont pad with lots of in preparation
  manuscripts
• Add a senior collaborator, if needed, to provide
  expertise you lack
• Literature Cited/Bibliography
• Be thorough, but critical, in citing previous
  work in the field
• Description/Project Abstract
• Most read part of the application
• Basis for referral to study section and funding
  Institute/Center
• Write it last, after the Research Plan is
  finished
• State problem, specific aims, types of methods to
  be used
• Letters of Collaboration
• Should be strong and definitively state what will
  be provided

39
Most Common Reasons for Unscored or Not
Recommended for Further Consideration

• Lack of new or original ideas
• Diffuse, superficial or unfocused research plan
• Lack of appreciation of published relevant work
• Lack of experience in essential methods
• Questionable reasoning in experimental approach
• Lack of sound rationale for hypothesis or methods
• Aims dont address hypothesis
• Unrealistically large amount of work proposed
• Lack of sufficient experimental detail
• Uncertainty about future directions of work
• Serious concerns about risks to human subjects or
  use of animals

40
NIH Program and Review Staff Can Help

• Know the NIH program officer(s) in your field
• Information about upcoming initiatives,
  opportunities, gap areas
• Information about potential collaborators, NIH
  resources
• Explain NIH policies, procedures, award
  mechanisms, eligibility requirements
• Advice in revising unfundable applications
• Know the Peer Review System and your SRA
• Review criteria and receipt/review schedules
• Explain NIH policies, procedures, award
  mechanisms, eligibility requirements
• Problems with referral or review
• Use the NIH and other websites to get latest
  information, forms, policies

41
Selected Web Sites of Interest

• National Institutes of Health (http//www.nih.gov)
  
• NIH Office of Extramural Research homepage, with
  links to the NIH Guide, grants policy
  information, and resources for new investigators
  http//grants1.nih.gov/grants/oer.htm
• Overview of NIH Extramural Research, with links
  to tools and FAQs http//grants1.nih.gov/grants/w
  elcome.htmintroduction
• Career Development Awards Information
  http//grants.nih.gov/training/careerdevelopmentaw
  ards.htm
• NIH Electronic receipt http//era.nih.gov/Electron
  icReceipt/
• NIH Center for Scientific Review
  (http//www.csr.nih.gov)
• Has links to Resources for Applicants, standing
  Study Section rosters, policy information, review
  procedures and review criteria, video of mock
  study section, and advice for investigators
  submitting clinical research applications