Title: The Review of Epidemiologic Research Applications at the National Institutes of Health: Process, Products, and Best Possible Science Investigator Viewpoint, from Idea to Submission, and Study Section Viewpoint, from Submission to Assessment Presented
1The Review of Epidemiologic Research Applications
at the National Institutes of Health Process,
Products, and Best Possible ScienceInvestigator
Viewpoint, from Idea to Submission, andStudy
Section Viewpoint, from Submission to
AssessmentPresented at the 44th Annual
Conference on Cardiovascular Disease Epidemiology
and Prevention of the American Heart Association,
March 3, 2004, San Francisco, CA
- J. Scott Osborne, III, Ph.D., M.P.H.
- Scientific Review Administrator
- Epidemiology of Chronic Diseases Study Section
- Center for Scientific Review, National Institutes
of Health - Department of Health and Human Services
2 - PROCESS OF REVIEW
- From Idea to Application, Submission, and
Assessment - PRODUCTS OF REVIEW
- Assessment of Scientific Merit and Summary
Statement - BEST POSSIBLE SCIENCE
3PROCESS OF REVIEW
- IDEA proposed in Application (PHS 398)
- SUBMISSION of APPLICATION to NIH Division of
Receipt and Referral, Center for Scientific
Review - ASSIGNMENT to Institute(s) and Study Section
(suggestions welcome) - REVIEW by NIH STUDY SECTION
- ASSESSMENT OF SCIENTIFIC MERIT (score and
percentile rank) for Institute Advisory Councils
and Center Advisory Boards
4PRODUCTS OF REVIEW
- PRE-REVIW APPLICATION development of an idea
into a meaningful project - REVIEW RECOMMENDATION (score percentile rank)
regarding ASSESSMENT of Scientific and Technical
Merit of Application - POST-REVIEW SUMMARY STATEMENT
5BEST POSSIBLE SCIENCEContributions of the
Process of Review
- PRODUCTS OF REVIEW LEAD TO BEST POSSIBLE SCIENCE.
- ASSESSMENTS regarding Scientific and Technical
Merit guide decision making. - SUMMARY STATEMENTS provide written feedback,
allowing improvement of projects and meaningful
revisions of applications for resubmissions. - NIH STAFF provide guidance.
6BEST POSSIBLE SCIENCEQUALITIES
- Moves field forward.
- Has clinical and pubic health significance.
- May provide clues in areas outside specific
framework of study.
7BEST POSSIBLE SCIENCEThe Impossible Dream
- Perfectly Justified Aims
- Perfect Clarity of Exposures, Confounders, Effect
Modifiers, and Outcomes - Perfect Design for a Perfect Sample
- Perfect Methods for Ascertainment and
Measurement, Perfect Validity and Reliability of
Data - Perfect Analysis
- Perfect Interpretation and Response
8BEST POSSIBLE SCIENCE
- CONVENIENCE vs. QUALITY Select the best (e.g.,
valid, generalizable, etc.) possible sample and
approach to collecting data do not opt for
convenience. - The Ideal or Gold Standard Fully explore
strengths and limitations of an approach, and
consider a selected approach in comparison to
alternatives. - Best Approximation Propose a study well
justified in the context of the current state of
the field.
93 TIERS OFEPIDEMIOLOGIC RESEARCH
- DESCRIPTION
- ETIOLOGY
- INTERVENTION (including clinical trial
epidemiology as well as primary, secondary, and
tertiary prevention)
10INVESTIGATORVIEWPOINT
11From Idea to Application IDEA
- IDEA Significance -- So what?
- IDEA Appropriately focused or too broad?
- IDEA Innovative, standard, or already in
progress or done? (Ask colleagues, and do a
thorough search of the literature. Check CRISP
at the NIH Website for funded NIH research.) - IDEA Evidence to support in literature (animal
and human studies)? Need pilot data to support? - IDEA Plausibility Mechanism
12INNOVATION
- New Question, New Approach
- New Question, Old Approach
- Old Question, New Approach
13From Idea to Application IDEA
- Is the NIH interested in this area? (Check for a
Program Announcement or Request for Applications.
Call NIH Program Staff at different relevant
Institutes to discuss your idea and how it fits
with their programs.) - SELECT APPROPRIATE MECHANISM
14From Idea to Application SUPPORT
- STANDARD NIH MECHANISMS OF GRANT SUPPORT REVIEWED
IN ECD - FELLOWSHIPS
- SMALL GRANTS (R03)
- R21
- R01
15From Idea to Application INVESTIGATIVE TEAM
- Investigative Team Convenience vs. Quality.
NETWORK!!! - The Principal Investigator should develop an
appropriate team. This may include one or more
Co-Principal Investigators as well as
Co-Investigators, Collaborators, and Consultants.
Persons on the investigative team are often from
different institutions.
16From Idea to Application INVESTIGATIVE TEAM
- INTELLECTUAL PROPERTY An application is
submitted by an applicant institution (e.g.,
your university) but is the intellectual
property of the investigators. - CONSULTANT ROLES vary according to the extent of
involvement, but they are generally viewed as a
service and do not involve a proprietary
intellectual contribution to a project. - SERVICES GENERALLY AVAILABLE (e.g., assays) are
not viewed as involving a proprietary
intellectual contribution to a project.
17From Idea to ApplicationENVIRONMENT
- General Institutional Support
- Specific Resources Available for Project
- Evidence of Departmental Level Support
- Potential for Success in Overall Environment
18From Idea to ApplicationMETHODS
- DESIGN
- SAMPLE
- REPRESENTATIVENESS AND GENERALIZABILITY (External
Validity) - POWER (and justification for Effect Size)
- DATA COLLECTION
- ANALYSIS OF DATA (linked to Specific Aims)
- POTENTIAL BIASES AND LIMITATIONS
19From Idea to ApplicationCritical Thinking
- Be your own worst critic. (About Everything!)
- So what? (Significance)
- Would you believe findings from this study?
(Methods)
20From Idea to ApplicationPHS 398 APPLICATION FORM
- RESEARCH PLAN
- SPECIFIC AIMS
- BACKGROUND AND SIGNIFICANCE
- PRELIMINARY STUDIES or PROGRESS REPORT
- RESEARCH DESIGN AND METHODS
- HUMAN SUBJECTS
21From Idea to ApplicationSUBMISSION
- RECEIPT DATES
- REQUEST STUDY SECTION ASSIGNMENT
22STUDY SECTION VIEWPOINT
- From SUBMISSION to ASSESSMENT OF SCIENTIFIC MERIT
23NIH SUPPORT
- CONTRACTS (Solicited with Request for Proposals
-- RFP) - COOPERATIVE AGREEMENTS
- SOLICITED SET-ASIDES RFA (Request for
Applications 1 submission, 1 review) - Program Announcements (PA)
- UNSOLICITED APPLICATIONS
24Epidemiology of Chronic Diseases (ECD) Study
Section
- Unsolicited applications
- 3 cycles (rounds) per year
- Submission/Receipt (FEB2004/June2004/Oct2004)
- Review (JUNE 2004/Oct2004/Feb2005)
- Council (OCT 2004/Jan2005/May 2005)
- Award by Institute to Applicant Institution vs.
Principal Investigator - New Investigator Status A new investigator
has not previously been a principal investigator
for an R01.
25APPLICATION ASSIGNMENT NUMBERS
- ORIGINAL SUBMISSION
- 1 R01 HL 012345-01
- REVISED SUBMISSIONS
- 1 R01 HL 012345-01A1
- 1 R01 HL 012345-01A2
- CONTINIUATIONS
- 2 R01 HL 012345-06
- 2 R01 HL 012345-06A1
- 2 R01 HL 012345-06A2
- SUPPLEMENTS (within project period)
- 3 R01 HL 012345-01, -01A1, -01A2
26Function of ECD Study Section
- ECD assesses NIH applications for scientific and
technical merit this is the first level of peer
review. - NIH Institute Advisory Councils assess
applications for relevance to NIH goals and
public health needs this is the second level of
peer review. - The Institute Directors use these assessments to
make funding decisions.
27Criteria to Evaluate Scientific and Technical
Merit
- Significance Is it? Why? Advance science?
- Approach Design/Population and Sample/Data
Collection/Analysis - Innovation Aims or approach?
- Investigator(s)
- Environment
28Study Section PerspectivePre-Meeting
- Submission/Receipt of Applications
- Self-Referral Request Study Section in cover
letter - SRA decides if appropriate for Study Section
after initial referral - Institutes determine their interest and roles as
primary, secondary, etc. - Additional Information (contact SRA)
- Human Subjects IRB approval not needed for
review
29Study Section PerspectivePre-Meeting
- Administrative Issues
- Assignments
- Chartered members (20 for 4 year terms each
about 5 members nominated by SRA each year Chair
nominated by SRA, usually for 2 years) - Special reviewers (as needed)
- Outside opinions/Mail Reviewers
30Confidentiality andConflict of Interest
- Confidentiality in perpetuity
- Conflict of Interest
- Assessed by SRA
- Assessed by reviewers
- Member Conflicts If a member is on your
application, it will be reviewed by a special
emphasis panel or another study section.
31Study Section PerspectiveMeeting Procedure of
Review
- Initial levels of enthusiasm from assigned
reviewers - Description and critique from first reviewer
(including assessment of potential risks to human
subjects and the inclusion of women, minorities,
and children) - Critiques from other reviewers
- Discussion by entire Study Section
- Final levels of enthusiasm
- Vote conscience
32Study Section PerspectiveMeeting Procedure of
Review
- After scoring
- Budget recommendations from reviewers, discussed
by entire study section - Discussion of Administrative issues (e.g.,
overlap) - Deferral and Site Visits
- An application may be deferred if it is viewed as
highly significant and if a small amount of
additional information (obtained by the SRA)
would address concerns. - Site visits are conducted ONLY if there is no
other way to acquire such needed information.
33Scoring
- Judgment by reviewers based on assessment of
scientific and technical merit - Criteria not equally weighted
- Budget not a factor
- Other administrative issues not factors
- HS/Gender/Minority/Children inclusion may be
factor (as related to aims)
34Scoring
- Streamlining (the UNscore) about 50 of
applications reviewed in a normal round - Range 1.0 - 5.0 (ideal median 3.0)
- Percentile Rank calculated using the current and
previous two rounds as a base - Exception F-series 1.0-5.0, NR
35Study Section PerspectivePost-Meeting
- Scores/codes/budgets verified by SRA and entered
in NIH computer system - Summary Statements prepared by SRA
- SRA advises investigators discusses review
- SRA represents the assessments of scientific and
technical merit by Study Section at the National
Advisory Councils of Institutes - Note We are receiving the next rounds
applications while preparing summaries for the
current round.
36Reorganization of Epidemiology Study Sections in
October 2003
- 1965-2003 Epidemiology and Disease Control-1
(EDC-1) Study Section and EDC-2 - 2001 Added EDC-3
- October 2003 reorganization
- EDC-1 ? Epidemiology of Chronic Diseases
(ECD) Study Section - EDC-2 ? Epidemiology of Cancer (EPIC) Study
Section - EDC-3 ? Epidemiology of Clinical Disorders and
Aging (ECDA) Study Section
37 Inside the NIH Grant Review Process Video
- CSR has developed a video of a mock study section
meeting to show how NIH grant applications are
reviewed.
http//www.csr.nih.gov/video/video.asp
38CSR Web Site http//www.csr.nih.gov
- News and Events
- Resources for Applicants
- Study Section Information
- Employment Opportunities
- Contact Information
39Information on the World Wide WebSelected Sites
of Interest
- National Institutes of Health (http//www.nih.gov)
- Office of Extramural Research (http//www.nih.gov/
grants/oer.htm) - Grants Policy (http//www.nih.gov/grants/policy/po
licy.htm) - Center for Scientific Review (http//www.csr.nih.g
ov) - Referral and Review (http//www.csr.nih.gov/refrev
.htm) - CSR Study Section Rosters (http//www.csr.nih.gov/
committees/rosterindex.asp) - Review Group Meeting Dates
(http//www. csr.nih.gov/committe
es/meetings/ssmeet1.asp) - CSR Reorganization News (http//www.csr.nih.gov/r
eview/reorgact.asp)