Office of Extramural Research, Education and Priority Populations (OEREP) Grantsmanship Session

1
Office of Extramural Research, Education and
Priority Populations (OEREP)Grantsmanship
Session

• Overview Francis Chesley, MD OEREP Director
• Prepare a Grant Application Debbie Rothstein,
  PhD, Senior Advisor to the Director - OEREP
• AHRQ Peer Review Process Kishena Wadhwani, PhD,
  MPH Div. Scientific Review Director, OEREP
• Human Subjects Protections and Inclusions
  Patrick McNeilly, PhD AHRQ IRB Official, OEREP

2
Preparing a Grant Application Steps to Success

• Debbie Rothstein, Ph.D.
• Senior Advisor, Extramural Research
• Office of Extramural Research,
• Education and Priority Populations
• Agency for Healthcare Research and Quality
• September 9, 2008

3
Getting Started

• Internet is funding gateway
• AHRQ and NIH use essentially the same application
  processes and grant mechanisms
• Keep abreast of open solicitations and Agency
  research priorities

4
AHRQs Website www.ahrq.gov
   AHRQ Research Agenda
   Funding Announcements
   Research Policies
   Grants Process
   Grant Application Basics
   Training and Education
   Contracts
5
AHRQ and NIH Grant Application Similarities

• Application forms SF 424(RR), PHS 398
• Application submission dates
• Use of internet as the funding gateway to keep
  abreast of research priorities and open
  solicitations, staff can provide technical
  assistance
• Announcement publication NIH Guide for Grants
  http//grants1.nih.gov/grants/guide/index.html
  and Grants.Gov www.http//grants.gov

6
AHRQ and NIH Grant Application Similarities
(contd)

• Receipt and referral of grant applications
• Application review procedures
• Grant mechanisms e.g., R01, R03, R13, R36, K01,
  K02, K08
• Transition to electronic submission of
  applications
• Some common Policy Notices and Funding
  Opportunity Announcements

7
AHRQ and NIH Grant Application Differences

• Detailed budget vs. modular budget
• Conference grants AHRQ has small and large
  opportunities
• AHRQ doesnt allow multiple PIs
• Budget limitations
• Large grants (R01, R18) - 300K vs. 500K
• Small grants (R03) - 100K vs. 50K
• Fewer grant mechanisms used at AHRQ

8
AHRQ Grant Opportunities

• AHRQ announces availability of grants in a
  Funding Opportunity Announcement (FOA)
• FOAs are published in the NIH Guide for Grants
  (http//grants.nih.gov/grants/guide/index.html)
  and posted on Grants.gov (http//www.grants.gov/
• Program Announcement (PA)
• Describes broad research interests
• Open over a period of time
• Request for Applications (RFA)
• More well defined area, specific program
  objectives
• Single application receipt date and amount of
  funds

9
Common AHRQ Grant Opportunities

• R01 large research grants
• Feb 5, June 5, Oct 5 application receipt dates
• up to 300K/yr
• R18 research demonstration grants
• Jan 25, May 25, Sept 25 application receipt dates
• up to 300K/yr
• R03 small research projects
• Feb 16, June 16, Oct 16 application receipt dates
• up to 100K in total costs
• Large and small conference grants (R13)
• Large Apr 12, Aug 12, Dec 12 receipt up to
  100K/yr in total costs
• Small Feb 20, Apr 20, Jun 20, Aug 20, Oct 20,
  Dec 20 receipt up to 50K

10
AHRQ Grant Opportunities Training/Career
Development

• Pre and Postdoctoral Training
• Institutional Training Programs (T32)
• Individual Pre-doctoral Fellowships to Promote
  Diversity (F31)
• Individual Post-doctoral Fellowships (F32)
• Dissertation Grants (R36)
• Career Development Awards
• Mentored Scientist Awards (K01)
• Independent Scientist Awards (K02)
• Mentored Clinical Scientist Awards (K08)
• Detailed information at
• http//www.ahrq.gov/fund/training/trainix.htm
• Send questions to training_at_ahrq.hhs.gov

11
AHRQ Staff Involved in Grants Process

• Referral Officer
• Review Staff Scientific Review Officer and
  Grants Management Specialist
• Program Staff Project Officer (PO)
• Grants Management Staff

12
Electronic Application and Submission

• Transition to Electronic Receipt began Dec, 2005
• Vast majority of all competing applications to
  AHRQ are now submitted electronically using SF
  424 (RR) application
• All major mechanisms used by AHRQ have
  transitioned except for the training grants

13
What is Grants.gov?

• Federal governments single, online portal to
  electronically
• Find Grant Opportunities
• Apply for Grants
• A cross-agency initiative spanning 900 grant
  programs from the 26 grant-making agencies, and
  over 350 billion in annual awards.

14
Electronic Submission of Grant Applications

• With this new process, its critical to read and
  follow all instructions in the announcement and
  application guide
• Applicant must prepare for e-submission by
  completing mandatory registrations
• Grants.gov registration for organization
• ERA Commons registration for organization and
  the PI

15
SF424 (RR) Application Form

• Provides a consistent electronic submission
  process through Grants.gov
• Consists of common data elements, arranged in
  components
• Not all components will be used for every Funding
  Opportunity Announcement (FOA)
• Each FOA will have the appropriate application
  package attached
• From data files, application image is generated
• Applicant should view and verify the application
  image

16
Electronic Application and Submission -
General Information and Help Links

• SF424 (RR) Application and Electronic Submission
  Information
• http//grants.nih.gov/grants/funding/424/index.htm
  
• General information on Electronic Submission of
  Grant Applications
• http//era.nih.gov/ElectronicReceipt/
• Finding Help
• http//era.nih.gov/ElectronicReceipt/support.htm

17
Paper No More, Use 424 (RR) OLD
NEW
18
What Determines Which Awards Are Made?

• Scientific merit
• Significance and originality
• Methods
• Program / Agency considerations
• What is uniquely AHRQ
• Existing research portfolio balance
• Anticipated IMPACT of research
• Availability of funds

19
Ingredients of a Successful Grant Application

• Good Idea
• Good Science
• Good Application
• Fits Agency Research Priorities

20
AHRQ PEER REVIEW PROCESS Kishena C.
Wadhwani, Ph.D., M.P.H. Director, Division of
Scientific Review (DSR) Kishena.wadhwani_at_ahrq.h
hs.gov Phone [phone] Office of
Extramural Research, Education and Priority
Population (OEREP) Agency for HealthCare
Research and Quality (AHRQ) Department of Health
and Human Services (DHHS)
10/09/2008
21
ORGANIZATIONAL CHART Office of Extramural
Research, Education and Priority Populations
(OEREP) Division of Scientific Review
(DSR) Current Study Review Group (SRG) Clusters
and the Corresponding SRO/GMS
OEREP Director Dr. Francis Chesley
Committee Management
Receipts Referrals
DSR Director Dr. Kishena Wadhwani
Study Section Clusters
Health Care Quality Effectiveness Research
(HCQER)
Health Care Technology and Decision Sciences
(HCTDS)
Health Care Research Training (HCRT)
Health Care System Research (HSR)
SRO Dr. Carl Ohata GMS Maureen Gallagher
SRO TBD GMS Ms. Carmen Colston
SRO Dr. Boris Aponte GMS Ms. Rebecca Trocki
SRO Ali Azadegan GMS Ms. Diane Manuel
22
AHRQ HEALTH SERVICES RESEARCH SRG AND ITS PRIMARY
RESEARCH FOCI
Health Care Systems Research (HSR) Health Care Technology and Decision Sciences (HCTDS) Health Care Quality Effectiveness Research (HCQER) Health Care Research Training (HCRT)
Translational Research Translational Research Translational Research Translational Research
Implementation Research Implementation Research Implementation Research Implementation Research
Access and Utilization Technology/HIT Assessment Healthcare Quality Effectiveness Training Career Development
Cost/Finance/Markets Information Sciences Dissemination Cost-effectiveness Research Training-related Conferences/Workshops
Qualitative Quantitative Methods Qualitative Quantitative Methods Qualitative Quantitative Methods Qualitative Quantitative Methods
Delivery Systems Clinical/Translational Decision-making Evidence-based Medicine Institutional Training Career Development
Organizational Studies Patient Utilities, Morbidity Function Quality of Care Research Demonstrations Evaluations
Infrastructure Building Research Outcome Research Access to Utilization of Care Outcome Research
Provider Workforce Healthcare Quality Cost Improvement Clinical Outcome Research Healthcare-related IT Training
23
SCHEDULE OF STUDY SECTION MEETINGS, FY08-FY09
(TENTATIVE) (Updated 09/01/2008)
Study Section ( Meeting Dates) October 2008 Meeting March 2009 Meeting June 2009 Meeting
HSR Ohata/Gallagher October 24, 2008 March 5-6, 2009 June 24-25, 2009
HCQER TBD/Colston October 30, 2008 February 23-24, 2009 June 18-19, 2009
HCTDS Aponte/Trocki October 29, 2008 February 24, 2009 June 18-19, 2009
HCRT Azadegan/Manuel October 23-24, 2008 March 5-6, 2009 June 24-25, 2009
24
Process of Review for New Individual Research
Project Grant Applications

• Three overlapping cycles per year
• Submit in February (June, October)
• - Review in June July (Oct -
  Nov, February - March)
• SLT in July - August (December, May -
  June)
• Earliest award in December (April, August -
  September)

25
Peer Review at NIH/AHRQ Types of Committees
(Study Sections)

• Standing Committees
• - Chartered multi-year commitment
• - Temporary members added as needed
• - Roster posted on NIH/AHRQ websites
• Special Emphasis Panels (SEPs)
• - All Temporary (Ad-hoc) members

26
Purpose of Peer Review

• To evaluate the scientific and technical
• merit of grant applications, providing
• information (Recommendations) used by the
  Agencies/Institutes and Centers to make funding
  decisions.

27
Reviewers consider

• Scientific and technical merit of proposed
  research/training
• Qualifications of Principal Investigator(s) and
  research team
• Availability of resources (Labs, Institutions)

28
Reviewers also consider (Cont.)

• Reasonableness of requested budget for work
  proposed
• Other factors (e.g., human subjects, animal
  welfare, inclusion policies/plans)

29
Reviewers do not consider

• Program relevance (Except if stated in FOA)
• Policy issues
• Funding levels
• Anticipated budget reductions
• Comparisons with other applications

30
Review Criteria

• For all R01, R03, R15, R21, and P01 subprojects
• Significance and Originality
• Approach Methods and Data
• Innovation
• Organization of the Project
• Investigators
• Environment Facilities and Resources

31
Sole Basis of Review Review Criteria
(The NIH 5 as an example)
Significance Does this study address an
important problem? If the aims of the application
are achieved, how will scientific knowledge be
advanced? What will be the effect of these
studies on the concepts or methods that drive
this field? Approach Are the conceptual
framework, design, methods, and analyses
adequately developed, well-integrated, and
appropriate to the aims of the project? Does the
applicant acknowledge potential problem areas and
consider alternative tactics? Innovation Does
the project employ novel concepts, approaches or
methods? Are the aims original and innovative?
Does the project challenge existing paradigms or
develop new methodologies or technologies?
Investigator Is the investigator appropriately
trained and well suited to carry out this work?
Is the work proposed appropriate to the
experience level of the Principal Investigator
and other researchers (if any)? Environment
Does the scientific environment in which the work
will be done contribute to the probability of
success? Do the proposed experiments take
advantage of the unique features of the
scientific environment or employ useful
collaborative arrangements? Is there evidence of
institutional support?
32
Also...

• Reviewers evaluate
• Protection of human subjects, the environment,
  and animal welfare

• Inclusion of women, minorities, and children

• Inclusion of AHRQ Priority Populations

• Budget

33
Research Involving Human Subjects

• Important Considerations that must be addressed
  in the application because they impact on
  priority score - considered to be part of the
  Approach
• Are there any risks to the human subjects?
• Are the protections adequate?
• Are there potential benefits to the subjects and
  to others?
• What is the importance of the knowledge to be
  gained?
• Are the plans for inclusion of minorities, both
  genders and children adequately addressed?
• Is the proposed study exempt from human subject
  review?
• No page limits

Risks include the possibility of physical,
psychological, social injury or safety resulting
from research.
34
AHRQ RequirementInclusion of Priority
Populations

• Inclusion of Children? - Rationale?
• Inclusion of Elderly? - Rationale?
• Inclusion of Rural? - Rationale?
• Inclusion of Inner City? - Rationale?
• Inclusion of Low Income? - Rationale?
• Inclusion of Disabled? - Rationale?
• Inclusion of Chronic Care? - Rationale?
• Inclusion of End of Life? - Rationale?

35
Review Criteria (Cont.)

• For FOAs (RFAs or PAs) , modified or additional
  criteria may be specified
• For other mechanisms, specific criteria apply

36
Peer Review Process

• Each application assigned to 3 reviewers for
  written comments
• Streamlined Review may be used
• Group discussion of each application, including
  budget recommendation
• Recommended score range
• Reviewers privately assign priority score

37
Peer Review Group Actions

• Unscored (Streamlined Review)
• Scientific Merit Rating (Priority Score)

38
Priority Scores/ Percentiles

• Priority scores range from
• 100 (Best) to 500
  (Worst)
  
• 100 x (1.0 5.0)
• In Streamlined Review, those in lower half are
  eliminated (unscored)
• Percentiles calculated to normalize scoring
  behavior across review groups

39
Priority Scores Assigned by Reviewers

• SCORED (stronger) 1.0 ? 1.4
  (Outstanding)
• 
  1.5 ? 2.0 (Excellent) (25)
• 
  2.0 ? 3.0 (25)
• UNSCORED (weaker ) 3.0 ? 4.0
• 4.0
  ? 5.0 (50)

40
HINTS PREPARING AN APPLICATION

• Start early
• Read and follow instructions
• Clarify any confusing instructions early
• Do not assume staff or reviewers will know what
  you mean
• Include well designed tables and figures
• Format - consider the reviewers
• Proofread and check before sending

41
PREPARING AN APPLICATION (cont.)

• Explicitly state the purpose of the proposed work
• Refer to the literature thoroughly but
  thoughtfully
• Present an organized, lucid write-up
• Be mindful of the review criteria to be used
• Get advice from colleagues

42
Art of Persuasion Assumptions

• that the reviewers are knowledgeable and
  committed to doing a thorough job of evaluating
  each application
• that the reviewers have less time to complete
  the task than desirable
• that good formatting will assist reviewers to
  remember your organization and the thrust of your
  arguments

43
Strategies to Strengthen Applications The Short
List

• Ask scientist with AHRQ support to critically
  review your application prior to submission
  more than once
• Talk with an Agency (AHRQ, etc.) program
  officer(s)

44
Strengthening Applications - The Long List

• Significance of problem Best if transparent yet
  not insultingly obvious
• Scientific Rationale New /or original ideas
  clearly identified
• Include a plan for acquiring needed expertise,
  if needed
• Discuss the limitations of the chosen approach
• Include abundant detail about the methodology
• Scope of work midway on the trivial grandiose
  continuum
• Describe the participants their involvement in
  the study in detail
• Include pilot or preliminary data
• Describe future research directions

45
Help Reviewers See the Merits

• Think like a reviewer
• Learn as much as possible about The System (e.g.,
  figure out the likely review group)
• Preempt criticism
• Include collaborators who can compensate for your
  deficiencies
• Learn as much as possible about what research
  projects in your area that have been funded by
  the NIH/AHRQ/HRSA/CDC/etc.
• CRISP Database
  (http//crisp.oit.nih.gov)

46
http//crisp.oit.nih.gov
47
HCRT Grant Applications Reviewed and Funded in FY08 HCRT Grant Applications Reviewed and Funded in FY08 HCRT Grant Applications Reviewed and Funded in FY08 HCRT Grant Applications Reviewed and Funded in FY08
Types Reviewed Funded Sucessful
F31 2 0 0
F32 8 2 25
K02 5 3 60
K08 12 6 50
R36 39 11 28
R01 1 1 100
R13 1 0 0
Total 68 23 34
48
In God We Trust.All Others Must Bring Data.
Olivia Bartlett, Ph.D. Chief, Research Programs
Review Branch Division of Extramural Activities,
NCI
49
BEST WISHES TO YOUR EDUCATION AND PROFESSIONAL
ENDEAVORS! ANY QUESTIONS?
50
Human Subjects Protections

• Overview for Applicants
• Patrick McNeilly, Ph.D.
• September 2008

51
Background

• Belmont Report
• - Respect for Persons
• - Beneficence
• - Justice
• Common Rule (45 CFR 46)
• http//www.hhs.gov/ohrp/humansubjects/guidance/45c
  fr46.htm

52
Common Rule Requires

• IRB Review
• Informed Consent
• Evaluations of all applications and proposals
  involving human subjects (45 CFR 46.120)

53
Peer Review vs IRB

• Peer review
• - Evaluate the application
• - Identify human subjects concerns in any
  portion of the
  application
• - Justification of exemption
• IRB
• - Initial and continuing review of research
• - Approve modifications to research

54
Human Subjects Concern

• Any actual or potential unacceptable risk, or
  inadequate protection against risk, to human
  subjects as described in any portion of the
  application.
• - Captured in Summary Statement
• - Administratively coded

55
Common Peer Review Issues

• No human subject protection plan
• Unidentified subjects
• Unidentified risks
• Informed consent issues

56
Resolution of Issues

• Written response from Principal Investigator to
  all human subjects concerns
• AHRQ Program Official will consult with human
  protections administrator on adequacy of response

57
Exemption from HHS Regulations

• Six categories of exemption
• Not all health services research is exempt
• Is there an appropriate justification of
  exemption?

58
Inclusion of Women Minorities

• Women and members of minority groups must be
  included
• Unless a clear and compelling rationale to
  exclude
• http//grants.nih.gov/grants/funding/women_min/wom
  en_min.htm

59
Priority Populations

• Inner-city rural low income minority women
  children elderly special health care needs
• Studies should consider including one or more
  AHRQ priority populations
• AHRQ specific requirement
• http//grants.nih.gov/grants/guide/notice-files/NO
  T-HS-03-010.html

60
Example 1

• The proposed study will provide a screening tool
  in the form of a Bruising Clinical Decision Rule
  (BCDR) for discriminating bruises caused by
  physical child abuse vs. accidental trauma.
  Success will result in a BCDR to function as a
  screening tool to identify children and infants
  with bruising who are at high risk for physical
  abuse and require further evaluation.

61
Example 1 (contd)

• The proposed study is a prospective
  observational study of bruising characteristics
  in children over 4 years of age. Data on
  bruising characteristics will be collected by
  pediatric emergency medicine physicians and child
  abuse experts on 1,000 children with bruising.

62
Example 2

• In an application focusing on the differences in
  treatment of prostate cancer for whites compared
  to non-whites.

63
Example 2 (contd)

• Reviewers Commented
• Given the applications focus, prostate cancer,
  the exclusion of women is appropriate. However,
  the low minority representation and poor
  recruitment strategies for minorities (85 White,
  7 Hispanic, 8 African American) would make it
  difficult for comparisons of treatment methods by
  racial and ethnic categories to be made .

64

• Questions?