Teenage Pregnancy Prevention: Replication of Evidence-based Programs (Tier 1)

1
Teenage Pregnancy Prevention Replication of
Evidence-based Programs (Tier 1)
U.S. Department of Health and Human
Services Office of Adolescent Health April 14,
2010 300-500pm ET
1
2
Purpose of Todays Call

• Review the Teenage Pregnancy Prevention
  Replication of Evidence-Based Programs Funding
  Opportunity Announcement, including the purpose,
  eligibility requirements, and how to apply for
  funds
• Will not be answering questions specific to
  individual applications

2
3
Introductions

• Office of Adolescent Health
• Evelyn Kappeler, Acting Director
• Alice Bettencourt, Acting Deputy Director
• Allison Roper, Public Health Analyst
• Amy Margolis, Public Health Analyst
• Jennifer Gannon, Program Specialist
• Miryam Gerdine, Public Health Analyst
• Office of Grants Management
• Karen Campbell, Director

April 14, 2010
3
4
Overview of the Office of Adolescent Health
Teenage Pregnancy Prevention Initiative
5
Office of Adolescent Health (OAH)

• Consolidated Appropriations Act, 2010 directed
  that a new OAH be established
• Responsible for implementing and administering
  new grant program to support evidence-based teen
  pregnancy prevention approaches
• Coordinates adolescent health programs and
  initiatives across the U.S. Department of Health
  and Human Services

April 14, 2010
5
6
Office of Grants Management

• Official signatory for obligating federal grant
  funds
• Official signatory for all grant business
• Monitor all business/financial transactions on
  grants for compliance to Federal Regulations
  (including interpretation of Federal Regulations)

7
(No Transcript)
8
Location
8
9
Teenage Pregnancy Prevention Initiative

• Consolidated Appropriations Act, 2010 (Public Law
  111-117)- 110 million
• 75 million - replicate program models proven
  effective through rigorous evaluation (Tier 1)
• 25 million - research and demonstration grants
  to develop, replicate, refine, and test
  additional models and innovative strategies (Tier
  2)
• 10 million - training and technical assistance,
  evaluation, outreach, and additional program
  support activities

9
10
Introduction to TPP Tier 1 Replication of
Evidence-based Programs Funding Announcement
11
Purpose of Tier 1 Funding Announcement

• To support the replication of evidence-based
  program models that are medically accurate, age
  appropriate, and have proven through rigorous
  evaluation to reduce teenage pregnancy,
  behavioral risks underlying teenage pregnancy, or
  other associated risk factors.

11
12
Target Populations

• Individuals 19 years of age or under at program
  entry
• Applicants should clearly define target
  populations by age groups or priority populations
  within a defined geographic area with high teen
  birth rates.

12
13
Whos eligible to apply?

• Funds made available shall be for making
  competitive contracts and grants to public and
  private entities (Consolidated Appropriations
  Bill, 2010)
• Nonprofit organizations
• For-profit organizations
• Universities and colleges
• Research institutions
• Hospitals
• Community-based organizations
• Faith-based organizations
• Federally recognized or state-recognized
• tribal governments
• State and local governments
• State and local school districts
• Political subdivisions of States

13
14
Funding Ranges for Tier 1

• Range A - 400,000 to 600,000 per yr
• Range B - 600,000 to 1,000,000 per yr
• Range C - 1,000,000 to 1,500,000 per yr
• Range D - 1,500,000 to 4,000,000 per yr
• Applicants may only apply for one funding range
  under this announcement.

14
15
Cost-Sharing or Matching

• Neither cost-sharing nor matching are required
• Applicants are encouraged to include
  participation by stakeholders in the community as
  an indicator of community support for the project
• An indication of institutional support from the
  applicant and its collaborators indicates a
  greater potential of success and sustainability
  of the project

15
16
Award Information

• Type of Award Cooperative Agreements
• Number of awards Up to 150
• Project Period Up to 5 years
• Funding Range 400,000 - 4,000,000
• Start Date no later than September 30, 2010
• Applicants may only submit one application for
  consideration

16
17

• Any Questions?

April 14, 2010
17
18
Identification of Evidence-based Programs
Eligible for Replication
19
Identification of Evidence-based Programs

• Independent, systematic review of evidence base
  conducted by Mathematica Policy Research, under
  contract to HHS
• Steps of the Review
• Find potentially relevant studies
• Screen studies to review
• Assess the quality of studies
• Assess the evidence of effectiveness
• More detailed information available at
  http//www.hhs.gov/ophs/oah

19
20
Finding Relevant Studies

• Review of reference lists from earlier research
  syntheses
• Search of relevant research and policy
  organizations websites
• Public call for studies to solicit new and
  unpublished research
• Keyword search of electronic databases

20
21
Screening Criteria

• Use quantitative data and statistical analysis
  and hypothesis testing to measure impacts
• Measure impact on at least one sexual risk
  behavior or its health consequences sexual
  activity (initiation, frequency, number of
  partners), contraceptive use, sexually
  transmitted infections, pregnancies, or births
• Focus on youth ages 19 or younger in the United
  States at start of program
• Been conducted or published since 1989

21
22
Assessing Study Quality

• High, Moderate, or Low Rating based on
• Study design - Reassignment
• Attrition - Confounding factors
• Baseline equivalence
• High rating random assignment studies with low
  attrition and no sample reassignment
• Moderate rating quasi-experimental designs with
  well-matched comparison groups at baseline
  certain random assignments that didnt meet all
  criteria for high rating

22
23
Assessing Evidence of Effectiveness

• Supported by at least one high- or moderate-rated
  study showing a positive, statistically
  significant impact on at least one priority
  outcome (delay in sexual activity increase in
  contraceptive use decrease in STIs, pregnancies,
  or births) for either the full study sample or a
  key subgroup (defined by gender or baseline
  sexual experience)

23
24
Program Models Eligible for Replication

• 28 Evidence-based program models currently
  identified as eligible for replication with Tier
  1 Funding
• http//www.hhs.gov/ophs/oah/prevention/research/pr
  ograms/index.html
• Appendix A of Funding Announcement
• All studies that were reviewed but didnt make
  the Tier 1 list and the rationale for why the
  study didnt meet the review criteria are
  available in a searchable database at
  www.hhs.gov/ophs/oah

24
25
Program Models Currently Eligible for Replication

• HIV Risk Reduction Among Detained Adolescents
• Horizons
• Its Your Game Keep it Real
• Making a Difference!
• Making Proud Choices!
• Project TALC
• Promoting Health Among Teens!
• Reducing the Risk
• Rikers Health Advocacy Program (RHAP)
• Safer Sex
• Seattle Social Development Project
• SiHLE
• Sisters Saving Sisters
• Teen Health Project
• Teen Outreach Program
• What Could You Do?

• Aban Aya Youth Project
• Adult Identity Mentoring (Project AIM)
• All4You!
• Assisting in Rehabilitating Kids (ARK)
• Be Proud! Be Responsible!
• Be Proud! Be Responsible! Be Protective!
• Becoming a Responsible Teen (BART)
• Childrens Aid Society (CAS) Carrera Program
• Comprehensive Abstinence and Safer Sex
  Intervention
• Cuídate!
• Draw the Line/Respect the Line
• FOCUS

26
Where to Learn More About the 28 Program Models

• Intervention Implementation Reports on OAH
  website (www.hhs.gov/ophs/oah)
• Intervention Name and Developer
• Program Description
• Target Population
• Curriculum Materials
• Training and Technical Assistance
• Research Evidence

27
Can an organization apply to replicate program
models not in Appendix A?

• Yes, but only if all of the following criteria
  are
• met
• Research or evaluation of program model was not
  previously reviewed (previously reviewed evidence
  will not be re-reviewed)
• Research or evaluation meet the screening and
  evidence criteria
• Application must include all relevant research
  and evaluation (not included in 100 page limit)
• Application must be received by May 17, 2010

27
28

• Any Questions?

April 14, 2010
28
29
Program Expectations
30
Overview of Program Expectations

• Implement an evidence-based program model
• Maintain fidelity to the program model
• Address the target population
• Ensure medical accuracy and age-appropriateness
• Engage in phased-in implementation period
• Collect and report performance measurement data
• Adhere to evaluation expectations

30
31
Maintaining Fidelity to Program Model

• Must maintain fidelity to the core components
  of the original evidence-based model that led to
  the outcomes associated with the program
• Core Components are those parts of the
  curriculum or its implementation determined by
  the developer to be the key ingredients related
  to achieving the outcomes associated with program

31
32
Can the Program Models be Adapted?

• Minimal adaptations are allowed
• Changing names or details in a role play
• Updating out-dated statistics
• Adjusting reading and comprehension levels
• Making activities more interactive
• May propose adaptations to make program more
  relevant to ethnic, racial, or linguistic
  characteristics of the population to be served as
  long as core elements arent affected
• Significant adaptations (adding activities,
  changing sequence of activities, replacing
  supplementary materials) are not allowed under
  Tier 1

32
33
Demonstrating Ability to Replicate Program Models
with Fidelity

• Demonstrate effectiveness of replication strategy
  
• Ensure facilitators delivering program have been
  or will be formally trained
• Receive training on acceptable adaptations or
  propose adaptations for approval
• Monitor and document program implementation to
  ensure fidelity
• Provide MOUs stating that all partners have
  agreed to implement with fidelity

April 14, 2010
33
34
Medical Accuracy Age-Appropriateness

• Funded programs will need to ensure that
  information provided is age appropriate, and
  scientifically and medically accurate
• Materials will be reviewed by OAH prior to
  implementation to ensure medical accuracy
• Full curricula should NOT be submitted with the
  application. Program materials will be submitted
  to OAH for review and approval during the
  phased-in implementation period of the first
  grant year.

34
35
Phased-In Implementation Period

• Funded recipients will engage in a planning,
  piloting, and readiness period for the first 6 to
  12 months of funding
• During this period, grantees will
• Continue to assess needs and resources
• Finalize goals, objectives, and logic model
• Assess program fit
• Build organizational capacity
• Finalize implementation plans
• Pilot test program
• The length of the phased-in implementation may
  vary by grantee depending on implementation
  readiness
• OAH approval is required before full-scale
  implementation

35
36
Evaluation Strategies

• Monitoring and reporting on program
  implementation and outcomes through performance
  measures for all grantees
• Grantee-level evaluations for projects funded in
  Ranges C and D
• Federal evaluation of a selected subgroup of all
  grantees

36
37
Performance Measures

• Developed by OAH during first year of program
• All grantees will be expected to collect and
  report on common set of performance measures to
  assess program implementation and outcomes
• Training and technical assistance will be
  provided by OAH
• Anticipated categories for measures
• Output measures
• Fidelity/adaptation
• Implementation and capacity building
• Outcome measures
• Community data

37
38
Evaluation for Funding Ranges A B

• Must be able to demonstrate ability to collect
  and report on common set of performance measures
  to assess program implementation and outcomes
• Not expected to conduct rigorous independent
  grantee-level evaluation

April 14, 2010
38
39
Evaluation for Funding Ranges C D

• Rigorous independent grantee-level evaluation
  design unique to proposed project
• Use either random assignment or
  quasi-experimental design
• OAH will review and assess proposed evaluation
  designs
• OAH approval required before implementing
  evaluation plan
• OAH will provide training TA on evaluation
  general project specific.
• See Appendix C in the FOA for detailed guidance
• Budget 20-25 to support evaluation activities,
  but not more than 500,000 per year

39
40
Federal-level Evaluation of TPP

• Grantees from all funding ranges are required to
  participate in Federal evaluation, if selected
• Will agree to follow all evaluation protocols
  established by HHS
• Will no longer be expected to have separate
  grantee-level evaluation and will be required to
  redirect evaluation budget to support activities
  related to the Federal-level evaluation
• Decisions of which grantees will participate in
  the Federal evaluation will be made by end of the
  1st grant year

40
41
Items Requiring OAH Approval

• OAH approval required for
• Medical accuracy of curricula and program
  materials
• Proposed adaptations
• Evaluation plans
• Full-scale implementation
• Full curricula should NOT be submitted with the
  application. Program materials will be submitted
  to OAH for review and approval during the
  phased-in implementation period of the first
  grant year.

April 14, 2010
42

• Any Questions?

April 14, 2010
42
43
Application Contents
44
Contents of Application Submission

• Abstract (one-page)
• Project Narrative (no more than 50 pages)
• Organizational Capability Statement
• Project Management
• Need Statement
• Model to be Replicated and Project Approach
• Target Population
• Program Goal(s), Objectives, and Activities
• Work plan and Timetable
• Collaborations and Description of MOUs
• Performance Measurement
• Evaluation
• Appendices
• Budget Narrative/Justification

No more than 100 pgs
44
45
Project Abstract (1 page)

• Clear, concise description of the project that
  can be understood without reference to other
  parts of the application. Should include
• Project title
• Applicant contact information
• Type of organization applying
• Overarching goal(s)
• Evidence-based program model to be replicated
• Geographic area to be served
• Target population

45
46
Project Narrative Formatting

• Double-spaced
• 8 ½ x 11 inch (letter-size) pages
• 1-inch or larger margins on top, bottom, and both
  sides
• At least 12 point font
• All pages, charts, figures, and tables should be
  numbered

46
47
Project NarrativeOrganizational Capability

• Current capability to organize and operate
  effectively and efficiently
• Decision-making authority and structure
• Organizations experience, expertise and previous
  accomplishments in the area of teen pregnancy
  prevention
• Previous partnerships and strategies used to
  address teen pregnancy prevention
• How various sites and outside resources will be
  managed logistically and programmatically

47
48
Project NarrativeProject Management

• Plans to govern and manage the execution of the
  overall program
• Governance structure, roles/responsibilities,
  operating procedures, composition of committees,
  workgroups, terms and associated leaders, and
  communications plans
• How plans and decisions are developed and
  documented and issues/risks managed
• Specify mechanisms to ensure accountability among
  community participants and incremental progress
  in achieving milestones

48
49
Project NarrativeNeed Statement

• Geographic area to be served
• Benefit for the target population
• Incidence of teen births in the area
• Documentation of sexually transmitted disease
  rates, socio-economic conditions including income
  levels, existing services and unmet needs in the
  service area
• Unique challenges and barriers facing proposed
  population

49
50
Project NarrativeModel to be Replicated

• Rationale for choosing program model for
  replication and how approach is based on previous
  practice and community needs assessment
• Lessons learned from previous projects of this
  type
• Implementation site(s)
• Plans to implement model with fidelity
• Proposed adaptations to program model
• Plans to train staff and obtain implementation
  materials
• Plans to coordinate, integrate, and link to
  existing services within the service area
• Program management plan

50
51
Project NarrativeTarget Population

• Describe target population using statistical data
  and other community factors
• Provide realistic estimates of overall number of
  program participants and number participating in
  proposed project site(s)
• Describe expected number of participants during
  first and second year of implementation break
  out by age, race, and ethnicity
• Describe age appropriateness of program model

April 14, 2010
51
52
Project Narrative Program Goal(s), Objectives,
and Activities

• Specific program goal(s) and up to 6 outcome
  objectives
• SMART objectives
• S Specific
• M Measurable
• A Achievable
• R Realistic
• T Time-framed
• Logic Model (5-year) visual representation of
  relationships between proposed resources, planned
  activities, and desired outputs and outcomes

April 14, 2010
52
53
Project NarrativeWork plan Timetable

• Detailed five-year work plan
• Concise, easy-to-read overview of goals,
  strategies, objectives, measures, activities,
  timeline, and those responsible
• Include each activity associated with the
  replication, the proposed time frame for the
  activity, and the responsible staff
• Timetable for first year of project

April 14, 2010
53
54
Project NarrativeCollaborations

• Detail intent to coordinate with other community
  agencies and not duplicate existing efforts
• Describe expertise and capabilities of other
  partnering agencies
• Identify community stakeholders
• Include MOUs from each participating site,
  stakeholders, and outside resources (if
  applicable) in the Appendices
• Detail the specific nature of involvement and
  level of commitment of each stakeholder

April 14, 2010
54
55
Project NarrativePerformance Measurement

• Monitor progress on uniform set of process and
  outcome performance measures
• Describe capability to implement monitoring and
  reporting systems to aid in internal data
  collection around metrics for successful
  achievement of performance measures

April 14, 2010
55
56
Project NarrativeEvaluation (Ranges A B)

• Applicants in Funding Ranges A B
• Describe capacity to collect and report on common
  set of performance measures to assess program
  implementation and outcomes
• Describe how data will be used for ongoing
  program improvements

April 14, 2010
56
57
Project NarrativeEvaluation (Ranges C D)

• Applicants in Funding Ranges C D only
• Provide clear and fully developed grantee-level,
  independent evaluation plan in accordance with
  criteria in Appendix C of the FOA
• Describe proposed project, experimental design,
  processes to be tested, theory upon which
  intervention is based, proposed
  questions/hypotheses, data collection
  instruments, sampling and data collection plan,
  and data analysis plan
• Describe how data will be used for ongoing
  program improvements
• Describe plans to ensure confidentiality of data
• Include a MOU and CV from the independent
  evaluator

April 14, 2010
57
58
Project NarrativeFederal Evaluation

• Successful applicants must agree, if selected, to
  participate in a Federal evaluation, conducted by
  an independent contractor through a separate
  competitive award process

April 14, 2010
58
59
Appendices

• Resumes for Project Director and detailed
  position descriptions
• Program logic model (see the Application Kit for
  example)
• Memoranda of Understanding from all participating
  sites
• Memorandum of Understanding with independent
  evaluator
• Curriculum Vitae of independent evaluator
• Memoranda of Understanding from all partners
• Organizational chart
• Applicant organizations Federal-Wide Assurance
• Proof of nonprofit status, if applicable
• For applicants seeking to replicate a program
  model not on the list in Appendix A, all
  materials that support the claim that the model
  meets evidence-related criteria (not counted in
  100 page limit)

59
60
Budget Narrative/Justification

• State the funding range requested
• Outline proposed costs that support all project
  activities
• Thoroughly describe how the proposed costs are
  derived
• Breakdown each line item and provide an
  explanation of the costs
• Personnel should include salary per person and
  percent of time each person will spend on grant
• Specify the source of in-kind contributions (if
  applicable), and how the valuation of that
  contribution was determined

60
61
Budget Information Explanation of Line Items
for 424a

1. Personnel
2. Fringe Benefits- based on bylaws of organization
   taxes social security insurance benefits
   percent varies by organization
3. Travel- include OAH annual meeting 3 regional
   trainings
4. Equipment- items over 5,000 per unit
5. Supplies- items under 5,000 per unit
6. Contractual- may include independent evaluator
7. Construction not allowed
8. Other

61
62
Funding Restrictions

• Funds may not be used
• For building alterations or renovations,
  construction, fundraising activities, political
  education or lobbying
• To supplant or replace current public or private
  funding
• To supplant on-going or usual activities of any
  organization involved in the project
• To purchase or improve land, or to purchase,
  construct, or make permanent improvements to any
  building
• To reimburse pre-award costs

April 14, 2010
62
63
Other Required Application Forms

• Application for Federal Assistance (SF 424)
• Budget Information for Non-Construction Programs
  (SF-424a)
• Project Abstract
• Project Narrative Attachment Form
• Budget Narrative Attachment Form
• Assurances for Non-Construction Programs
  (SF-424B)
• Disclosure of Lobbying Activities (SF-LLL)
• HHS Certifications (08-2007)

63
64

• Any Questions?

April 14, 2010
64
65
Application Submission Instructions
66
Receipt Deadlines

• Letter of Intent May 3, 2010
• Include descriptive title, funding range, and
  contact information
• Submit via fax to [phone] or via email to
  oah.gov_at_hhs.gov
• Tier 1 Applications
• Electronic submission June 1, 2010 by 1100pm
  ET
• Paper submission June 1, 2010 by 500pm ET
• Applications to replicate a program not on the
  list in Appendix A May 17, 2010 by 500pm ET
  for paper applications or 1100pm ET for
  electronic applications (must include all
  relevant new evidence with the application
  evidence will not be counted in 100 page limit)

66
67
Electronic Submissions

• Grants.gov- http//www.grants.gov/
• GrantSolutions- https//www.grantsolutions.gov/
• Register well in advance
• Must be submitted no later than 1100pm Eastern
  Time on June 1,2010
• All required hardcopy original signatures and
  mail-in items must be received by the Office of
  Grants Management Grant Application Center no
  later than 500pm Eastern Time on June 2, 2010
• Applications are not considered valid until all
  electronic components, hardcopy original
  signatures, and mail-in items are received

67
68
Paper Submissions

• Must be received no later than 500pm Eastern
  Time on June 1,2010
• Address all materials to
• Office of Grants Management
• Office of Public Health and Science (OPHS)
• Department of Health and Human Services (DHHS)
• c/o Grant Application Center
• 1515 Wilson Blvd., Suite 100
• Arlington, VA 22209
• Attention Office of Adolescent Health
• Include the following on the lower left corner of
  your mailing envelope ATTENTION OFFICE OF
  ADOLESCENT HEALTH TIER 1 (NEW GRANT
  APPLICATION)

April 14, 2010
68
69
Application Tracking Form

• Applicants are strongly encouraged to complete
  and submit this form at the time of application
  submission
• Fax to [phone], or
• Email to oah.gov_at_hhs.gov
• Used by the OAH to track grant applications
• Form available in Appendix D of FOA or on the OAH
  website under Current Grant Announcements -
  Application Kit

April 14, 2010
69
70
Application Tips

• If submitting a hardcopy by mail or hand
  delivering
• Do NOT staple or otherwise bind your application
• Do NOT send any extraneous materials such as
  videos, books, etc.
• Do NOT send double-sided information/ pamphlets
• Do NOT submit the proposed curriculum or
  educational materials- only include the Table of
  Contents in the application
• DO double space on 8 ½ X 11 paper
• DO submit an original and two copies

70
71
Application Tips

• Read the entire FOA and application kit BEFORE
  writing
• Write the One Page Summary - AFTER the entire
  narrative is complete
• Read the review criteria
• Read the application instructions
• Do NOT exceed the 50 page limit for the project
  narrative OR the total page limit of 100 pages
  for the full application
• Turn in application no later than 5pm ET (hard
  copies) or 11pm (electronic copies) on June 1,
  2010
• Allow time for any unforeseen difficulties with
  the on-line application process, etc.

71
72
Application Tips

• FOA is the primary guide to programmatic
  requirements
• Follow what the FOA says

April 14, 2010
72
73
Review Selection Criteria
74
The Application Review Process Review Criteria

• Project Approach and Work Plan (35 points)
• Organizational Capacity and Experience (25
  points)
• Project Management and Staffing (15 points)
• Evaluation Plans and Performance Measurement (15
  points)
• Budget (10 points)

74
75
Selection Process

• Objective Review Committee
• Expert peer reviewers
• Federal Staff
• Applications reviewed according to the published
  review criteria

76
Final Award Decision Criteria

• Score and rank order given by the Objective
  Review Committee
• Availability of funds
• Representation of programs across communities,
  including varied types of interventions and
  evidence-based strategies
• Geographic distribution of projects
• Inclusion of communities of varying sizes
• Feasibility of evaluation plan
• Inclusion of range of populations
  disproportionately affected by teenage pregnancy

77
Notification of Funding

• Notice of Grant Award notifies successful
  applicant of selection
• Includes any conditions on the award

78

• Any Questions?

78
79
Thank you!

• Office of Adolescent Health
• Website - www.hhs.gov/ophs/oah
• Email oah.gov_at_hhs.gov
• Phone [phone]
• Office of Grants Management
• Email karen.campbell_at_hhs.gov
• Phone [phone]

79