PLANNING FOR AND EVALUATING THE IMPACT OF PUBLIC HEALTH PROGRAMS AND POLICIES: APPLYING REAIM TO DIS

1
PLANNING FOR AND EVALUATING THE IMPACT OF
PUBLIC HEALTH PROGRAMS AND POLICIES APPLYING
RE-AIM TO DISSEMINATION ISSUES

• Russell E. Glasgow, Ph.D.
• Kaiser Permanente Colorado
• CDC 11/15/06

2
OVERVIEW

• Issues in evaluating public health impact
• Examples of using RE-AIM to evaluate impact
• Issues specific to health policies
• Applying RE-AIM to health policies
• Future directions and discussion

3
The law of halves a story
4
ULTIMATE IMPACT OF MAGIC DIET PILL
Dissemination Step Concept Impacted
50 of Clinics Use Adoption 50
50 of Clinicians Prescribe Adoption 25
50 of Patients Accept Medication Reach 12.5
50 Follow Regimen Correctly Implementation 6.2
50 of Those Taking Correctly Benefit Effective
ness 3.2
50 Continue to Benefit After 6
Months Maintenance 1.6
5
(No Transcript)
6
RE-AIM TO HELP PLAN, EVALUATE, AND REPORT STUDIES

• R Increase Reach
• E Increase Effectiveness
• A Increase Adoption
• I Increase Implementation
• M Increase Maintenance

Glasgow, et al. Ann Behav Med 200427(1)3-12
7
PURPOSES OF RE-AIM

• To broaden the criteria used to evaluate programs
  to include external validity
• To evaluate issues relevant to program adoption,
  implementation, and sustainability
• To help close the gap between research studies
  and practice by
• Informing design of interventions
• Providing guides for adoptees
• Suggesting standard reporting criteria

8
RE-AIM DIMENSIONS AND DEFINITIONS
www.re-aim.org
9
RE-AIM DIMENSIONS AND DEFINITIONS (cont.)
www.re-aim.org
10
RECOMMENDED PURPOSE OF TRANSLATION/EFFECTIVENESS
RESEARCH
To determine the characteristics of interventions
that can

• Reach large numbers of people, especially those
  who can most benefit
• Be widely adopted by different settings
• Be consistently implemented by staff members with
  moderate levels of training and expertise
• Produce replicable and long-lasting effects (and
  minimal negative impacts) at reasonable cost

11
(No Transcript)
12
(No Transcript)
13
REACH WHICH IS BETTER?
In-office attracted 50 (vs. 41) of eligible
participants
PERCENT
14
ADOPTION WHICH IS BETTER?
Linked, separate approach recruited 47 of PCPs
in-office PCP team delivery approach recruited
6 of PCPs
PERCENT
15
WHICH PROGRAM IS BETTER?
See www.re-aim.org for displays and evaluation
questions
Glasgow et al. AJPM 30(1)67-73
16
NEW RE-AIM SUMMARY METRICS THAT ADDRESS

• Health disparities e.g., who participates and
  who benefits
• Costs and cost-effectiveness
• Effects of different interventionists
• Combining different factors to produce composite
  outcomes

Glasgow, et al. Using RE-AIM Metrics to Evaluate
DiabetesAJPM 200630(1)67-73
17
POLICY ISSUES
The first priority is to develop better tools to
assess the effects of policies, to guide policy
development, and to prioritize policy choices.
Schmid TL, Pratt M, Whitmer L, in J Phys Activity
Health 20063(Suppl 1)S20-S29)
18
ISSUES IN IDENTIFYING INTENDED POLICY AUDIENCE(S)

• Who makes the Decision or Policy (Adopting
  Organization)
• Who is Responsible for Enforcing the Policy
• Who is Responsible for Following or Adhering to
  the Policy?

19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Jilcott S, Ammerman A, Sommers J, Glasgow RE.
Assessing the Public Health Impact of Policy
Change Under Revision for Am J Public Health
24
(No Transcript)
25
(No Transcript)
26
STEPS IN IDENTIFYINGTHE DENOMINATOR
ANDEVALUATING REACH AND ADOPTION

• Who is excluded at recruitment (both
  intentionally and unintentionally)
• Participation rate among those targeted passive
  vs. active policies
• Representativeness of those who participate
• Characteristics of participants vs. those
  declining
• Characteristics of participants vs. general
  target audience in that area using secondary data

27

• Assess if Implementation Varies
• Over time
• Across intervention implementation staff
• Across population subgroups

28
CAVEATS AND COMPLEXITIES

• REACH Impact on health disparities and who is
  reached are critical
• EFFECTIVENESS - Policy outcomes should usually
  change over time
• ADOPTION Are there adequate funds for
  enforcement?

29
CAVEATS AND COMPLEXITIES (cont.)

• IMPLEMENTATION Consistency across enforcing
  agents and population subgroups
• MAINTENANCE Here as well as throughout,
  contextual factors are critical and qualitative
  as well as quantitative data are helpful

30
APPLICATIONS OF RE-AIMTO HEALTH POLICIES

• When planning and writing new policies
• When comparing policy alternatives
• Considering policies relative to other
  alternatives
• Evaluating the impact of specific policies
• Reviewing the literature on policy impact

31
To every complex question, there is a simple
answer and it is wrong. H. L. Mencken
32
RE-AIM AND RELATED GENERALIZATION ISSUES

• The 3 Rs of Integrating Research into Practice
• Representativeness (Reach, Adoption)
• Robustness (Effectiveness across
  subgroupsespecially re disparities)
• Cronbachs generalization across persons,
  time, measures
• Replicability (Implementation) in
  representative settings

Cronbach LH, et al. The dependability of
behavioral measurements Theory of
generalizability for scores and profiles. New
York, John Wiley Sons, 1972 Shadish WR, et al.
Experimental and quasi-experimental design for
generalized causal inference. Boston Houghton
Mifflin, 2002
33
SUMMARY

• Need to Plan for Implementation and Dissemination
  from the outset, for both programs and policies
• Attend to all barriers to dissemination
• Track adaptation over timelearn from it
• Context is critical, and often best addressed by
  combination of quantitative and qualitative
  assessment

34
Questions, Counterpoint, Discussion