Title: Common Measures of Implementation Fidelity: SPF SIG Crosssite Workgroup A Briefing for the Western C
1 Common Measures of Implementation Fidelity SPF
SIG Cross-site Workgroup A Briefing for the
Western CAPT
- Presented by
- Roy M. Gabriel, Ph.DCross-site Workgroup Chair
and Washington SPF SIG EvaluatorApril 1, 2009
2Workgroup Purpose
- Develop instruments and methods for assessing
fidelity of implementation at community level for
use in state- and national-level evaluation - Goal A scoring method to represent variability
among SPF SIG communities in their
quality/fidelity of implementation
3Workgroup Members
- Cross-site Evaluation
- Rob Orwin, Westat
- Ann Landy, Westat
- Al Stein-Seroussi, PIRE
- Doug Piper, PIRE
- Elisabeth Duchesneau, Westat
- Melissa Gutierrez, Westat
- Janis Wolford, Westat
- Kristi Pettibone, Maya Tech
- State
- Christine Owens, MO
- Jeffrey Metzger, NH
- Suzanne Kennedy-Leahy, CO
- David Currey, NV
- Mary Dugan, MO
- Susan Depue, MO
- Roy Gabriel, WA (Chair)
4The Strategic Prevention Framework Steps 1-5
- 1. Profile needs, resources and readiness
- 2. Mobilize and Build Capacity
- 3. Develop Strategic Prevention Plan
- 4. Implement EBPs and Infrastructure Development
Activities - Stage 1 Selecting, Planning for EBPs
- Stage 2 Implementing EBPs
- 5. Monitor process, evaluate effectiveness and
sustain
5Approach
- Specify the key components or activities within
each strategy (What needs to happen for each
strategy to be done well?) - Develop rating scales to assess the quality of
implementation of each of these activities - Develop scoring rubrics to arrive at a composite
rating of implementation - Provide guidance for using these tools
6Addressing the Implementation of Environmental
Strategies
- Consulted with cross-site Environmental
Strategies work group - They had conducted a survey of SPF SIG states
planning of environmental strategies - 40 environmental strategies reported
- Categorized into four groups
- Media campaigns/strategies
- Law enforcement interventions
- Policy change
- Targeted education/training
7Developing Fidelity Assessment Rubrics for
Environmental Strategies
- Back to the Implementation Fidelity workgroup
- Level of specificity of fidelity assessment
needed - Unique to each strategy? (n40)
- Common for those in same group? (n4)
- Common to all environmental strategies? (n1)
- Initial Workgroup approach focus on group-level
- Identify core activities
- Devise rating scales for rubric
- Began with Media Campaigns/Strategies
8Group 1 Media Campaigns/Strategies
- Types of Media Campaigns/Strategies
- Social Marketing Campaigns
- Social Norms Marketing Campaigns
- Mass Media Campaigns
- Media Advocacy
9What is Social Marketing?
- Definition used on the Cross-site Evaluation
Community-Level Instrument - Using commercial marketing techniques to develop,
implement, and evaluate programs designed to
influence the behavior of a target audience. - Social marketing often relies on the use of mass
media and involves identifying the needs of a
specific group, supplying information so people
can make informed decisions, offering services
that meet needs, and assessing how well the needs
were met (CLI definition).
10What is Social Norms Marketing?
- gathers credible data from a target population
and then, using various health communication
strategies, consistently tell the truth about its
actual norms of health, protection, and the
avoidance of risk behaviors (Haines et al.,
2005).
11What is a Mass Media Campaign?
- Any promotion activity that uses the mass media
(including print, television, radio, and the
internet) to communicate a message.
12Overlapping Strategies
- Social norms marketing is a subset of social
marketing - Mass media campaigns are one tool that could be
used within a social marketing or social norms
marketing campaign
13What is Media Advocacy?
- an approach to media that helps empower people
to tell their own story proactively rather than
waiting passively for the media to get it right
(Wilbur Stewart, 1999).
14Core Activities of Social Marketing/Social Norms
Marketing/Mass Media Campaigns
- Plan in place
- Formative research conducted
- Use of local data
- Multiple media used
- Frequency of exposure
- Repetition conducted
15Core Activities of Media Advocacy
- Get to know media contacts
- Monitor the media
- Frame the issue
- Stay on message
- Link message to issue in public consciousness
- Use local statistics to tell story
16Rating Scales for Key Components (generic)
- 0Missing or not done
- 1 Weak Fidelity
- 2Moderate Fidelity
- 3Strong Fidelity
17Identifying Evidence-based Environmental
Strategies The Environmental Strategies SPF SIG
cross-site workgroup
- Reviewed prevention literature for articles that
- Described the environmental strategy
- Included guidelines for implementation
- Identified output measures for evaluation
- Evaluated the effectiveness of the strategy for
reducing substance abuse - Produced a resource notebook including 27
environmental strategies 23 of which had
sufficient implementation guidance to identify
core activities or components
18Environmental Strategy Fidelity Assessment
Materials
- Assessment rubrics developed for 23 environmental
strategies - Core activities w/rationale for their inclusion
- Fidelity rating scales for each core activity
- Fidelity materials reviewed by
- Expert Technical Advisory Group to SPF SIG
cross-site evaluation - Selected state-level SPF SIG evaluators
- Selected prevention practitioners
19Major issues arising in reviews
- This is not an exhaustive list!
- Criteria for inclusion not always met
- Why is evaluation not included as a core
activity? when indicated in literature? - Evaluation is understood as SPF Step 5
- Assessing fidelity of implementation over time?
- Necessary
- How do you account for adaptations?
- As with other EBPs, fidelity is addressed
w/respect to how community planned to implement
the strategy, rather than how it was designed,
i.e., incorporating adaptations
20Status of Environmental Strategies Fidelity
materials
- Following reviews and revisions, updated draft
posted on SPF SIG Web Board - Will be organized by Intervening Variable1 they
address, rather than general strategy type - Retail Access
- Social Access
- Social Norms
- Promotion
- Perceived Risk
- Pricing
- Enforcement
- Other
- 1 From Birckmayer, Holder, et al. (2004). A
general causal model to guide alcohol,
tobaccoand illicit drug prevention Assessing
the research evidence. Journal of Drug Education. -
21Again, why are we doing this?
- Fidelity of implementation getting increased
attention (Fixsen et al, 2004) - Lack of attention to it dubbed a Type III error
in research methodology (Dobson Cook, 1980),
clouding the interpretation of outcomes - Hypothesis that variations in implementation
fidelity and quality are associated with program
effectiveness (Longhi, 2005)
22Summary Plot Washington SIG Community Outcomes
by Implementation Level
2000-2002 Decreases in Substance Use among 8th
Graders by Poor Average and Best Implementers
NOTE Correlation between higher levels of
implementation and better outcomes (substance
abuse decreases) was .94.