Common Measures of Implementation Fidelity: SPF SIG Crosssite Workgroup A Briefing for the Western C - PowerPoint PPT Presentation

1 / 22
About This Presentation
Title:

Common Measures of Implementation Fidelity: SPF SIG Crosssite Workgroup A Briefing for the Western C

Description:

Developing Fidelity Assessment Rubrics for Environmental Strategies ... Fidelity of implementation getting increased attention (Fixsen et al, 2004) ... – PowerPoint PPT presentation

Number of Views:188
Avg rating:3.0/5.0
Slides: 23
Provided by: royga
Category:

less

Transcript and Presenter's Notes

Title: Common Measures of Implementation Fidelity: SPF SIG Crosssite Workgroup A Briefing for the Western C


1
Common Measures of Implementation Fidelity SPF
SIG Cross-site Workgroup A Briefing for the
Western CAPT
  • Presented by
  • Roy M. Gabriel, Ph.DCross-site Workgroup Chair
    and Washington SPF SIG EvaluatorApril 1, 2009

2
Workgroup Purpose
  • Develop instruments and methods for assessing
    fidelity of implementation at community level for
    use in state- and national-level evaluation
  • Goal A scoring method to represent variability
    among SPF SIG communities in their
    quality/fidelity of implementation

3
Workgroup Members
  • Cross-site Evaluation
  • Rob Orwin, Westat
  • Ann Landy, Westat
  • Al Stein-Seroussi, PIRE
  • Doug Piper, PIRE
  • Elisabeth Duchesneau, Westat
  • Melissa Gutierrez, Westat
  • Janis Wolford, Westat
  • Kristi Pettibone, Maya Tech
  • State
  • Christine Owens, MO
  • Jeffrey Metzger, NH
  • Suzanne Kennedy-Leahy, CO
  • David Currey, NV
  • Mary Dugan, MO
  • Susan Depue, MO
  • Roy Gabriel, WA (Chair)

4
The Strategic Prevention Framework Steps 1-5
  • 1. Profile needs, resources and readiness
  • 2. Mobilize and Build Capacity
  • 3. Develop Strategic Prevention Plan
  • 4. Implement EBPs and Infrastructure Development
    Activities
  • Stage 1 Selecting, Planning for EBPs
  • Stage 2 Implementing EBPs
  • 5. Monitor process, evaluate effectiveness and
    sustain

5
Approach
  • Specify the key components or activities within
    each strategy (What needs to happen for each
    strategy to be done well?)
  • Develop rating scales to assess the quality of
    implementation of each of these activities
  • Develop scoring rubrics to arrive at a composite
    rating of implementation
  • Provide guidance for using these tools

6
Addressing the Implementation of Environmental
Strategies
  • Consulted with cross-site Environmental
    Strategies work group
  • They had conducted a survey of SPF SIG states
    planning of environmental strategies
  • 40 environmental strategies reported
  • Categorized into four groups
  • Media campaigns/strategies
  • Law enforcement interventions
  • Policy change
  • Targeted education/training

7
Developing Fidelity Assessment Rubrics for
Environmental Strategies
  • Back to the Implementation Fidelity workgroup
  • Level of specificity of fidelity assessment
    needed
  • Unique to each strategy? (n40)
  • Common for those in same group? (n4)
  • Common to all environmental strategies? (n1)
  • Initial Workgroup approach focus on group-level
  • Identify core activities
  • Devise rating scales for rubric
  • Began with Media Campaigns/Strategies

8
Group 1 Media Campaigns/Strategies
  • Types of Media Campaigns/Strategies
  • Social Marketing Campaigns
  • Social Norms Marketing Campaigns
  • Mass Media Campaigns
  • Media Advocacy

9
What is Social Marketing?
  • Definition used on the Cross-site Evaluation
    Community-Level Instrument
  • Using commercial marketing techniques to develop,
    implement, and evaluate programs designed to
    influence the behavior of a target audience.
  • Social marketing often relies on the use of mass
    media and involves identifying the needs of a
    specific group, supplying information so people
    can make informed decisions, offering services
    that meet needs, and assessing how well the needs
    were met (CLI definition).

10
What is Social Norms Marketing?
  • gathers credible data from a target population
    and then, using various health communication
    strategies, consistently tell the truth about its
    actual norms of health, protection, and the
    avoidance of risk behaviors (Haines et al.,
    2005).

11
What is a Mass Media Campaign?
  • Any promotion activity that uses the mass media
    (including print, television, radio, and the
    internet) to communicate a message.

12
Overlapping Strategies
  • Social norms marketing is a subset of social
    marketing
  • Mass media campaigns are one tool that could be
    used within a social marketing or social norms
    marketing campaign

13
What is Media Advocacy?
  • an approach to media that helps empower people
    to tell their own story proactively rather than
    waiting passively for the media to get it right
    (Wilbur Stewart, 1999).

14
Core Activities of Social Marketing/Social Norms
Marketing/Mass Media Campaigns
  • Plan in place
  • Formative research conducted
  • Use of local data
  • Multiple media used
  • Frequency of exposure
  • Repetition conducted

15
Core Activities of Media Advocacy
  • Get to know media contacts
  • Monitor the media
  • Frame the issue
  • Stay on message
  • Link message to issue in public consciousness
  • Use local statistics to tell story

16
Rating Scales for Key Components (generic)
  • 0Missing or not done
  • 1 Weak Fidelity
  • 2Moderate Fidelity
  • 3Strong Fidelity

17
Identifying Evidence-based Environmental
Strategies The Environmental Strategies SPF SIG
cross-site workgroup
  • Reviewed prevention literature for articles that
  • Described the environmental strategy
  • Included guidelines for implementation
  • Identified output measures for evaluation
  • Evaluated the effectiveness of the strategy for
    reducing substance abuse
  • Produced a resource notebook including 27
    environmental strategies 23 of which had
    sufficient implementation guidance to identify
    core activities or components

18
Environmental Strategy Fidelity Assessment
Materials
  • Assessment rubrics developed for 23 environmental
    strategies
  • Core activities w/rationale for their inclusion
  • Fidelity rating scales for each core activity
  • Fidelity materials reviewed by
  • Expert Technical Advisory Group to SPF SIG
    cross-site evaluation
  • Selected state-level SPF SIG evaluators
  • Selected prevention practitioners

19
Major issues arising in reviews
  • This is not an exhaustive list!
  • Criteria for inclusion not always met
  • Why is evaluation not included as a core
    activity? when indicated in literature?
  • Evaluation is understood as SPF Step 5
  • Assessing fidelity of implementation over time?
  • Necessary
  • How do you account for adaptations?
  • As with other EBPs, fidelity is addressed
    w/respect to how community planned to implement
    the strategy, rather than how it was designed,
    i.e., incorporating adaptations

20
Status of Environmental Strategies Fidelity
materials
  • Following reviews and revisions, updated draft
    posted on SPF SIG Web Board
  • Will be organized by Intervening Variable1 they
    address, rather than general strategy type
  • Retail Access
  • Social Access
  • Social Norms
  • Promotion
  • Perceived Risk
  • Pricing
  • Enforcement
  • Other
  • 1 From Birckmayer, Holder, et al. (2004). A
    general causal model to guide alcohol,
    tobaccoand illicit drug prevention Assessing
    the research evidence. Journal of Drug Education.

21
Again, why are we doing this?
  • Fidelity of implementation getting increased
    attention (Fixsen et al, 2004)
  • Lack of attention to it dubbed a Type III error
    in research methodology (Dobson Cook, 1980),
    clouding the interpretation of outcomes
  • Hypothesis that variations in implementation
    fidelity and quality are associated with program
    effectiveness (Longhi, 2005)

22
Summary Plot Washington SIG Community Outcomes
by Implementation Level
2000-2002 Decreases in Substance Use among 8th
Graders by Poor Average and Best Implementers
NOTE Correlation between higher levels of
implementation and better outcomes (substance
abuse decreases) was .94.
Write a Comment
User Comments (0)
About PowerShow.com