Lessons Learned About Fidelity

1
Lessons Learned About Fidelity Adaptation from
the Massachusetts SIG

• Wayne Harding, Ed.M., Ph.D.
• Social Science Research Evaluation, Inc.
• 21-C Cambridge Street
• Burlington, Massachusetts 01803
• 781-273-4206, wayneh_at_gis.net

2
Topics

• Why is fidelity adaptation an issue?
• Massachusetts SIG (MassCALL) policies and
  procedures concerning fidelity and adaptation.
  
• Lessons learned.
• Fidelity monitoring tool.
• Massachusetts Collaboration for Action,
  Leadership, and Learning.

3
Why Is Fidelity / AdaptationAn Issue?

• Replication can be hard.
• Limited supply of science-based programs makes
  adaptation likely.
  
• Adaptation happens.

4
Adaptation Happens

• Between 23 and 81 of program activities may be
  omitted during implementation (Durlak, 1998).
  
• Only 19 of schools implemented research-based
  curricula with fidelity (Hallfors and Godette,
  2002).
  
• Only 75 of students received at least 60 of the
  Life Skills Training sessions (Botvin et al.,
  1995).
  

5
MassCALL Pre-award Policy Monitoring
Procedures

• Pre-RFP meetings discussed science-based
  prevention, importance of selecting appropriate
  programs, and guidelines for adaptation.
  

6
MassCALL Pre-award Policy Monitoring
Procedures

• The RFP requested information about adaptations
  planned and rationales for them.
  

7
Post-award Policy Monitoring Procedures

• Required evaluation plans that identified any
  adaptations and plans to monitor program
  implementation.
  
• Site visits to all subrecipients included
  importance of fidelity, identification of
  adaptations, alternatives to adaptation.

8
Post-award Policy Monitoring Procedures

• MassCALL implemented an approval process for
  adaptations.
  
• Biannual reports from subrecipients included
  information on any adaptations and rationales for
  them.
  

9
Lessons Learned Adaptation Is Likely
Types and Frequency of Adaptations Among
MassCALL Subrecipients (Formica Harding 2002)

10
Lessons LearnedAdaptation Is Likely

• All 21 program sites made one or more program
  adaptations.
  
• Adaptations occurred for all program types.
• The adaptations applied to all aspects of the
  programs (content, delivery methods, target
  population, setting, delivery agent).
  
• Conservatively, 83 adaptations were made.

11
Lessons Learned

• Adaptation may continue.
• Important to track adaptation to both original
  and proposed program.
  
• Important to track adaptation to evaluation
  procedures.

12
Lessons Learned

• Adaptation can be tracked accurately.
• Asking about the variety of types of adaptations
  may increase accuracy with which they are
  reported.
  

13
Fidelity Tool

• Assess fidelity to original or model program and
  to the proposed (adapted) program.
  
• Assess fidelity at intervals.
• Assess multiple aspects of the program (e.g.,
  content, delivery methods, population served,
  setting, delivery agent).
  
• Assess the rationale for adaptation.
• Assess adaptations to evaluation methods (in both
  original and proposed program).
  
• Permits assessment/comparisons across program
  types.
  

14
References

• Botvin, G.J., Baker, E., Dusenbury, L., Botvin,
  E.M., Diaz, T. (1995). Long-term follow-up
  results of a randomized drug abuse prevention
  trial in a white middle-class population.
  Journal of the American Medical Association, 273
  1106-1112.
•  
• Durlack, J.A. (1998) Why Program Implementation
  Is Important. Journal of Prevention
  Intervention in the Community. 17(2) 5-18.
  
• Formica, S. and Harding, W. M. (2001).
  Documentation of adaptations to science-based
  programs implemented by MassCALL sub-recipient
  communities. Massachusetts Collaboration for
  Action, Leadership, Learning (MassCALL),
  Massachusetts Department of Public Health, Bureau
  of Substance Abuse Services, Boston, MA.
• Hallfors, D. and Godette, D. (2002) Will the
  "Principles of Effectiveness' Improve Prevention
  Practice? Early Findings from a Diffusion Study.
  Health Education Research. 17(4) 461-470.
•  
•