Implementing What Works

1
Implementing What Works
Daniel Perkins Brian Bumbarger Pennsylvania
State University
2
Importance of Research Standards

• There could be no wiser investment in our
  country than a commitment to foster the
  prevention of mental disorders or problem
  behaviors and the promotion of mental health
  through rigorous research with the highest of
  methodological standards. Such a commitment would
  yield the potential for healthier lives for
  countless individuals and the general advancement
  of the nation's well-being.
• Institute of Medicine- 1994

3
What are Evidence-based Programs?The Gold
Standard

• Strong evidence of effectiveness
• Randomized controlled trials, well designed and
  implemented
• Trials showing effectiveness in two or more
  settings (including a setting similar to that of
  school/classroom implementing the program) (at
  least 300 students or 50-60 classrooms.)
• Quality Quantity Strong Evidence

4
Why is a Randomized Clinical Trial (RCT)
Convincing?

• We know unequivocally if a program is effective
• Not due to pre-test differences
• Not due to other changes that might explain
  effects
• Replication of effects using RCT greatly
  increases confidence that the program causes the
  changes
• Examples Nurse Family Partnership Program PATHS
• Life Skills Training SFP 10-14 Early Head
  Start

5
What is Convincing?

• The choice of research methodologies is a major
  issue in examining preventive interventions and
  research trials designed to determine their
  outcomes. It determines whether evidence is
  compelling. The ideal design is a randomized
  controlled trial.
• Institute of Medicine- 1994

6
RCTs, Service, Ethics

• Tension between rigorous science and providing
  services.
• Without evidence of positive impacts then we
  cannot be certain that what we are doing is
  working, or worse, is not causing harm.
• Not providing services to all feels like we are
  not being truthful to community.
• Agencies who provide innovative programs almost
  uniformly believe that their programs work
• There is a need for a clear framework of
  accountability
• The answer is compromise in terms of providing
  the best service and at the same time doing the
  most rigorous possible science.

7
Why is a Randomized Clinical Trial Not Sufficient
by Itself?

• There is a need for replication
• There is a need to show effects across different
  populations
• Ethnicity, Urban/Rural, Levels of Education,
  Types of Communities
• There is a need for a carefully developed set of
  training procedures to ensure fidelity when
  disseminated
• There is a need to learn how to flexibly adapt
  some aspects of the model to the culture of
  different communities

8
(No Transcript)
9
In the past

• 20 years ago, there were NO empirically-validated
  prevention programs
• Efforts were guided primarily by good
  intentions and gut instinct
• Hundreds of millions of dollars were spent
  without any accountability
• Prevention was considered more art than
  science

10
Now

• Two decades of rigorous scientific research have
  informed our knowledge of epidemiology, etiology,
  methodology, and prevention practice
• We have learned more about what causes and what
  works to preventing youth problem behaviors and
  promoting positive youth development in the last
  20 years than we did in the previous 200 years
• We have tested theories of changes (public health
  model) that guide our program
• Today, there are many programs that have been
  proven effective in well-designed studies and
  have been independently replicated
• There is clearly a science of prevention!

11
Why Evidence-based Programs?

• Required use of scientifically-based research
  to decide which interventions to use and those
  that will be funded
• Accountability
• To ensure the smart use of LIMITED resources

12
When are Evidence-based Programs Needed?

• When you want to increase the likelihood that
  your program will have expected impacts
  (long-term).
• When there is support to implement an
  evidence-based program with rigor by the
  collaborators. (Evidence-based programs take a
  lot a time to implement if done right)

13
The Impact of Programs that Workmodelprograms.sam
hsa.gov

• Life Skills Training cut tobacco, alcohol, and
  marijuana use 50 - 75
• Nurse Home Visitation reduced alcohol use by 56
  in children 15 years after the intervention
• Project TND found a 26 reduction in regular hard
  drug use
• All-Stars reduced poly-drug use 40-60 at
  immediate post-test
• Project Alert reduced marijuana use initiation by
  30 and regular marijuana use by 60

14
Prevention is Cost-effectivewww.wa.gov/wsipp(me
asured benefits and cost per youth)
15
Still more work to do

• Most prevention programs being utilized are not
  EBIs
• Research has shown that most are not being
  implemented with fidelity
• There is tension between rigorous science design
  and providing services to all

16
Next Steps With Research-Based Programs Ensure
Implementation Quality

• When Communities Adopt Research-Based Programs
  the Central Concerns Are
• Maintaining High Fidelity
• Understand What Factors Influence Implementation
  Quality
• Leads to A New Generation of Research Questions
• What factors influence the quality of
  implementation?
• How does implementation quality effect outcome?

17
Our work must emphasize deliberate investment
in positive factors that research has shown to be
closely tied to reduced levels of negative
behaviors as well as increased levels of
thriving resiliency attitudes and
behaviors. (Blyth, 2000)
18
Replication of Effective ProgramsPAs
Blueprints Initiative
19
Risk-focused Prevention Planning (the CTC model)
Collect local data on risk and protective factors
Use data to identify priorities
Select and implement evidence-based program that
targets those factors
Re-assess prevalence of risk and protective
factors
20
Issues Challenges

• Readiness/Program Selection
• Understanding of program and what is required
• Buy-in of key stakeholders
• Training availability/access
• Cost, timeliness and turnover
• Fidelity
• Ongoing TA
• Monitoring/measurement
• Measurement of program impact
• Sustainability

21
Why does fidelity matter?

• Research has clearly linked fidelity with
  positive outcomes
• Higher fidelity is associated with better
  outcomes across a wide range of programs and
  practices (PATHS, MST, FFT, TND, LST and others)
• Fidelity enables us to attribute outcomes to the
  intervention, and provides information about
  program feasibility

22
The reality.

• While possible, fidelity is not a naturally
  occurring phenomenon adaptation (more
  accurately program drift) is the default
• Most adaptation is reactive rather than proactive
• Most adaptation weakens rather than strengthens
  the likelihood of positive outcomes

23
Adaptation happens

• Between 23 and 81 of program activities may be
  omitted during implementation. (Durlak, 1998)
• Only 19 of schools implement research-based
  curricula with fidelity. (Hallfors Godette,
  2002)
• Only about 75 of the students received 60 or
  more of the Life Skills Training Program.
  (Botvin, et al., 1995)

24
Adaptation as a Function of Training (formal
training by the developer)
25
Is adaptation inevitable/necessary?

• Research shows that a high degree of fidelity is
  attainable (Project TND, PROSPER, Blueprints)
• There is little empirical support for cultural
  adaptation of EVPs
• Most have shown similar effects across gender,
  ethnicity/race, SES
• Studies of prospective cultural adaptations have
  failed to yield positive outcomes

26
Improving fidelity locally

• What gets measured matters
• Improve practitioner knowledge of prevention
  science
• Use adaptation discussion as a tool for training
  on the logic model of an intervention
• Build a sustainable infrastructure for monitoring
  implementation fidelity and quality
• Build internal capacity AND desire

27
Building internal capacity and motivation

• Approach fidelity from a practical,
  accountability perspective dont make it a
  research issue
• The goal is to develop local intrinsic motivation
  for monitoring fidelity and quality of program
  delivery it must be tied to outcomes
• Involve local practitioners/implementers in the
  development and conduct of evaluation
• Process evaluation is fidelity monitoring

28
Practical strategies

• Peer coaching, peer observation
• Schedule regular opportunities for reflective
  practice and de-briefing
• Never let the initial training be the only
  training
• Data in must ALWAYS require data out create
  feedback loops and safe environments for
  reflection
• Foster internal competition
• Emphasize the importance of a clear understanding
  of a programs logic model

29
Where to find evidence-based interventions

• The What Works Clearinghouse
• (http//www.,w-w-c.org/)
• SAMHSA National Registry of Effective Prevention
  Programs
• ((http//www.modelprograms.samhsa.gov))
• The Promising Practices Network
  (http//www.promisingpractices.net/)
• Blueprints for Violence Prevention
  (http//www.colorado.edu/cspv/blueprints/index.htm
  l)
• The International Campbell Collaboration
• (http//www.campbellcollaboration.org/Fralibrary.
  html)
• Safe and Sound An Educational Leaders Guide to
  Evidence-Based Social and Emotional Learning
  Programs (http//www.CASEL.org)
• Social Programs that Work (http//www.excel.gov.or
  g/displayContent.asp?KeywordprppcSocial)
• Center for Disease Control Effective Programs
• (http//www.cdc.gov?healthyYouth.partners/registr
  ies.htm)

30

• When one has no stake in the way things are, when
  ones needs and opinions are provided no forum,
  when one sees oneself as the object of unilateral
  actions, it takes no particular wisdom to suggest
  that one would rather be elsewhere.
• -S. Sarason, 1990

31
Successful Community Engagement

• Use data about strengths and needs of community
  to inform your selection strategies
• Agencies and staff buy-in is critical
• Participation in decision-making understanding
  of overall logic model
• Support for the community member engagement on
  the management teams
• Advisory board that engages the support of local
  Champions and community leaders
• Opportunities for community members to complete
  meaningful tasks

32

• Ongoing communication among staff and agencies
• Learning Communities
• Recognize successes
• Social Marketing Strategy to obtain support of
  citizens (timing)

33
Critical Elements of Youth on Management teams

• Adult support
• Youth-friendly environment
• Opportunities to complete meaningful tasks
• Opportunities to learn and use new skills.

34
We are Guilty of many errors and many faults, but
our worst crime is abandoning the children,
neglecting the fountain of life. Many of the
things we need can wait. The child cannot. Right
now is the time his bones are being formed, his
blood is being developed.To him we cannot answer
Tomorrow.His name is Today.Gabriela
Mistral, Nobel Prize-winning Poet