Evidencebased school programs targeted at adolescent substance use: Suspicious stories of success

1
Evidence-based school programs targeted at
adolescent substance use Suspicious stories
of success

• Hilde Pape
• Norwegian Insitute for Alcohol and Drug Research

2
An expert group formed by the public
authorities on health and education recently
concluded that 1 of 15 drug prevention programs
used in Norwegian schools Youth
Intoxication was evidence-based (i.e. its
effects on adolescent drinking had been
demonstrated)
Nordahl T, Gravrok Ø, Knudsmoen H et al. (2006)
Forebyggende innsatser i skolen. Oslo
Utdanningsdirektoratet / Sosial- og
helsedirektoratet.
3
The evaluation of Youth Intoxication was
based on a solid research design and the short
term effects were published
in a leading scientific journal.

Wilhelmsen BU, Laberg JC, Klepp KI.
Evaluation of two student and teacher involved
alcohol prevention programmes. Addiction 1994
89 1157-65
4

• However, no reliable evidence
• of reduced adolescent drinking was found
• in the publications from the evaluation of
• Youth Intoxication
• Pape, H. e tal. (2007)
• Villedende om rusforebygging i skolen.
  Tidsskrift for Den norske lægeforening, 127 1399.

5

• The evaluation of Youth Intoxication
• Sample 8th graders (i.e. 13-14 year olds)
• - 2 intervention groups Low role
  specification (LRS)
  High role
  specification (HRS)
• - Control group (no intervention)
• The reported effects on adolesent drinking after
  2 months were
• a significant increase in the LRS programme,
  
  a significant decrease in the HRS programme
  Wilhelmsen et al (1994) p. 1162
• Longer term effects
  
  No differences between the
  groups 11 months after the intervention
  (Wilhelmsen Laberg, 1996)

6
Only one behavioural outcome was assessed

• How often do you drink alcohol at the
  present time?
• (3 questions, response scale 0-3)

An additive index (beer wine hard liquor) was
used in the analyses
7
Other limitations of the evaluation of Youth and
Intoxication

• The roles of program developer and program
  evaluator were mixed
• The program was tested out in a small scale (8
  schools), and the
  context of the implementation was extraordinary
• Individual rather than aggregate level data were
  analysed
• No replications of the evaluation study were
  carried out
• Such limitations seem to characterize several
  evaluation studies with good news about the
  effectiveness of
  school-based drug prevention programs

8
Mixing the roles of program developer and
evaluator

• A recent review of studies on school-based
  programs targeted at adolescent substance use
  (published in peer-reviewed journals, n246)
  revealed that...
• 78 included the program developer as author
• 89 included the program developer as author
  and/or authors connected to the program developer
  (Gorman Conde 2007)
• It has also been demonstrated that
• Studies performed by programme developers
  yield considerably stronger effects than studies
  performed by others (Borman et al 2003)
  
  
• Independent outside evaluations have often
  fail to confirm evaluations by programme
  developers (Skager 2007)
• Conflict of interest is an ignored issue in
  this reseach literature

9
Efficacy rather than effectiveness studies

• School-based programs with documented effects
  have typically been implemented under ideal
  conditions
• Whether these programs work when implemented in a
  large scale and under real life conditions is
  highly questionable
• Incomplete light versions of preventive
  programs are typically used in ordinary school
  settings (Dusenbury et al 2003 )
•  ..small efficacy trials conducted by programme
  developers provide insufficient evidence of
  effectiveness (Hallfors et al 2006 )

10
However, few school-based programs have been
tested beyond small efficacy trials conducted by
program developers and implemented under ideal
conditions

• Hallfors D., et al. (2006). Effecacy vs.
  effectiveness trial results of an indicated
  model substance abuse
• program Implications for public
  health. American Journal of Public Health 96
  2254-2259.

11
The evaluation studies behind
apparently effective school-based
programs have typically been
confined to assessments
of short- term effects

• and long-term effects are much harder to achieve
  (if
  possible at all)

12
Widely used strategies that enhance the
probablity of finding effects of school-based
programs (Gorman 2003, Gandhi et al
2007, Brown 2001)

• Individuals rather than interventions groups as
  unit of analyses
• Post-hoc sample refinement
  
  (e.g., excluding groups in which the programme
  was poorly implemented)
• Separate analyses of multiple outcome measures
  (of
  which only a few yield statistically significant
  results)
• Use of one-tailed tests for statistical
  significance (doubling
  the likelihood of finding effects)
  
  

13
In their critical review of studies
behind proven school based interventions,
Gandhi et al (2007)
concludedOur findings ... raise the
question of whether any school based prevention
programme will substantially reduce the number of
teens who experiment with drugsGandhi AG,
Murphy-Graham E, Petrosino A m.fl. The devil is
in the details.
Examining the evidence for proven
school-based drug abuse prevention programs.
Evaluation Review
200731, 43-74
14
Another point in this sad story is about
publication bias

• Numerous reveiws of the research literature
  have concluded school-based drug prevention
  programs rarely have the desired effect
  on adolescent substance use
• Even so, the literature is probably biased in
  favour of publications with good news about the
  effects of such programs
  (cf. McCambridges article on Tobler et al.s
  meta-analyses of school-based programs)
• McCambridge J. (2007) A case study of
  publication bias in an influential series of
  reviews of drug education.
  Drug Alcohol Review 2
  463-368.

15
Biased reporting of results also occurs

• For instance, indications of no effect or
  counterproducive results may be toned down by the
  authors
• e.g. An article on the evaluation of Youth
  Intoxication was entitled
  Primary prevention targeted at drinking in
  junior high school
  Positive short-term effects)
• Wilhelmsen BU, Laberg JC. (1996)
  Primærforebyggende alkoholundervisning i
  ungdomsskolen
  Positive korttidseffekter. Norsk Epidemiologi 6
  91-6.

16
Implications

• Many of the studies which conclude that
  
  school-based drug prevention programs works
  must be
  taken with a pinch of salt
• Evaluation studies should..
• be carried out by researchers with no vested
  interest
• .. assess effects of programs implemented in a
  real life context
• include long-term follow-ups
• publish the results whatever they indicate