Promising Results from the National Evaluation of the Gang Resistance Education and Training (G.R.E.A.T.) Program

1
Promising Results from the National Evaluation of
the Gang Resistance Education and Training
(G.R.E.A.T.) Program

• Finn Esbensen
• E. Desmond Lee Professor of Youth Crime
  Violence
• Chair, Department of Criminology and Criminal
  Justice
• University of Missouri-St. Louis

2
Presentation Overview

• Why is there interest in youth violence and gang
  prevention?
• How do we know what works?
• What do terms like evidence-based, model
  programs and effective really mean?
• Use the G.R.E.A.T. program as an example of how
  question of program effect is addressed.

3
NYGS Jurisdictions Reporting Youth Gang Problems
in One or More Years, 1999-2001(Ngt1400)
4
Gang Violence

• Widely documented finding that gang members are
  responsible for a disproportionate amount of
  crime.
• Thornberry and Burch (1997), for example,
  reported that gang members accounted for 86
  percent of all serious offenses in the Rochester
  Youth Development Study.
• Youths have higher rates of offending during
  active gang membership than they do either before
  or after gang involvement

5
How do we know what works?

• Anecdotal evidence is NOT sufficient
• Good intentions are NOT sufficient
• Saying that something works or that something is
  effective, no matter how many times you say it,
  does NOT make it so
• Rigorous evaluation is required

6
Promising Programs forViolence Prevention

• The Blue Prints Program Center for the Study
  and Prevention of Violence at CU-Boulder
• Reviewed over 600 programs
• 11 were identified as model programs
• proven scientifically to be effective in reducing
  youth aggression, violence, other delinquent
  behavior, and substance abuse
• An additional 21 designated as promising
• That means 570 were not effective or not
  evaluated

7
Model Program Designation

• Sound program or theoretical model
• Random assignment
• Sustained program effect (12 month post
  treatment)
• Replicated in at least one study

8
What is known about responses to gangs?

• Very little
• No shortage of programs - lack of evaluations of
  existing programs or lack of positive outcomes
• G.R.E.A.T. as example

9
What is G.R.E.A.T.?

• 13 week in-school general prevention program
• Taught by uniformed law enforcement officer
• Original program developed in 1991 by Phoenix
  area officers 9 lessons and mostly didactic
• Several efforts to review the curriculum
• null findings from longitudinal study
• Curriculum review conducted from 1999 2000

10
What is G.R.E.A.T.? - 2

• Program revision 2001 piloted in 2002
• Seattle Social Development Model - classroom
  management, cooperative learning, interactive
  teaching
• Life Skills Training self-management and social
  skills development
• Interactive and skills building focus
• Especially important that program be taught as
  designed lessons build on each other
• Important to assess program fidelity
• Implementation of Revised G.R.E.A.T. in 2003

11
G.R.E.A.T. LESSONS

• Lesson 1 Welcome to G.R.E.A.T. - acts as the
  introduction to the program and introduces the
  relationship between gangs, violence, drug abuse,
  and crime.
• Lesson 2 Whats the Real Deal - consists of
  message analysis skills and facts and fictions
  about gangs.
• Lesson 3 Its About Us - focuses on different
  communities and how students are a part of these,
  including their responsibilities to their
  community or communities.
• Lesson 4 Where Do We Go From Here - introduces
  students to the concept of goals and how to set
  realistic and achievable aspirations.
• Lesson 5 Decisions, Decisions, Decisions -
  focuses on decision-making, in which students
  learn the G.R.E.A.T. decision making model and
  the impact their decisions have on their goals
  students are able to practice making positive
  decisions.

12
G.R.E.A.T. LESSONS - II

• Lesson 6 Do You Hear What Im Saying? - teaches
  the importance of listening to others and the
  difference between verbal and non-verbal
  communication.
• Lesson 7 Walk in Someone Elses Shoes -
  instructs students in active listening skills and
  how to identify others emotional states through
  empathy-building techniques.
• Lesson 8 Say It Like You Mean It - teaches
  refusal skills so students may resist
  peer-pressure to engage in deviant or delinquent
  acts this includes learning about body language
  and tone of voice.
• Lesson 9 Getting Along Without Going Along -
  consists of recognizing peer pressure and other
  influences that may push students into
  delinquency.
• Lesson 10 Keeping Your Cool - teaches students
  to keep calm in the face of anger with anger
  management tips and practicing the cooling-off
  technique.

13
G.R.E.A.T. LESSONS - III

• Lesson 11 Keeping It Together - consists of
  recognizing anger in others and learning to calm
  them.
• Lesson 12 Working It Out - teaches students to
  work through problems without fighting and
  provides tips for conflict resolution, practice
  of such, and information about where to go for
  help in their communities.
• Lesson 13 Looking Back - consists of a program
  review and the presentation and discussion of
  student final projects.
• http//www.great-online.org/

14
Program Goals

• Measuring Explicit Goals of Program
• Avoid gang membership
• Do you consider your group of friends to be a
  gang?
• Are you now in a gang?
• Prevent violence and criminal activity
• Self-reported delinquency measures
• Develop positive relationships w/law enforcement
• Global attitudinal scale
• G.R.E.A.T.-specific attitudes

15
Lesson Specific Outcomes

• Impulsivity and risk taking
• Anger management
• Self centeredness
• Peer pressure
• Negative peer commitment
• Peer delinquency
• Neutralization (hitting)
• Conflict resolution

• Refusal skills
• Empathy
• Active listening
• Problem solving
• Collective efficacy
• Attitudes about gangs
• Altruism

16
Risk Factors

• Analogy with medical field
• Genetic and lifestyle factors associated with
  likelihood of particular illness
• Presence of risk factors does not mean that
  person will be ill only increased probability
• Adopt this approach with violent offending and
  gang membership
• Risk factors in 5 domains community, school,
  family, peer, individual

17
Risk factors

• The more risk factors and in multiple domains
  the greater the risk of gang joining.
• No silver bullet not all youth living in
  high-risk neighborhoods join gangs.
• Addressing even one of these risk factors reduces
  the odds of gang joining and violent offending.

18
Cumulative Effect of Multiple Risk Factors
Number of Risk Factors
19
Does the program work?

• Is the program implemented with fidelity?
• Can the program be adopted in a variety of
  settings?
• Is the program effective?

20
Process and Outcome Evaluation of the G.R.E.A.T.
Program (June 2006 December 2012)

• This research was made possible, in part, by the
  support and participation of seven school
  districts, including the School District of
  Philadelphia. This project was supported by Award
  No. 2006-JV-FX-0011 awarded by the National
  Institute of Justice, Office of Justice Programs,
  U.S. Department of Justice. The opinions,
  findings, and conclusions or recommendations
  expressed in this presentation are those of the
  authors and do not necessarily reflect the views
  of the Department of Justice or of the seven
  participating school districts.

21
G.R.E.A.T. II Evaluation Team

• Finn Esbensen, Ph.D., Principal Investigator
• University of Missouri-St. Louis
• Terrance J. Taylor, Ph.D., Investigator
• University of Missouri-St. Louis
• Dana Peterson, Ph.D., Investigator
• University at Albany
• Wayne Osgood, Ph.D., Investigator
• Pennsylvania State University
• Kristy Matsuda, Ph.D., Research Associate
• University of Missouri-St. Louis
• Adrienne Freng, Ph.D., Site Coordinator
• University of Wyoming
• Dena Carson, Site Coordinator

22
Site Selection

• Site selection
• Geographic and demographic diversity
• Nature of community gang problems
• Law enforcement agency and Program size
• G.R.E.A.T. taught at least one year
• No program saturation
• Obtained agreements from law enforcement and
  public school districts
• Process
• Created preliminary list of 56 cities (identified
  by at least one RA, IIR or BJS staff member)
• Contacted agencies about G.R.E.A.T. program
• Resulted in final list of 7 hoped to recruit 6
• Albuquerque, NM Chicago, IL DFW area city, TX
  Greeley, CO Nashville, TN Philadelphia, PA
  Portland, OR

23
Process Evaluation

• To what extent is G.R.E.A.T. delivered as
  intended?
• If effect found, can it be attributed to the
  program?
• If no effect found, is it a case of program
  failure or failed program?
• Triangulation
• GOT observations
• Classroom observations
• Officer Supervisor interviews
• G.R.E.A.T. Officer survey
• School personnel questionnaires

24
GOT Observations

• Observations of Training
• Observed nine GOTs (1 week and 2 week sessions)
• Purpose
• Learn new curriculum as officers are learning it
  
• Assess quality and consistency of training within
  and across GOTs
• Well-designed and implemented
• Reasonable to expect that officers are prepared
  to teach the program with fidelity

25
Classroom Observations (2006-2007 school year)

• Purpose
• Assess program delivery regarding
• Dosage
• Adherence to intended lessons
• Quality of instruction
• How?
• 502 in-class observations of program delivery
• 108 non-G.R.E.A.T. classes observed
• Control classrooms and G.R.E.A.T. when program
  not taught
• 33 different officers
• Each lesson observed between 27 and 49 times
  allows us to assess lesson quality in addition to
  officer implementation fidelity

26
Findings Implementation

• We were able to provide feedback on some
  important lesson-related issues (e.g., time
  management, student interest, role of teacher,
  lesson content)
• Importantly, of 33 total officers, 27 implemented
  the lessons in average/above average manner
• 3 officers were judged not to have delivered the
  program
• Evidence of program fidelity so if effects are
  found, they can be reasonably attributed to the
  program.

27
Outcome Evaluation

• To what extent is G.R.E.A.T. achieving its stated
  goals?
• Determine both short- and long-term effects
• Compare students who receive program to students
  who do not, in an experimental design
• Multiple, diverse locations

28
Outcome Evaluation

• Seven cities
• Albuquerque, Chicago, DFW area, Greeley,
  Nashville, Philadelphia, Portland
• 31 schools total
• 4 schools in five cities, 5 in Portland, 6 in
  Chicago
• 195 classes total (102 GREAT, 93 Control)
• 24 - 35 classes per city
• 3 - 12 classes in each school (half receive
  G.R.E.A.T.)
• 3,820 students
• 457 - 614 students per city
• Between 59 and 186 students per school

29
Outcome Evaluation

• Random Assignment at classroom level
• Equivalent comparison groups
• No bias in assignment
• Active Parental Consent
• Students are the program target, so we need their
  assessments. How do we do this?
• Confidential self-report questionnaires
• Questionnaires read aloud in group-administered
  setting
• Researchers ensure privacy, answer questions
• Pre- and post-tests
• 4 annual follow-up surveys

30
Outcome Student Surveys

• Site selection
• Aug and Sept 2006
• Active consent process
• Sept Oct 2007 Jan Feb 2008 (two added
  schools)
• Survey administration
• Pre-tests - Sept 2006 Feb 2007 Feb March
  2008
• Post-tests - Nov 2006 May 2007 May 2008
• Wave 3 Oct 2007 March 2008 Dec 2008 Feb
  2009
• Wave 4 Oct 2008 March 2009 Nov 2009 Feb
  2010
• Wave 5 Oct 2009 March 2010 Nov 2010 March
  2011
• Wave 6 Oct 2010 March 2011 October 2011
• Data entry (and verification)
• Preparation of analysis file and outcome analyses

31
Completion Rates

• Active consent rate 78 (N3,820)
• Pretest completion 98
• Post-tests 95
• Wave 3 87
• Wave 4 83
• Wave 5 75
• Wave 6 73
• Wave 5 surveyed students in 216 difference
  schools
• 66 different schools in Philly

32
Outcome Results - I

• Compare G.R.E.A.T. students with non-G.R.E.A.T.
  students
• Wave 2 (post-test) and
• Wave 3 (one year after program)
• A number of significant differences were found
• 11 of 33 significant at plt.05
• 3 significant at plt.10

33
Program Goals - Outcomes

• Avoid gang membership
• 39 lower odds of gang membership
• Develop positive relationships w/law enforcement
• More positive attitudes to police (ES.076)
• Teaching specific role of police (ES.204)
• Prevent violence and criminal activity
• No significant differences

34
Proximate outcomes

• Favorable G.R.E.A.T. outcomes plt.05
• Less positive attitudes about gangs (ES.114)
• More us of refusal skills (ES0.90)
• Resistance to peer pressure (ES.079)
• Higher collective efficacy (ES.125)
• Less hitting neutralizations (ES.105)
• Fewer delinquent peers (ES.083)
• Less self centeredness (ES.054)
• Less anger (ES.057)

35
Proximate outcomes

• Favorable G.R.E.A.T. outcomes plt.10
• Less use of lie neutralization
• More pro-social peers
• More pro-social involvement
• So, why are these proximate outcomes important?
• Remember, reducing one or two risk factors
  reduces odds of gang joining

36
Bottom Line

• At 12 month post treatment, 2 of the three
  program goals were met (reduction in gang joining
  and improvement in attitudes towards police)
• A number of proximate factors suggested
  G.R.E.A.T. has a positive effect
• But, effect sizes are modest no effect on
  delinquency
• Will these effects be sustained across time?
• Are these findings replicated across cities?
• Are there differential effects based on initial
  level of risk?

37
Reports

• Website http//www.umsl.edu/ccj/html_files/great
  _evaluation.html
• Reports to Schools Communities
• 2007 Evaluation Overview.
• 2008 Anti-Social Norms among a Sample of
  Middle-School Students.
• 2009 Program Implementation and Preliminary
  Outcome Results.
• 2010 Report to Schools and Communities School
  Safety and Victimization.
• Results from Surveys and Interviews with
  G.R.E.A.T.-trained Officers.
• Observing the Implementer Description of
  Findings, Recommendations, and Conclusions from
  GREAT Program Implementation Observations.
• School Personnel Survey Report.
• G.R.E.A.T. Officer Training (GOT) Report.

38
Thats all, folks

• Thanks for listening
• Any questions?