The School Counseling Research Base: Identifying Interventions That Work

1
The School Counseling Research Base Identifying
Interventions That Work

• Massachusetts School Counselors Association
• May 8, 2006
• John Carey Carey Dimmitt
• National Center for School Counseling Outcome
  Research

2
Download Power Point at National Center for
School Counseling Outcome Research Website

• www.cscor.org

3
Factors Leading to Need Data Skills in School
Counseling

• Accountability Movement
• Standards-Based Education Movement (NCLB)
• School Reform Movement
• New Models of School Counseling Practice
• Education Trust Transforming School Counseling
  Initiative
• ASCA National Model

4
A Model of Evidence-Based Practice
Practitioners Individual Expertise
Best Evidence
Client Values and Expectations
EBP
Adapted from Shlonsky and Gibbs (2004), Will
the Real Evidence-Based Practice Please Stand Up?
Teaching the Process of Evidence-Based Practice
to the Helping Professions. In Brief Treatment
and Crisis Intervention, 4(2), 137-153.
5
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
6
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals Knowing what is needed
EBP
7
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence Knowing what
generally works
Assessment of Intervention Targets and
Goals
EBP
8
Evidence-Based Practice in School Counseling
Outcome Research Competence
Intervention and Program EvaluationCompetence
Knowing how studentschanged
Assessment of Intervention Targets and Goals
EBP
9
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
10
Terms

• Data-Based Decision Making use of school data to
  determine problems that need to be addressed.
• Outcome Research use of the scientific method to
  discover generalizable truth about the
  effectiveness of interventions.
• Evaluation use of the scientific method to
  improve local decision-making by determining
  whether it was likely that an intervention
  resulted in desired changes in behavior.

11
How These Fit Together

• Data-Based Decision Making
• 8th graders in Carey Middle school are doing
  poorly on the state test
• Outcome Research
• Research yields strong evidence that Student
  Success Skills can increase test scores by
  teaching self-management and enhancing motivation
• Program Evaluation
• Carey Middle School students learn self
  management and improve grades and test scores
  after SSS

12
Critical Skills

• Surveying the Outcome Research Literature
• Evaluating Selected Interventions (Action
  Research)

13
Evidence for Important Causal Relationships in
Outcome Research

• Not all research is equal
• Research Quality is related to the confidence
  with which causal inferences can be made
• Research Quality is related to
• Internal validity
• External validity

14
Pre-Evidence-Based Education Pre-ASCA National
Model
Career Development Outcomes
School Counseling Interventions
Academic Development Outcomes
Personal/Social Development Outcomes
15
Post-Evidence-Based Education Post-ASCA
National Model
Career Development Outcomes
School Counseling Interventions
Academic Development Outcomes
Personal/Social Development Outcomes
16
Post-Evidence-Based Education Post-ASCA
National Model
Career Development Outcomes
School Counseling Interventions
Academic Development Outcomes
Personal/Social Development Outcomes
Strong Evidence
Promising Evidence
Weak Evidence
17
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 1. Measurement
• Principle Important academic, career and/or
  personal/social outcomes are measured using
  reliable and valid instruments.
• Strong Evidence
• Outcomes measures have established high
  reliability and validity characteristics.
• Outcome measures are established to be
  appropriate for the population under study.
• Promising Evidence
• Outcome measures have been used in previous
  studies.
• Reliability characteristics are evaluated in the
  study and show adequate reliability.
• Logical argument supports the appropriateness of
  the measures for the population under study.

18
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 2. Comparison Groups
• Principle Comparison groups with adequate
  controls are included so that resulting group
  differences can be attributed to the
  intervention.
• Strong Evidence
• Active Comparison Groups (alternative treatment)
  with Adequate Controls (attention, placebo) are
  included in an outcome study.
• Initial Group Equivalence is assures through
  random assignment.
• Group Equivalence in Mortality/Attrition is
  established
• Promising Evidence
• Groups equated through matching or statistical
  procedures (e.g. ANCOVA) or strong pre-post-test
  designs are used with adequate controls.

19
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 3. Statistical Analyses of Outcome Variables
• Principle Statistical analysis documents low
  probability of Type 1 error and potency of
  intervention.
• Strong Evidence
• Statistically significant finding using
  appropriate test
• Control for Experimentwise error rate
• Adequate N
• At least a Moderate Effect size for critical
  outcome variable
• Promising Evidence
• Statistically significant finding using
  appropriate test
• Control for Experimentwise error rate
• Adequate N
• At least a Small Effect size for critical outcome
  variable

20
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 4. Implementation Fidelity
• Principle Intervention can be delivered with
  fidelity across contexts and is not contaminated
  with implementer.
• Strong Evidence
• Intervention is extensively documented (manual or
  protocol) so that it can be reliably replicated.
• Intervention is delivered by multiple people with
  adequate training and checks for adherence to
  protocol.
• Promising Evidence
• Intervention is standardized and can be delivered
  across contexts.
• Intervention is delivered by multiple people with
  adequate training.

21
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 5. Replication
• Principle The same intervention independently
  implemented with an equivalent population results
  in equivalent outcomes.
• Strong Evidence
• Independent evaluators find equivalent outcomes
  with a similar population.
• Promising Evidence
• Same evaluator finds equivalent outcomes with
  same population

22
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 6. Ecological Validity
• Principle The intervention can be implemented
  effectively in a public school with consistent
  effects across all student subgroups or with
  known differences between student subgroups.
  Limitations of the generalizability of results
  are clearly explicated.
• Strong Evidence
• Study conducted in a diverse public school.
• Outcomes are assesses across different subgroups
  of students or clearly specified as valid for a
  specific subgroup.
• Promising Evidence
• Study conducted in a private, laboratory, or
  charter school or in a public school with limited
  diversity.

23
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 7. Persistence of Effect
• Principle The intervention results in a lasting
  effect on an important outcome measure.
• Strong Evidence Treatment-Comparison Group
  Differences are demonstrated to persist for a
  practically significant time period.
• Promising Evidence Treatment-Comparison Group
  Differences are demonstrated to persist beyond
  the immediate implementation.

24
Methods for Weighing Evidence

• Research Briefs
• Analytic Reviews
• Meta-Analytic Reviews
• Expert Panels, Centers and Institutes
• Research Literature Searches School-Based Teams

25
Research Briefs

• Center for School Counseling Outcome Research

26
Analytic Reviews

• Whiston and Sexton, 1994
• McGannon, Carey and Dimmitt, 2005

27
Meta-Analytic Reviews

• Gene Glass
• Select studies
• Compute effect size
• Group studies
• Interventions
• Outcomes
• Quality

28
Meta-Analytic Reviews

• Effect Size (Welkowitz, Ewen Cohen)
• .20Small
• .50Medium
• .80Large

29
Meta-Analysis of School Counseling Interventions

• Whiston et al (2005)

30
Information on Studies

• 117 studies
• 69.2 published
• 30.8 thesis/dissertations
• Participants
• 16,296 total
• Age
• Elementary 59 studies (50.4)
• Middle/Junior High 21 studies (17.9)
• High School 29 studies (24.8)
• Mixture 7 studies (6)
• Unreported 1 study (.9)

31
Interventions

• 153 school counseling interventions
• Types of interventions
• Guidance Curriculum 44 studies/57 interventions
• Individual Planning 9 studies/10 interventions
• Responsive Services 58 studies/74 interventions
• Program-Wide Services 8 studies/9 interventions
• Other 2 studies/3 interventions
• Length of interventions
• Number of sessions M12.44 , SD 15.53, Range
  1-120
• Number of hours M8.41 , SD 6.87, Range .42-40
• Duration M71.76 , SD 87.52, Range 1-720
• Providers of interventions
• Experienced counselor 37.9 (58 interventions)
• Counselor in training 9.8 (15 interventions)
• Teacher 8.5 (13 interventions)
• Student peer 3.3 (5 interventions)
• Student independent 5.9 (9 interventions)
• Other 5.9 (9 interventions)
• Combination 13.7 (21 interventions)

32
Outcomes

• 785 outcomes
• Cognitive 12.2 (96 outcomes)
• Behavioral 34.8 (273 outcomes)
• Affective 39.0 (306 outcomes)
• Effective Role Functioning 9.4 (74 outcomes)
• Satisfaction .1 (1 outcome)
• Global evaluation of school counseling program
  .4 (3 outcomes)

33
Effect Sizes

• Unweighted .46
• Weighted .29 (based on 153 interventions)
• CI .26 to .32
• Q 616.124, p lt .001

34
Grade Level and Type of Intervention

• Grade level effect sizes
• Elementary ES .25
• Middle School ES .39
• High School ES .34
• Type of intervention
• Guidance Curriculum ES .33 (44 studies/57
  interventions)
• Individual Planning ES .27 (9 studies/10
  interventions)
• Responsive Services ES .35 (58 studies/74
  interventions)
• Program Wide Evaluation .19 (8 studies/9
  interventions)

35
Program component by Grade level

• Guidance Curriculum
• Elementary ES .29
• Middle School ES .41
• High School ES .39
• Individual Planning
• Elementary ES -.137 (1 intervention)
• Middle School ES 1.01 (3 interventions)
• High School ES .22 (6 interventions
• Responsive Services
• Elementary ES .40 (45 interventions)
• Middle School ES .22 (10 interventions)
• High School ES .34 (20 interventions)

36
Cognitive Outcomes

• Overall ES .19 (94 outcomes)
• GPA ES .19 (49 outcomes)
• Academic Achievement tests ES .16 (16 outcomes)
• Career knowledge ES .61 (12 outcomes from 3
  studies)
• Other ES .73 (19 outcomes from 11 studies)

37
Behavioral Outcomes

• Overall ES .39 (274 outcomes)
• Behavior rating scales ES .24 (123 outcomes)
• No study examined change in enrollment (e.g.,
  taking more difficult classes) as an outcome
  measure
• Attendance ES .30 (19 outcomes)
• Interpersonal or social skills ES .34 (55
  outcomes)
• Aggressive behavior/fights ES .26 (12 outcomes)
• Problem-solving skills ES .96 (26 outcomes)
• Peer counseling skills ES 1.12 (15 outcomes)
• Number of disciplinary referrals ES .82 (9
  outcomes)
• Other ES .25 (14 outcomes)

38
Affective Outcomes

• Overall ES .22 (306 outcomes)
• Self-esteem ES .18 (172 outcomes22 of total
  outcomes)
• Personal/social adjustment ES .24 (27 outcomes)
• Anxiety ES .39 (35 outcomes)
• Depression ES .35 (6 outcomes)
• Other ES .24 (66 outcomes)

39
Effective Role Functioning

• Overall ES .12 (74 outcomes)
• Career maturity/decidedness ES .20 (30
  outcomes)
• Academic functioning ES -.17 (9 outcomes)
• Other ES .12 (35 outcomes)

40
Conclusion

• School counseling interventions moderately
  effective
• Individual planning at middle school level has a
  large effect size (small number of studies)
• Guidance curriculum more effective at middle and
  high school level whereas, responsive services
  have a larger effect size at elementary level
• Large effects
• Career knowledge
• Problem solving
• Disciplinary referrals
• Contributes to academic achievement
• Moderately effective in reducing anxiety and
  depression, increasing interpersonal social
  skills, and positively influencing attendance

41
Expert Panels Centers and Institutes

• National Panel for Evidence-Based School
  Counseling Practice

42
Second Step EBP Protocol Summary

• Protocol Domain
• 1. Measurement
• 2. Comparison Groups
• 3. Statistical Analysis of Outcome Variables

• Second Step
• Strong Evidence
• Strong Evidence
• Strong Evidence

43
Second Step EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• Second Step
• Strong Evidence
• Strong Evidence
• Strong Evidence
• Strong Evidence

44
SSS EBP Protocol Summary

• Protocol Domain
• Measurement
• Comparison Groups
• Statistical Analysis of Outcome Variables

• SSS
• Strong Evidence
• Promising Evidence
• Strong Evidence

45
SSS EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• SSS
• Strong Evidence
• Promising Evidence
• Strong Evidence
• Weak Evidence

46
Other Resources

• What Works Clearing House
• CDC/DASH Programs That Work
• SDFS Exemplary and Promising Programs
• SAMHSA National Registry of Evidence-Based
  Programs and Practices
• Collaborative for Academic Social and Emotional
  Learning

47
School-Based Teams

• Best done in the Context of Data-Based Decision
  Making
• Team includes key perspectives and resources
• Team has identified problem data, (academic,
  career, psychosocial) and vision data (what
  problem data should look like in the future.
• Now the focus is on what evidence-based
  interventions are most likely to result in a
  change in the problem behavior

48
School-based Teams

• What research-based interventions are out there?
• Does the evidence support a link between the
  intervention and the desired outcomes?
• Does the evidence support intervention
  effectiveness with the desired target students?
• Can the intervention be implemented with fidelity
  in your setting?
• Is the intervention feasible in your setting?
• Are evaluation materials included in the
  intervention (if not, how will it be evaluated)?

49
School-Based Teams

• Online tutorial

50
School-Based Teams

• Sample Matrix of Interventions
• Feasibility Checklist

51
Questions and Discussion
52
National Center for School Counseling Outcome
Research

• Thank You

www.cscor.org