ResearchBased and Promising School Counseling Practices

1
Research-Based and Promising School Counseling
Practices

• ASCA National Conference
• June 26, 2006
• National Panel for Evidence-Based School
  Counseling

2
Download PowerPoint at National Center for School
Counseling Outcome Research Website

• www.cscor.org

3
National Panel Members

• John Carey
• University of Massachusetts, Amherst
• Carey Dimmitt
• University of Massachusetts, Amherst
• Trish Hatch
• San Diego State University
• Rich Lapan
• University of Missouri, Columbia
• Susan Whiston
• University of Indiana, Bloomington

4
Factors Leading to Need for Data Skills in School
Counseling

• Accountability Movement
• Standards-Based Education Movement (NCLB)
• School Reform Movement
• New Models of School Counseling Practice
• Education Trust Transforming School Counseling
  Initiative
• ASCA National Model

5
A Model of Evidence-Based Practice
Practitioners Individual Expertise
Best Evidence
Client Values and Expectations
EBP
Adapted from Shlonsky and Gibbs (2004), Will
the Real Evidence-Based Practice Please Stand Up?
Teaching the Process of Evidence-Based Practice
to the Helping Professions. In Brief Treatment
and Crisis Intervention, 4(2), 137-153.
6
Evidence-Based Practice in School Counseling
Intervention and Program Evaluation
Outcome Research
Data-Based Decision-Making
EBP
7
Terms

• Data-Based Decision Making use of school data to
  determine which problems need to be addressed
  assessment of intervention targets and goals
• Outcome Research use of the scientific method to
  discover generalizable truth about the
  effectiveness of interventions
• Evaluation use of the scientific method to
  improve local decision-making by determining
  whether it was likely that an intervention or
  program resulted in desired changes or outcomes

8
Evidence-Based Practice in School Counseling
Intervention and Program Evaluation
Outcome Research
Data-Based Decision-Making Knowing what
is needed
EBP
9
Evidence-Based Practice in School Counseling
Intervention and Program Evaluation
Outcome Research Knowing what generally
works
Data-Based Decision-Making
EBP
10
Evidence-Based Practice in School Counseling
Outcome Research
Intervention and Program Evaluation Knowing
how studentschanged
Data-Based Decision-Making
EBP
11
Evidence-Based Practice in School Counseling
Intervention and Program Evaluation
Outcome Research
Data-Based Decision-Making
EBP
12
How These Fit Together

• Data-Based Decision Making
• 8th graders in Carey Middle school are doing
  poorly on the state test
• Outcome Research
• Research yields strong evidence that Student
  Success Skills can increase test score by
  teaching self-management and enhancing motivation
• Program Evaluation
• Carey Middle School students learn
  self-management and improve grades and test
  scores after SSS

13
Critical Skills

• Surveying the Outcome Research Literature
• Where is the research located?
• Which research is relevant?
• Which research is well-done?
• Which research is applicable in my setting?

14
Critical Skills

• Evaluating Selected Interventions (Action
  Research)
• How will we know that the intervention has
  worked?
• What outcomes am I trying to impact?
• How do I measure these outcomes?

15
Evidence for Important Causal Relationships in
Outcome Research

• Not all research is equal
• Research Quality is related to the confidence
  with which causal inferences can be made. Just
  because two things happen at the same time
  doesnt mean that one caused the other.
• Research Quality is related to
• Internal validity
• External validity

16
Methods for Weighing Evidence

• Research Briefs
• Analytic Reviews
• Meta-Analytic Reviews
• Expert Panels, Centers and Institutes
• Research Literature Searches
• School-Based Teams

17
Research Briefs

• Center for School Counseling Outcome Research

18
Analytic Reviews

• Whiston and Sexton, 1994
• McGannon, Carey and Dimmitt, 2005

19
Meta-Analytic Reviews

• Gene Glass
• Select studies
• Compute effect size
• Group studies
• Interventions
• Outcomes
• Quality

20
Meta-Analytic Reviews

• Effect Size (Welkowitz, Ewen Cohen)
• .20Small
• .50Medium
• .80Large

21
Meta-Analysis of School Counseling Interventions

• Whiston et al. (2005)
• Summary of findings are part of the Panels 2005
  presentation at ASCA www.umass.edu/schoolcounseli
  ng/PDFs/EBP/EBP_Panel_Report_ASCA_2005(color).pdf

22
Expert Panels Centers and Institutes

• National Panel for Evidence-Based School
  Counseling Practice

23

• 1. Measurement
• Principle Important academic, career and/or
  personal/social outcomes are measured using
  reliable and valid instruments.
• Strong Evidence
• Outcomes measures have established high
  reliability and validity characteristics.
• Outcome measures are established to be
  appropriate for the population under study.
• Promising Evidence
• Outcome measures have been used in previous
  studies.
• Reliability characteristics are evaluated in the
  study and show adequate reliability.
• Logical argument supports the appropriateness of
  the measures for the population under study.

24

• 2. Comparison Groups
• Principle Comparison groups with adequate
  controls are included so that resulting group
  differences can be attributed to the
  intervention.
• Strong Evidence
• Active Comparison Groups (alternative treatment)
  with adequate Controls (attention, placebo) are
  included in an outcome study.
• Initial Group Equivalence is assures through
  random assignment.
• Group Equivalence in Mortality/Attrition is
  established.
• Promising Evidence
• Groups equated through matching or statistical
  procedures (e.g. ANCOVA) or strong pre-post-test
  designs are used with adequate controls.

25

• 3. Statistical Analyses of Outcome Variables
• Principle Statistical analysis documents low
  probability of Type I error and potency of
  intervention.
• Strong Evidence
• Statistically significant finding(s) using
  appropriate test
• Control for Experimentwise error rate
• Adequate N
• At least a Moderate Effect size for critical
  outcome variable
• Promising Evidence
• Statistically significant finding using
  appropriate test
• Control for Experimentwise error rate
• Adequate N
• At least a Small Effect size for critical outcome
  variable

26

• 4. Implementation Fidelity
• Principle Intervention can be delivered with
  fidelity across contexts and is not contaminated
  with implementer.
• Strong Evidence
• Intervention is extensively documented (manual or
  protocol) so that it can be reliably replicated.
• Intervention is delivered by multiple people with
  adequate training and checks for adherence to
  protocol.
• Promising Evidence
• Intervention is standardized and can be delivered
  across contexts.
• Intervention is delivered by multiple people with
  adequate training.

27

• 5. Replication
• Principle The same intervention independently
  implemented with an equivalent population results
  in equivalent outcomes.
• Strong Evidence
• Independent evaluators find equivalent outcomes
  with a similar population.
• Promising Evidence
• Same evaluator finds equivalent outcomes with
  same population

28

• 6. Ecological Validity
• Principle The intervention can be implemented
  effectively in a public school with consistent
  effects across all student subgroups or with
  known differences between student subgroups.
  Limitations of the generalizability of results
  are clearly explicated.
• Strong Evidence
• Study conducted in a diverse public school.
• Outcomes are assessed across different subgroups
  of students or clearly specified as valid for a
  specific subgroup.
• Promising Evidence
• Study conducted in a private, laboratory, or
  charter school or in a public school with limited
  diversity.

29

• 7. Persistence of Effect
• Principle The intervention results in a lasting
  effect on an important outcome measure.
• Strong Evidence Treatment-Comparison Group
  Differences are demonstrated to persist for a
  practically significant time period.
• Promising Evidence Treatment-Comparison Group
  Differences are demonstrated to persist beyond
  the immediate implementation.

30
Research About PeaceBuilders

• Embry, D.D., Flannery, D.J., Vazsonyi, T.T.,
  Powell, K.E., Atha, H (1996). PeaceBuilders A
  theoretically driven, school-based model for
  early violence prevention. American Journal of
  Preventive Medicine, Supplement to 12 (5),
  91-100.
• Flannery, D.J., Liau, A.K., Powell, K.E.,
  Vesterdal, W., Vazsonyi, A.T., Guo, S., Atha, H.,
  Embry, D. (2003). Initial behavior outcomes
  for the PeaceBuilders universal school-based
  violence prevention program. Developmental
  Psychology, 39(2), 292-308.
• Vazsonyi, A.T., Bellison, L.M, Flannery, D.J.
  (2004). Evaluation of a school-based, universal
  violence prevention program Low-, medium-, and
  high-risk children. Youth Violence and Juvenile
  Justice, 2, 185-206.

31
Domain 1 Measurement

• PeaceBuilders has Promising Evidence
• Some outcome measures used have established high
  reliability and validity characteristics
  (Aggressive Behavior subscale of the Achenbach
  Teacher Report Form, and Walker-McConnell Scale
  of Social Competence and School Adjustment)
• Some measures were developed for this study
  (child self-report of aggressive behavior,
  prosocial behavior and peace-building behavior)
  and had not been used before
• All measures used had reliability and validity
  evaluated in the study, with alphas of .58 to .95

32
Domain 2 Comparison Groups

• PeaceBuilders has Promising Evidence
• Quasi-experimental design
• Random assignment by school--schools matched by
  geography, somewhat by demographics
• 4 Tier 2 schools received PB intervention a year
  later
• Outcomes relational but not causal

33
Domain 3 Statistical Analyses

• PeaceBuilders has Strong Evidence
• Used hierarchical linear model to examine change
  in behavior assessed at 4 points in time over 2
  years
• N 4000
• Small to Moderate Effect Sizes, with larger ES
  for students with lower baseline scores

34
Domain 4 Implementation Fidelity

• PeaceBuilders has Promising Evidence
• Intervention was standardized, though not
  manualized
• PB developer Embry did all training and coaching
  to ensure fidelity
• PB was delivered by multiple people with adequate
  training
• Fidelity was checked through an 8-item
  self-report survey by teachers at time 4
• There were not checks for adherence to protocol

35
Domain 5 Replication

• PB has Weak Evidence
• There is only one study of PeaceBuilders, though
  there are 4 research articles based on the study
  
• Each article uses either a specific part of data
  (nurses logs, behavioral measures) or reanalyzes
  the data (by gender and by risk-level)

36
Domain 6 Ecological Validity

• PeaceBuilders has Strong Evidence
• Study was conducted in public schools with
  diverse populations (included Latino,
  African-American, Asian, Caucasian, Native
  American range of SES grades K-5)
• Outcomes were consistent across groups, with the
  exception of gender.

37
Domain 7 Persistence of Effect

• PeaceBuilders has Promising Evidence
• Differences were found between Time 1 and Time 4,
  which is across 2 school years.

38
PeaceBuilders EBP Protocol Summary

• Protocol Domain
• Measurement
• Comparison Groups
• Statistical Analysis of Outcome Variables

• PeaceBuilders
• Promising Evidence
• Promising Evidence
• Strong Evidence

39
PeaceBuilders EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• PeaceBuilders
• Promising Evidence
• Weak Evidence
• Strong Evidence
• Promising Evidence

40
Open Circle

• Designed to explore effects of previous exposure
  to a social skills lessons in 6th grade students
• Consists of lessons taught by teachers on self
  control, social problem-solving etc.
• 2x week for 15-30 weeks
• Teachers taught to implement during a four day
  training program (NO Mention of school
  counselors).
• N277

41
Open Circle EBP Protocol Summary

• Protocol Domain
• Measurement
• Comparison Groups
• Statistical Analysis of Outcome Variables

• Open Circle
• Weak Evidence
• Weak Evidence
• Weak Evidence

42
Open Circle EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• Open Circle
• Weak Evidence
• Weak Evidence
• Weak Evidence
• Weak Evidence

43
Open Circle Major Concerns

• Measurement no measurement used in previous
  studies
• Comparison Groups They divided the groups into
  those that remembered having received the lessons
  for two or more years and those that indicated
  they either did not recall having the lessons or
  that they had the lessons 1 year or less.
• SAOV (Statistical Analysis of Outcome Variables
  60 results listed 2 had lt0.05 effectiveness.
  It is more likely that a significant difference
  will be found by chance alone, therefore, the
  results are only suggestive.

44
Open Circle Major Concerns

• Fidelity no standardization of services (great
  variance)
• Replication none
• Ecological Validity 79 white, high SES, 96 had
  parent who went to college
• Persistence of Effect none
• Admissions of poor research
• No actual records of participation
• Some records reviews were conducted but more
  were corrupted, so they were considered
  unreliable and not used

45
Bully Busters

• Effectiveness of Bully Prevention programs on
  teachers' knowledge of intervention skills,
  teacher self efficacy, students classroom
  behavior and disciplinary referrals.
• Major Issues Contamination of treatments with
  outcome measurements same teachers who
  participated in study rated effectiveness of
  intervention as well as provided and measured the
  discipline.
• Study was more concerned with teachers than
  students - and so what? Did students change?

46
Bully Busters EBP Protocol Summary

• Protocol Domain
• Measurement
• Comparison Groups
• Statistical Analysis of Outcome Variables

• Bully Busters
• Weak Evidence
• Promising Evidence
• Weak Evidence

47
Bully Busters EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• Bully Busters
• Promising Evidence
• Weak Evidence
• Promising Evidence
• Weak Evidence

48
Student Success Skills EBP Protocol Summary

• Protocol Domain
• Measurement
• Comparison Groups
• Statistical Analysis of Outcome Variables

• SSS
• Strong Evidence
• Promising Evidence
• Strong Evidence

49
Student Success Skills EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• SSS
• Strong Evidence
• Promising Evidence
• Strong Evidence
• Weak Evidence

50
Second Step EBP Protocol Summary

• Protocol Domain
• 1. Measurement
• 2. Comparison Groups
• 3. Statistical Analysis of Outcome Variables

• Second Step
• Strong Evidence
• Strong Evidence
• Strong Evidence

51
Second Step EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• Second Step
• Strong Evidence
• Strong Evidence
• Strong Evidence
• Strong Evidence

52
Next Steps in Research

• Based on these studies what do you think needs
  to be researched next?
• Research implications?
• Rich.???

53
Questions and Discussion
54
National Center for School Counseling Outcome
Research

• Thank You

www.cscor.org