Evaluation is not a Four Letter Word: How to Know Whether Or Not What Your Are Doing is Working - PowerPoint PPT Presentation

1 / 26
About This Presentation
Title:

Evaluation is not a Four Letter Word: How to Know Whether Or Not What Your Are Doing is Working

Description:

... delivered in a standardized fashion consistent with the developer's designs? ... curriculum goals (self efficacy, self-management, career decision-making) ... – PowerPoint PPT presentation

Number of Views:75
Avg rating:3.0/5.0
Slides: 27
Provided by: uma7
Category:

less

Transcript and Presenter's Notes

Title: Evaluation is not a Four Letter Word: How to Know Whether Or Not What Your Are Doing is Working


1
Evaluation is not a Four Letter Word How to Know
Whether Or Not What Your Are Doing is Working
  • Colorado School Counselors Association
  • April 20, 2006
  • John Carey,
  • National Center for School Counseling Outcome
    Research

2
Factors Leading to Need Data Skills in School
Counseling
  • Accountability Movement
  • Standards-Based Education Movement (NCLB)
  • School Reform Movement
  • New Models of School Counseling Practice
  • Education Trust Transforming School Counseling
    Initiative
  • ASCA National Model

3
A Model of Evidence-Based Practice
Practitioners Individual Expertise
Best Evidence
Client Values and Expectations
EBP
Adapted from Shlonsky and Gibbs (2004), Will
the Real Evidence-Based Practice Please Stand Up?
Teaching the Process of Evidence-Based Practice
to the Helping Professions. In Brief Treatment
and Crisis Intervention, 4(2), 137-153.
4
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
5
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals Knowing what is needed
EBP
6
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence Knowing what
generally works
Assessment of Intervention Targets and
Goals
EBP
7
Evidence-Based Practice in School Counseling
Outcome Research Competence
Intervention and Program EvaluationCompetence
Knowing how studentschanged
Assessment of Intervention Targets and Goals
EBP
8
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
9
Terms
  • Date-Based Decision Making use of school data
    to determine problems that need to be addressed
  • Outcome Research use of the scientific method to
    discover generalizable truth about the
    effectiveness of interventions.
  • Evaluation use of the scientific method to
    improve local decision-making by determining
    whether it was likely that an intervention was
    resulted desired changes in behavior

10
How These Fit Together
  • Data-Based Decision Making
  • 8th graders in Carey Middle school are doing
    poorly on the state test
  • Outcome Research
  • Research yields strong evidence that Student
    Success Skills can increase test score by
    teaching self-management and enhancing motivation
  • Program Evaluation
  • Carey Middle School students learn self
    management and improve grades and test scores
    after SSS

11
Evaluation is Not Reserch
  • Research use of the scientific method to
    discover generalizable truth about the
    relationships between variables.
  • Evaluation use of the scientific method to
    improve local decision-making

12
Evaluation is not Research
  • Research is concerned with eternal truth.
  • Research seeks to prove causality by eliminating
    all but one cause of an outcome.
  • Research seeks to minimize false positives by
    setting very conservative criteria. (alpha
    level).
  • Research seeks to maximize generalizability by
    including a broad, representative sample.

13
Research
14
Evaluation is not Research
  • Evaluation is concerned with pragmatic truth.
  • Evaluation seeks to establish that an given
    outcomes is likely to result from an
    intervention.
  • Evaluation is designed to minimize false
    negatives by setting relatively liberal criteria
    (alpha level)
  • Evaluation is concerned only with the context at
    hand.

15
Evaluation Issues
  • Outcomes Measurement
  • Immediate
  • Proximal
  • Distal
  • Intervention Fidelity and Dose (Formative
    Evaluation)
  • Summative Evaluation Design
  • Single Group
  • Comparison Groups
  • Statistical Analysis

16
Formative Evaluation Intervention Fidelity
  • Is the intervention being delivered in a
    standardized fashion consistent with the
    developers designs?
  • Good Manual
  • Adequate Training
  • Frequent Supervision
  • Periodic Content Checks
  • Periodic Student Learning Checks

17
Formative Evaluation
  • Are students receiving a strong dose.
  • Expertise of the People
  • Quality of the Intervention
  • Amount of Engaged Time.
  • Length of Time over which the Intervention Takes
    Place.

18
Summative Evaluation
  • 1) Did students learn what was intended?
    (Immediate Outcomes)
  • 2) Did students change in ways the predict long
    term changes in school behavior and performance?
    (Proximal Outcomes)
  • 3) Did students show long term changes in
    behavior and/or performance? (Distal Outcomes)

19
Summative Evaluation Example
  • Immediate Outcome After a training session
    students can name five effective conflict
    resolution strategies.
  • Proximal Outcome After training students show
    an increase in effective conflict resolution
    skills.
  • Distal Outcome After training the school shows
    a decrease in disciplinary referrals and an
    increase in safety scores on a school climate
    survey.

20
Outcome Measures Immediate Outcomes
  • Are the students learning what the intervention
    is intended to teach?
  • Most interventions include instruments tied to
    learning objectives.
  • May need to construct surveys.
  • Administer pre and post if possible.
  • Also useful as formative evaluation information.

21
Outcome Measures Proximal Outcomes
  • Do students show expected gains related to school
    counseling curriculum goals (self efficacy,
    self-management, career decision-making)?
  • Some interventions include psychometrically
    instruments related to proximal outcomes.
  • Free Internet resources for psychometrically
    sound instruments are useful.
  • Buros Test Reviews online.
  • Build your own only as a last resort.

22
Outcome Measures Distal Measures
  • So students show expected gains on important
    school performance measures (achievement test
    scores, discipline referrals, graduation rates) ?
  • Typically measured in institutional student
    data-base.
  • May need to add data to student data base.

23
Summative Evaluation Design
  • Single Group Pre-Post
  • Weaker design in terms of Interval Validity
  • Comparison Group
  • Random assignment seldom feasible
  • Closer matching yields stronger assurance of
    Internal Validity
  • Statistical equating possible if pretest
    differences are noted

24
Statistical Analysis
  • Create summary tables
  • Use statistical tests to determine confidence
    that observed differences are real.
  • Generally use less conservative alpha since the
    goal is better decisions not universal truth.
  • EZAnalyse is free software keyed to ASCA National
    model

25
Questions and Discussion
26
National Center for School Counseling Outcome
Research
  • Thank You

www.cscor.org
Write a Comment
User Comments (0)
About PowerShow.com