Title: Evaluation is not a Four Letter Word: How to Know Whether Or Not What Your Are Doing is Working
1Evaluation is not a Four Letter Word How to Know
Whether Or Not What Your Are Doing is Working
- Colorado School Counselors Association
- April 20, 2006
- John Carey,
- National Center for School Counseling Outcome
Research
2Factors Leading to Need Data Skills in School
Counseling
- Accountability Movement
- Standards-Based Education Movement (NCLB)
- School Reform Movement
- New Models of School Counseling Practice
- Education Trust Transforming School Counseling
Initiative - ASCA National Model
3A Model of Evidence-Based Practice
Practitioners Individual Expertise
Best Evidence
Client Values and Expectations
EBP
Adapted from Shlonsky and Gibbs (2004), Will
the Real Evidence-Based Practice Please Stand Up?
Teaching the Process of Evidence-Based Practice
to the Helping Professions. In Brief Treatment
and Crisis Intervention, 4(2), 137-153.
4Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
5Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals Knowing what is needed
EBP
6Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence Knowing what
generally works
Assessment of Intervention Targets and
Goals
EBP
7Evidence-Based Practice in School Counseling
Outcome Research Competence
Intervention and Program EvaluationCompetence
Knowing how studentschanged
Assessment of Intervention Targets and Goals
EBP
8Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
9Terms
- Date-Based Decision Making use of school data
to determine problems that need to be addressed - Outcome Research use of the scientific method to
discover generalizable truth about the
effectiveness of interventions. - Evaluation use of the scientific method to
improve local decision-making by determining
whether it was likely that an intervention was
resulted desired changes in behavior
10How These Fit Together
- Data-Based Decision Making
- 8th graders in Carey Middle school are doing
poorly on the state test - Outcome Research
- Research yields strong evidence that Student
Success Skills can increase test score by
teaching self-management and enhancing motivation - Program Evaluation
- Carey Middle School students learn self
management and improve grades and test scores
after SSS
11Evaluation is Not Reserch
- Research use of the scientific method to
discover generalizable truth about the
relationships between variables. - Evaluation use of the scientific method to
improve local decision-making
12Evaluation is not Research
- Research is concerned with eternal truth.
- Research seeks to prove causality by eliminating
all but one cause of an outcome. - Research seeks to minimize false positives by
setting very conservative criteria. (alpha
level). - Research seeks to maximize generalizability by
including a broad, representative sample.
13Research
14Evaluation is not Research
- Evaluation is concerned with pragmatic truth.
- Evaluation seeks to establish that an given
outcomes is likely to result from an
intervention. - Evaluation is designed to minimize false
negatives by setting relatively liberal criteria
(alpha level) - Evaluation is concerned only with the context at
hand.
15Evaluation Issues
- Outcomes Measurement
- Immediate
- Proximal
- Distal
- Intervention Fidelity and Dose (Formative
Evaluation) - Summative Evaluation Design
- Single Group
- Comparison Groups
- Statistical Analysis
16Formative Evaluation Intervention Fidelity
- Is the intervention being delivered in a
standardized fashion consistent with the
developers designs? - Good Manual
- Adequate Training
- Frequent Supervision
- Periodic Content Checks
- Periodic Student Learning Checks
17Formative Evaluation
- Are students receiving a strong dose.
- Expertise of the People
- Quality of the Intervention
- Amount of Engaged Time.
- Length of Time over which the Intervention Takes
Place.
18Summative Evaluation
- 1) Did students learn what was intended?
(Immediate Outcomes) - 2) Did students change in ways the predict long
term changes in school behavior and performance?
(Proximal Outcomes) - 3) Did students show long term changes in
behavior and/or performance? (Distal Outcomes)
19Summative Evaluation Example
- Immediate Outcome After a training session
students can name five effective conflict
resolution strategies. - Proximal Outcome After training students show
an increase in effective conflict resolution
skills. - Distal Outcome After training the school shows
a decrease in disciplinary referrals and an
increase in safety scores on a school climate
survey.
20Outcome Measures Immediate Outcomes
- Are the students learning what the intervention
is intended to teach? - Most interventions include instruments tied to
learning objectives. - May need to construct surveys.
- Administer pre and post if possible.
- Also useful as formative evaluation information.
21Outcome Measures Proximal Outcomes
- Do students show expected gains related to school
counseling curriculum goals (self efficacy,
self-management, career decision-making)? - Some interventions include psychometrically
instruments related to proximal outcomes. - Free Internet resources for psychometrically
sound instruments are useful. - Buros Test Reviews online.
- Build your own only as a last resort.
22Outcome Measures Distal Measures
- So students show expected gains on important
school performance measures (achievement test
scores, discipline referrals, graduation rates) ? - Typically measured in institutional student
data-base. - May need to add data to student data base.
23Summative Evaluation Design
- Single Group Pre-Post
- Weaker design in terms of Interval Validity
- Comparison Group
- Random assignment seldom feasible
- Closer matching yields stronger assurance of
Internal Validity - Statistical equating possible if pretest
differences are noted
24Statistical Analysis
- Create summary tables
- Use statistical tests to determine confidence
that observed differences are real. - Generally use less conservative alpha since the
goal is better decisions not universal truth. - EZAnalyse is free software keyed to ASCA National
model
25Questions and Discussion
26National Center for School Counseling Outcome
Research
www.cscor.org