Using Measurable Outcomes to Evaluate Tutor Programs

1
Using Measurable Outcomes to Evaluate Tutor
Programs

• Innovative Educators Webinar
• December 1, 2010
• Jan Norton, Presenter

2
Webinar Contents

• Terminology
• The Black Box
• Timetables
• Data/Measurement
• Data/Measurement Cautions
• Program Outcomes Examples
• Student Outcomes Examples
• Closing Questions

3
Terminology

• Visions
• Missions
• Values
• Goals
• Objectives
• Outcomes
• Program vs. Student Outcomes
• Learning Outcomes

4
The Black Box

• .

Something Happens
5
Inside the Black Box

• Actions leading to specific, observable results
• What students can expect to learn
• How students or programs will be different as a
  result of what happens in the box
• What knowledge, skills, and abilities will be
  improved
• What measurable changes occur in tutoring
  programs

6
Timetables

• Scope of assessment
• 6 years
• 2 years
• 1 year
• 1 semester
• 1 course
• 1 assignment
• 1 tutoring session

7
Timetables

• Planning for Assessment
• Continual processes
• Multiple times per semester
• Once or twice per semester
• Two or three times per year
• Once per year
• Every two years
• Every 4 years

8
Data/Measurement

• Why Measure
• What to measure
• Before (Benchmarks)
• During
• After (Outcomes)

9
Data/Measurement

• How to measure
• Program-generated data
• Course-specific data
• Campus data
• Campus surveys
• Standardized tools
• Locally developed tools

10
Data/Measurement

• More about locally developed tools
• Surveys
• Likert-type scales
• Explicated scales
• Behavioral rubrics
• Self assessment by student
• Tutor assessment
• Observer assessment

11
Data/Measurement Cautions

• Direct vs. Indirect Measures
• Data access and consistency
• Realistic limitations
• Cooperation
• The Baseline Challenge
• Time money
• (and staff and comfort and )

12
Program Outcomes

• Possible Areas of Assessment Focus
• Establishing baselines
• Usage
• Satisfaction or quality
• Level/variety of services
• Staff development
• Reaching certification or meeting standards

13
Program Outcomes Examples

• Outcome Statement
• As a result of the new marketing strategy, the
  number of students attending group tutoring
  sessions will increase by 10 in Spring 2011.
• Timetable
• One semester
• Starting Measure (Benchmark)
• Average of past attendance
• Ending Measure
• Workshop attendance records

14
Program Outcomes Examples

• Outcome Statement
• During 2010-11, the number of tutors who earn
  CRLA Level I certification will increase by 5.
• Timetable
• One academic year
• Starting Measure (Benchmark)
• Number of tutors achieving Level I in previous
  year(s)
• Ending Measure
• Number of tutors achieving Level I in 2010-11

15
Program Outcomes Examples

• Outcome Statement
• Outside reviewers using the NADE self-study
  process will score the tutoring program at 85 or
  above on all sections.
• Timetable
• One year
• Starting Measure (Benchmark)
• Internal review self-study
• Ending Measure
• External review

16
Program Outcomes Examples

• Outcome Statement
• As a result of tutor training sessions, tutors
  will demonstrate an improved understanding of
  Blooms Taxonomy.
• Timetable
• One semester or year, depending on training
  schedule
• Starting Measure (Benchmark)
• Pre-test
• Ending Measure
• Post-test

17
Student Outcomes

• Types

• Measures

• Affective
• Build confidence
• Improve attitude
• Reduce anxiety
• Cognitive
• Mastery of material
• Conscious strategies use

• Qualitative
• Interviews focus groups
• Opinion surveys
• Observations or self reports
• Quantitative
• Assignment course grades
• Instrument scores
• Observation rubrics

18
Student Outcomes Examples

• Outcome Statement
• After being tutored for at least one hour,
  clients will be more confident in their ability
  to accomplish the academic task they focused on
  in tutoring.
• Timetable
• One tutoring session
• Starting Measure (Benchmark)
• Client self-report
• Ending Measure
• Client self report

19
Student Outcomes Examples

• Outcome Statement
• Students who work with the study skills tutors
  will demonstrate an increased ability to use
  memorization strategies.
• Timetable
• One tutoring session
• Starting Measure (Benchmark)
• Tutor or observer checklist
• Ending Measure
• Tutor or observer checklist

20
Student Outcomes Examples

• Outcome Statement
• 75 of the students who work with a reading tutor
  for at least two sessions will score at least 75
  on the main ideas reading test.
• Timetable
• Within three weeks
• Starting Measure (Benchmark)
• Reading quiz at start of tutoring
• Ending Measure
• Chapter reading test in class

21
Student Outcomes Examples

• Outcome Statement
• Students who attend at least 10 group tutoring
  sessions for Algebra Review 090 will earn an
  average grade of B or above in the course.
• Timetable
• One semester, length of course
• Starting Measure (Benchmark)
• Placement score into course
• Ending Measure
• Course grade

22
Student Outcomes Examples

• Outcome Statement
• After attending mandatory tutoring for all 5 of
  their developmental education courses, students
  will be retained until the start of their junior
  year at a rate at least 25 higher than
  developmental students for whom tutoring was not
  required.
• Timetable
• Two years
• Starting Measure (Benchmark)
• Developmental course placement
• Ending Measure
• Enrollment level

23
Student Outcomes Examples

• Outcome Statement
• Students attending SI for World Religions will
  apply previously understood information,
  concepts, and experiences to a new situation or
  setting. (CAS)
• Timetable
• Within one semester
• Starting Measure (Benchmark)
• Pre-survey
• Ending Measure
• Post-survey

24
Closing Questions

• Check with your institutional research or
  assessment office
• Review early drafts of outcomes statements with
  your supervisor
• Pre-test any locally created measurement tool
• Be prepared for outcomes that are not met or
  wildly exceeded
• Remember the black box we do not own or control
  the box, and our ability to see into it is very
  limited