An Integrated Model for Measuring the Impact of Course, Program, and Institutional Learning Outcomes

1
An Integrated Model for Measuring the Impact of
Course, Program, and Institutional Learning
Outcomes
Keith Wurtz, Senior Research Analyst, Chaffey College Giovanni Sosa, Research Analyst, Chaffey College Jim Fillpot, Director of Institutional Research, Chaffey College

Strengthening Student Success Conference October 9th, 2009

2
Measuring the Impact of Learning Outcomes
Good evidence, then, is obviously related to the
questions the college has investigated and it can
be replicated, making it reliable. Good evidence
is representative of what is, not just an
isolated case, and it is information upon which
an institution can take action to improve. It
is, in short, relevant, verifiable,
representative, and actionable.
Guide to Evaluating Institutions ACCJC, August
2008
3
Measuring the Impact of Learning Outcomes
How do we collectively examine disparate learning
outcomes assessment efforts? (representativeness
and replicability) How do we provide evidence
about these disparate efforts? (verifiable) How
can we use evidence to improve upon learning
outcomes practices? Upon district practices?
(actionable)
4
Measuring the Impact of Learning Outcomes

• Assessment often stands alone relationships are
  often not explored between
• Courses From Different Programs
• Diverse Programs (within Instruction and
• Between Instruction and Student Services)
• Course/Program Learning Outcomes and Their
• Contribution to Institutional Learning
  Outcomes
• Types of Learning Outcomes (SLOs AUOs)
• Differing Methods of Assessment
• Learning Outcomes and College Goals

5
Measuring the Impact of Learning Outcomes
Statistical Analysis

• How do we statistically assess impact?
• Tests of statistical significance are not
  necessary
• Greatly influenced by sample size (mean 10.0
  10.1)
• Do not speak to the magnitude of the difference
• Not well understood even by experts
• Effect Size as measure of practical significance
• Unstandardized
• Standardized (d, r)
• Cohens conventions d .20 small .50
  moderate .80 large

6
Statistical Significance
7
Effect Size (unstandardized)
8
Effect Size (standardized)
9
Meta-Analysis

• Combines research results across all identified
  studies
• Combines results using Effect Size (i.e., d) -
  generates an Average d
• Average d may vary across study characteristics
  (i.e., Moderators)
• Number of Assessment Cycles
• Type of Assessment Instrument

10
Moderators(Grouping Variables)

• Term term data collection occurred
• Academic Year (AcadYr) Academic year (SU, FA,
  SP) data collection occurred
• Learning Outcome (LearnOutcome)
• AUO Administrative Unit Outcome
• SLO Student Learning Outome
• Researcher person who conducted research
• Program department
• Course

11
Effect Size by Academic Year
Number of Studies 1 9 15 44 48
12
Effect Size by Learning Outcome Type (AUO or SLO)
Number of Studies 2 115
13
Effect Size by Program
Number of Studies 3 1 14 1 2 11 10 5 11 59
14
Effect Size by Course
Number of Studies 1 21 12 12 14 3 1 4 2 4 2
15
Example of Meta Analysis Output from
Comprehensive Meta Analysis
16
Moderators (Continued)(Grouping Variables -
Measure)

• Criterion How was the Criteria for Success set
  by the program?
• Population A measure obtained from the
  population (e.g. critical thinking, Nursing
  Licensing Exam, etc.)
• Pre-assessment score The pre-assessment score
  in a pre-post assessment
• Program The program set a specific criterion
  (e.g. PE, Communication Studies, etc.)
• PrePost The assessment was a post-assessment or
  pre-post
• Standardized Measure used to assess outcome was
  standardized or unstandardized
• Instrument
• Rubric
• True/False
• Multiple choice
• Matching
• Likert Scale
• Anchored Scale
• Unduplicated used to identify number of
  students impacted by learning outcomes processes.
  From 2004-005 to 2008-2009 6,714 students have
  been impacted by learning outcomes.
• Assessor
• Self-Assessment (e.g. student self-assessed)
• External Evaluation (e.g. student evaluated by
  teacher)

17
Effect Size by Criterion Type
Number of Studies 1 31 85
18
Effect Size by Pre-Post or Post-Assessment
Number of Studies 85 32
19
Effect Size by whether the Instrument was
Standardized or Unstandardized
Number of Studies 10 107
20
Effect Size by Measurement Type
Number of Studies 7 22 2 8 75 3
21
Effect Size by whether Student Self-Assessed
Learning or Learning was Assessed by External
Evaluator (e.g. teacher)
Number of Studies 82 35
22
Moderators (Continued)(Grouping Variables -
Planning)

• EndsPolicy1
• (1) Instructional and Student Services, (2)
  Comprehensive Education Program, (3)
  Collaborative Partnerships, (4) Continuous
  Improvement, (5) Learning Outcomes, (6) Core
  Competencies
• EndsPolicy2
• (1) Foundation Skills Courses, (2) Occupational
  Programs, (3) Transfer Level Programs, (4)
  Outreach, (5) Student Services, (6) Library and
  Learning Support Services
• EndsPolicy3
• (1) College Facilities, (2) Technology, (3) Human
  Resources
• EndsPolicy4
• (1) Balanced Budget, (2) Achievement of Planned
  Enrollment Growth, (3) External Funding, (4) Bond
  Reserve
• EndsPolicy5
• (1 Qualified Personnel and Professional
  Development, (2) Commitment to Diversity, (3)
  Employment Agreement, (4) Professional Ethics

23
Effect Size by Ends Policy 1 Learning Centered
College
Number of Studies 1 7 23 86
24
Effect Size by Ends Policy 2 Institutional
Effectiveness
Number of Studies 1 92 22 1
25
Moderators (Continued)(Grouping Variables
Learning Outcomes)

• KSA Knowledge, Skill, or Ability (e.g.
  critical thinking, speech, confidence,
  self-efficacy, etc.)
• KSACateg KSA Categorized by Core Competency and
  Instruction
• Communication Students will demonstrate
  effective communication and comprehension skills.
• Critical Thinking Students will demonstrate
  critical thinking skills in problem solving
  across the disciplines and in daily life
• Community/Global Awareness and Responsibility
  Students will demonstrate knowledge of
  significant social, cultural, environmental and
  aesthetic perspectives
• Personal, Academic, and Care Development
  Students will assess their own knowledge, skills
  and abilities set personal, educational, and
  career goals work independently and in group
  settings identify lifestyle choices that promote
  self reliance, financial literacy and physical,
  mental and social health
• Instruction Students will demonstrate KSA
  specific to course subject (e.g. theory
  identification, research methods, etc.)
• Cycles Number of cycles program has assessed
  outcome
• Level
• Course
• Program
• Institution

26
Effect Size by Knowledge, Skill, or Ability
Number of Studies 65 9 12 20 11
27
Effect Size by Number of Times Program has
Assessed Outcome
Number of Studies 74 34 8 1
28
Effect Size by Course and Program Level Learning
Outcomes
Number of Studies 77 40
29
Limitations

• Database only includes results from outcomes
  evaluated by the Office of Institutional Research
• How do we connect information to student success
  (e.g. goals, course success, transfer, etc.)?

30
Resources

• Microsoft Excel Spreadsheets that calculate
  effect size statistics http//www.stat-help.com/s
  preadsheets.html
• Effect Size Calculator with Confidence Intervals
  http//www.cemcentre.org/renderpage.asp?linkID303
  25017