Title: An Integrated Model for Measuring the Impact of Course, Program, and Institutional Learning Outcomes
1An Integrated Model for Measuring the Impact of
Course, Program, and Institutional Learning
Outcomes
Keith Wurtz, Senior Research Analyst, Chaffey College Giovanni Sosa, Research Analyst, Chaffey College Jim Fillpot, Director of Institutional Research, Chaffey College
Strengthening Student Success Conference October 9th, 2009
2Measuring the Impact of Learning Outcomes
Good evidence, then, is obviously related to the
questions the college has investigated and it can
be replicated, making it reliable. Good evidence
is representative of what is, not just an
isolated case, and it is information upon which
an institution can take action to improve. It
is, in short, relevant, verifiable,
representative, and actionable.
Guide to Evaluating Institutions ACCJC, August
2008
3Measuring the Impact of Learning Outcomes
How do we collectively examine disparate learning
outcomes assessment efforts? (representativeness
and replicability) How do we provide evidence
about these disparate efforts? (verifiable) How
can we use evidence to improve upon learning
outcomes practices? Upon district practices?
(actionable)
4Measuring the Impact of Learning Outcomes
- Assessment often stands alone relationships are
often not explored between - Courses From Different Programs
- Diverse Programs (within Instruction and
- Between Instruction and Student Services)
- Course/Program Learning Outcomes and Their
- Contribution to Institutional Learning
Outcomes - Types of Learning Outcomes (SLOs AUOs)
- Differing Methods of Assessment
- Learning Outcomes and College Goals
5Measuring the Impact of Learning Outcomes
Statistical Analysis
- How do we statistically assess impact?
- Tests of statistical significance are not
necessary - Greatly influenced by sample size (mean 10.0
10.1) - Do not speak to the magnitude of the difference
- Not well understood even by experts
- Effect Size as measure of practical significance
- Unstandardized
- Standardized (d, r)
- Cohens conventions d .20 small .50
moderate .80 large
6Statistical Significance
7Effect Size (unstandardized)
8Effect Size (standardized)
9Meta-Analysis
- Combines research results across all identified
studies - Combines results using Effect Size (i.e., d) -
generates an Average d - Average d may vary across study characteristics
(i.e., Moderators) - Number of Assessment Cycles
- Type of Assessment Instrument
10Moderators(Grouping Variables)
- Term term data collection occurred
- Academic Year (AcadYr) Academic year (SU, FA,
SP) data collection occurred - Learning Outcome (LearnOutcome)
- AUO Administrative Unit Outcome
- SLO Student Learning Outome
- Researcher person who conducted research
- Program department
- Course
11Effect Size by Academic Year
Number of Studies 1 9 15 44 48
12Effect Size by Learning Outcome Type (AUO or SLO)
Number of Studies 2 115
13Effect Size by Program
Number of Studies 3 1 14 1 2 11 10 5 11 59
14Effect Size by Course
Number of Studies 1 21 12 12 14 3 1 4 2 4 2
15Example of Meta Analysis Output from
Comprehensive Meta Analysis
16Moderators (Continued)(Grouping Variables -
Measure)
- Criterion How was the Criteria for Success set
by the program? - Population A measure obtained from the
population (e.g. critical thinking, Nursing
Licensing Exam, etc.) - Pre-assessment score The pre-assessment score
in a pre-post assessment - Program The program set a specific criterion
(e.g. PE, Communication Studies, etc.) - PrePost The assessment was a post-assessment or
pre-post - Standardized Measure used to assess outcome was
standardized or unstandardized - Instrument
- Rubric
- True/False
- Multiple choice
- Matching
- Likert Scale
- Anchored Scale
- Unduplicated used to identify number of
students impacted by learning outcomes processes.
From 2004-005 to 2008-2009 6,714 students have
been impacted by learning outcomes. - Assessor
- Self-Assessment (e.g. student self-assessed)
- External Evaluation (e.g. student evaluated by
teacher)
17Effect Size by Criterion Type
Number of Studies 1 31 85
18Effect Size by Pre-Post or Post-Assessment
Number of Studies 85 32
19Effect Size by whether the Instrument was
Standardized or Unstandardized
Number of Studies 10 107
20Effect Size by Measurement Type
Number of Studies 7 22 2 8 75 3
21Effect Size by whether Student Self-Assessed
Learning or Learning was Assessed by External
Evaluator (e.g. teacher)
Number of Studies 82 35
22Moderators (Continued)(Grouping Variables -
Planning)
- EndsPolicy1
- (1) Instructional and Student Services, (2)
Comprehensive Education Program, (3)
Collaborative Partnerships, (4) Continuous
Improvement, (5) Learning Outcomes, (6) Core
Competencies - EndsPolicy2
- (1) Foundation Skills Courses, (2) Occupational
Programs, (3) Transfer Level Programs, (4)
Outreach, (5) Student Services, (6) Library and
Learning Support Services - EndsPolicy3
- (1) College Facilities, (2) Technology, (3) Human
Resources - EndsPolicy4
- (1) Balanced Budget, (2) Achievement of Planned
Enrollment Growth, (3) External Funding, (4) Bond
Reserve - EndsPolicy5
- (1 Qualified Personnel and Professional
Development, (2) Commitment to Diversity, (3)
Employment Agreement, (4) Professional Ethics
23Effect Size by Ends Policy 1 Learning Centered
College
Number of Studies 1 7 23 86
24Effect Size by Ends Policy 2 Institutional
Effectiveness
Number of Studies 1 92 22 1
25Moderators (Continued)(Grouping Variables
Learning Outcomes)
- KSA Knowledge, Skill, or Ability (e.g.
critical thinking, speech, confidence,
self-efficacy, etc.) - KSACateg KSA Categorized by Core Competency and
Instruction - Communication Students will demonstrate
effective communication and comprehension skills. - Critical Thinking Students will demonstrate
critical thinking skills in problem solving
across the disciplines and in daily life - Community/Global Awareness and Responsibility
Students will demonstrate knowledge of
significant social, cultural, environmental and
aesthetic perspectives - Personal, Academic, and Care Development
Students will assess their own knowledge, skills
and abilities set personal, educational, and
career goals work independently and in group
settings identify lifestyle choices that promote
self reliance, financial literacy and physical,
mental and social health - Instruction Students will demonstrate KSA
specific to course subject (e.g. theory
identification, research methods, etc.) - Cycles Number of cycles program has assessed
outcome - Level
- Course
- Program
- Institution
26Effect Size by Knowledge, Skill, or Ability
Number of Studies 65 9 12 20 11
27Effect Size by Number of Times Program has
Assessed Outcome
Number of Studies 74 34 8 1
28Effect Size by Course and Program Level Learning
Outcomes
Number of Studies 77 40
29Limitations
- Database only includes results from outcomes
evaluated by the Office of Institutional Research - How do we connect information to student success
(e.g. goals, course success, transfer, etc.)?
30Resources
- Microsoft Excel Spreadsheets that calculate
effect size statistics http//www.stat-help.com/s
preadsheets.html - Effect Size Calculator with Confidence Intervals
http//www.cemcentre.org/renderpage.asp?linkID303
25017