An Overview of Educational Assessment in the

1
An Overview of Educational Assessment in the
Industrial and Management Engineering
Undergraduate Program Charles J.
Malmborg Department of Decision Sciences and
Engineering Systems
2
Overview The 12 Educational Objectives of the
IME Undergraduate Program Development and
Refinement of the IME Educational Objectives
1998-2005 How the Assessment Process Works The
Assessment Matrices The Assessment
Instruments The Assessment Software Continuous
Improvement and Process Maintenance Application
of the Assessment System Overview of the AY
2003-2004 Curriculum Development Process Lessons
Learned
3
Ability to apply a total integrated systems
perspective Ability to apply knowledge of
manufacturing and service systems Ability to
apply in-depth knowledge of computing Ability
to manage people and systems Ability to design
innovative products, services, facilities,
equipment, processes and systems Ability to
identify, model, analyze and solve challenging
real-life problems Solid foundation in
mathematics and science Strong written and
interpersonal communication skills Ability to
perform effectively on diverse teams as leader
and contributor Informed citizens broadly
educated in the humanities and social
sciences Prepared to practice engineering in a
socially responsible and ethical manner Learning
in a stimulating environment fostering continued
growth
4
Chronology of Activities Culminating in
Development of the 12 Educational Objectives Ja
nuary 1998 Special DSES Faculty Retreat on
Educational Objectives Response to Announcement
of the ABET 2000 Initiative Decision to Focus
on Differentiating Program Strengths Students,
Faculty, Employers and Alumni as Initial
Constituents
5
AY 1999-2000 DSES Performs Two Alumni Surveys
on Preliminary Objectives Industrial Advisory
Board Discussion and Feedback SoE Assessment
Committee Formed to Prepare for ABET
2000 Develops a Standard Template for SoE
Programs Tasks Programs with Populating
Assessment Templates and Soliciting Faculty
Input Collects Transcript and Survey Data to
Support the First Round of Assessment Using
Spreadsheet Based Analyses
6
AY 2000-2001 SoE Assessment Committee Completes
the First Round of Assessment Calculations Sugg
ests Process Improvements Coordinates Self
Study Reports for Fall 2001 ABET Visit
7
AY 2001-2002 SoE Assessment Committee Fall
2001 ABET Visit Yields Mixed Results Criticism
of Process Centralization, Perceived Lack of
Buy-In SoE Programs Encouraged to Personalize
Learning Objectives and Assessment
Processes SoE Continues to Support School Wide
Annual Alumni Survey With Some Sections
Personalized by Program
8
AY 2002-2003 DSES Undergraduate Advisory
Committee Proposes Expanded List of 12
Educational Objectives Solicits Faculty,
Student, Alumni and Industrial Advisory Board
Input Finalizes Assessment Matrices Formal
Faculty Vote on Final Objective List Taken in
March 2003 Formal Advisory Board Endorsement in
May 2003 Appoints DSES Faculty Member as
Assessment Coordinator
9
Summer 2003 DSES Assessment Coordinator and
Staff Assistant Update Data Collection
Instruments for Revised Objectives Datamining
of 2001, 2002, 2003 Assessment Instruments Proce
ss Transcript Data on IME Program
Graduates Design/Code Assessment Software for
Re-Designed Instruments Populate Supporting
Databases
10
AY 2003-2004 DSES Undergraduate Advisory
Committee Applies Assessment Software in
Committee Meetings Live Use of Assessment
Results for What-If Analyses Recalibration of
Target Performance Levels Major Curriculum
Revisions Guided by Assessment Results First
Round Changes to the Assessment System
11
Summer 2004 DSES Assessment Coordinator and
Staff Assistant Process Transcript Data on 2004
IME Program Graduates Program Software Upgrades
for First Round of Changes IDEA Form
Data CO-OP Evaluation Forms Update System
Databases with 2004 Data
12
AY 2004-2005 DSES Undergraduate Advisory
Committee Applies Assessment Software Detail
ed Performance Analysis Informed by
Assessment Transcript Analyses More
Difficult Performance Problems Further
Refinement of the Assessment Process Senior
Project Sponsor Evaluation Form
13
Plans for AY 2005-2006 DSES Undergraduate
Advisory Committee Long Term Planning for
Assessment Expanding Program Constituency
(Parents, Grad Schools) Profiling the Next
Generation Assessment System Assuring That
Refinement Does Not Stifle Innovation Closing
the Loop Between Course and Program Level
Objectives Adapting a More Proactive
Hypothesis Testing Approach Focusing on Direct
Measures
14
How the DSES ABET Assessment System
Works Assessment Matrices Annual Data
Collection Software Features Maintenance
Requirements
15
How the DSES ABET Assessment System Works
Assessment Matrices Original ABET 2000
Template Adapted/Refined for the IME
Program For Each Educational Objective,
Assessment Matrices Specify Statement of
Program Outcomes Pertinent to the
Objective Assessment Implementation
Strategy Performance Criteria Assessment
Methods, Timeline and Feedback Loop
16
How the DSES ABET Assessment System Works
Assessment Matrices Objective 2
Manufacturing and Service Systems Program
Outcomes Understanding Key Parallels
Differences Implementation Strategy
Transcripts, IDEA, Alumni, Senior
Surveys Criteria Minimum Acceptable
Distribution of Survey Results,
GPAs Methods/Timeline Annual Survey
Instruments Feedback Loop Undergraduate
Committee, Department Chair
17
How the DSES ABET Assessment System Works
Assessment Instruments Objective 2
Manufacturing and Service Systems Transcripts
of IME Program Graduates IDEA Form Special
Question Implemented Each Semester Alumni
Survey Senior Survey
18
How the DSES ABET Assessment System Works
Assessment Software Objective 2
Manufacturing and Service Systems 80 of
Graduates Achieve at Least 2.5 in Associated
Courses Students Respond Affirmatively to the
IDEA Form Question This Course Has Enhanced my
Knowledge of Manufacturing and Service
Systems 80 of Alumni Report That They Gained
an Understanding of Manufacturing and Service
Systems Through The Undergraduate Experience 80
of Graduating Seniors Agree That They Have an
Understanding of Manufacturing and Service Systems
19
How the DSES ABET Assessment System Works
Assessment Software Objective 2
Manufacturing and Service Systems Demonstration
of the Assessment Software Annual Maintenance
Requirements Implementation of Senior and
Alumni Surveys Collection of Objective Specific
Rating Forms Senior Project Industry Sponsor
Forms IDEA Form Extra Questions in IME
Courses Collection of Co-OP Evaluation
Forms Entry of Transcript Data/Database
Updates Software Maintenance
20
Application of the Assessment System AY
2003-2004 Overview Objective 1 The Systems
Approach Program is Exceeding Target
Performance No Change Objective 2
Manufacturing and Service Systems Underperforman
ce on Senior Survey, IAB Recommendations New
Course Requirement in Supply Chain Management
21
Application of the Assessment System AY
2003-2004 Overview Objective 3 Knowledge and
Applications of Computing Underperformance in
Grades, Senior Survey, Alumni Survey Change in
Programming Course Requirements Additional
Computing Modules in Three Existing
Courses Additional Course in Discrete Event
Simulation
22
Application of the Assessment System AY
2003-2004 Overview Objective 4 Managing People
and Systems IAB Recommendation,
Underperformance in Grades Additional
Management Elective Course Objective 5 Design
Skills Some Underperformance in Alumni Survey
Results Further Study Required New Survey
Instrument Introduced
23
Application of the Assessment System AY
2003-2004 Overview Objective 6 Problem Solving
Skills Generally Strong Performance No
Immediate Action Required Objective 7
Foundation in Mathematics and Science Significan
t Underperformance in Senior Survey Results and
Grades Recommendations Still Under
Development
24
Application of the Assessment System AY
2003-2004 Overview Objective 8 Communication
Skills Contradictory Results from Student and
Alumni Surveys New Survey Instrument Introduced
to Gain More Specific Information No Action
Pending Institute Level Changes in Core
Requirements Objective 9 Team Skills Program
is Generally Meeting Target Performance Levels
25
Application of the Assessment System AY
2003-2004 Overview Objective 10 Broad Based
Education Program is Exceeding Target
Performance No Change Objective 11
Ethics Program Appears to Meet Target
Performance Levels No Change Objective 12
Stimulating Learning Environment Program is
Exceeding Target Performance No Change
26
Lesson Learned Delineation of Assessment
Responsibilities is Crucial, Assessment
Coordinators Should Be Knowledgeable of and
Closely Aligned with the Degree Program Willing
to Take a Long Term Perspective Detail Oriented
When Necessary Able to Gain Cooperation from
Faculty Colleagues Open to Dealing with the
Non-Faculty Constituents (Students, Alumni,
Industrial Advisory Boards, etc.) Committed (no
pun intended)
27
Lesson Learned Long Cycle Process Curricular
Changes Can Be Slow to Impact Assessment
Results Early Rounds of the Assessment Process
Support Recalibration of Target Performance
Levels Obvious and Readily Addressable
Curricular Changes Design Changes in the
Assessment System Subsequent Rounds of the
Assessment Process Support More Subtle and
Difficult Curricular and Learning
Issues Incremental Improvements in the
Assessment System Modifications to the Initial
Assessment-Driven Changes Planning for the Next
Generation Assessment System
28
Lesson Learned Assessment Takes Significant Time
and Effort Faculty Time Commitment Assessment
Coordinator Degree Program Committee Course
Instructors Implementing Assessment
Instruments Staff Time Commitment Implementat
ion Support (Surveys, IAB Presentations,
etc.) Information Systems Management Data
Processing and Entry
29
Lesson Learned Resources Are Necessary to
Sustain the Assessment Process Recognition of
Assessment Leadership and Participation in
Annual Faculty Performance Evaluations Summer
Support for Assessment Coordinators Staff
Support for the Assessment Process
30
Lesson Learned Assessment Has Significant
Benefits Transformation of the ABET Process
From Periodic Burden to a Functional Process for
Curriculum Improvement More Explicit Statement
and Improved Communication of Program Learning
Goals and Objectives Increased Regularity and
Enforcement of Curriculum Review Improved
Coordination Between Course and Program
Development Increased Focus on Non-Faculty
Program Constituents