ABET Assessing Program Outcomes

1
ABETAssessing Program Outcomes

• Amir Rezaei

2
Outline

• Context of Assessment
• Process of Assessment
• Similarities and differences between classroom
  and program assessment
• Developing measurable outcomes
• Developing scoring rubrics
• Assessment methods
• Direct measures/Indirect measures
• Data collections
• Closing Loops
• Writing the report

3
Context of Assessment
What are we doing with the inputs?
What comes into the system?
What is the effect?
How many?
Input Progress Output Outcomes
Student Background Program services offered population served Student grades graduation rates employment statistics What have students learned what skills have they gained attitudes developed?
Faculty background Faculty teaching loads/class size Publication numbers/Faculty development activities credit hrs delivered Faculty publication citations data faculty developement
Educational Resources Policies, procedures, governance Statistics on resource availability, participation rate Student learning and growth
Input Progress
Student Background Program services offered population served
Faculty background Faculty teaching loads/class size
Educational Resources Policies, procedures, governance
Output
Student grades graduation rates employment statistics
Publication numbers/Faculty development activities credit hrs delivered
Statistics on resource availability, participation rate
Outcomes
What have students learned what skills have they gained attitudes developed?
Faculty publication citations data faculty developement
Student learning and growth
4
Process of Assessment
Collect data when they graduate

• Can we demonstrate that student have learned
  outcomes xx to an appropriate level by the time
  of graduation?
• Can we demonstrate that we have added value to
  student learning of outcome xx to an appropriate
  level by the time of graduation?

Pre and post data collection
5
ABET uses these terms, Use same language to
reduce the confusion
ABET Terms Definition Some other terms for same concepts
Objectives Broad statement that describe the career and professional accomplishments that the program is preparing graduates to achieve Goals, outcomes, purpose, etc
Outcomes Statement that describe what students are expected to know and able to do by the time graduation Objectives, standards, etc
Performance Criteria Specific, measurable statements identifying the performance(s) required to meet the outcome evidence Standards, rubrics, specifications, metrics, outcomes, etc.
Assessment Process that identify, collect, use and prepare data that can be used to evaluate achievement Evalautions
Evaluation Process of reviewing the results of data collection and analysis and making a determination of the value of finding and action to be taken Assesments
ABET Terms
Objectives
Outcomes
Performance Criteria
Assessment
Evaluation
6
Similarities and differences between classroom
and program assessment

• Degree of complexity
• Time span
• Accountability for the assessment process
• Cost
• Level of faculty buy-in
• Level of precision of the measure

7
Classroom Assessment
Context Subject matter Faculty member Pedagogy Stu
dent Facility
Concepts Forces in 2d 3D Moment f a
force about.. Eq in 2D 3D FBD Trusses, Frames
and Machines Friction
Topics Statics of particles Equivalent
system of forces Eq. Rigid bodies Equilibrium Str
uctures Friction
Subject Statics
Assessment Focus Evaluate individual
studentperformance (grades)Evaluate
teaching/learning
8
Program Assessment
Student Pre-college Traits
Classroom Experience Pedagogy Facility Faculty
student Characteristics
9
Developing measurable outcomes
Performance Criteria
Outcomes
Researches and gathers information Fulfill
duties of team roles Shares work equally Listen
to other teammates
Objetive
Ability to function on
multi-disciplinary team
Work Effectively with other
Makes contributions Takes responsibility Value
other viewpoints
10
Developing scoring rubrics

• A rubric is a set of categories which define and
  describe the important components of the work
  being completed, critiqued, or assessed
• Purposes
• Information to/about individual student
  competence (Analytic)
• Overall examination of the status of the
  performance of a group of students (Holistic)

11
Developing scoring rubrics

• Generic
• Big picture approach
• Element of subjectivity
• Task-specific
• Single task
• Focused approach
• Less subjective
• You dont have to develop a rubric for every
  outcomes

-Note Rubric template
12
Assessment methods

• Written surveys and questionnaires
• Exit and other interviews
• Commercial, standardized exams
• Locally developed exam
• Archival records
• Focus group
• Portfolios
• Simulations
• Performance Appraisal
• External examiner
• Oral exam
• Behavioral observations

No rubric Rubric Rubric No rubric No rubric No
rubric Rubric Rubric Rubric Rubric Rubric Rubric
13
Direct and Indirect Measures

• Direct measures
• Direct measures provide for the direct
  examination or observations of student knowledge
  or skills against measurable learning outcomes
• Indirect Measures
• Indirect measures are those that ascertain the
  opinion of self-report of the extent or value of
  learning experiences

14
Direct Indirect

• Exit and other interviews
• Standardized exams
• Locally developed exams
• Portfolios
• Simulations
• Performance Appraisal
• External examiner
• Oral exam
• Behavioral observation

• Written surveys and questionnaires
• Exit and other interviews
• Archival records
• Focus groups

15
Sampling And Data collection

• For program assessment, sampling is acceptable
  and desirable for programs of sufficient size

Year 1
Year 2
Year 3
Year 4
Define Outcomes/ Map curr.
Data Collection
Evaluation Design of implementation
Implement Improvements Data collec.
16
Closing the Loop
Evaluation committee receive and evaluate all
data makes report and refers recommendations to
appropriate areas
Institute acts on the recommendations of the
Eval. Comm. Report of actions taken by the
Institute and the targeted areas are returned to
the Eval Comm. For iterative evaluation
Institute assessment comm. Prepares reports for
submission to dept. Heads of the collected data
(e.g. survey, e-portfolio rating)