School of Engineering and Computer Science Program Assessment and Improvement

1
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Engineering an
• Assessment Program
• An Overview of the SECS
• Assessment Process
• March 7, 2007

2
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Once upon a time.
• Engineering program assessment was based on
  delivering content, and as long as a program
  could show that students took classes that
  totaled the proper number amount of hours in the
  proper content areas, accreditation was assured.
• Life was good.

3
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• In 2000, the accreditation requirements for
  engineering programs changed.
• And abruptly, people who were very comfortable
  with a long-established, well-defined and
  well-understood process based on concrete
  requirements
• were challenged to change their thinking and
  adapt to a system of self-defined goals and
  assessment.

4
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• The goals were to be written, with input from all
  constituent groups, as sets of skills that were
  to be measured, with feedback directed to improve
  future measurements.
• These skill sets consist of two types
• Goals to be demonstrated before students
  graduate program outcomes
• Goals to be demonstrated after graduation, as
  students work in the profession program
  educational objectives

5
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• With looming deadlines and very limited guidance,
  the SECS faculty began the process of developing
  an assessment and continuous improvement process
• We attended numerous workshops designed to
  facilitate moving to an assessment-based system
• We looked to technology to solve all of our
  problems
• We made a LOT of mistakes
• And, finally, we realized that we do this all the
  time, just not in a formalized manner

6
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Principles
• Assessment process to be applied SECS-wide
• Faculty-driven, designed to minimize impact on
  faculty time
• As much as possible, data to be taken, analyzed
  and stored online with documentation in
  Assessment Website

7
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• All SECS courses have course objectives, designed
  to introduce, reinforce and accomplish the
  program outcomes, a set of skills necessary for
  professional practice which SECS students are
  required to measurably demonstrate in key courses
  before graduation. Thus, the SECS assessment
  question is
• Do SECS students demonstrate achievement of the
  program outcomes before graduation?

8
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• The process relies on a limited amount of highly
  relevant data, systematically collected and
  analyzed, that directly measures the ability of
  students to demonstrate the program outcomes in
  key courses.
• Direct measures of student performance with
  respect to the program outcomes are backed up by
  several indirect survey measurements.

9
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Assessment data is evaluated and used by faculty
  committees charged with making recommendations
  for the improvement of the program
• Every step of the assessment process is
  documented in a central online database, for ease
  of recording, review and recall

10
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• As courses are given, student work is assigned
  which can be used, if required, as demonstrations
  of the development of, or even the achievement
  of, the program outcomes.
• Every instructor, in their end-of-course summary,
  is required to document and describe how the work
  the students performed relates to each program
  outcome, and the level to which the outcome has
  been addressed.

11
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• In certain courses in each program, key courses,
  the student work is collected and examined by
  external evaluators, who judge and document how
  well they believe each outcome is demonstrated.
• The direct external evaluation of program
  outcomes is the primary assessment tool of the
  SECS process.

12
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• The School and program curriculum committees
  examine the results of these external
  evaluations, along with the results of other,
  secondary assessment tools
• Course Evaluations
• Senior Exit Surveys
• Alumni input
• Employer input
• to recommend changes to the programs in order to
  improve the demonstration of the program
  outcomes.

13
School of Engineering and Computer
ScienceProgram Assessment and Improvement
SECS Courses
External Evaluation of Program Outcomes
Key Courses

• Course Objectives
• Program Outcomes

• Student End-of-Course Evaluations
• Faculty End-of-Course Summaries

Department SECS Undergraduate Affairs Committees
Program Educational Objectives

• Exit Survey
• Alumni Input
• Employer Input

SECS/Department Advisory Boards
14
School of Engineering and Computer
ScienceProgram Assessment and Improvement
SECS Courses
External Evaluation of Program Outcomes
Key Courses

• Course Objectives
• Program Outcomes

• Student End-of-Course Evaluations
• Faculty End-of-Course Summaries

Department SECS Undergraduate Affairs Committees
Program Educational Objectives

• Exit Survey
• Alumni Input
• Employer Input

SECS/Department Advisory Boards
15
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Current SECS Program Outcomes
• an ability to apply knowledge of mathematics,
  science, and engineering
• an ability to design and conduct experiments, as
  well as to analyze and interpret data
• an ability to design a system, component, or
  process to meet desired needs within realistic
  constraints such as economic, environmental,
  social, political, ethical, health and safety,
  manufacturability, and sustainability
• an ability to function on multi-disciplinary
  teams
• an ability to identify, formulate, and solve
  engineering problems

16
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Current SECS Program Outcomes (continued)
• an understanding of professional and ethical
  responsibility
• an ability to communicate effectively
• the broad education necessary to understand the
  impact of engineering solutions in a global,
  economic, environmental, and societal context
• a recognition of the need for, and an ability to
  engage in life-long learning
• a knowledge of contemporary issues
• an ability to use the techniques, skills, and
  modern engineering tools necessary for
  engineering practice.

17
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Links to course objectives
• Sample syllabus

18
School of Engineering and Computer
ScienceProgram Assessment and Improvement
SECS Courses
External Evaluation of Program Outcomes
Key Courses

• Course Objectives
• Program Outcomes

• Student End-of-Course Evaluations
• Faculty End-of-Course Summaries

Department SECS Undergraduate Affairs Committees
Program Educational Objectives

• Exit Survey
• Alumni Input
• Employer Input

SECS/Department Advisory Boards
19
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Course Evaluations and Summaries
• Completely online
• Each course has custom evaluation forms for both
  students and faculty
• Students notified via e-mail (with two reminders
  for those who need them) during last two weeks of
  semester with direct link to website

20
School of Engineering and Computer
ScienceProgram Assessment and Improvement
SECS Courses
External Evaluation of Program Outcomes
Key Courses

• Course Objectives
• Program Outcomes

• Student End-of-Course Evaluations
• Faculty End-of-Course Summaries

Department SECS Undergraduate Affairs Committees
Program Educational Objectives

• Exit Survey
• Alumni Input
• Employer Input

SECS/Department Advisory Boards
21
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• External Evaluation of Program Outcomes
• The main assessment tool in the SECS process
• Key Courses are
• Required of all students in the program
• Cannot be transferred from other institutions
• Usually are at the top of the programs
  curriculum
• Chosen so that each program outcome can be
  demonstrated at least twice

22
School of Engineering and Computer
ScienceProgram Assessment and Improvement
SECS Courses
External Evaluation of Program Outcomes
Key Courses

• Course Objectives
• Program Outcomes

• Student End-of-Course Evaluations
• Faculty End-of-Course Summaries

Department SECS Undergraduate Affairs Committees
Program Educational Objectives

• Exit Survey
• Alumni Input
• Employer Input

SECS/Department Advisory Boards
23
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• SECS Program Educational Objectives
• The undergraduate program in SECS will provide
  educational experiences aimed toward producing
  graduates who
• are able to design, implement and test systems or
  component (program-specific)
• can apply laboratory and computer skills to
  engineering analysis and design
• can adapt and contribute to new technologies and
  methods and use these in engineering design
• are prepared to pursue successfully graduate
  study in engineering or a related discipline

24
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• SECS Program Educational Objectives (continued)
• can function successfully in the automotive and
  other global industries
• can be effective in a variety of roles such as
  developing and implementing solutions to problems
  with technical and non-technical elements,
  serving as a team member and leading others
• are proficient in written and oral communication
  and
• have high standards of personal and professional
  integrity and ethical responsibility.

25
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Program Educational Objectives assessed through
• Online senior exit survey
• Alumni and employer input
• Formerly online surveys
• Currently face-to-face meetings with small groups

26
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• The success of the SECS Assessment and Continuous
  Improvement Process can best be demonstrated
  through two examples
• Development of a multidisciplinary senior design
  experience
• Development of a new engineering core curriculum

27
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Development of a multidisciplinary senior design
  experience
• Need for change was based on feedback gathered in
• Senior design end-of-course evaluations and
  summaries
• Senior exit surveys and face-to-face interviews
• Alumni surveys
• Many students reported frustrating lack of
  knowledge of other branches of engineering and
  computer science that made it virtually
  impossible to perform meaningful design projects
  that were confined only to their majors.

28
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Development of a multidisciplinary senior design
  experience
• As a result, all of the senior design courses in
  SECS were combined into a single,
  multidisciplinary design experience
• Every student group contains electrical,
  computer, mechanical and systems engineering
  majors and computer science majors
• Projects can be explored that are truly
  meaningful and require the latest modern
  engineering skills, techniques and standards
• Workplace-style student interaction grades
  result from successful presentations of
  competitive designs
• Continuous revision and improvement, driven by
  assessment data

29
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Development of a multidisciplinary senior design
  experience
• Over 2000 people attended the project fair in
  Winter 2006
• OU Provost The SECS Senior Design Experience is
  a vital undergraduate research program and
  provides 20k/year in support

30
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Development of a re-focused engineering core
  curriculum
• Changes based on feedback gathered in
• Senior design end-of-course evaluations and
  summaries
• Senior exit surveys and interviews
• Alumni and employer surveys and interviews
• Discussions with Advisory Board members
• All the above assessment data reinforced the
  notion that a re-focus to a systems approach to
  engineering would better serve our students
  during their academic and professional careers

31
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Development of a new engineering core curriculum
• It became clear that all engineers need
• a Broad view of engineering topics to allow them
  to see the forest and not just the trees and to
  be better prepared to confront unknown future
  challenges
• to learn the Essential fundamentals of basic
  engineering topics
• a Springboard to lifelong learning
• Provided access to modern engineering Tools

32
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Development of a new engineering core curriculum
• A task-force of faculty designed a new core
  curriculum consisting of 6 new courses that would
  allow students to
• be exposed to a broad view of engineering early
  in the curriculum,
• acquire the fundamentals of computer, electrical,
  mechanical, industrial and systems engineering,
• develop problem solving skills,
• develop laboratory skills,
• use modern engineering tools early in the
  curriculum,
• develop and practice oral and written
  communication skills,
• participate in the design process through a
  significant team design experience during the
  sophomore year.

33
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Development of a new engineering core curriculum
• The changes in the Core have required that all
  courses, and all course sequences, in all
  programs be re-examined and revised
• This work continues as the revised Core is
  implemented and adjusted to better deliver its
  intended content and systems philosophy

34
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• SECS assessment documentation
• Online assessment and improvement process
  overview and schedules.
• Results of Exit, Alumni and Employer Surveys.
• Custom customized external evaluation form
  generator
• Online documentation of assessment activities.
• Online documentation and review of external
  evaluation assessments.
• Online documentation of feedback implementation
  and program improvement.

35
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Look to the future - Improvements for a better
  system
• Obtain alumni and employer feedback with
  different instruments our surveys have not
  worked well at all.
• Meet with company representatives in small
  groups, and with small groups of alumni to find
  out well they are served by our programs.
• Minutes of these meetings take the place of the
  poor response to the employer and alumni surveys.

36
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Looking to the future - A need for a better
  system
• The documentation of our current system can be
  greatly improved. Industry recently has had to
  develop a way to deal with this.
• Set up a ISO-9000 type of system, with multiple
  layers of extensive and complete documentation
  (definitions, descriptions, mission, standardized
  reports through templates) electronically stored
  for easy submission and retrieval
• Require closing the loop meetings in each
  program. Minutes to be stored via the system
  described above

37
School of Engineering and Computer
ScienceProgram Assessment and Improvement

• Looking to the future
• The SECS Faculty are committed to continuing to
  develop an assessment system that will serve us
  better, and allow us to continuously improve our
  programs based on real, measured data.