Assessment Plan and Notes on the ABET Program evaluator training - PowerPoint PPT Presentation


PPT – Assessment Plan and Notes on the ABET Program evaluator training PowerPoint presentation | free to download - id: 20e212-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Assessment Plan and Notes on the ABET Program evaluator training


Issues and questions a program evaluator may ask for each accreditation criterion ... 2 Olds et-al Op. cit. October 27, 2005. ECE Department Meeting. 16 ... – PowerPoint PPT presentation

Number of Views:113
Avg rating:3.0/5.0
Slides: 22
Provided by: INEL8
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Assessment Plan and Notes on the ABET Program evaluator training

Assessment Plan and Notes on the ABET Program
evaluator training
  • J. Fernando Vega-Riveros, Ph.D.Dept. of
    Electrical and Computer EngineeringUniversity of
    Puerto Rico

  • Objectives of presentation
  • Assessment and Evaluation
  • ABET workshop learning objectives
  • Key points of the workshop
  • Issues and questions a program evaluator may ask
    for each accreditation criterion
  • Discussion

Objective of presentation
  • Present issues and comparative analysis of
    assessment approaches and ABET criteria
  • Promote discussion of issues as they pertain to
    Department INEL/ICOM
  • Make a decision about department assessment
    instruments and methods

  • A unified assessment approach for our two
    programs would would reduce administrative and
    academic burden on faculty, students and staff.
  • Same instruments and tools used for the whole

Accreditation timeline
INEL and ICOM accredited till 2009
Request re-accreditation to ABET
Submit Self-Study Report
ABET Visit
Design metrics, thresholds and sampling method
Define assessment instruments
Conduct assessment and close the loop
Assessment Plan
December 2005
Summer 2006
Summer 2007
July 2008
January 2008
December 2009
Assessment and evaluation1
  • Assessment act of collecting data or evidence
    that can be used to answer classroom, curricular.
    Or research questions.
  • Assessment method Procedures used to support the
    collection of data for assessment purposes.
  • Evaluation refers to the interpretations that
    are made of the evidence collected about a given

1 Olds, B.M. Moskal, B.M. and Miller, R.
Assessment in engineering education evolution,
approaches and future collaborations. ASEE
Journal of Engineering Education. Vol. 94 No. 1.
Jan. 2005. pp 13-25
Types (components?) of assessment
Assessment of outcome achievement in a given
academic term (snapshots)
Formative Assessment
Transversal Assessment
Summative Assessment
Longitudinal Design
Tracks outcome achievement over time
Goal of assessment
  • Measure outcome achievement (a-k) and how these
    outcomes are developing along and across the
  • Validate interventions, e.g. curricular
    revisions, improved or novel pedagogies, new
    courses, course restructuring, etc.
  • Collect data for quality assurance and continuous
    quality improvement
  • Realistic and minimize burden

Program Educational Objectives
  • Program will allow the graduates of the program
  • Obtain a broad educational experience necessary
    to understand the impact of electrical/computer
    engineering problems and solutions within a
    global and societal context
  • Possess a combination of knowledge and
    analytical, computational, and experimental
    skills necessary to solve practical
    electrical/computer engineering problems
  • Have adequate communication skills both as an
    individual and as part of a team
  • Value the importance of lifelong learning
  • Be aware of contemporary issues and thus be able
    to make decisions taking into consideration
    professional and societal needs and ethical

Program educational outcomes
  • Graduates of the program will have the
  • Ability to apply knowledge of mathematics,
    science and engineering necessary to carry out
    analysis and design appropriate to
    electrical/computer engineering problems
  • Ability to design and conduct experiments as well
    as analyze and interpret data
  • Ability to design computer systems to meet
    desired needs
  • Ability to function in multidisciplinary teams
  • Ability to identify, formulate, and solve
    engineering problems

Program educational outcomes (cont)
  • Understanding of professional and ethical
  • Ability to communicate effectively
  • Broad education necessary to understand the
    impact of engineering solutions in a
    global/societal context
  • Recognition of the need for and ability to engage
    in lifelong learning
  • Knowledge of contemporary issues
  • Ability to use the techniques, skills, and modern
    engineering tools necessary for engineering

Current Assessment Approach (ICOM)
  • Instruments
  • GPA (Transcript Analysis)
  • Student, alumni and employer surveys
  • Metrics and satisfaction thresholds
  • Items related to grades are considered
    satisfactory it they are above 3.0 (B grade)
  • Items related to satisfaction rating are
    satisfactory if
  • Above 3.0 (weight of a satisfied response in
    surveys) or
  • Above average satisfaction of all items in same

Issues of current approach
  • Minimum burden on faculty
  • High burden on accreditation team (coordinator)
  • Uses surveys, practice that ABET discourages for
    outcome assessment. Surveys can be used as
    supportive information but depends on perception
    not on actual outcome achievement3
  • Some thresholds move depending on population and
    may not be set according to more explicit program

3 ABET 2005 Program Evaluator Training Materials
(EAC Training, Approved December 2004).
Issues of current approach (Continued)
  • GPA does not discriminate among outcomes of a
    given course it gives some indication of
    achievement of the set of outcomes set for a
    given course or courses.
  • ABET has some concerns about use of course

4 Guidelines to Institutions, Team Chairs and
Program Evaluators on Interpreting and Meeting
the Standards Set Forth in Criterion 3 of the
Engineering Accreditation Criteria.
http// Documents-UPDATE/Progra
m Docs/EAC Guidelines or Criterion3.pdf.
Assessment methods2
  • Descriptive designs
  • Surveys
  • Interviews and focus groups
  • Observation
  • Conversational analysis
  • Ethnographic studies
  • Experimental designs
  • Randomized controlled trials
  • Matching
  • Baseline data
  • Longitudinal design
  • Post-test-only design

2 Olds et-al Op. cit.
Some approaches to outcome assessment
  1. Student portfolios manageable with small student
    populations. How do we store and organize
    portfolios (and make them easily retrievable)?
    Who is responsible for portfolio repository?
  2. Periodic comprehensive (Standardized) tests
    administered in key core courses (could be pre or
    post). How can we motivate students to take them
    seriously? How do we assess outcomes d, g, i?
    Professional examination could be one of these
    tests. Who designs tests?
  3. Cornerstone and capstone courses. Do we have
    cornerstone courses? Can we identify those
  4. Assessment across course materials (tests,
    evidence of presentations and reports, when
    needed). Requires collaboration of faculty and
    material sampling and metrics not just doing a
    test or presentation suffices. Need assessment
    criteria from faculty but need to show some unity
    of criteria.

Approaches to outcome assessment
Alternative Pros Cons
Student Portfolios Unified assessment for all students and all academic termsComprehensive Manageable with small student populations.How do we store and organize portfolios?Who is responsible for portfolio repository?
Periodic comprehensive tests StandardizedUnified assessment for all students and all academic termsProfessional licensure could be one of the testsWhat if student achieves outcome but fails the course or vice versa? How do we assess criteria d, g and i?How and when would we give them?How can we motivate students to take them seriously?Who designs the tests?Standardized tests measure how well students learned information but may not demonstrate how well they solve problems
Approaches to outcome assessment (Continued)
Alternative Pros Cons
Cornerstone and capstone courses ComprehensiveAssessment instruments could be more standardized than in content courses (no need to come up with new questions for tests or exams) What are the cornerstone courses? Do we have any?We need to identify candidate courses and/or create them
Course material-based assessment We may already have many necessary elements and evidence materialsProbably the smallest additional burden on facultyAllows student progress monitoringAllows multiple measures of each outcome Need to do a mapping between materials and outcomesHow to sample materials (statistically significant)Not comprehensive
Approaches to outcome assessment (Continued)
Alternative Pros Cons
Professional Licensure test (Reválida) Almost comprehensiveCan be used for comparative analysis with rest of the nationShows outcomes of the program Does not allow assessment of outcomes d, g and i.Only one point measurement and does not let us see how students develop skills to achieve outcomesLong response time to any curricular changes
Assessment Proposal
  • Establish cycles of assessment for courses
  • Sampling of course material (no need to be
    exhaustive, just representative and statistically
  • Mapping of course material and educational
    outcomes (may need collaboration of course
  • Measurement of outcome achievement using course
    material and outcome mapping.
  • No course is assessed every semester (unless
    something exceptional is found). Round-Robin
  • Not all sections of a course need to be assessed.
    Assessment plan will establish cycle of sections
    assessment to minimize burden on faculty members
    and distribute assessment load as evenly as
  • Will use professional licensure examination as a
    component of summative assessment and
  • Results reviewed by area committees and final
    results send to steering committees

Assessment Proposal developing the assessment
  • Determine who we will assess
  • Educational objectives graduates between 1 and 3
  • Educational outcomes current undergraduate
    students in the Department.
  • Establish a Schedule for Assessment. Proposed
    re-accreditation timeline.
  • Determine who will interpret results Area
  • Interpret how results will inform
    teaching/learning and decision making Steering
  • Determine how and with whom we will share
    interpretations Department, College, Office of
    Continuous Improvement and Assessment, ABET,
    Middle States, CES through periodic reports and
    self studies.
  • Decide how our department and institution will
    follow-up on implemented changes periodic
    assessment and continuous program quality

Assessment Schedule
Publish Assessment Schedule
1st Assessment Sampling
2nd Asmt Smpl and start closing loop
3rd Asmt Smpl and continue closing loop
4th Asmt Smpl finish asmt. cycle
Committee Evaluations
Committee Evaluations
Committee Evaluations
Submit Self-Study Report
Request re-accreditation to ABET
ABET Visit
July 2008
December 2009
January 2008
January 2006
June 2006
December 2006
June 2007
December 2007
Questions, discussion and decision