Assessment Plan and Notes on the ABET Program evaluator training - PowerPoint PPT Presentation

Loading...

PPT – Assessment Plan and Notes on the ABET Program evaluator training PowerPoint presentation | free to download - id: 20e212-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Assessment Plan and Notes on the ABET Program evaluator training

Description:

Issues and questions a program evaluator may ask for each accreditation criterion ... 2 Olds et-al Op. cit. October 27, 2005. ECE Department Meeting. 16 ... – PowerPoint PPT presentation

Number of Views:113
Avg rating:3.0/5.0
Slides: 22
Provided by: INEL8
Learn more at: http://ece.uprm.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Assessment Plan and Notes on the ABET Program evaluator training


1
Assessment Plan and Notes on the ABET Program
evaluator training
  • J. Fernando Vega-Riveros, Ph.D.Dept. of
    Electrical and Computer EngineeringUniversity of
    Puerto Rico

2
Outline
  • Objectives of presentation
  • Assessment and Evaluation
  • ABET workshop learning objectives
  • Key points of the workshop
  • Issues and questions a program evaluator may ask
    for each accreditation criterion
  • Discussion

3
Objective of presentation
  • Present issues and comparative analysis of
    assessment approaches and ABET criteria
  • Promote discussion of issues as they pertain to
    Department INEL/ICOM
  • Make a decision about department assessment
    instruments and methods

4
Rationale
  • A unified assessment approach for our two
    programs would would reduce administrative and
    academic burden on faculty, students and staff.
  • Same instruments and tools used for the whole
    department

5
Accreditation timeline
INEL and ICOM accredited till 2009
Request re-accreditation to ABET
Submit Self-Study Report
ABET Visit
Design metrics, thresholds and sampling method
Define assessment instruments
Conduct assessment and close the loop
Assessment Plan
December 2005
Summer 2006
Summer 2007
Today
July 2008
January 2008
December 2009
6
Assessment and evaluation1
  • Assessment act of collecting data or evidence
    that can be used to answer classroom, curricular.
    Or research questions.
  • Assessment method Procedures used to support the
    collection of data for assessment purposes.
  • Evaluation refers to the interpretations that
    are made of the evidence collected about a given
    question.

1 Olds, B.M. Moskal, B.M. and Miller, R.
Assessment in engineering education evolution,
approaches and future collaborations. ASEE
Journal of Engineering Education. Vol. 94 No. 1.
Jan. 2005. pp 13-25
7
Types (components?) of assessment
Assessment of outcome achievement in a given
academic term (snapshots)
Formative Assessment
Transversal Assessment
Summative Assessment
Longitudinal Design
Tracks outcome achievement over time
8
Goal of assessment
  • Measure outcome achievement (a-k) and how these
    outcomes are developing along and across the
    curriculum
  • Validate interventions, e.g. curricular
    revisions, improved or novel pedagogies, new
    courses, course restructuring, etc.
  • Collect data for quality assurance and continuous
    quality improvement
  • Realistic and minimize burden

9
Program Educational Objectives
  • Program will allow the graduates of the program
    to
  • Obtain a broad educational experience necessary
    to understand the impact of electrical/computer
    engineering problems and solutions within a
    global and societal context
  • Possess a combination of knowledge and
    analytical, computational, and experimental
    skills necessary to solve practical
    electrical/computer engineering problems
  • Have adequate communication skills both as an
    individual and as part of a team
  • Value the importance of lifelong learning
  • Be aware of contemporary issues and thus be able
    to make decisions taking into consideration
    professional and societal needs and ethical
    implications

10
Program educational outcomes
  • Graduates of the program will have the
  • Ability to apply knowledge of mathematics,
    science and engineering necessary to carry out
    analysis and design appropriate to
    electrical/computer engineering problems
  • Ability to design and conduct experiments as well
    as analyze and interpret data
  • Ability to design computer systems to meet
    desired needs
  • Ability to function in multidisciplinary teams
  • Ability to identify, formulate, and solve
    engineering problems

11
Program educational outcomes (cont)
  • Understanding of professional and ethical
    responsibility
  • Ability to communicate effectively
  • Broad education necessary to understand the
    impact of engineering solutions in a
    global/societal context
  • Recognition of the need for and ability to engage
    in lifelong learning
  • Knowledge of contemporary issues
  • Ability to use the techniques, skills, and modern
    engineering tools necessary for engineering
    practice.

12
Current Assessment Approach (ICOM)
  • Instruments
  • GPA (Transcript Analysis)
  • Student, alumni and employer surveys
  • Metrics and satisfaction thresholds
  • Items related to grades are considered
    satisfactory it they are above 3.0 (B grade)
  • Items related to satisfaction rating are
    satisfactory if
  • Above 3.0 (weight of a satisfied response in
    surveys) or
  • Above average satisfaction of all items in same
    category

13
Issues of current approach
  • Minimum burden on faculty
  • High burden on accreditation team (coordinator)
  • Uses surveys, practice that ABET discourages for
    outcome assessment. Surveys can be used as
    supportive information but depends on perception
    not on actual outcome achievement3
  • Some thresholds move depending on population and
    may not be set according to more explicit program
    criteria.

3 ABET 2005 Program Evaluator Training Materials
(EAC Training, Approved December 2004).
14
Issues of current approach (Continued)
  • GPA does not discriminate among outcomes of a
    given course it gives some indication of
    achievement of the set of outcomes set for a
    given course or courses.
  • ABET has some concerns about use of course
    grades4.

4 Guidelines to Institutions, Team Chairs and
Program Evaluators on Interpreting and Meeting
the Standards Set Forth in Criterion 3 of the
Engineering Accreditation Criteria.
http//www.abet.org/Linked Documents-UPDATE/Progra
m Docs/EAC Guidelines or Criterion3.pdf.
15
Assessment methods2
  • Descriptive designs
  • Surveys
  • Interviews and focus groups
  • Observation
  • Conversational analysis
  • Ethnographic studies
  • Experimental designs
  • Randomized controlled trials
  • Matching
  • Baseline data
  • Longitudinal design
  • Post-test-only design

2 Olds et-al Op. cit.
16
Some approaches to outcome assessment
  1. Student portfolios manageable with small student
    populations. How do we store and organize
    portfolios (and make them easily retrievable)?
    Who is responsible for portfolio repository?
  2. Periodic comprehensive (Standardized) tests
    administered in key core courses (could be pre or
    post). How can we motivate students to take them
    seriously? How do we assess outcomes d, g, i?
    Professional examination could be one of these
    tests. Who designs tests?
  3. Cornerstone and capstone courses. Do we have
    cornerstone courses? Can we identify those
    courses?
  4. Assessment across course materials (tests,
    evidence of presentations and reports, when
    needed). Requires collaboration of faculty and
    material sampling and metrics not just doing a
    test or presentation suffices. Need assessment
    criteria from faculty but need to show some unity
    of criteria.

17
Approaches to outcome assessment
Alternative Pros Cons
Student Portfolios Unified assessment for all students and all academic termsComprehensive Manageable with small student populations.How do we store and organize portfolios?Who is responsible for portfolio repository?
Periodic comprehensive tests StandardizedUnified assessment for all students and all academic termsProfessional licensure could be one of the testsWhat if student achieves outcome but fails the course or vice versa? How do we assess criteria d, g and i?How and when would we give them?How can we motivate students to take them seriously?Who designs the tests?Standardized tests measure how well students learned information but may not demonstrate how well they solve problems
18
Approaches to outcome assessment (Continued)
Alternative Pros Cons
Cornerstone and capstone courses ComprehensiveAssessment instruments could be more standardized than in content courses (no need to come up with new questions for tests or exams) What are the cornerstone courses? Do we have any?We need to identify candidate courses and/or create them
Course material-based assessment We may already have many necessary elements and evidence materialsProbably the smallest additional burden on facultyAllows student progress monitoringAllows multiple measures of each outcome Need to do a mapping between materials and outcomesHow to sample materials (statistically significant)Not comprehensive
19
Approaches to outcome assessment (Continued)
Alternative Pros Cons
Professional Licensure test (Reválida) Almost comprehensiveCan be used for comparative analysis with rest of the nationShows outcomes of the program Does not allow assessment of outcomes d, g and i.Only one point measurement and does not let us see how students develop skills to achieve outcomesLong response time to any curricular changes
20
Assessment Proposal
  • Establish cycles of assessment for courses
  • Sampling of course material (no need to be
    exhaustive, just representative and statistically
    significant)
  • Mapping of course material and educational
    outcomes (may need collaboration of course
    instructor)
  • Measurement of outcome achievement using course
    material and outcome mapping.
  • No course is assessed every semester (unless
    something exceptional is found). Round-Robin
    strategy.
  • Not all sections of a course need to be assessed.
    Assessment plan will establish cycle of sections
    assessment to minimize burden on faculty members
    and distribute assessment load as evenly as
    possible.
  • Will use professional licensure examination as a
    component of summative assessment and
    benchmarking
  • Results reviewed by area committees and final
    results send to steering committees

21
Assessment Proposal developing the assessment
plan
  • Determine who we will assess
  • Educational objectives graduates between 1 and 3
    years.
  • Educational outcomes current undergraduate
    students in the Department.
  • Establish a Schedule for Assessment. Proposed
    re-accreditation timeline.
  • Determine who will interpret results Area
    committees.
  • Interpret how results will inform
    teaching/learning and decision making Steering
    committees.
  • Determine how and with whom we will share
    interpretations Department, College, Office of
    Continuous Improvement and Assessment, ABET,
    Middle States, CES through periodic reports and
    self studies.
  • Decide how our department and institution will
    follow-up on implemented changes periodic
    assessment and continuous program quality
    improvement.

22
Assessment Schedule
Publish Assessment Schedule
1st Assessment Sampling
2nd Asmt Smpl and start closing loop
3rd Asmt Smpl and continue closing loop
4th Asmt Smpl finish asmt. cycle
Committee Evaluations
Committee Evaluations
Committee Evaluations
Submit Self-Study Report
Request re-accreditation to ABET
ABET Visit
July 2008
December 2009
January 2008
January 2006
June 2006
December 2006
June 2007
December 2007
23
Questions, discussion and decision
About PowerShow.com