Assessment Techniques for Curricular Improvement - PowerPoint PPT Presentation

1 / 39
About This Presentation
Title:

Assessment Techniques for Curricular Improvement

Description:

Assessment Techniques for Curricular Improvement Roxanne Canosa, Rajendra K. Raj Department of Computer Science Rochester Institute of Technology – PowerPoint PPT presentation

Number of Views:256
Avg rating:3.0/5.0
Slides: 40
Provided by: ComputerScienceDep96
Category:

less

Transcript and Presenter's Notes

Title: Assessment Techniques for Curricular Improvement


1
Assessment Techniques for Curricular Improvement
  • Roxanne Canosa, Rajendra K. Raj
  • Department of Computer Science
  • Rochester Institute of Technology

2
Overview
  • What is Assessment?
  • Analytic vs. Holistic Approaches
  • Assessment Grading?
  • Terminology
  • Assessment vs. Accreditation
  • Outcomes vs. Objectives
  • Performance Criteria
  • Direct vs. Indirect
  • Evaluation and Continuous Improvement

3
What is Assessment?
  • Assessment is one or more processes that
    identify, collect, and prepare data to evaluate
    the achievement of program outcomes and
    educational objectives
  • 2006-2007 Criteria for Accrediting Computing
    Programs Appendix A (Proposed Changes)
  • From Section II.D.1 of the ABET Accreditation
    Policy and Procedure Manual

4
Analytic vs. Holistic Approaches
  • Analytic approach
  • All students/courses analyzed to diagnose areas
    in need of improvement
  • Holistic approach
  • Focus on overall performance of the program
  • Input from employers, alumni, advisory board
  • Develop efficient and effective processes
  • Lean, mean assessment machine
  • Dont commit random acts of assessment
  • Gloria Rogers

5
What is Your Assessment Goal?
  • Assessing all students or specific groups of
    students?
  • Assessing students, department, or program?
  • Assessing for short-term improvement or long-term
    effect?
  • Assessing for formative or summative purposes?

6
Grading vs. Assessing
  • Grading
  • Measures extent to which a student meets faculty
    requirements and expectations for a course
  • Can grades infer students achievement of an
    outcome?
  • Factors
  • Student knowledge
  • Work ethic
  • Faculty variance in course content, grading
    components, beliefs, bias,
  • Assessing
  • Measures extent to which a student achieves each
    course (program) outcome
  • Can we leverage grading components for
    assessment?
  • Use rubrics, which are pre-announced performance
    criteria

7
Assessment vs. Accreditation
  • Institutional accreditation through Middle
    States, SACS, etc. are increasingly requiring
    direct assessment of program objectives and
    outcomes
  • Jargon may be different, but the essential ideas
    are the same

8
Terminology (Jargon)
From ABET perspective
9
Terminology Lessons
  • Use terminology for your situation
  • Sometimes dictated by institutional accreditation
    (SACS, Middle States)
  • Sometimes dictated by program accreditation
    (ABET)
  • Keep a glossary of terms handy for any external
    evaluators
  • Stick to your terminology
  • Terms are not fungible without causing too much
    grief

10
Proposed Changes toABET Criteria for Computing
  • Old criteria
  • Intents and Standards
  • New criteria (2008-2009 cycle)
  • General
  • Program Specific

11
New ABET Criteria
  • 8 General Criteria
  • Students
  • Program Educational Objectives
  • Program Outcomes (a) through (i)
  • Assessment and Evaluation
  • Curriculum
  • Faculty
  • Facilities
  • Support
  • CS Program Specific Criteria
  • Outcomes and Assessment (a) and (b)
  • Faculty Qualifications
  • Curriculum (a), (b), and (c)
  • IT/IS Program Specific Criteria

12
Program Audit Concern
  • Concern
  • A criterion is currently satisfied however,
    potential exists for this situation to change in
    the near future such that the criterion may not
    be satisfied. Positive action is required to
    ensure full compliance with the Criteria.

13
Program Audit Weakness
  • Weakness
  • A criterion is currently satisfied but lacks
    strength of compliance that assures that the
    quality of the program will not be compromised
    prior to the next general review. Remedial action
    is required to strengthen compliance with the
    Criteria.

14
Program Audit Deficiency
  • Deficiency
  • A criterion is not satisfied. Therefore, the
    program is not in compliance with the Criteria
    and immediate action is required.

15
Program Objectives
  • Program educational objectives are broad
    statements that describe the career and
    professional accomplishments that the program is
    preparing graduates to achieve.
  • Long-term goals
  • Should be distinct to your program
  • Should be publicly available
  • Must be measurable!

16
Program Outcomes
  • Program outcomes are narrower statements that
    describe what students are expected to know and
    be able to do by the time of graduation. These
    relate to the skills, knowledge, and behaviors
    that students acquire in their matriculation
    through the program.
  • Should be publicly available
  • Must be measurable!

17
Objectives vs. Outcomes
  • Example objective
  • Graduates will exhibit effective communication
    skills
  • Example outcomes
  • By the time of graduation, students will
  • demonstrate effective written communication
    skills
  • demonstrate effective oral communication skills

- Gloria Rogers
18
Performance Criteria
  • Define and describe progression toward meeting
    important components of work being completed,
    critiqued, or assessed
  • Student provides adequate detail to support
    his/her solution/argument
  • Student uses language and appropriate word choice
    for the audience
  • Student work demonstrates an organizational
    pattern that is logical and conveys completeness
  • Student uses the rules of standard English
  • Provide solid evidence of progression

19
What is Solid Evidence?
  • Direct Evidence
  • Easier to measure
  • Familiar to most faculty - exam or project
    grades, presentation skills, etc.
  • Indirect Evidence
  • Difficult to measure
  • Attitudes or perceptions
  • For example, a desired outcome of a course may
    include improving students appreciation of team
    work

20
Direct vs. Indirect Assessment
  • The assessment process should include both
    indirect and direct measurement techniques
  • A variety of sources should be used
  • Employers, students, alumni, etc.
  • Converging evidence from multiple sources can
    reduce the effect of any inherent bias in the data

21
Direct Assessment
  • Direct examination or observation of student
    knowledge or skills using stated, measurable
    outcomes
  • Faculty typically assess student learning
    throughout a course using exams/quizzes,
    demonstrations, and reports
  • Sample what students know or can do
  • Provide evidence of student learning

22
Direct Assessment of PEOs
  • Employment statistics
  • Promotions and career advancement of graduates
  • Job titles, advanced degrees earned, additional
    course work taken after graduation, etc.
  • PEOs must be assessed separately from POs

23
Direct Assessment of POs
  • Common final exams
  • Locally developed exit exams
  • Standardized regional or national exit exams
  • External examiner
  • Co-op reports from employers
  • Portfolios of student work

24
Indirect Assessment
  • Indirect assessment of student learning
    ascertains the perceived extent or value of
    learning experiences
  • Assess opinions or thoughts about student
    knowledge or skills
  • Provides information about student perception of
    their learning and how this learning is valued by
    different constituencies

25
Indirect Assessment Measures
  • Exit and other kinds of interviews
  • Archival data
  • Focus groups
  • Written surveys and questionnaires
  • Industrial advisory boards
  • Employers
  • Job fair recruiters
  • Faculty at other schools

26
Survey of Assessment Methods
27
Direct and Indirect
  • Duality of some instruments, e.g., an exit
    interview
  • Indirect
  • Survey of opinions about the perceived value of
    the program components
  • Direct
  • If person asking the questions uses it as a way
    of assessing students skills (e.g., oral
    communication), then the survey is being used as
    a direct measure of the achievement of that
    outcome

28
Evaluation
  • Evaluation is one or more processes for
    interpreting the data and evidence accumulated
    through assessment practices. Evaluation
    determines the extent to which program outcomes
    or program educational objectives are being
    achieved, and results in decisions and actions to
    improve the program.

29
Continuous Improvement
  • Accreditation boards are moving towards
    outcomes-based assessment of CS, IS, and IT
    programs
  • Programs must have an established outcomes-based
    assessment plan in place (or at least be making
    progress in that direction)
  • Process must be documented
  • Process must show continuous improvement (both
    quantitatively and qualitatively)

30
Faculty Responsibility
  • All faculty must have a commitment to and be
    directly involved in the evaluation of program
    educational objectives and program outcomes, as
    well as the process for continuous improvement of
    the program

31
Need for Faculty And Staff Buy-In
  • What makes most academics tick?
  • Rewards
  • Money?
  • Fun?
  • Appreciation?
  • Recognition?
  • How to encourage involvement?
  • We all resent any extra work!

32
Where to Begin?
  • Define your Mission Statement
  • Define your Program Educational Objectives (PEOs)
  • Define your Program Outcomes (POs)
  • Define Course Outcomes (COs)
  • Include specific course outcomes on each course
    syllabus
  • Make publicly available

33
Then What?
  • Show how course outcomes map to program outcomes
  • Show how program outcomes map to program
    educational objectives
  • Choose measurement tools, both direct and
    indirect
  • Collect data

34
Finally
  • Present data to faculty in an easily digestible
    form
  • Charts, graphs, tables, etc.
  • Faculty evaluates the data
  • Are students actually learning the material that
    the faculty believe (and claim) they are
    learning?
  • Faculty make recommendations for improvement as
    necessary

35
The Big Picture
Performance Criteria
Mission Statement
Stakeholders(students, alumniemployers
faculty, )
Course Outcomes
Program Objectives
Program Outcomes
Assess Collect and Analyze Evidence
Assess Collect and Analyze Evidence
Revise
Evaluate Interpret Evidence Take Action
Educational Practices/Strategies
36
The Big Picture
  • Show relationship between mission statement,
    objectives, and outcomes
  • Assess and evaluate objectives and outcomes
    independently
  • Map program outcomes to program objectives
  • Map course outcomes to program outcomes
  • Identify weaknesses and implement focused
    improvements in targeted areas

37
Issues
  • All assessment methods have their limitations and
    contain some bias
  • Meaningful analysis requires both direct and
    indirect measures from a variety of sources
  • Students, alumni, faculty, employers, etc.
  • Multiple assessment methods provides converging
    evidence of student learning

38
Assessment Lessons
  • Cannot do everything at once
  • Try an approach for first round learn and refine
  • Having data isnt all there is to it!
  • Easy to generate lots of bad data
  • One size fits all NOT!
  • Programs, courses, instructors all differ
  • Be ready to compromise
  • Perfection is neither possible nor desirable
  • Faculty evaluation and promotion
  • Do not tie to data generated from assessment

39
Resources
  • http//www.cs.rit.edu/rlc/Assessment/
  • http//www.abet.org/assessment.shtml
Write a Comment
User Comments (0)
About PowerShow.com