CS-87: Evaluating Program Effectiveness Using Student Assessment Data: Basic Steps for Beginners

1
CS-87 Evaluating Program Effectiveness Using
Student Assessment Data Basic Steps for
Beginners

• Retta E. Poe, PhD, Western Kentucky University
• Dennis K. George, PhD, Western Kentucky
  University
• SACS COC Annual Meeting, Orlando, FL.
• Monday, 12/11/2006, 215 -315 PM
• Sun C, Convention Center

2
Goals for this workshop

• Clarify distinctions between student and program
  assessment
• Clarify distinctions between course and program
  outcome statements
• Show how data from student assessment products
  may be used for program assessment
• Provide practice in modifying typical student
  assessment methods to yield program assessment
  data

3
Why Do It?

• Kill two birds with one stone, or in other words,
  earn double mileage
• Use a process with which faculty are already
  familiar (student assessment)
• Save ourselves some work!

4
Student vs. Program Assessment

• Similarities
• Unit of measurement (the student or some product
  produced by the student)
• Tools/Instruments of assessment
• Differences
• Overall purpose of assessment
• Primary utilization of results

5
Student Assessment

• Purpose Students assessed with respect to
  mastery of material presented ensure capability
  or competency of each student
• Assessment tools written examinations, essays,
  papers, presentations, etc.
• Use of results determine grades or initiate
  some remedial action (tutoring, extra work
  assignment, etc.)
• Unit of remediation individual student

6
Program Assessment

• Purpose judge and improve overall instructional
  performance provide data for making program
  improvements
• Assessment tools written examinations, essays,
  papers, presentations, etc. - BUT data are
  aggregated
• Use of Results program improvements ensure
  that the program is accomplishing its objectives
• Unit of remediation program

7

• Suppose that the assessment data you and your
  colleagues have collected show that one or more
  of your program outcomes were not achieved that
  students didnt know or couldnt do what you had
  established as your program outcomes.
• What are some of the actions that you might
  consider taking to address the problem?

8
Program Remediation Strategies

• Curricular changes
• New courses
• Revised courses
• Revised course sequence
• Admission criteria changes
• Instructional methodology changes

9
Institutional Effectiveness

• SACS focuses on Program Assessment
• Institutional Effectiveness The institution
  identifies expected outcomes for its educational
  programs and its administrative and educational
  support services assesses whether it achieves
  these outcomes and provides evidence of
  improvement based on analysis of those results.
• (Comprehensive Standard 3.3.1)

10
Institutional Effectiveness

• Documentation of Institutional Effectiveness
  requires
• Evidence that outcomes for educational programs
  and administrative and educational support
  services are being achieved and that improvements
  are the results of assessment procedures
  (Handbook for Reaffirmation of Accreditation
  p. 62)

11
Student Assessment Feeds IntoProgram Assessment

• Aggregate Student Assessment data for Program
  Assessment
• Student or some product produced by the student
  is unit of assessment for both
• Broader view of data indicates effectiveness of
  overall program
• Supplements other program assessment data

12
Other Data for Program Assessment

• Standardized Exams (e.g., ETS Major Field, ACAT)
• Alumni Surveys
• Exit Surveys
• Employment Data
• Employer Surveys
• Graduate School Admissions Data

13
Assessment Terminology

• What to call that which is most important in a
  discipline?
• Core Competencies
• Educational Objectives
• Student Learning Outcomes
• Strategic Learning Goals
• Fundamental Knowledge in the Major Field
• Educational Standards

14
Assessment Terminology

• SACS Principles use term outcomes
• Term adopted at WKU for our campus-wide
  Assessment Plan Model

15
What is an Outcome?1

• What Educational Outcomes are NOT
• Individual course outcomes or course learning
  objectives.
• Individual student assessment within the context
  of a given course.
• What Educational Outcomes ARE
• Over-arching end results of the entire academic
  program.
• Facultys view of the most important attributes
  of the finished product.

16
Preparing Statements of Educational Outcomes1

• 1. Consistent with institutional mission and
  expanded statement of purpose
• 2. Reasonable given the ability of the students
• 3. Reflect key concepts of program
• 4. Focus on what students will know, think, or be
  able to do (not what faculty do).
• 5. Clearly stated
• 6. Accomplishment is ascertainable
• 7. Rotate when validated

17
Sources of Student Outcomes

• Course syllabi
• Professional associations
• Accrediting bodies
• Colleagues/programs at other campuses
• Informal or formal descriptions from employers
  about expectations of graduates
• Discussions with current and former students
• Faculty discussions about what are
  characteristics of the ideal graduate

18
Faculty Discussions

• Stimulate faculty creativity
• Outcomes owned by the program faculty
• Faculty has relative freedom to develop, i.e.,
  not meeting any externally determined student
  performance standards

19
Short List/Long List Concept

• Statements of
• Intended Educational Outcomes
• Statements of what is currently being
  assessed
• __________________
• __________________
• __________________
• __________________
• __________________

• Other Valid Statements of Intended Educational
  Outcomes
• __________________
• __________________
• __________________
• 40. _________________

20
Program Outcomes

• What a student should
• Know
• Think (i.e. attitudinal, how to think)
• Be able to do
• upon completion of an academic
• program1
• SACS requires that outcomes be identified by each
  academic program

21
Student Assessment Feeds IntoProgram Assessment

• KEY ISSUE!
• To the extent that a given student assessment
  measures or relates to a programmatic outcome, it
  can potentially be aggregated and used for
  program assessment.

22

• Imagine that you are creating the syllabus for a
  new course that you will teach. You have the
  content all laid out. The course objectives are
  determined.
• What are the tools you will use to assess
  individual student learning in this course?

23
Commonly-used Student Assessments That May Be
Used for Program Assessment

• Comprehensive exams
• Internships
• Essays
• Portfolios
• Oral presentations
• Juried performances
• Theses
• Etc.

24
Comprehensive Exam as Student Assessment

• Given in Capstone Course
• Consists of 50 multiple choice questions worth 2
  points each

25
Comprehensive Exam as Student Assessment

• Standard grade scale for individual students
• 90 100 A
• 80 89 B
• 70 79 C
• 60 69 D
• lt 60 F

26
Comprehensive Exam as Student Assessment

• Criterion for success for individual student
• Must achieve a 70 overall (C or better)
• Remediation for individual student
• Students scoring less than 70 will receive
  general tutoring

27
How Can We Use the Comprehensive Exam as Program
Assessment?

• Outcomes for Environmental Health and Safety
  (EHS) Program (see handout)
• Outcome 1 Application
• Outcome 2 PPE
• Outcome 3 Monitoring
• Outcome 4 Agents
• Outcome 5 LEV

28
How Can We Use the Comprehensive Exam as Program
Assessment?

• Link exam questions to outcomes
• Questions 1 10 assess Outcome 1
• Questions 11 20 assess Outcome 2
• Questions 21 30 assess Outcome 3
• Questions 31 40 assess Outcome 4
• Questions 41 50 assess Outcome 5

29
How Can We Use the Comprehensive Exam as Program
Assessment?

• Criteria for Success for Program
• The average grade on the Senior Exam will be no
  less than 70
• On no individual outcome area will the average be
  less than 14 points

30
Comprehensive Exam as Program Assessment
(Aggregate of Student Data)
31
Comprehensive Exam as Program Assessment
(Aggregate of Student Data)
32
Comprehensive Exam as Program Assessment

• Program Remediation
• Changes in course content to emphasize LEV in
  certain courses
• Changes in course sequence
• Creation of new course
• Re-visit importance of LEV as a Program Outcome
• Look carefully at the exam

33
Comprehensive Exam as Program Assessment

• Key Points
• Correlate specific questions with specific
  outcomes and aggregate data
• Comprehensive Exam contributes to grade in
  Capstone Course
• Course-embedded assessment
• Helps solve student motivation problem

34
Internship as Means of Program Assessment

• Co-op or internship experience required in many
  UG and grad programs
• Extremely useful for student and program
  assessment
• External evaluator
• Uses data from evaluations by on-site supervisors

35
Internship as Means of Program Assessment

• Sample program outcome for MA Clinical Psychology
  program
• Graduates will demonstrate masters-level skills
  in psychological assessment.
• Evaluated by on-site supervisors using rating
  sheet listing six aspects of assessment skills,
  rated from 1 (poor) to 5 (excellent)

36
Sample assessment dimensions to be evaluated in
internship

• Behavioral observation skills
• Interviewing skills
• Test selection skills
• Standardized test administration skills
• Test interpretation skills
• Report writing skills

37
Internship as Means of Program Assessment

• Key Issue
• Feedback from supervisors MUST relate to outcomes
  to be useful in program assessment

38
Internship as Means of Program Assessment

• Sample overall program criteria for success with
  aggregated data
• On no individual outcome will more than 20 of
  students receive a rating of 3 or below.
• The average rating for all students will be at
  least 22 out of 30.
• Recognize multi-rater consistency issues

39
Sample Internship Evaluation Data
40
Essay as Means of Student Assessment

• Course objective is to evaluate a students
  ability to provide a persuasive argument that
  clearly supports a given resolution to an ethical
  dilemma3
• Means of Assessment
• Written essay given in a course at the senior
  level or other appropriate point in the
  curriculum
• Criterion for Success Individual Student
• Determined by the course instructor

41
Essay as Means of Program Assessment

• Assume ethics is identified as a programmatic
  outcome.
• Outcome Ethics
• Graduates of the program will be able to provide
  a persuasive argument that clearly supports a
  given resolution to an ethical dilemma2

42
Essay as Means of Program Assessment

• Criteria for Success Program
• Average score (as determined by a panel of
  faculty) will be at least 70 out of a possible
  100 points on a rubric designed to score the
  written essay (see handout from Ron Miller,
  Colorado School of Mines)
• On no individual rubric category will the average
  score be less than 60 of the possible points

43
Essay as Means of Program Assessment

• Notes
• All essays need not be scored for program
  assessment a sample may be sufficient
• A group of faculty should be involved in scoring
  the essays
• The same rubric used to score each essay for
  individual student assessment may also be used in
  program assessment

44
Essay as Means of Program Assessment

• Notes (cont)
• The same student product (essay) is used twice
  once for individual student assessment and again
  for program assessment
• Course-embedded assessment overcomes student
  motivation problem

45
Essay as Means of Program Assessment

• Key
• The means of student assessment (e.g. essay) MUST
  relate to or measure in some way the PROGRAM
  outcome!

46
Portfolio as Means of Student Assessment

• Example WKU Teacher Work Sample, completed
  during student teaching
• Student Outcome
• Teacher candidate must demonstrate use of
  multiple assessment modes and approaches aligned
  with learning goals to assess student learning
  before, during, and after instruction.

47
Portfolio as Means of Student Assessment

• Means of assessment
• Teacher Work Sample, scored by supervising
  faculty member (see handout).
• Criteria for success
• The student must achieve an average of 2.5 on all
  five elements to pass. Any element scored less
  than 2.0 must be re-done.

48
Portfolio as Means of Program Assessment

• Program Outcome
• Teacher candidates must demonstrate ability to
  use multiple assessment modes and approaches
  aligned with learning goals to assess student
  learning before, during, and after instruction.

49
Portfolio as Means of Program Assessment

• Means of assessment
• A sample of TWS portfolios is selected. Each
  portfolio is scored by at least two raters, using
  the TWS rubric.
• Criteria for success
• The average rating must be 2.5 or higher on the
  five assessment planning elements.
• On no element will the average rating be less
  than 2.3.

50
Portfolio as Means of Program Assessment
51
Oral Presentation as Means of Student Assessment

• Sample course outcome Students are expected to
  be able to deliver an effective and polished
  persuasive or informative oral presentation.
• Means of assessment assignment to give 5-minute
  oral presentation.
• Presentation is evaluated using a rubric (see
  handout) that includes verbal and non-verbal
  dimensions of effective presentations.

52
Oral Presentation as Means of Program Assessment

• Program outcome Graduates will demonstrate
  ability to deliver a polished presentation
  appropriate to topic and audience.

53
Oral Presentation as Means of Program Assessment

• Means of assessment Videotapes of oral
  presentations are recorded. A sample is selected
  for evaluation on eight variables.
• Criteria for success The average overall rating
  will be at least 24, with no student receiving
  less than a 3 on any variable.

54
Juried Performance as Means of Student Assessment

• Programs such as those in music and art
  frequently require senior recital or exhibition
• Course objective (example) Students are
  expected to demonstrate proficiency in tone,
  intonation, accuracy, rhythm, technique, and
  interpretation or phrasing.

55
Juried Performance as Means of Program Assessment

• Outcome
• Program graduates will be able to deliver a
  polished and technically correct performance.
• Means of assessment
• rubric scored by jury (see handout)

56
Juried Performance as Means of Program Assessment

• Criteria for success
• Average ratings will be at least 3.0 across all
  elements on the rubric.
• On no element will the average be less than 2.5.

57
Juried Performance as Means of Program Assessment
58
Thesis as Means of Program Assessment

• Sample program outcome
• Program graduates will demonstrate skills in
  planning, conducting, and communicating the
  results of independent research.

59
Thesis as Means of Program Assessment

• Means of assessment masters thesis and oral
  thesis defense
• Thesis committee members complete rubric
  assessing the thesis and oral defense on quality
  of literature review, research design and
  methodology, description and analysis of results,
  quality of discussion of results, quality of
  writing, quality of oral presentation, etc.

60
Thesis as Means of Program Assessment

• Rubric provides for evaluating each dimension on
  a scale of 1 (poor) to 5 (excellent).
• Sample criteria for success
• The average overall rating will be at least 3.75.
• No student will score less than 3 on any single
  dimension.

61
Student and Program Assessment

• Key Issues for Program Assessment
• Means of assessment must relate to or measure the
  Program Outcome(s)
• Criteria for Success must be stated in terms of
  averages or aggregates of individual student
  scores
• Scoring must involve more than one faculty member

62
Using Student Assessment Data for Program
Assessment

• Advantages
• Can be accomplished with many types of
  course-embedded student products
• Is something each student is already doing as
  part of course that relates to a program outcome
• Can be transparent to students
• Criteria for success trigger program improvement

63
Using Student Assessment Data for Program
Assessment

• Guidelines
• Select a representative sample (some, most, all)
  to be scored in some fashion
• Use multiple faculty (even external evaluators,
  such as on-site supervisors)
• Can even block off names for privacy

64
References

• Nichols, J.O., Nichols, K.W. (2000). The
  Departmental Guide and Record Book for Student
  Outcomes Assessment and Institutional
  Effectiveness (3rd ed.). New York Agathon Press.
  
• Ron Miller and Barbara Olds, Colorado School of
  Mines, Golden Colorado

65
Contact Information

• Retta E. Poe retta.poe_at_wku.edu, [phone]
• Dennis K. George dennis.george_at_wku.edu, (270)
  745-8723