Designing Grading Rubrics to Assess Student Learning

1
Designing Grading Rubrics to Assess Student
Learning

• The DIY (Do It Yourself) Workshop
• Mary T. Blackinton PT, MS, EdD
• Director Transition DPT Program
• Chair, PT Outcomes Committee
• Nova Southeastern University

2
Agenda

• Introduction to Grading
• Myths Principles of Effective Grading
• Choosing an Assignment for a Rubric
• Analyzing Grading Rubrics
• Small group discussion
• Developing Rubric
• Definitions
• Primary Trait Analysis
• Creating Using Rubrics
• Design
• Pitfalls

3
Objectives

• Upon completion of the workshop, faculty will be
  able to
• Describe the role of grading rubrics in
• Clarifying teacher expectations
• Evaluating student learning
• Providing detailed feedback
• Improving instruction
• Performing outcome assessment
• Evaluate the strengths and weaknesses of grading
  rubrics as a student assessment tool in health
  professions education
• Integrate the principles of primary trait
  analysis (PTA) into designing a rubric for an
  assignment or performance skill
• Create a grading rubric for an assignment,
  project, or exam

4
Grading Challenge 214 Students Perceptions
5
Why do we Grade?

• Grading1
• The process used by faculty to assess student
  learning via assignments and exams, including
• Relating test items or assignments to learning
  objectives
• Establishing criteria/standards (SAFETY)
• Helping students acquire needed knowledge/skills
• Facilitating student motivation
• Giving feedback about performance
• Communicating about what has been learned
• Using results to influence teaching and
  curriculum

• 1 BE Walvoord, VJ Anderson (1998). Effective
  grading A tool for learning and assessment.
  Jossey-Bass San Francisco

6
Student Assessment in Health Professions

• Faculty charged to assess competency in patient
  care
• Responsibility to protect public
• Other stakeholders accrediting bodies,
  profession, students, parents
• Professionals integrate 3 domains of learning
• Cognitive-thinking/reasoning
• Affective-feeling, valuing, ethics
• Psychomotor-doing, manipulating, performing,
  administering
• Assessment strategies reflect demands
  professional ed
• Must include more than multiple choice assessment
  to determine learning
• Often have multiple faculty grading student
  performance

7
Myths About Grading

• Grading
• Is a necessary EVIL
• Must be objective
• Should be easy
• Should not take detract from teaching
• Use knowledge of experts to evaluate novices
• Is assessment only, doesnt impact learning
• With clear expectations means you are
  spoon-feeding
• Cannot be used in program assessment

8
Rubrics as Assessment of Program Outcomes
9
Principles of Effective Grading1

• Appreciate the complexity of grading
• Substitute judgment for objectivity
• Distribute time effectively
• Be open to change
• Listen observe
• Communicate collaborate with students
• Integrate grading with other key processes
• Seize the teachable moment (feedback)
• Make student learning the primary goal
• Be a teacher FIRST, gatekeeper LAST
• Encourage learning-centered motivation
• Emphasize student involvement

• 1 Walvoord Anderson (1998)

10
Types of Grading in Health Professions Student
Assessment Methods

• Multiple choice examination
• Short answer questions on tests
• Essay questions
• Annotated bibliographies
• Literature reviews
• Case Studies
• Oral Examinations
• Practical/Performance Examinations
• Clinical Assessments
• Journals
• Portfolios
• Lab Conclusions

Student-Constructed Responses
2LC Jacobs, CI Chase (1992). Developing and
using tests effectively. Jossey-Bass San
Francisco
11
Methods of Grading Student-Constructed Responses
12
What are Assessment Rubrics?Small Group
Discussion

• In small group, look at the sample rubric
  packet. Then, answer the following
• Discussion Questions
• Which rubrics seem the MOST clear to you? Why?
• Which rubrics are not clear? Why?
• What was common among the rubrics you viewed as
  clear?
• Based on samples, how do you define assessment
  rubrics?

13
What is a Grading Rubric?

• Method of articulating expectations for an
  assignment by listing the criteria, or what
  counts, describing levels of quality from
  excellent to poor3
• Type of assessment that specifies gradations of
  quality from excellent to poor4
• A criterion-referenced method of grading using
  highly specific grading criteria that are linked
  to objectives

3 HG Andrade, Y Du (2005). Students perspectives
on rubric-referenced assessment. Practical
Assessment, Research Evaluation, Vol 10 (3). 4
HG Andrade ((2005). Teaching with rubrics The
good, the bad, and the ugly. Coll Teaching, 53
(1)
14
What is a Grading Rubric?
Gradation excellent-poor
Categories important to the teacher/class
Dimensions also called criteria
Weighted Points
15
(No Transcript)
16
How do Students Perceive Rubrics?

• Andrad Yu, 2005
• Investigated how students use grading rubrics
• Focus group / qualitative design
• Students reported that they used rubrics to
• Help them determine faculty expectations
• To plan an approach to assignment
• Check/revise work before handing in
• Help reflect on their learning-see
  strengths/weaknesses clearly
• Perceived results of rubric use
• Better, fairer grades
• Improvements in quality across classes
• Less anxiety

17
Using Primary Trait Analysis1,4 To Develop a
Grading Rubric

• Rationale
• Analyze traits / characteristics of student
  learning and then clearly articulate them,
  leading to
• Assignment specific
• Explicit criteria
• Adds objectivity consistency to holistic
  scoring
• Lets students know in ADVANCE how will be graded

1 Walvoord Anderson (1998) 4R Lloyd-Jones
(1977). Primary Trait Scoring in C. Cooper L.
Odell (Eds), Evaluating Writing Describing,
measuring, judging. Urbana IL National Council
of Teachers
18
Constructing a Primary Trait Analysis Scale1

• Choose assignment/test that matches course
  objective
• Identify all traits that will count for scoring
  in assessment
• Body Language vs Thoroughness vs Accuracy
• Build scale for scoring performance, gradations
  of skill
• Scale usually ranges from 2-5 points
• Include what should be demonstrated and what
  should be avoided
• Build a range that discriminates A from A- from
  B
• Run scale by colleague, graduate, teaching
  assistant
• Weight items content spelling accuracy
  efficiency, etc
• Evaluate performance against criteria
• Try scale with sample revise as needed

19
Example Designing a Rubric

• Select assignment appropriate for PTA
• Taking a Patient History (clinical performance)
• Identify ALL traits you look for in patient
  history
• Body language eye contact (X-1)
• Thoroughness (X-2)
• Data gathering (X 1.5)
• Weight them each trait

20
Building a Scale Body Language

• Level 3-Excellent
• Consistently had good eye contact with patient
• Communicated at patients eye-level
• Maintained an open posture, leaning toward
  patient
• Facial expression interested, non-judgmental
• Confident and relaxed, not arrogant
• Level 2-Good
• At least 4 of the above behaviors noted
• OR, all 5 qualities demonstrated but not
  consistently
• Level 1-Fair
• Inconsistent in several (2-3) areas, for example,
  confident through some portions of history but
  not others or, eye contact missing at end
• Level 0-Poor
• Arrogance, avoidance, lack of confidence
  detracted from relationship

Gives students specific behaviors to demonstrate!
Identifies behaviors to avoid!
21
Building a Scale Thoroughness

• 4 Extremely Thorough
• History includes all the following core areas 1.
  all body systems 2. history current
  illness/problem 3. family history 4.
  medications 5. educational background / learning
  style 6. lifestyle (diet, exercise, habits) 7.
  living/ work/social environment 8.
  assistive/adaptive devices (glasses, cane, etc)
  9. prior level of function
• 3 Thorough
• All items (1-9) included but may have missed some
  sub-categories (ie missed 2 body systems) or,
  only missed 1-2 of the 9 core areas above
• 2 Somewhat Thorough
• Missing 3-4 of the 9 core areas, or, missing only
  1-2 areas but was also superficial in review of
  body systems
• 1 Incomplete
• Less than half of the 9 core areas were covered
  incomplete data

22
Building a Scale Data Gathering

• Excellent-4
• 1. Uses open-ended, broad questions 2.
  encourages patient to tell story chronologically
  3. allows patient to talk without interrupting
  4. encourages by using phrases such as tell me
  more or what else? 5. summarizes what was
  heard 6. Approach was organized
• Good-3
• Demonstrates at least 4 of 6 skills noted above
  throughout the history, or, used all 5 skills but
  inconsistently.
• Fair-2
• Demonstrates 3 of the core probing skills, or,
  3-4 skills used however did so inconsistently
  multiple missed opportunities to probe
  disorganized at times
• Poor-1
• Did not consistently demonstrate probing skills,
  questions were narrow and rarely open-ended,
  interrupted or cut patient off, rarely encouraged
  more information with phrases such as tell me
  more.

23
DIY Build Your Own PTA
Assignment or Test_______________________________
_____________________ Related Class Objectives
________________________________________________
24
Step 4 Evaluate Performance Using Criteria

• Test the Scale
• Use it with an past paper, videotape, case
  report, etc.
• What changes are needed in the rubric?
• Were the criteria easy to follow?
• Did the overall grade reflect performance?
• Are any changes needed in description or in the
  point allotment?
• Did an A performance score better than a B?
• Is it easy? Hard? Too cumbersome?

25
Applying the Rubric

• Distribute ( or post) rubric to students in
  advance
• Have conversations about expectations
• Ask students to attach rubric to assignment
• Helps students pay attention!
• Use rubric to grade
• Match written comments to phrases in rubric
• Revise after use
• Make changes soon after grading for next time
• Answer Curriculum Questions
• Did the students learn? To what degree were
  objectives met?
• If not, was it the teaching? The assignment?
  Background skills?
• What, if any, changes should be made in the class
  or curriculum? Rubric?

26
Advantages of Grading Rubrics1

• Save time in grading process
• Makes process of grading reliable/fair
• Clarifies expectations for students
• Reinforces key concepts - help faculty relate to
  objectives
• Students are participants as expectations are
  known
• Student peer review
• Works well if team-teaching
• Share across courses or over curriculum
• Basis for departmental/program assessment
• Which of these advantages pertain to your
  teaching in the health professions?

• 1 Walvoord Anderson, 1998

27
CAUTION Common Rubric Pitfalls

• Rubric does not correspond with class or program
  outcomes
• Example Entire rubric focused on writing quality
  not content
• Scale does not have enough gradations or levels
• Not distinguishing the As from the Bs
• All traits are given equal weight regardless of
  complexity
• Grammar Content
• Too broad, not enough content described
• Words like breadth and depth used in lieu of
  specifics
• Students still not sure, other grading faculty
  still not clear
• Too long/too complicated
• Faculty students get lost in the rubric

28
Classroom Assessment Technique

• Write down ONE thing you will commit to
  incorporate regarding the use of grading rubrics
• Write down ONE unanswered question
• Contact maryb_at_nova.edu