Conditions for summative portfolio assessment in medical education

1
Conditions for summative portfolio assessment in
medical education

• Hanno van Keulen, Edith ter Braak, Jobien
  Olijhoek
• Utrecht University, The Netherlands
• j.vankeulen_at_ivlos.uu.nl
• The first international residential seminar
• The Oxford Belfry, Oxfordshire, UK, 9th - 11th
  October 2006
• Researching and Evaluating
• Personal Development Planning and
• e-Portfolios

2
Outline

• A short introduction
• Presentation of some research findings
• Some proposals on portfolio procedures that
  stimulate reflection and learning
• Discussion and exchange

3
Introduction

• The context of this project
• Whats so great about summative use of
  portfolios?
• The questions What do we want to know?

4
Educational context

• Medical School
• Year three
• First Internship (Internal Medicine)
• Six week full time course
• Learning objectives
• Clinical Reasoning
• Status writing
• Students take in new patients
• Anamnesis (Medical Interviewing)
• Physical Examination
• Diagnosis
• Suggestions for subsequent (laboratory)
  examinations
• Suggestions for therapy and prognosis
• Contribution to the status of the patient
• Students compose a portfolio consisting of
• Patient reports
• Reflections

5
Portfolios focus of this presentation

• We use portfolios
• To stimulate professional development
• Clinical judgment
• Writing
• To stimulate reflection on these professional
  skills
• To provide feedback
• To assess
• Our portfolios are not
• Purely formative
• On personal development
• On generic/academic skills

6
The Patient Report (page 1)
7
The Patient Report (page 2)
8
Assessment Procedure

• Students are required to compose a portfolio of
  at least 10 Patient Reports
• The Portfolio is assessed halfway and at the end
  (after 5 and 10 patient reports) in 30 minute
  interviews by a Resident
• Some aspects of medical proficiency are assessed
  otherwise (professional attitude certain medical
  skills)
• Portfolios are either handwritten or electronic

9
Assessment grid
10
Prediction of (future) success A simple model
Personal Characteristics
Knowledge Skills
Behaviour
Achievements
11
Characteristics of Portfolio assessment

• Strengths
• Outcomes, not (just) processes
• No need to observe
• Volume and case mix
• Day-to-day performance (Miller shows/does)
• Shows development
• Evidence from various sources (360 degree)
• Enables feedback
• Positive impact on learning behaviour
• Preparing for assessment (writing and analysing
  a patient report) is congruent with learning
  (doing a patient)
• Problems
• Lots of work for the student
• Assessment is not objective
• Student hides weaknesses
• In-direct

12
The Research Questions

• Is this assessment procedure sufficiently
  reliable?
• Is the action-reflection-feedback-reaction cycle
  effective?
• What can we do to make it more reliable,
  effective, and acceptable?

13
The issue of reliability

• Our grading is not reliable on a 4-10 point scale
  (as expected)
• But it is on pass-fail decisions!
• Numerical grading conflicts with feedback!
• Personal rater bias is welcome (within bounds)!
• Strategies for optimal reliability
• Entrance selection
• Holistic approach rather than analytic
• Triangulation
• Training of raters

14
The action-feedback cycle some findings

• Our Portfolio is highly structured
• Conflict with theory?
• Feedback is provided on two (conflicting) levels
• Medical feedback
• Feedback on meta-perspectives
• Clinical thinking
• Structure in writing
• Medical feedback should be provided immediately
  and on each patient report
• Meta-feedback should be provided on the whole
  sequence of reports

15
Acceptance ICT

• Hand-writing culture
• Quick and intuitive, but
• Difficult to provide feedback in time
• Doesnt link with electronic information
• E-Portfolio
• Problematic history
• Hospital safety procedures conflict with free
  internet access from outside
• Use of ELO (WebCT) complicated and inflexible
• We now use Word and E-mail!
• Ambitions and experiments
• Including Sound Vision
• Medical symbols and feedback palets
• Digital from the start (PDAs small laptops
  tablets)