Conditions for summative portfolio assessment in medical education

1
Conditions for summative portfolio assessment in
medical education

• Hanno van Keulen, Edith ter Braak, Jobien
  Olijhoek
• Utrecht University, The Netherlands
• j.vankeulen_at_ivlos.uu.nl
• The first international residential seminar
• The Oxford Belfry, Oxfordshire, UK, 9th - 11th
  October 2006
• Researching and Evaluating
• Personal Development Planning and
• e-Portfolios

2
Introduction

• Portfolios not only are potent tools for personal
  planning and development
• They are also very useful for summative
  assessment of professional development
• The question is What kind of usefulness do we
  expect?

3
Educational context

• We are focusing on the Medical School of Utrecht
  University, The Netherlands, .to the year three
  Internal Medicine clerkship
• This is a six week full time course, with
  learning objectives on Clinical Reasoning and
  Status writing.
• Students take in new patients. They do the
  Anamnesis (Medical Interviewing) and the Physical
  Examination.
• After this, they write down a (differential)
  diagnosis suggestions for subsequent
  (laboratory) examinations and suggestions for
  therapy and prognosis.
• Students compose a portfolio consisting of these
  Patient reports (see next two slides) and their
  reflections

4
The Patient Report (page 1)
5
The Patient Report (page 2)
6
Portfolios educational focus

• We use portfolios
• To stimulate professional development
• Clinical judgment
• Writing
• To stimulate reflection on these professional
  skills
• To provide feedback
• To assess
• Our portfolios are not
• Purely formative
• On personal development
• On generic/academic skills

7
Assessment Procedure

• Students are required to compose a portfolio of
  at least 10 Patient Reports
• The Portfolio is assessed halfway and at the end
  (after 5 and 10 patient reports) in a 30 minute
  interview by a Resident
• Some aspects of medical proficiency (professional
  attitude certain medical skills) are also
  assessed during this course but not by means of a
  portfolio
• Portfolios are either handwritten or electronic
• An example of the assessment and feedback form is
  shown in the next slide

8
Assessment form
9
The rationale of portfolio assessment

• Clearly, direct observation of the clinical
  process from anamnesis to diagnosis and prognosis
  is richer and therefore more valid than indirect
  and restricted observation through a portfolio.
• However, portfolio assessment is much less time
  consuming for staff, it allows for more patient
  variety, and it shows professional development
  through time.
• In this way, portfolio assessment is problably
  more rational than other instruments for
  assessing clinical competenties such as Mini-CEX
  or OSCE.

10
Characteristics of Portfolio assessment

• Strengths
• Outcomes, not (just) processes
• No need to observe
• Volume and case mix
• Day-to-day performance (Miller shows/does)
• Reveals development
• Evidence from various sources (360 degree)
• Enables feedback
• Positive impact on learning behaviour
• Preparing for assessment (writing and analysing
  a patient report) is congruent with learning
  (doing a patient)
• Problems
• Lots of work for the student
• Assessment is not completely objective
• Student hides weaknesses
• In-direct

11
The Research Questions

• We ask ourselves the following questions
• Is this assessment procedure sufficiently
  reliable?
• Is the action-reflection-feedback-reaction cycle
  effective?
• What can we do to make it more reliable,
  effective, and acceptable?

12
The issue of reliability

• Evidence so far indicates that grading is not
  reproducable on a 4-10 point scale (as expected)
• But it is with regard to pass-fail decisions!
• We also find that numerical grading conflicts
  with feedback
• Personal rater bias can be a problem but also is
  welcome (within bounds) since student can learn
  different things from different faculty (e.g. the
  rigorous and systematic approach versus an
  approach that has minimum impact to the patient)
• The following strategies are important to improve
  and maintain reliability
• Pre-entrance selection of students
• Holistic approach rather than analytic
• Triangulation
• Training of raters

13
The action-feedback cycle some findings

• An assessment portfolio necessarily is far more
  structured than a PDP-portfolio. We think this is
  not in conflict with theory.
• Feedback is provided on two (conflicting) levels
• Medical feedback
• Feedback on meta-perspectives
• Clinical thinking
• Structure in writing
• Medical feedback should be provided immediately
  and on each patient report, if possible.
• Meta-feedback should be provided on the whole
  sequence of reports.

14
Acceptance ICT

• Hand-writing culture
• Quick and intuitive, but
• Difficult to provide feedback in time
• Doesnt link with electronic information
• E-Portfolio
• Problematic history
• Hospital safety procedures conflict with free
  internet access from outside
• Use of ELO (WebCT) complicated and inflexible
• We now use Word and E-mail!
• Future ambitions and experiments
• Including Sound Vision
• Medical symbols and feedback palets
• Digital from the start (PDAs small laptops
  tablets)

15
Selected literature

• Delandshere, G. and A. R. Petrosky (1998).
  "Assessment of complex performances limitations
  of key measurement assumptions." Educational
  Researcher 27(2) 14-24.
• Lettus, M. K., P. H. Moessner, et al. (2001).
  "The clinical portfolio as an assessment tool."
  Nursing Administration Quarterly 25(2) 74-79.
• Miller, G. E. (1990). "The assessment of clinical
  skills/competence/performance." Academic Medicine
  65(9) 63-67.
• Johnston, B. (2004). "Summative assessment of
  portfolios an examination of different
  approaches to agreement over outcomes." Studies
  in Higher Education 29(3) 395-412.
• Pitts, J., C. Coles, et al. (2002). "Enhancing
  reliability in portfolio assessment discussions
  between assessors." Medical Teacher 24(2)
  197-201.
• Schutz, A. and P. A. Moss (2004). Reasonable
  decisions in portfolio assessment evaluating
  complex evidence of teaching. Education Policy
  Analysis Archives. 12.
• Vleuten, C. P. M. van der and L. W. T. Schuwirth
  (2005). "Assessing professional competence from
  methods to programmes." Medical Education in
  press.
• Wass, V. (2006). The assessment of clinical
  competence in high stakes examinations - Are we
  justified in abandoning old methods in favour of
  the new? Thesis, Maastricht University,
  Maastricht.