PEER ASSESSMENT (PA) OF MEDICAL PROFESSIONALISM BY MEDICAL STUDENTS

1
PEER ASSESSMENT (PA) OF MEDICAL PROFESSIONALISM
BY MEDICAL STUDENTS

• AFMC PROFESSIONALISM RESOURCE GROUP MEETING
• A Keith W Brownell MD
• London April 29, 2006

2
OVERVIEW

• References
• Background to My Interest in the Topic
• General Comments About PA
• Specific Comments on PA of Professionalism by
  Medical Students
• Some Canadian Experiences
• Discussion

3
DISCLOSURE/DISCLAIMER

• None of this material is original and most of it
  comes directly or indirectly from one of the key
  references which are listed on Slide 5

4
BACKGROUND TO INTEREST IN THIS TOPIC
5
KEY REFERENCES

• John Norcini Peer Assessment of Competence.
  Medical Education 200337539-543.
• Deirdre Lynch et al Assessing Professionalism
  A Review of the Literature. Medical Teacher
  200426(4)366-373.
• Carolyn Shue et al Maximizing Participation in
  Peer Assessment of Professionalism the Students
  Speak. Academic Medicine 200580(10
  Suppl)S1-S5.
• Elaine Daddefer et al Peer Assessment of
  Professional Competence. Medical Education
  200539713-722.
• Louise Arnold David Stern Content and Context
  of Peer Assessment in Measuring Medical
  Professionalism Edited by DT Stern Oxford
  University Press 2006

6
GENERAL COMMENTS ABOUT PA

• The act of making judgments on the performance of
  ones peers is ubiquitous and has formed the
  basis of the referral process in medicine and
  other professions for centuries.
• Throughout this long history, peers have been
  deployed in a variety of ways to make judgments
  on the competence of their colleagues.

7
NATURE OF JUDGMENTS

• Judgments about structured tasks versus global
  impressions.
• Judgments about occurrence, quality or
  suitability.

8
JUDGMENTS ABOUT STRUCTURED TASKS VERSUS GLOBAL
IMPRESSIONS

• Global impressions judgments are more common than
  judgments about structured tasks.
• Each type has strengths and weaknesses so for
  example
• Global impressions influenced more by halo
  effect.
• Structured tasks are limited by the number of
  encounters being assessed.

9
JUDGMENTS ABOUT OCCURRENCE, QUALITY OR
SUITABILITY

• Occurrence can be assessed by check lists.
• Quality of performance is more commonly assessed
  but limited by number of observations.
• Is the performance suitable/satisfactory this
  involves determining if the performance was of
  good quality and also good enough for the purpose
  of the evaluation. More difficult to do.

10
ASPECTS OF COMPETENCE ASSESSED

• Technical /Cognitive
• Technical ability
• Basic science knowledge
• Clinical knowledge
• Judgment
• Problem solving

• Relationship/Non-cognitive
• Peer relations
• Patient relations
• Reliability
• Industry
• Personal appearance
• Reaction to pressure

11
FACTORS INFLUENCING THE QUALITY OF PA

• Reliability
• Relationships
• Stakes
• Equivalence

12
FACTORS INFLUENCING THE QUALITY OF PA (I)

• Reliability
• This is influenced by the
• Number of relevant observations
• Number of peers involved
• Number of aspects of competence involved
• Influenced little by wording of questions, number
  of points on rating scales, whether to describe
  all points on scale etc.

13
FACTORS INFLUENCING THE QUALITY OF PA (II)

• Relationships
• Are the peers competing with each other?
• Are the rating peers friends?
• Are there financial relationships?

14
FACTORS INFLUENCING THE QUALITY OF PA (III)

• Stakes
• Likely to be influenced by use to which it will
  be put with high stakes then tendency to give
  higher ratings. This can be mitigated to some
  degree by ensuring anonymity of raters.
• Ensure that peer is being asked to rate on
  quality of performance rather than suitability
  for something.

15
FACTORS INFLUENCING THE QUALITY OF PA (IV)

• Problems can be minimized by
• Increasing the number of peers doing the
  assessment
• Providing the peers with clear criteria for
  making their judgments

• Equivalence
• This means
• being judged on same activities
• being judged by peers who all have the same
  degree of stringency (need to avoid hawks and
  doves)

16
STEPS IN IMPLEMENTATION

• State purpose of assessment, preferably in
  writing.
• Assessment criteria must be developed and
  communicated to the participants.
• Training should be provided to all participants.
• The results of the assessment should be monitored
  throughout the implementation process.
• Feedback should be provided to the participants.

17
NOW TO MEDICAL PROFESSIONALISM

• AND
• PEER ASSESSMENT OF MEDICAL STUDENTS

18

• Why Do This?
• Medical teachers have less time to spend in
  direct contact with students.
• As a result opportunities for learners to
  demonstrate a wide variety of professional and
  unprofessional behaviors are more likely to occur
  among peers who spend more time with one another
  and work closely together as a team.
• Professional behaviors like responsibility,
  effective communication, interpersonal respect,
  thoroughness and altruism have a direct impact on
  peers or reflect values that peers might be able
  to infer and observe in actions of their peers.

19

• The medical student peers are individuals who
  have attained the same level of training or
  expertise, exercise no formal authority over each
  other, and share the same hierarchical status in
  an institution.
• These non-hierarchical relationships can promote
  both authentic behavior and genuine feedback
  among peers while reducing the biasing influence
  of social desirability.
• In PA, peers are asked to judge each others
  characteristics or behaviors relevant to an
  evaluation task.

20

• PA particularly useful in situations in which
• Peers are afforded a unique view of one anothers
  behavior
• Peers are capable of accurately perceiving and
  interpreting one anothers behaviors
• There is a need to improve the effectiveness of
  assessment of group members behavior
• Peers not only observe different behaviors but
  they also provide a different perspective on the
  same behaviors observed by others.

21
Methods of PA (I)

• Ratings involve each member of a group
  rendering a judgment about other members on a
  specified set of behaviors, performances or
  characteristics using a scale.
• Nominations consist of group members naming a
  certain number of group members as the best along
  a particular performance dimension or quality.
• Rankings involve each group member ordering all
  other group members from best to worst on
  specified behavioral dimensions or
  characteristics.

22
Methods of PA (II)

• Peer voting assign learners a number of votes
  equal to the number of group members and directs
  learners to apportion their votes across the
  group members according to the degree to which
  each member exhibits a behavior or quality
  relevant to the evaluation.
• Qualitative comments.

23
PSYCHOMETRIC CHARACTERISTICS OF DIFFERENT METHODS
USED IN PA
24
RATINGS

• Most common method used and most supported with
  evidence of reliability and validity.
• Validity includes face, content, construct,
  concurrent and predictive validity.
• Method of choice when specific information about
  each group member is desired.
• Especially useful for formative assessments which
  provide feedback.
• They are subject to problems of bias and may be
  less valid because they judge across the entire
  range of performances or qualities.

25
CONTEXT OF PA

• The reaction of learners to PA and the procedures
  used for initiating, maintaining, and reporting
  peer assessments can affect the integrity of
  evaluations reported by peers.
• The use to which PA results are put may influence
  the peers willingness to provide genuine
  evaluations.
• Medical students seem to be resistant to the idea
  of PA.

26
PA MOST LIKELY TO SUCCEED IF STUDENTS KNOW

• How the results of it will be used.
• That it is anonymous.
• That it is being done at an appropriate time.
• It is mandatory.
• It covers areas in which peers see themselves as
  particularly perceptive. Or in other words that
  the content reflects the ideas and the lives of
  those who will participate in the system.

27
PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE
CONTEXT OF A SUPPORTIVE ENVIRONMENT WHICH WOULD
INCLUDE (I)

• School responsiveness to peer reports.
• Behavioral standards on professionalism that
  Faculty consistently enforce.
• Student groups whose members are amenable to
  teaching each other and exploring professionalism
  issues.
• Close relationships between students and Faculty.
• Approachable administrators who trust students.
• Value placed on assessment aimed at learners
  improvement.

28
PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE
CONTEXT OF A SUPPORTIVE ENVIRONMENT WHICH WOULD
INCLUDE (II)

• Education which explores the meaning of
  professionalism.
• Specifies expectations for professional behavior.
• Provides training for feedback on professionalism
  and conflict resolution.
• Highlights the importance of PA.
• Provides for faculty modeling of PA.

29
SURVEY

• January sent e-mail to everyone on the AFMC
  Professional Resource Group Mailing List.
• No names listed from Francophone Schools, NOSM or
  Memorial on the AFMC mailing list thus no
  contacts were made with any of these schools.
• Arranged telephone contacts with the people who
  responded.

30
CANADIAN EXPERIENCES WITH MEDICAL STUDENT PA

• University of British Columbia Niamh Kelly
  PBL.
• University of Alberta Lorraine Breault
  Interdisciplinary HSC Course.
• University of Calgary Janet Wright Rod
  Crutcher Well Physician Course.
• University of Western Ontario Peter Flanagan
  PCL.
• University of Ottawa Walter Hendelman
  Professionalism Program.
• Others.

31
DISCUSSION

• Is peer assessment a good thing to do?
• If yes, why are we not doing more of it?
• What could this Resource Group do to move PA
  forward in our schools?
• Interest in making peer assessment of medical
  professionalism a focus of this resource group?
• Other issues?