Experience Examined

1
Experience Examined

• Feedback and Evaluation in Clinical Learning
• Catherine Lucey, MD

2
Goal

• Increase the comfort with which participants
  assess, provide feedback to, and complete
  evaluations on their learners

3
Clinical Learning is Experience Examined
Ende and Davidoff, 1992
4
Two Critical Roles

• Coach Feedback/ Formative Evaluation
• Information provided during the rotation,
  describing performance, with the intent to guide
  future performance.
• Judge Summative Evaluation
• Judgement provided at the close of a rotation
  assessing whether the learner met performance
  standards.

5
Feedback and Evaluation Linked but Different
6
Residents Help!
How often do you get?
Isaacson et al. Cleveland Clinic Foundation, 1995
7
Programs Help!

• 2/3s of Program Directors feel that accuracy of
  faculty evaluations are major concerns
• APDIM Precourse, April 1999
• 50 of Program Directors had a recent troubling
  feedback/eval session
• APDIM Plenary, October 1999

8
What are the Barriers?

• Brainstorming Session

9
Barriers Fairness, Uncertainty, Time

• Will my evaluation be reliable?
• I dont know what learners today are supposed
  to be able to do at this level...
• Will my evaluation be valid?
• I didnt see them do x enough to be able to
  judge fairly
• I cant remember

10
Barriers Sensitivity Concerns

• Will my feedback be well received?
• what if they cry?
• There goes my teaching award.
• How will my feedback and evaluation be used?
• I dont want to ruin a promising career

11
Overcoming Barriers

• Knowledge
• Expectations of the rotation
• Understanding of the tools
• Attitudes
• Faith in the process and the learner
• Skills
• Targeted observation and
• Constructive Feedback

12
Road Map for A Successful Rotation
Expectations
Formative Eval
Summative Eval
13
Case Study
Program University of Anywhere Learner Terry, a
PGY-1 Preceptor Dr. Andersen, a general
internist in practice on the eastern shore of
Maryland. S/he knows Terrys program director
from residency. The program director remembers
Dr. Andersen as a smart doctor and good teacher.
14
Beginning of the Month

• Choose Your Focus
• Choose The Right Tools
• Announce your feedback intentions

15
1. Choose your Focus Rotation Specific Objectives
Overall Clinical Competence
16
Examples

• Knowledge
• Offer a prioritized differential diagnosis for
  common ambulatory complaints such as CP, SOB,
  Dizziness, Edema.
• Skills
• Use and document telephone encounters for dx and
  thx follow-up...

17
2. Choose Your Evaluation Tools

• Standard Tools For Assessment
• Supervised Direct Patient Care
• Case Presentations
• Standard Tool for Measurement
• Global Rating Scales
• Behaviorally Anchored Scales
• Developmentally Anchored Scales

18
BARS The Final Pathway
Behaviorally Anchored Rating Scales

• ABIM
• 123 456 789
• unsatisfactory satisfactory superior
• Dartmouth
• poor fair good excellent

Must be improved
Room for improvem
Adequate Skills
Superior Skills
19
Developmental Scales
Difficulty despite
Does with assistance
Often does w/o assist
Usually does w/o assist
Always expert, w/o assist
20
Other Tools to Consider

• Medical Record Audit
• Chart stimulated recall
• Mini CEX
• Standardized Patients
• Nursing and Patient Evaluations

21
3. Announce Your Feedback Intentions

• Why will feedback be given?
• Be all you can be...
• When will feedback be given?
• Micro-feedback
• Macro-feedback or Formative Evaluation
• How will feedback be given?

22
Heart of the Rotation

• Use Micro-feedback skills to coach the learner
• What did they do well?
• Where do they need to improve?
• What is the next step for them?

23
Micro-feedback shapes behavior

• Timely Case by Case, 1st Hand
• Focused Choose your pearls
• Expected Part of the culture
• Applied Next step, next patient, goal directed
• Documented Progress Notes

24
How to give effective micro-feedback

• Look for behaviors to reinforce as well as
  correct
• Be specific about what you saw
• Avoid speculating on intentions
• Use nonjudgmental language
• Encourage self evaluation
• Identify next steps

25
Reinforcing Feedback

• I was impressed that you gave Mrs.. Jones plenty
  of time to tell her story. (specific and
  reinforcing)
• When you are patient like that, you often get a
  more accurate history (nonjudgmental and
  reinforcing)
• The next step for you is to practice using
  clarifying questions for symptoms such as chest
  pain.

26
Corrective Feedback

• I found a very large liver on Mr. Smith, why do
  you think you underestimated its size?
  (specific, encourages self evaluation)
• When you examine for a liver, it is important to
  undress the patient. (nonjudgmental, forces self
  evaluation)
• As a next step, lets go over how you position
  your hands

27
Embarrassment and Humiliation

• Humiliation Externally driven
• generally negative experience
• avoidable with skilled feedback techniques
• Shame/Embarrassment Internally driven
• neither entirely avoidable nor entirely negative

28
Progress Notes
29
Feedback Progress Notes

• Action Document case by case impressions
• Goal Identify patterns of strengths and
  weaknesses in multiple encounters
• Outcome More specific formative and summative
  evaluations

30
Group Exercise

• Using the scenarios in your handout, role play
  micro-feedback and progress note completion
• Switch roles to allow each individual to
  experience the role of feedback provider

31
Midpoint

• Formative Evaluation
• Macro Feedback
• Coaching Role Continues

?
32
Macro-Feedback

• Use patterns identified during micro-feedback
• Concretely compare to the agreed upon goals
• Plan of action to achieve the end goals
• more specific more achievable

33
Skill Set Communicating Evaluations

• Ask How am I doing as a teacher?
• Ask What do you see as your strengths then
  weaknesses?
• Tell Summarize your observations and ideas
• Ask What is the take home message, next steps?

34
Group Exercise

• Practice the Ask, Ask,Tell, Ask format of leading
  an evaluative session
• Check your action plans for specificity

35
End of the Month

• Summative Evaluation
• Did they meet standards?

Met
Exceeded
Failed
36
Skill Set Pitfalls in Evaluation

• Range Restriction
• Hawk/Dove
• Halo Effect

37
Group Exercise

• Complete the Attached evaluation form, using the
  notes that you have taken during the role play
  sessions.

38
Conclusions

• Feedback and Evaluation are critical components
  of successful clinical learning
• Optimal performance of these skills requires
  collaboration between the program and preceptor

39
Conclusions

• Less is Frequently More
• Choose your focus for evaluation
• Provide feedback frequently, in small bits of
  manageable information
• Next steps should be measurable
• Feedback is a two way street