A Practical Approach to Evaluation in the Ambulatory Setting in the Era of the New ACGME General Com

1
A Practical Approach to Evaluation in the
Ambulatory Setting in the Era of the New ACGME
General Competencies

• Eric S. Holmboe Stephen Huot
• Yale University School of Medicine
• Yale Primary Care Residency Program

2
ACGME Core Competencies

• Medical knowledge
• Patient care
• Practice-based learning and improvement
• Interpersonal and communication skills
• Professionalism
• Systems-based practice

3
Workshop Objectives

• Understand the importance of the outpatient
  setting for assessment of clinical skills
• Appreciate importance of directly observing
  residents interacting with patients
• Discuss practical strategies for focused direct
  observation

4
Workshop Elements

• Mini-Lectures
• Basic Premises
• Ambulatory clinical skills
• Faculty rating accuracy
• Direct observation exercises
• Performance dimension exercise
• Videotape evaluation exercises

5
Basic Premises

• Accurate resident evaluation important
• Decision-making summative
• Feedback formative
• Professional obligation
• Resident observation
• Traditional and vital

6
Ambulatory Clinical Skills

• History taking
• Focused physical examinations
• Counseling and education
• Reflective practice

7
Importance of Sound Clinical Skills

• Physician behaviors and communication
• Accuracy / completeness of data gathering
• Patient satisfaction and compliance
• Clinical outcomes
• Legal implications
• Contribution of History PE to
  decision-making
• 80 to 90 diagnoses made by H P
• Cost-effective use of health care resources

8
Clinical Skills

• Stillman (1990)
• Wide variability in MS4 clinical skills
• Sachdeva (1995)
• Wide variability in intern skills
• Wray (1983) / Johnson (1986)
• High frequency of errors
• Mangione (1997)
• Deficient cardiac auscultatory skills

9
Clinical Skills

• Suchman (1997)
• Poor communication / humanistic skills
• Ramsey (1998)
• Incomplete history-taking / preventive health
  screening
• Braddock (1999)
• Of gt 1000 patient visits, less than 15 fulfilled
  core elements of informed decision making

10
Resident Clinical Skills Themes

• Deficiencies exist across continuum
• Specific skills more error-prone
• Not detected by other evaluation methods
• Basic clinic skills dont correlate with other
  competence dimensions
• Residents aware of importance and under-emphasis
• Without detection cannot be corrected

11
ACGME and Direct Observation

• Direct Observation crucial to evaluate
• Patient care
• History taking, Pexam, counseling
• Interpersonal and communication skills
• Patient/peer/colleague interactions
• Professionalism

12
Faculty Observation / Rating Skills

• Thompson (1990)/Haber (1994)
• Significant halo effect with ratings
• Ratings based mostly on perceived knowledge and
  personality
• Kalet (1992)
• Poor reliability interpersonal skills
• Poor validity and predictive value
• Rater training ineffective

13
Faculty Observation / Rating Skills

• Herbers (1989) / Noel (1992)
• Structured gt open-ended form
• Brief training video not effective
• Increased accuracy ? discriminative ability
• Kroboth (1992)
• Poor inter-rater reliability
• Rater training ineffective

14
Faculty as Raters Key Issues

• Faculty do not observe actual performance
• Faculty ratings lack
• Reliability
• Accuracy
• Content specificity

15
Faculty as Raters - Solutions

• Step 1 Getting faculty to observe
• Required by the ACGME
• Focused observations are logistically possible
• 5 to 10 minute observations are valuable
• Build into existing clinic schedule
• Build on faculty epiphany
• The You will not believe what I saw today
  experience

16
Mini - CEX Tool

• Structured approach to direct observation
• Direct assessment of actual patient care
• Incorporation of CEX into daily activities
• High satisfaction among housestaff

17
Logistics GIMC

• One mini-CEX per intern per day per week
• One attending observes portion of first visit of
  the day
• Interview, physical exam, counseling
• Minimizes disruption of resident clinic
• Perform over course of academic year
• Easy to obtain 6-8 Mini-CEXs per year per intern

18
Faculty as Raters - Solutions

• Step 2 Improving reliability
• Multiple brief observations
• Perform over time outpatient setting allows for
  longitudinal observation
• Involve multiple faculty
• MiniCEX sufficient reliability for pass/fail
  determinations after just 4 observations

19
Direct ObservationYale PGY-2 Resident
20
Videotape

• Watch the following videotape and then complete a
  Mini-CEX evaluation on the clinical skills of
  this resident

21
Faculty as Raters - Solutions

• Step 3 Improve accuracy and validity
• Most difficult step
• Improved with structured rating forms
• Can be improved with rater training, but
• Brief training interventions do not work

22
Can You Train Faculty?

• Performance Appraisal Literature
• Can reduce rating errors
• Can improve discriminative ability
• Can improve accuracy

23
Summary of Rater Training

• Performance Dimension Training
• Frame of Reference Training
• Behavioral Observation Training

24
Performance Dimension Training

• Involves familiarizing faculty with the specific
  dimensions of competence
• Should involve discussion of the qualifications
  required for each dimension
• Use the ACGME competencies and the ABIM portfolio
  to calibrate faculty

25
Frame of Reference Training

• Goal is to improve judgment and accuracy
• Steps in FOR training
• 1. Raters given descriptions of each dimension -
  discuss qualifications needed for each
  dimension (PDT)
• 2. Review of clinical vignettes describing
  critical incidents of performance unsatisfactory
  to average to superior

26
Frame of Reference Training

• 3. Raters used vignettes to then provide ratings
  on a behaviorally anchored rating scale (BARS) -
  think ABIM eval form
• 4. Session trainer provides feedback on what
  true ratings should be along with rationale
• 5. Discussion ensues about discrepancies between
  trainers ratings and the participants ratings

27
Frame of Reference Training

• Most difficult aspect of FOR
• Setting the actual performance standards
• Reaching agreement and consensus among teaching
  faculty

28
Behavioral Observation Training

• Two main strategies
• 1. Increase the amount of sampling
• - More observations lead to more accurate
  evaluations.
• 2. Use of observational aides
• - Behavioral diary to record observed
  performance.

29
Structuring the Observation

• Prepare for the observation
• Minimize intrusiveness correct positioning
• Minimize interference with the resident-patient
  interaction
• Avoid distractions
• Possible solution
• Allow for habituation by consistent observation

30
Direct Observation Challenges

• Like all skills, requires training and practice
• Faculty calibration important
• Agreeing on metrics of performance
• Faculty comfort with own skills
• Faculty training
• How, when, who, what, where

31
Observation Summary

• Sample parts of the visit
• History-taking
• Physical examination
• Counseling
• Perform longitudinally
• No need to do it all at once
• Agree on performance metrics with ambulatory
  faculty