An Example of Teaching and Assessing Resident Competence in PracticeBased Learning and Improvement

1
An Example of Teaching and Assessing Resident
Competence in Practice-Based Learning and
Improvement

• Laura J. Morrison, MD, Greg Ogrinc, MD, MS,
• Tina Foster, MD, MPH, MS, Linda A. Headrick, MD,
  MS
• MetroHealth Medical Center, Case Western Reserve
  University
• White River Junction VA, Dartmouth-Hitchcock
  Medical Center

2
Outline

• Practice Based Learning and Improvement -
  defining, teaching, assessing
• Resident Elective
• - aim, design, intervention, measures
• - QIKAT
• Results
• Next Steps and Conclusions

3
Practice-Based Learning and Improvement

• involves investigation and evaluation of
  residents own patient care, appraisal and
  assimilation of scientific evidence, and
  improvements in patient care

ACGME, 1999
4
Teaching PBLI

• Professional knowledge must be combined with
  knowledge for improvement.
• It is helpful to combine didactic and
  experiential learning.
• We can be guided by IHIs Eight Knowledge Domains
  for the Improvement of Healthcare.

5
Assessing PBLI

• Common outcomes self-report of satisfaction and
  lessons learned
• Performance-based assessments project reports,
  story boards, individual reflection
• ACGME Toolbox of Assessment Methods multiple
  assessment options
• Standardized, objective assessments of PBLI
• do not yet exist in the literature

6
Aim

• Design, implement, and evaluate a
• learning opportunity for residents about
• the improvement of healthcare that
• is easily reproducible
• is meaningful
• includes assessment of the application of quality
  improvement knowledge

7
Design
Internal medicine residents at MetroHealth and
Dartmouth
1 month QI elective offered
Subjects (N11)
Controls (N22)
Baseline QI knowledge assessed
Elective month
Completion of elective
Final QI knowledge assessed
Comparison of groups
8
Intervention

• Didactic sessions with QI faculty
• Appropriately trained faculty member
• Planned as 1-2 hours per week
• Readings from texts and articles
• Emphasis on application of principles and methods
  to residents improvement project

9
Intervention

• Improvement project with a project sponsor
• Small piece of an existing project
• Resident initiated project

10
Didactic Curriculum Overview

• Week 1 Linking Professional Knowledge to
  Improvement Knowledge
• Week 2 Systems and Processes
• Week 3 Measurement and Variation
• Week 4 Recommending Changes,
• Benchmarking, and Holding the Gains

11
Outcome Measures

• QIKAT (QI Knowledge Application Tool) (pre/post)
• Structured evaluation of project synthesis
  presentation
• Knowledge and Skills Self-assessment
• (pre/post, 6-8 month f/u)
• Satisfaction survey
• Time logs for residents and faculty

12
The QIKAT

• Brief case-based scenario and instructions
• Please answer each of the following questions as
  if you were developing a program to investigate
  and improve the problem presented above.
•  
• 1)   What would be the aim?
•  
• 2)   What would you measure to assess the
  situation?
•  
• 3)   Identify one change that might be worth
  testing

13
Results
14
MetroHealth Sickle Cell Disease

• Clinic follow-up after discharge from
  hospitalization for acute pain crisis
• lt14 days is recommended standard
• Analyzed 15 months of admissions and discharges
  for pain crisis
• lt50 had follow-up scheduled at time of discharge
• Recommended template for appointment scheduling
  after hospitalization for crisis

15
DHMC Needle Stick Injuries

• Nurses have highest number of injuries
• Resident looked at rate over 18 months
• Leaners are an at-risk population
• Found a spike in rate in August 2000
• No needlestick training for medical students or
  residents
• Standardized training now part of resident
  orientation

16
QIKAT scores
17
Resident Presentation Evaluation
18
Knowledge and Skills Self-Assessment
19
Resident Satisfaction with Learning Objectives
20
Cost Time

• Subjects averaged 119 hours (range 41-232) over
  the four weeks
• QI faculty averaged 6 hours (range 3.3 - 8) over
  the four weeks
• Project sponsors averaged 1-2 hours/week

21
Limitations

• Self-selected subjects
• Small sample size
• Only internal medicine residents
• Unclear whether curriculum would be effective
  with other specialties
• MetroHealth and Dartmouth have very active
  cultures of improvement

22
Challenges

• Moving to required model
• now part of required ambulatory rotation at VA
  WRJ
• QI faculty
• Resident projects
• Continuing to learn from our experiences

23
Moving Forward

• Study group expanded to 4 sites
• Dartmouth/White River Junction VA, MetroHealth,
  U Missouri, Cleveland VA
• Elective offered to other specialties, fellows
• Simple Rules conceived to give flexibility in
  elective design at all sites while maintaining
  key components (ie, combined with other rotations)

24
Simple Rules

• Building on the concept of simple rules that
  organize complex behavior
• Proposed rules to guide us in designing and
  assessing resident experiences in PBLI
• We will use and evaluate these simple rules as we
  move on

25
Simple Rules

• The core of the elective is work on a defined
  project. Given the short time frame, preplanning
  is essential so that the resident can actually do
  the work, rather than simply planning it.
• The elective includes a core curriculum with core
  content which is covered in didactic sessions,
  with discussion centering on the residents
  project.

26
Simple Rules

• QI faculty will take responsibility for the
  didactic sessions and will be available to the
  resident for time-sensitive problems. A project
  sponsor may also be needed.
• Resident evaluation will include a structured
  evaluation of the project synthesis presentation.
  The QIKAT can be used to assess application of
  QI knowledge.

27
Integration

• Ultimately this learning and experience should be
  part of every residents daily work
• Some of these residents will become teachers of
  improvement

28
Conclusions

• It is possible for residents to gain meaningful
  Practice-Based Learning and Improvement
  experience in a one month elective.
• We developed a standardized curriculum that was
  implemented successfully at two sites.
• The time commitment was reasonable for both
  residents and faculty.
• Resident projects can provide a significant
  contribution to institutional improvement efforts.

29
QI Knowledge Assessment Tool

• Psychometric Performance (2 studies)
• Non-QI trained vs. QI trained (N38)
• 7.4 vs. 10.7 (0-15 pt scale) p lt 0.01
• Kappa 0.50 0.64
• CWRU QI grad course pre vs. post (N12)
• 7.9 vs. 11.8 (0-15 pt scale) p lt 0.0001
• Kappa 0.51

30
Scoring
31
QIKAT Waiting times in clinic

• Aim
• Decrease time from check-in to triage
• Measurement
• Time from check-in to check-out
• Intervention
• Have docs order house charts the day before clinic

• Aim
• Decrease time from check-in to triage
• Measurement
• Time from check-in to triage
• Intervention
• Trial of new vital sign machine for the nurses

Score 3
Score 5
32
Examples of Teaching Improvement to Medical
Students

• Dartmouth
• Students worked with preceptors 11 to identify
  problems and propose changes over 1 year
• Preceptors showed variable buy in and students
  were limited in their ability to direct change

33
Examples of Teaching Improvement to Medical
Students

• University of Connecticut
• Groups of 2-4 students picked a topic related to
  DM care
• 3 year longitudinal experience with serial chart
  audits
• Improved care for pts by DM outcome measures

34
Examples of Teaching Improvement to Residents

• Residents as part of hospital QI committees
• Residents improving diabetic care as part of
  primary care practice at SUNY-Buffalo
• Residents working with teams to lower unnecessary
  IV usage at U Minn