Defining and measuring benchmarks to evaluate competence in practicebased learning and systemsbased

1
Defining and measuring benchmarks to evaluate
competence in practice-based learning and
systems-based practice

• Robert Englander, MD, MPH
• Carol Carraccio, MD

2
Learning Objectives

• Learn the 4 step process to developing a
  competency-based system of education
• Review benchmarks for domains of competence in
  practice-based learning and systems-based
  practice
• Review criteria for evaluating competence in
  practice-based learning systems-based practice

3
Paradigm Comparison
4
The ProcessStep 1

• Defining broad competencies
• (ACGME)
• Refining competencies
• (specialty quads)
• Developing benchmarks or performance indicators
  for each of the competencies
• (attempted by us)

5
Step 1 Defining the Competencies

• Patient care
• Medical Knowledge
• Professionalism
• Interpersonal skills and communication
• Practice-based learning and improvement
• Systems-based practice

6
Step 1 Refining the Competencies

• PATIENT CARE
• Residents must be able to provide family
  centered patient care that is developmentally age
  appropriate, compassionate, and effective for the
  treatment of health problems and the promotion of
  health.

7
Step 2- Phase 1 Defining Benchmarks

• Competency Use information technology, peer
  review, and self assessment to promote life-long
  learning
• Knowledge/Skills/Attitudes Benchmarks
• Uses feedback to identify areas for improvement
• Seeks opportunities to strengthen deficits in
  knowledge/skills (e.g. specific electives,
  didactic opportunities, information technology
  resources, etc)

8
Step 2- Phase 2 Setting Thresholds

• Setting Thresholds
• Defining types of thresholds
• Applying the appropriate type of threshold to
  each benchmark under each competency

9
Achieving Consensus

• Survey sent to all members of the Association of
  Pediatric Program Directors
• Consensus defined as gt 50 of directors agreeing
  on a specific threshold
• For those benchmarks with lt 50 agreement at
  single level, the threshold was dropped to the
  next level down until gt 50 consensus was reached

10
Step 2- Phase 3 Consensus on Thresholds

• For the majority of benchmarks there was clear
  consensus regarding the threshold for the
  benchmark based on level of experience
• Most discrepancies occurred in defining
  thresholds within the first year
• Consensus reached for 95 of benchmarks at PGY-2
  level and 100 at PGY-3

11
The Process Step 3

• Evaluating Competence
• Creating a portfolio
• Three tools were developed to address competence
  in PBL I and SBP

12
Practice based Learning and Improvement

• Practice Improvement Project
• 15 total benchmarks
• Project measures half of the benchmarks
• Team QA project in a continuity clinic setting
• Detailed instructions for learners
• Criteria for judging competence for faculty
  preceptors

13
Systems-based Practice

• Two Projects
• Together they measure half of the 17 benchmarks
• Detailed instructions for learners
• Criteria for judging competence for faculty

14
Systems-based Practice

• Navigating the System
• Builds on practice improvement project with the
• Resident outlines a plan to further improve
  quality by using potential resources outside of
  the immediate system

15
Systems-based Practice

• System Errors
• Learner identifies a system error and outlines
  strategies to rectify it which are at a system
  level

16
Next Steps

• Study the reliability and validity of the newly
  created tools
• Create additional tools and venues for residents
  to demonstrate competence in the remaining
  benchmarks

17
The Process Step 4

• Evaluating the Process
• Reviewing each of the prior steps for efficacy
• (Phase 4 of the ACGME Outcomes Project 2011)