Assessing Clinical Competence A Satellite View

1
Assessing Clinical CompetenceA Satellite View

• University of Saskatchewan
• April 24 2007

2
Professor Gordon PageFaculty of
MedicineUniversity of British Columbia
Vancouver Canada
3
(No Transcript)
4
CLINICAL COMPETENCE
Knowledge
Skills
Attitudes
Effective Patient Care
Newble, 2000
5
Royal College of Physicians and Surgeons of
Canada -- CanMEDS Competencies

• Medical Expert
• Communicator
• Scholar
• Collaborator
• Manager
• Health Advocate
• Professional

6
(No Transcript)
7
CLINICAL COMPETENCE
Knowledge
Skills
Attitudes
Effective Patient Care
Newble, 2000
8
Why Focus on Knowledge?

• Norcini (ASME, 2003) -- Of all assessment
  techniques, knowledge test scores have shown to
  be the best predictors of future clinical
  performance and of clinical outcomes.

Tamblyn et al (JAMA, 2002,1998)
licensing/certification examination scores in
Canada show sustained relationships with
effectiveness indices of preventive care and
acute and chronic disease management
9
Clinical Competence and Knowledge

• Knowledge a cornerstone of clinical competence

10
Outline of Presentation

• In tests of knowledge, what should we test?
• What features of written tests contribute to
  their effectiveness in assessing clinical
  competence?
• What are the emerging trends in assessing
  clinical performance

11
Outline of Presentation

• In tests of knowledge, what should we test?
• What features of written tests contribute to
  their effectiveness in assessing clinical
  competence?
• What are the emerging trends in assessing
  clinical performance

12
(No Transcript)
13
Clinical Decision-Making
Reduced
(empty mind)
Dispersed
(cluttered mind)
Elaborated
(deductive thinker)
Compiled
(recall/recognition)
(Bordage, G. Academic Medicine, 1994, 1999)
14
Clinical Decision-Making
Reduced
(empty mind)
Dispersed
(cluttered mind)

Elaborated
(deductive thinker)
Compiled
(recall/recognition)
(Bordage, G. Academic Medicine, 1994, 1999)
15
In assessing clinical competence, what should we
examine?

• Application of knowledge
• Not recall of knowledge
• Not thoroughness of data gathering
• In early training, clinical reasoning
  (elaborated thinking -- application of basic
  sciences knowledge)
• In later training, clinical decision making
• Key steps in the resolution of clinical problems

16
Application of Knowledge?

• Mrs. Hamada is a 62 year old woman who has a
  herniated disc in her cervical spine.
• What would you expect to find on your examination
  of Mrs. Hamada?
• What symptoms would you expect Mrs. Hamada to be
  experiencing?

17
A 63-year-old diabetic is seen in the emergency
department with a 24 hour history of right loin
pain. His abdomen is soft and urinalysis shows
blood. Which one of the following is the MOST
APPROPRIATE IMAGING INVESTIGATION to
arrange?  A. Ultrasound of the
abdomen. B.(Y) Computed tomography (CT) scan
of the abdomen. C. Right retrograde
pyelogram. D. Intravenous pyelogram. E. Plain
abdominal radiograph.
Application of Knowledge?
18
Recall vs Application
Top-Down
Bottom-Up
Problem
Problem
?
?
Clinical Features
Clinical Features
19
Outline of Presentation

• In tests of knowledge, what should we test?
• What features of written tests contribute to
  their effectiveness in assessing clinical
  competence?
• What are the emerging trends in assessing
  clinical performance

20
Test of Factual Recall
What is the most likely renal abnormality in
children with nephrotic syndrome and normal renal
function?
acute poststreptococcal glomerulonephritis
(A)
hemolytic-uremic syndrome
(B)
minimal change nephrotic syndrome
(C)
nephrotic syndrome due to focal and segmental
glomerulosclerosis
(D)
Schönlein-Henoch purpura with nephritis
(E)
Case SM, Swanson DB. Constructing Written Test
Questions for the Basic and Clinical Sciences,
1996. Page 58-9.
21
Same Question, using a brief case description
A 2-year-old boy has a 1-week history of edema.
Blood pressure is 100/60 mm Hg, and there is
generalized edema and ascites. Serum
concentrations are creatinine 0.4 mg/dL, albumin
1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis
shows 4 protein and no blood. What is the most
likely diagnosis?
22
Same question, using a longer case description
A 2-year-old black child developed swelling of
his eyes and ankles over the past week. Blood
pressure is 100/60 mm Hg, pulse 110/min, and
respirations 28/min. In addition to swelling of
his eyes and 2 pitting edema of his ankles, he
has abdominal distension with a positive fluid
wave. Serum concentrations are creatinine 0.4
mg/dL, albumin 1.4 g/dL, and cholesterol 569
mg/dL. Urinalysis shows 4 protein and no blood.

23
Non Vignette
What is the most likely renal abnormality in
children with nephrotic syndrome and normal renal
function?
acute poststreptococcal glomerulonephritis
(A)
hemolytic-uremic syndrome
(B)
minimal change nephrotic syndrome
(C)
nephrotic syndrome due to focal and segmental
glomerulosclerosis
(D)
Schönlein-Henoch purpura with nephritis
(E)
Short Vignette
A 2-year-old boy has a 1-week history of edema.
Blood pressure is 100/60 mm Hg, and there is
generalized edema and ascites. Serum
concentrations are creatinine 0.4 mg/dL, albumin
1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis
shows 4 protein and no blood. What is the most
likely diagnosis?
Long Vignette
A 2-year-old black child developed swelling of
his eyes and ankles over the past week. Blood
pressure is 100/60 mm Hg, pulse 110/min, and
respirations 28/min. In addition to swelling of
his eyes and 2 pitting edema of his ankles, he
has abdominal distension with a positive fluid
wave. Serum concentrations are creatinine 0.4
mg/dL, albumin 1.4 g/dL, and cholesterol 569
mg/dL. Urinalysis shows 4 protein and no blood.

24
Discrimination is it affected by the
authenticity of the question stem?
What is the most likely renal abnormality in
children with nephrotic syndrome and normal renal
function?
A B C D E 1 0 99 0 0 8 1 90 1 0
Overall P-Value 94
A 2-year-old boy has a 1-week history of edema.
Blood pressure is 100/60 mm Hg, and there is
generalized edema and ascites. Serum
concentrations are creatinine 0.4 mg/dL, albumin
1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis
shows 4 protein and no blood. What is the most
likely diagnosis?
0 0 98 2 0 5 2 82 8 1
88
A 2-year-old black child developed swelling of
his eyes and ankles over the past week. Blood
pressure is 100/60 mm Hg, pulse 110/min, and
respirations 28/min. In addition to swelling of
his eyes and 2 pitting edema of his ankles, he
has abdominal distension with a positive fluid
wave. Serum concentrations are creatinine 0.4
mg/dL, albumin 1.4 g/dL, and cholesterol 569
mg/dL. Urinalysis shows 4 protein and no blood.
0 1 98 1 0 10 9 66 10 5
84
Case SM, Swanson DB. Constructing Written Test
Questions for the Basic and Clinical Sciences,
1996. Page 58-9.
25
Does the format of question (e.g., MCQ, short
answer) affect its ability to effectively
identify students at different levels of clinical
competence?
26
Discrimination a function of the number of
options?
PERCENT OF STUDENTS
PERCENT CORRECT SCORE
27
Free Response vs Selected Response Scores
MCC/RACGP
28
Length of Tests and Numbers of Questions?
29
Case Specificity
Problem 1
30
Case Specificity

• Inter-case correlation .1 - .3
• Each case presents unique challenges (Arthritis ?
  Anemia ? Crohns ? Diabetes)

31
Reliability of Test Scores
Domain of Competence
?
?
32
Reliability as a function of testing time
1Norcini et al., 1985 2Stalenhoef-Halling et al.,
1990 3Swanson, 1987
4Was et al., under editorial review 5Newble
Swanson, 1987 6Ram et al., 1999
33
Conclusions For written examinations to
effectively assess clinical competence, they
should

• Test the (bottom-up) application of knowledge in
  the context of authentic, high-fidelity
  clinical challenges which replicate as closely as
  possible what clinicians are challenged to do in
  their clinical practice
• Make use of question formats that accurately
  discriminate among examinees at varying levels of
  competence (e.g., more options, short answer)
• Adequately and representatively sample the domain
  of competence of interest

34
Outline of Presentation

• In tests of knowledge, what should we test?
• What features of written tests contribute to
  their effectiveness in assessing clinical
  competence?
• What are the emerging trends in assessing
  clinical performance?

35
Clinical Performance Assessment

• OSCEs (standardized patients, skills stations, )
• Long and short cases (ideally observe trainee
  with a patient)
• In-training assessment forms
• Mini Clinical Evaluation Exercise (Mini-CEX)
• Case-based discussion
• Direct Observation of Procedural Skills (DOPS)

36
A Story of Performance Assessment Since 1970

• Disenchantment with (long and short case) oral
  examinations/professional judgment of examiners
  notoriously unreliable scores
• Development of the OSCE the same set (and
  larger numbers) of simulated cases for all
  examinees, objective checklists for scoring
  better reliability of scores

37
Objective Structured Clinical Examination (OSCE)

• Many (10-20) stations students rotate from
  station to station
• In each station, students take a history from,
  examine, or counsel a Standardized patient a
  person trained to simulate a patient
• Students performance is observed by a faculty
  member
• Check lists and rating forms are used to assess
  students performance

38
A 30-Year Story of Performance Assessment

• Increased reliability of scores from OSCEs shown
  to be a function of better sampling across cases
  and not of the objective checklists.
• Global ratings (professional judgments) on OSCEs
  shown to provide more reliable scores than
  checklist scores

39
A 30-Year Story of Performance Assessment

• Global ratings on less structured/less
  standardized assessments also now shown to
  possess reliability comparable to OSCEs, for
  similar testing times.

40
A 30-Year Story of Performance Assessment

• Conclusions
• Sampling across cases, not objectivity provided
  by checklists, is the main contributor to better
  reliability (accuracy) of scores
• Global (professional) judgements, in contrast to
  checklist scores, seem to be more reliable.
• Reliable, formal assessment in real clinical
  situations is both achievable and desirable.

41
Observing Trainees in Real Patient Encounters

• Is a more authentic method of assessing many
  components of clinical competence
• Should be included as a component of the clinical
  curriculum
• Is an addition to, not a substitute for
  examinations (e.g., OSCEs)
• Can employ short and long case formats, or a
  Mini-CEX

42
Mini-Clinical Evaluation Exercise (Mini-CEX)

• 10-20 minute exercise
• Observation of a trainee performing a specific
  task with a patient
• Accompanied by a rating form eliciting judgements
  re history, physical, communication,
  diagnostic/management conclusions, professional
  behaviour,
• Designed to contribute to summative assessment
  (think of as an OSCE station) and to provide
  immediate, specific feedback to trainee

43
UBC Department of Medicine Mini CEX Assessment
Form   EVALUATOR DATE __RESIDENT
____________________ PATIENT PROBLEM/DX
AGE GENDER PROBLEM COMPLEXITY
LOW MODERATE HIGH   1. MEDICAL
INTERVIEWING SKILLS ( Not Observed) 1 2 3
4 5 6 7 8 9
Unsatisfactory Satisfactory
Superior 2. PHYSICAL EXAMINATION SKILLS
( Not Observed)  1 2 3 4 5 6 7 8
9 Unsatisfactory
Satisfactory Superior  3.
PROFESSIONALISM/ HUMANISTIC QUALITIES  1 2 3 4 5
6 7 8 9  4. CLINICAL
JUDGMENT ( Not Observed)  1 2 3 4 5 6 7 8
9 Unsatisfactory
Satisfactory Superior  5.
ORGANIZATION/EFFICIENCY  1 2 3 4 5 6 7 8
9 Unsatisfactory
Satisfactory Superior  OVERALL
CLINICAL COMPETENCE  1 2 3 4 5 6 7 8
9 Unsatisfactory
Satisfactory
Superior  Grade the residents performance Fail B
orderline Pass
44
Reliability of the Mini-CEX Overall Clinical
Competence Ratings
45
Reliability of ITER Average Composite Ratings
46
Mini-CEX Study Results

• Correlations between
• Mini-CEX and MCQ scores .335
• Mini-CEX and OSCE scores .372
• Mini-CEX and ITER scores .724
• Mini-CEX and 360 (overall) .570

47
Conclusions For a performance assessment system
to effectively assess clinical competence, it
should

• Include observations and ratings of performance
  with real patients authentic assessment
• Be based on an adequate sample of performance
  case specificity
• Employ several different methods (OSCE, ITER,
  long case, mini-CEX, ) do not rely on one
  method only

48
Summary In assessing clinical competence, What
are some key themes?

• Test application, not recall
• Focus on key features, not thoroughness
• Present authentic clinical challenges (real
  data, clinical decision-making)
• Use discriminating question formats
• Ensure adequate sampling
• Observe performance with real patients
• Employ multiple methods

49
(No Transcript)