Judgment analysis: a method for quantitative evaluation of trainee surgeons judgments of operative r

1
Judgment analysis a method for quantitative
evaluation of trainee surgeons judgments of
operative risk

• R Jacklin1, N Sevdalis2, C Harries3, A Darzi1, C
  Vincent1
• 1Imperial College London, 2Imperial College
  London and the National Patient Safety Agency,
  3University College London
• r.jacklin_at_imperial.ac.uk

2

• A good surgeon knows how to operate a better
  one knows when to operate and an excellent one
  knows when not to operate

3
Outline

• Background
• To studying surgical judgment
• To judgment analysis methodology
• Research question
• Methods
• Application of judgment analysis
• Results
• Conclusions

4
(No Transcript)
5
4 key attributes of (preoperative) surgical
judgement

• Accurate risk estimation
• Handling of probabilistic information
• Management of uncertainty
• Ability to integrate this information with the
  patients preferences

6
Judgment analysis - background

• Egon Brunswik
• The environment is an uncertain, probabilistic
  one
• "probabilistic functionalism"

7
The Lens Model

• is used in several fields to study how observers
  correctly and incorrectly use objective cues to
  perceive physical or social reality1

• cues have a probabilistic (uncertain) relation to
  the actual objects
• one cue can signal several objects in the
  environment
• cues are (often) redundant

1Judgment Analysis. Theory, methods and
applications. Ray W Cooksey. Academic Press Inc,
California USA 1996
8
Previous use in surgery

• Used to evaluate prioritisation decisions1,2
• No gold standard
• Practice variation explained by differential
  weighting of criteria
• 1Judgment Analysis of Surgeons Prioritisation of
  Patients for Elective General Surgery. MacCormick
  et al, Med Decis Making 2006 26 255
• 2Urgency and priority for cardiac surgery a
  clinical judgment analysis. Kee et al, BMJ 1998
  316 925-929

9
The Lens Model
10
Clinical example risk of conversion to open
during laparoscopic cholecystectomy

• Common
• Practice variation
• Risk given as a percentage during consent process
• Consent taken by junior surgeons
• Risk factors for conversion known

11

Aim/research question

• Is judgment analysis a potential methodology for
  assessment of expertise at estimating surgical
  risk?

12
Methods 1. Designing the cases

• Attempt to preserve some ecological validity
• Format of case presentation
• Representative cases
• Selection of cues to incorporate
• Relevant to judgments identified from literature
• Available on documentation

13
2. Cues selected

• Biliary history
• Previous abdominal surgery
• Age/comorbidity
• Race
• Sex
• Obesity

14
3. Overall task design

• Minimum of 5-10 cases per cue1
• Orthogonal design using SPSS 64 cases
• 20 cases selected at random to be repeated
• Repeats mixed through the design cases
• Task time estimated to be one hour

1Judgment Analysis. Theory, methods and
applications. Ray W Cooksey. Academic Press Inc,
California USA 1996
15
Sample case material
16
4. Gold standard

• Derived from A risk score for conversion from
  laparoscopic to open cholecystectomy
• Kama et al AJS 181(2001) 520 525
• Each case scored from chart
• Regression procedure applied to scores

17
5. Participants

• Initially single intermediate group
• Selection criteria
• Ethical approval
• NHS RD approval

18
Results

• Consistency (Spearmans rho)
• Mean 0.77
• Range 0.47-0.98
• Model fit (adjusted R2) Mean 0.53 Range
  0.12-0.76
• Comparison to gold standard (Spearmans rho)
• Mean 0.48
• Range 0.080.72

19
Cue usage by Surgeon A
20
Cue usage by Surgeon B
21
Cue usage across the group
22
Conclusions

• Method allows detailed quantitative evaluation of
  judgments of risk
• Consistency
• Influence of different variables (cues)
• Cue usage very variable between individuals
• Construct validity?
• Potential for formative feedback from display of
  results

• Time consuming
• Narrow focus
• Gold standard
• Modelling the cases
• Transferability of models

23
Thank you