Title: Structured Assessment Format for Evaluating Operative Reports (SAFE-OR) in General Surgery
1Structured Assessment Format for Evaluating
Operative Reports (SAFE-OR) in General Surgery
- Ashley Vergis1 MD, Lawrence Gillman1 MD, Jason
Park1 MD, Samuel Minor MD FRCSC2, Mark Taylor1 MD
FRCSC - 1University of Manitoba, Winnipeg, Canada
- 2Dalhousie University, Halifax, Nova Scotia,
Canada
2- Medicine is an information intensive and reliant
profession whose practice is dominated by how
well information is processed or reprocessed,
retrieved and communicated
Murphy et al. (1992). Good European Health
Record. Centre for Health Informatics and
Multi-professional Education University College
London 1-62.
3Background
- Operative dictations essential for
- Caregiver communication
- Quality assurance
- Research
- Medico-legal
? Must be comprehensive and concise!
Eichholz et al. (2004). Obstetrics and
Gynecology 103(2) 342-6.
4Background
- Multiple previous attempts to judge the quality
of OR notes - Third party billing criteria
- Comparison to consensus criteria
- Overall quality appears poor
- Critical details often omitted
- Superfluous aspects detailed
5Background
- However
- No validated tools that examine quality of
operative dictations - Consequences
- Cannot currently measure the adequacy of an
operative report - Unclear if operative reports meet their stated
objectives
6Objective
- Evaluate the Structured Assessment Format for
Evaluating Operative Reports (SAFE-OR) in
General Surgery - Construct validity
- Inter-rater agreement
- Internal consistency
7Methods
- Subjects
- University of Manitoba general surgery residents
- Novices PGY 1-3
- Experienced PGY 4-6
8Methods
- Design
- View and dictate a videotaped laparoscopic
sigmoid colectomy - Dictation transcribed
9Methods
- Design
- Evaluated by 2 blinded, independent faculty
general surgeons - SAFE-OR instrument
- Structured Assessment Format
- Adapted from CAGS clinical practice committee
guidelines - Global Quality Rating scale
10Methods
- Structured Assessment Format
- 19 Categorical items (yes/no)
- E.g., Date of Procedure
- 7 Ordinal items (5 point anchored Likert Scale)
- E.g., Findings on Exploration
11Structured Assessment Form
12Methods
- Global Quality Rating Scale
- 5 point anchored Likert scale
- 9 categories
- 1. Readability 6. Clarity
- 2. Operative Indications 7. Knowledge
- 3. Operative Steps 8. Reproducibility
- 4. Operative Findings 9. Overall Quality
- 5. Succinctness
13Global Quality Ratings Scale
14Analysis
- Construct Validity
- Novice and experienced groups
- compared (Independent samples T test)
- Inter-rater Agreement
- Intra-class correlation coefficient (ICC)
- gt0.8 desirable
- Internal Consistency
- Cronbachs alpha
Wanzel, et al. (2002). Current Problems in
Surgery 39(6) 573-659.
15Results Construct Validity
Mean Score Mean Score
Novice (n13) Experienced (n8) p
Structured Assessment (out of 44) 23.3 5.2 34.1 6.0 0.001
Global Quality (out of 45) 25.6 4.7 35.9 7.0 0.006
16Results Inter Rater Reliability Internal
Consistency
Overall ICC (95 CI) Range Cronbachs Alpha
Structured Assessment 0.98 (0.96 - 0.99) 0.73 - 0.90 0.85
Global Quality 0.93 (0.83 - 0.97) 0.65 - 0.91 0.96
Ordinal items, Kappa 0.73 - 1.00 for
categorical items
17Conclusion
- SAFE-OR
- Satisfies construct validity
- Excellent inter-rater reliability
- Excellent internal consistency
18Project Significance
- SAFE-OR is the first tool that assesses the
quality of an operative report in a valid and
reliable manner
19Project Significance
- Means to more objectively measure operative
report characteristics - Assess degree to which reports meet stated
functions (quality assurance) - If unsatisfactory, may employ methods to improve
reporting
20Project Significance
- Potential to develop methods of improving
dictations in surgical education - Implement SAFE-OR for assessment in training
programs - Formative and Summative
- Feedback (self and faculty)
- Global attributes
- Specific details
21Project Significance
- Develop, monitor and refine curriculum for
operative dictation - Improve communication
- ? Implications for patient care
22Future Investigation
- Analyze characteristics of real world operative
dictations - National survey (Cdn) of resident and program
directors training perceptions - Develop and assess specific templates for
operative dictation - Assess computerized dictation templates (WebSMR)
- Edhemovic et al., Annals of Surgical
Oncology, 11(10)941947.
23Acknowledgements
- Royal College of Physicians and Surgeons of
Canada Medical Education Research Grant
24(No Transcript)
25(No Transcript)
26(No Transcript)
27Structured Assessment Format for Evaluating
Operative Reports (SAFE-OR) in General Surgery
- Ashley Vergis1 MD, Lawrence Gillman1 MD, Jason
Park1 MD, Samuel Minor MD FRCSC2, Mark Taylor1 MD
FRCSC - 1University of Manitoba, Winnipeg, Canada
- 2Dalhousie University, Halifax, Nova Scotia,
Canada