Title: Refinement and Validation of the AHRQ Patient Safety Indicators
1Refinement and Validation of the AHRQ Patient
Safety Indicators
- Developed by UC-Stanford Evidence Based Practice
Center - Funded by the Agency for Healthcare Research and
Quality
EPC Team (PSI Development) PI Kathryn McDonald,
M.M., Stanford Patrick Romano, M.D., M.P.H, UC
Davis Jeffrey Geppert, J.D., Ed.M.,
Stanford Sheryl Davies, M.A., Stanford Bradford
Duncan, M.D., M.A., Stanford Kaveh G. Shojania,
M.D., UCSF
Support of Quality Indicators PI Kathryn
McDonald, M.M., Stanford Patrick Romano, M.D.,
M.P.H, UC Davis Jeffrey Geppert, J.D. Ed.M.,
Stanford Sheryl Davies, M.A., Stanford Mark
Gritz, PhD, Battelle Greg Hubert,
Battelle Denise Remus, Ph.D., RN, AHRQ
2Acknowledgments
- Funded by AHRQ
- Contract No. 290-97-0013
- Support of Quality Indicators Contract No.
290-02-0007 - Data used for analyses
- Nationwide Inpatient Sample (NIS), 1995-2000.
Healthcare Cost and Utilization Project (HCUP),
Agency for Healthcare Research and Quality - State Inpatient Databases (SID), 1997 (19
states). Healthcare Cost and Utilization Project
(HCUP), Agency for Healthcare Research and
Quality -
- For more information
- http//www.qualityindicators.ahrq.gov
3Acknowledgments
- We gratefully acknowledge the data organizations
in participating states that contributed data to
HCUP and that we used in this study the Arizona
Department of Health Services California Office
of Statewide Health and Development Colorado
Health and Hospital Association CHIME, Inc.
(Connecticut) Florida Agency for Health Care
Administration Georgia Hospital Association
Hawaii Health Information Corporation Illinois
Health Care Cost Containment Council Iowa
Hospital Association Kansas Hospital
Association Maryland Health Services Cost Review
Commission Massachusetts Division of Health Care
Finance and Policy Missouri Hospital Industry
Data Institute New Jersey Department of Health
and Senior Services New York State Department of
Health Oregon Association of Hospitals and
Health Systems Pennsylvania Health Care Cost
Containment Council South Carolina State Budget
and Control Board Tennessee Hospital
Association Utah Department of Health
Washington State Department of Health and
Wisconsin Department of Health and Family Service.
4Rationale for the PSIs
- Background Perceived need for an inexpensive
patient safety surveillance system based on
readily available data - UC-Stanford EPC charge To review and improve the
evidence base related to potential patient safety
indicators (PSIs) that can be ascertained from
data elements in a standardized, multi-state
health data system, the Healthcare Cost and
Utilization Project (HCUP).
5Literature review to find candidate indicators
- MEDLINE/EMBASE search guided by medical
librarians at Stanford and NCPCRD (UK) - Few examples described in peer reviewed journals
- Iezzoni et al.s Complications Screening Program
(CSP) - Miller et al.s Patient Safety Indicators
- Review of ICD-9-CM code book
- Codes from above sources were grouped into
clinically coherent indicators with appropriate
denominators
6Structure of indicators
- All definitions were created using ICD-9-CM
diagnosis and procedure codes (along with DRG,
MDC, sex, age and procedure dates) - Numerator of each indicator is the number of
cases with the complication of interest (e.g.,
Postop DVT/PE) - Denominator of each indicator is the number of
hospitalizations (or patients) considered to be
at risk (e.g. elective surgical patients) - Exclusions were defined to restrict the
denominator to patients for whom the complication
was less likely to have been present at
admission, and more likely to have been
preventable - The indicator rate is the numerator/denominator
7PSI assessment methods
- Literature review to gather data on coding and
construct validity - ICD-9-CM coding consultant review (face validity)
- Clinical panel review (face validity)
- Empirical analyses of nationwide rates, hospital
variation, impact of risk adjustment, and
relationships among indicators
8Clinical panel review
- Intended to establish consensual validity
- Modified RAND/UCLA Appropriateness Method
- Physicians of various specialties/subspecialties,
nurses, other specialized professionals (e.g.,
midwife, pharmacist) - Potential indicators were rated by 8
multispecialty panels surgical indicators were
also rated by 3 surgical panels - All panelists rated all assigned indicators (1-9)
on - Overall usefulness
- Likelihood of identifying the occurrence of an
adverse event or complication (i.e., not present
at admission) - Likelihood of being preventable (i.e., not an
expected result of underlying conditions) - Likelihood of being due to medical error or
negligence (i.e., not just lack of ideal or
perfect care) - Likelihood of being clearly charted in the
medical record - Extent to which indicator is subject to bias due
to case mix
9Evaluation framework
Medical error and complications continuum
Medical error
Nonpreventable Complications
- Pre-conference ratings and comments/suggestions
- Individual ratings returned to panelists with
distribution of ratings and other panelists
comments/suggestions - Telephone conference call moderated by PI and
attended by note-taker, focusing on
high-variability items and panelists suggestions
(90-120 mins) - Suggestions adopted only by consensus
- Post-conference ratings and comments/ suggestions
10Final selection of indicators
- Retained indicators for which overall
usefulness rating was Acceptable or
Acceptable- - Median score 7-9
- Definite or indeterminate agreement
- Excluded indicators rated Unclear, Unclear-,
or Unacceptable - Median score lt7, OR
- At least 2 panelists rated the indicator in each
of the extreme 3-point ranges
11PSIs reviewed
- 48 indicators reviewed in total
- 37 reviewed by multispecialty panel
- 15 of those reviewed by surgical panel
- 20 accepted based on face validity
- 2 dropped due to operational concerns
- 17 experimental or promising indicators
- 11 rejected
12Accepted PSIs
- Selected postoperative complications
- Postoperative thromboembolism
- Postoperative respiratory failure
- Postoperative sepsis
- Postoperative physiologic and metabolic
derangements - Postoperative abdominopelvic wound dehiscence
- Postoperative hip fracture
- Postoperative hemorrhage or hematoma
- Selected technical adverse events
- Decubitus ulcer
- Selected infections due to medical care
- Technical difficulty with procedures
- Iatrogenic pneumothorax
- Accidental puncture or laceration
- Foreign body left in during procedure
- Other
- Complications of anesthesia
- Death in low mortality DRGs
- Failure to rescue
- Transfusion reaction
- Obstetric trauma and birth trauma
- Birth trauma injury to neonate
- Obstetric trauma vaginal delivery with
instrument - Obstetric trauma vaginal delivery without
instrument - Obstetric trauma cesarean section delivery
13National trends in PSI rates
- Nationwide Inpatient Sample (NIS), 1995-2000
- 7.5 million discharges/1,000 hospitals/28 States
- Approximates 20 sample of nonfederal acute care
hospitals - Discharge level weights applied to generate
national estimates for each year - Adjusted for age, gender, age-gender
inter-actions, comorbidities, and DRG clusters - 1,121,000 potential safety-related events
affecting 1,070,000 hospitalizations
14Estimated cases in 2000
Indicator Frequency 95 CI Rate per 100
Postoperative septicemia 14,055 1060 1.091
Postoperative thromboembolism 75,811 4,156 0.919
Postoperative respiratory failure 12,842 938 0.359
Postoperative physiologic or metabolic derangement 4,003 419 0.089
Decubitus ulcer 201,459 10,104 2.130
Infection due to medical care 54,490 2,658 0.193
Postoperative hip fracture 5,207 327 0.080
Accidental puncture or laceration 89,348 5,669 0.324
Iatrogenic pneumothorax 19,397 1,025 0.067
Postoperative hemorrhage/hematoma 17,014 968 0.206
15Impact of patient safety events in 2000 (Zhan and
Miller, JAMA 2003)
Indicator Excess LOS (days) Excess charge ()
Postoperative septicemia 10.9 57,700
Postoperative thromboembolism 5.4 21,700
Postoperative respiratory failure 9.1 53,500
Postoperative physiologic or metabolic derangement 8.9 54,800
Decubitus ulcer 4.0 10,800
Selected infections due to medical care 9.6 38,700
Postoperative hip fracture 5.2 13,400
Accidental puncture or laceration 1.3 8,300
Iatrogenic pneumothorax 4.4 17,300
Postoperative hemorrhage/hematoma 3.9 21,400
16Estimated cases in 2000
Indicator Frequency 95 CI Rate per 100
Birth trauma 27,035 5,674 0.667
Obstetric trauma cesarean 5,523 597 0.593
Obstetric trauma - vaginal without instrumentation 249,243 12,570 8.659
Obstetric trauma - vaginal w instrumentation 60,622 3,104 24.408
Failure to rescue 267,541 5,056 17.424
Postoperative abdominopelvic wound dehiscence 3,858 289 0.193
Transfusion reaction 138 49 0.0004
Complications of anesthesia 5,305 455 0.056
Death in low mortality DRGs 5,912 433 0.043
Foreign body left during procedure 2,710 204 0.008
17Impact of patient safety events in 2000 (Zhan and
Miller, JAMA 2003)
Indicator Excess LOS (days) Excess charge ()
Birth trauma -0.1 (NS) 300 (NS)
Obstetric trauma cesarean 0.4 2,700
Obstetric trauma - vaginal without instrumentation 0.05 -100 (NS)
Obstetric trauma - vaginal w instrumentation 0.07 220
Postoperative abdominopelvic wound dehiscence 9.4 40,300
Transfusion reaction 3.4 (NS) 18,900 (NS)
Complications of anesthesia 0.2 (NS) 1,600
Foreign body left during procedure 2.1 13,300
18National trends 1995-2000
Romano, PS, Geppert, JJ, Davies, SM, Miller, M et
al. A National Profile of Patient Safety in US
Hospitals Based on Administrative Data, Health
Affairs 200322(2)154-166.
19National trends 1995-2000
Romano, PS, Geppert, JJ, Davies, SM, Miller, M et
al. A National Profile of Patient Safety in US
Hospitals Based on Administrative Data, Health
Affairs 200322(2)154-166.
20National trends 1995-2000
Romano, PS, Geppert, JJ, Davies, SM, Miller, M et
al. A National Profile of Patient Safety in US
Hospitals Based on Administrative Data, Health
Affairs 200322(2)154-166.
21National trends 1995-2000
Romano, PS, Geppert, JJ, Davies, SM, Miller, M et
al. A National Profile of Patient Safety in US
Hospitals Based on Administrative Data, Health
Affairs 200322(2)154-166.
22National trends 1995-2000
Romano, PS, Geppert, JJ, Davies, SM, Miller, M et
al. A National Profile of Patient Safety in US
Hospitals Based on Administrative Data, Health
Affairs 200322(2)154-166.
23Research/Policy Question
- Why are some PSIs increasing in incidence over
time while others are decreasing? - Selective changes in coding practice
- Changes in severity of illness or underlying risk
of potential safety-related events - True changes in quality due to technical
improvements in surgical or nursing technique,
counterbalanced by inadequate staffing to
prevent some complications
24Standard deviation of hospital effects 1997 SID
25Ratio of hospital-level signal to total hospital
variation 1997 SID
26Year-to-year correlation of hospital effects
1996-97 Florida SID
27Risk adjustment methods
- Must use only administrative data
- APR-DRGs and other canned packages may adjust for
complications - Final model
- DRGs (complication DRGs aggregated)
- Modified Comorbidity Index based on list
developed by Elixhauser et al. - Age, Sex, Age-Sex interactions
28Hospital level variationImpact of bias, 1997
SID (summary)
High Bias Medium Bias Low Bias
Failure to rescue (44 change 2 deciles) Postop respiratory failure (11) Postop abdominopelvic wound dehiscence (4)
Accidental puncture or laceration (24) Postop hip fracture (8) Obstetric trauma cesarean birth (2)
Decubitus ulcer (26) Iatrogenic pneumothorax (14) Postop hemorrhage or hematoma (4)
Postop thromboembolism (14) Postop physio/metabolic derangement (5) Complications of anesthesia (lt1)
Death in low mortality DRGs (13) Obstetric trauma vaginal birth with instrumentation (5) Obstetric trauma vaginal birth without Instrumentation (lt1)
Postop sepsis (11) Selected infections due to medical care (10) Birth trauma (0)
29PSIs loading on catheter-related and technical
complications (factor 1)
30PSIs loading on post/intraoperative
complications (factor 2)
31PSIs loading on neither factor (lt1 variance
explained)
32Conclusions
- Administrative data are appealing, but the
development of indicators is time-consuming - Variations across hospitals and over time merit
further exploration - Potentially useful screening tool for providers,
provider associations, and health data agencies
to identify possible safety problems - Ongoing support and validation work expected to
offer many more insights into opportunities and
obstacles in using administrative data for
patient safety surveillance