Use of REMIND Artificial Intelligence Software for Rapid Assessment of Adherence

1
Use of REMIND Artificial Intelligence Software
for Rapid Assessment of Adherence to Disease
Specific Management Guidelines in Acute Coronary
Syndromes
Ali F. Sonel, MD, C. Bernie Good, MD MPH, Harsha
Rao, MD, Alanna Macioce, BS, Lauren J. Wall, BS,
Radu Stefan Niculescu, PhD, Sahtyakama Sandilya,
PhD, Phan Giang, PhD, Sriram Krishnan, PhD,
Prasad Aloni, MS, MBA, Bharat Rao, PhD
Center for Health Equity Research and Promotion,
VA Pittsburgh Healthcare System and the
Cardiovascular Institute, University of
Pittsburgh Pittsburgh, PA, Siemens Medical
Solutions, USA, Malvern, PA
ABSTRACT Introduction Manual extraction of data
for Quality Improvement is tedious, requiring
significant individual training and careful
attention to the HIPAA Privacy Rule. Automated
chart abstraction is an alternative approach that
saves time and costs. We compared manual chart
abstraction from an electronic medical record (VA
CPRS EMR System) to automated extraction using
the REMIND artificial intelligence software in
327 consecutive patients admitted with unstable
angina or non-ST elevation myocardial infarction.
Methods All patient features required by
ACC/AHA guidelines for determining eligibility
for class I recommendations to use aspirin,
beta-blockers, heparin, glycoprotein IIb/IIIa
receptor antagonists, and ACE inhibitors were
extracted by both methods. Manual extraction was
carried out by well-trained, qualified chart
abstractors with prior experience in manual chart
abstraction. When both extraction results were
identical, the result was assumed correct.
Disagreements were manually adjudicated based on
pre-determined definitions. Results Manual
extraction and data entry required 176 hours
compared to 4 hours using the Siemens REMIND
software. A total of 5232 data elements were
identified, with agreement in 4385 (84) and
disagreement in 847 (16), involving 2.5-35 of
patients for various parameters. REMIND was found
to be correct in 642 disagreements (76) and
manual extraction was correct in the remaining
24 (205). Based on adjudication, REMIND
identified adherence compared very favorably to
manual extracted as well as adjudicated guideline
adherence for aspirin (83 vs. 88 vs. 85), beta
blockers (78 vs.82 vs. 81), heparin (53 vs.
51 vs. 54), glycoprotein IIb/IIIa receptor
antagonists (35 vs. 38 vs. 40) and ACE
inhibitors (69 vs. 78 vs. 76). Conclusions
REMIND can assess disease specific management
guideline adherence at least as accurately as
manual chart abstraction. Use of REMIND for
Quality Improvement and research can result in
significant savings, better resource utilization,
and may improve data extraction quality.
METHODS
RESULTS

• Patient Population
• 327 patients admitted with high-risk
  non-ST-segment elevation myocardial infarction
  were included in the study
• Data Collection
• Records were extracted from VA CPRS Electronic
  Medical Record System
• Manual extraction of predefined variables was
  performed by a trained abstractor with expertise
  in ACS data abstraction for research purposes
• An artificial intelligence model developed by
  Siemens, the REMIND automated data extraction
  tool, was used to extract the same information
  electronically
• Medical information required to determine
  eligibility and the presence of absence of
  contraindications for Class I treatment
  recommendations in the ACC/AHA guidelines was
  collected for the following medications
• Aspirin in all patients
• Beta-blockers in all patients
• Heparin in all patients
• Angiotensin converting enzyme (ACE) inhibitors
  or angiotensin receptor blockers (ARB) in
  patients with diabetes mellitus, congestive heart
  failure, left ventricular dysfunction or
  hypertension
• Glycoprotein IIb/IIIa receptor antagonists in
  patients in whom an early invasive management
  strategy is planned
• Data Analysis
• We compared the results of the two methods for
  accuracy
• When both extraction methods were in agreement,
  the result was assumed to be correct. When
  extracted results differed, disagreements were
  manually adjudicated based on pre-determined
  definitions, using the source documents of each
  extraction method
• Accuracy was defined as the number of patients
  where there was agreement with adjudication as to
  whether the patient was compliant or not, divided
  by the total number of patients in the study
• Compliance is defined as the number of patients
  eligible and not contraindicated to that
  medication, who actually received the medication,
  divided by the number of patients who are
  eligible and have no contraindication to that
  medication.

• Complete data extraction required 176 hours of
  manual extraction, compared to 4.5 hours with
  REMIND automated extraction

Table 3 Accuracy of Compliance Assessment with
REMIND Compared to Manual Extraction
Table 1 Determination of Contraindications and
Eligible Patients for Processes of Care
TREATMENT ACCURACY () N327 ACCURACY () N327
TREATMENT REMIND MANUAL
Aspirin 319 (98) 314 (96)
Beta Blockers 319 (98) 316 (97)
Heparin 315 (96) 296 (91)
ACE Inhibitors/ARB 300 (92) 310 (95)
Glycoprotein IIb/IIIa Receptor Antagonists 300 (92) 290 (89)
TREATMENT (N327) Patients with Contraindications for Processes of Care () Patients with Contraindications for Processes of Care () Patients with Contraindications for Processes of Care () Ideal Patients for Processes of Care () Ideal Patients for Processes of Care () Ideal Patients for Processes of Care ()
TREATMENT (N327) REMIND MANUAL ADJUDICATED REMIND MANUAL ADJUDICATED
Aspirin 82 (25) 77 (24) 86 (26) 245 (75) 250 (77) 241 (74)
Beta Blockers 229 (70) 186 (57) 233 (71) 98 (30) 141 (43) 94 (29)
Heparin 82 (25) 77 (24) 86 (26) 245 (75) 250 (77) 241 (74)
ACE Inhibitors/ARB 142 (43) 128 (39) 152 (46) 146 (45) 125 (38) 125 (38)
Glycoprotein IIb/IIIa Receptor Antagonists 99 (30) 93 (28) 106 (32) 103 (32) 79 (24) 86 (26)
BACKGROUND

• Research and quality improvement projects involve
  large amounts of data collection through review
  of medical records
• Manual data collection requires a significant
  amount of training and is time consuming
• Automated data extraction methods could save time
  and improve resource utilization
• Little is known about the accuracy of automated
  systems for record extraction

Accuracy defined as true positives plus true
negatives divided by the total number of patients
CONCLUSION

• REMIND can determine ACC/AHA guideline adherence
  for non-ST-elevation acute coronary syndromes at
  least as accurately as manual chart abstraction.

Table 2 Assessment of Compliance with Guideline
Recommended Therapies
TREATMENT COMPLIANCE BY REMIND EXTRACTION () COMPLIANCE BY MANUAL EXTRACTION () COMPLIANCE FOLLOWING ADJUDICATION ()
Aspirin 202/245 (83) 220/250 (88) 206/241 (85)
Beta Blockers 76/98 (78) 116/141 (82) 76/94 (81)
Heparin 131/245 (53) 127/250 (51) 131/241 (54)
ACE Inhibitors/ARB 101/146 (69) 98/125 (78) 95/125 (76)
Glycoprotein IIb/IIIa Receptor Antagonists 36/103 (35) 30/79 (38) 34/86 (40)
IMPLICATIONS

• Use of REMIND for quality improvement and
  research related applications in facilities with
  electronic medical records can result in
  significant savings and better resource
  utilization.
• Use of REMIND can enable evaluation of very large
  sets of medical information that would otherwise
  be impractical by manual extraction

SPECIFIC AIMS

• Compare the accuracy of data collection in a
  large and complex medical record set using manual
  extraction and REMIND automated extraction tool
• Compare the level of adherence to ACC/AHA
  guideline recommendations for treatment of non-ST
  elevation acute coronary syndromes (ACS) using
  manual extraction and REMIND automated data
  extraction tool