Clinical Decision Support Systems

1
Clinical Decision Support Systems

• Syed Tirmizi, M.D.
• Medical Informatician
• Veterans Health Administration

2
Clinical Decision Support Systems

• Definition (What)
• Business case (Why)
• Use Cases (How)
• Usability testing Evaluations

3
Decision Support Systems

• Decision support systems are a class of
  computer-based information systems including
  knowledge based systems that support decision
  making activities.
• -Wikipedia

4
Decision Support Systems

• A passive DSS is a system that aids the process
  of decision making, but that cannot bring out
  explicit decision suggestions or solutions.
• An active DSS can bring out such decision
  suggestions or solutions.
• A cooperative DSS allows the decision maker (or
  its advisor) to modify, complete, or refine the
  decision suggestions provided by the system,
  before sending them back to the system for
  validation.
• 
  Haettenschwiler

5
Clinical Decision Support Systems

• computer software employing a knowledge base
  designed for use by a clinician involved in
  patient care, as a direct aid to clinical
  decision making
• a set of knowledge-based tools that are fully
  integrated with both the clinician workflow
  components of a computerized patient record, and
  a repository of complete and accurate data
• providing clinicians or patients with clinical
  knowledge and patient-related information,
  intelligently filtered and presented at
  appropriate times, to enhance patient care
• Clinical Decision Support in Electronic
  Prescribing Recommendations and an Action Plan
• Report of the Joint Clinical Decision Support
  Workgroup
• JONATHAN M. TEICH, MD, PHD, JEROME A. OSHEROFF,
  MD, ERIC A. PIFER, MD, DEAN F.SITTIG, PHD,
• ROBERT A. JENDERS, MD, MS, THE CDS EXPERT REVIEW
  PANEL
• J Am Med Inform Assoc. 200512365376.

6
Patient Safety Quality Gaps Acknowledged

• Virtually Every Patient Experiences a Gap
  Between the Best Evidence and the Care They
  Receive
• (IOM, 2001)

• 98,000 Hospital Patients Die Yearly Because of
  Adverse Events
• (IOM, 1999)

7
Outpatient Adverse Drug Events

• Overall
• 25 of outpatients incurred an ADE
• 39 were preventable
• Antidepressants and antihypertensives were
  largest contributors
• Elderly (over 65)
• Adverse Events in 5 of population per year
• 28 preventable

Gandhi et al, NEJM 2003348(16)1556-1564
Gurwitz et al, JAMA 20032891107-16
8
Chances of Receiving Appropriate Preventive Care
is about 50 -NEJM
9
Employer/Payor business case for CDS - Diabetes

• Estimated avg 21,000/year per diabetic employee
  in absenteeism, disability and medical costs
  (study of 6 employers with 375,000 employees
• Glycemic control is associated with 1000-2000
  medical costs savings/year to payor
• Currently, we are reimbursed to measure HgA1c
  annually (captured claim for test ordered)
• Will soon be reimbursed for maintaining control
  through test result surveillance, goal is lt 7
• Tonya Hongsermeier, MD, MBA
• Partners Healthcare Systems

10
Knowledge Processing Required for Care Delivery

• Medical literature doubling every 19 years
• Doubles every 22 months for AIDS care
• 2 Million facts needed to practice
• Genomics, Personalized Medicine will increase the
  problem exponentially
• Typical drug order today with decision support
  accounts for, at best, Age, Weight, Height, Labs,
  Other Active Meds, Allergies, Diagnoses
• Today, there are 3000 molecular diagnostic tests
  on the market, typical HIT systems cannot support
  complex, multi-hierarchical chaining clinical
  decision support

Covell DG, Uman GC, Manning PR. Ann Intern Med.
1985 Oct103(4)596-9
11
Drilling for the Best Information
12
Links Reminder
With Actions
With Documentation
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
Suggest Use of Thiazide

• Set up the reminder dialog so that if the patient
  is a reasonable candidate for a thiazide and not
  currently on one, then suggest use of a thiazide.
• Suppressed by Crgt2.0, Calciumgt10.2, Nalt136 or
  allergy.

23
Standard HTN dialog copied from the national
reminder
24
Insert section at the top if the patient is a
candidate for use of a thiazide
25
(No Transcript)
26
Clinical Reminders Performance Measures

• Clinical Reminders
• Real time decision support
• Targeted to specific patient cohort
• Targeted to specific clinic/clinicians
• Reminder Dialogs
• Standard documentation
• Capture of data (HF, encounter data, etc)
• Reminder Reports
• Performance improvement/scheduled feedback
• Identification of best practices
• Targeting low scorers for educational
  intervention
• Patient recall if missed intervention

27
Clinical Reminder Reports

• Multiple Uses for Reminder Reports
• Patient care
• Future Appointments
• Which patients need an intervention?
• Past Visits
• Which patients missed an intervention?
• Action Lists
• Inpatients
• Which patients need an intervention prior to
  discharge?

28
Clinical Reminder Reports

• Identify patients for case management
• Diabetic patients with poor control
• Identify patients with incomplete problem lists
• Patients with () Hep C test but no PL entry
• Identify high risk patients
• on warfarin, amiodarone
• Track annual PPD due (Employee Health)

29
Clinical Reminder Reports

• Quality Improvement
• Provide feedback (team/provider)
• Identify ( share) best practices
• Identify under-performers (develop action plan)
• Track performance
• Implementation of new reminders or new processes
• Identify process issues early (mismatch of
  workload growth versus staffing)
• Provide data for external review (JCAHO)

30
Clinical Reminder Reports

• Management Tool
• Aggregate reports
• Facility / Service
• Team (primary care team)
• Clinic / Ward
• Provider-specific reports
• Primary Care Provider
• Encounter location
• If one provider per clinic location

31
Reminder/Dialogs Other Uses

• Examples Reminder dialogs linked to note title
• Present ordering dialogs
• Medications Orders
• Sildenafil/levitra (screening for risk factors)
• Clopidogrel (Plavix) (updated criteria)
• Discharge Order
• Support medication reconciliation (when
  pharmacists are not available to review meds)
• Gather information for display on Health Summary
• Non VA surgery

32
Computerized Patient Record System CPRS

• Improve healthcare outcomes
• Translate Clinical Practice Guidelines into
  clinical activities
• Real time decision support for clinicians at
  point of care reminders, alerts
• Prevent patient from falling through the cracks
• Avoid reliance on memory, vigilance
• Reduce errors (omissions, transcriptions, etc)
• Facilitate documentation for performance
  measurement and improvement efforts

33
However

• This is NOT about technology
• It is about RESULTS
• Improved Health Care Quality
• Improved Health Outcomes

34
How Do We Compare to non-VA Providers? VHA
Continues to exceed HEDIS in the vast majority of
17 common measures
CLINICAL PERFORMANCE INDICATOR VA FY 05 HEDIS Commercial 2004 HEDIS Medicare 2004 HEDIS Medicaid 2004
Breast cancer screening 86 73 74 54
Cervical cancer screening 92 81 Not Reported 65
Colorectal cancer screening 76 49 53 Not Reported
LDL Cholesterol lt 100 after AMI, PTCA, CABG Not Reported 51 54 29
LDL Cholesterol lt 130 after AMI, PTCA, CABG Not Reported 68 70 41
Beta blocker on discharge after AMI 98 96 94 85
Hypertension BP lt 140/90 most recent visit 77 67 65 61
Follow-up after Hospitalization for Mental Illness (30 days) 70 76 61 55
HEDIS Health Plan Employer Data Information
Set From the National Committee on Quality
Assurance (NCQA)
35
How Do We Compare to non-VA Providers? VHA
Continues to exceed HEDIS in the vast majority of
17 common measures
CLINICAL PERFORMANCE INDICATOR VA FY 05 HEDIS Commercial 2004 HEDIS Medicare 2004 HEDIS Medicaid 2004
Diabetes HgbA1c done past year 96 87 89 76
Diabetes Poor control HbA1c gt 9.0 (lower is better) 17 31 23 49
Diabetes Cholesterol (LDL-C) Screening 95 91 94 80
Diabetes Cholesterol (LDL-C) controlled (lt100) 60 40 48 31
Diabetes Cholesterol (LDL-C) controlled (lt130) 82 65 71 51
Diabetes Eye Exam 79 51 67 45
Diabetes Renal Exam 66 52 59 47
CLINICAL PERFORMANCE INDICATOR VA FY 2005 HEDIS Commercial 2004 HEDIS Medicare 2004 BRFSS 2004
Immunizations influenza, (note patients age groups) 75 (65 and older or high risk) 39 (50-64) 75 (65 and older) 68 (65 and older)
Immunizations pneumococcal, (note patients age groups) 89 (all ages at risk) Not Reported Not Reported 65 (65 and older)
36
FY99-04 Changes in Total, Major and Minor
Age-Adjusted Amputation Rates Among Patients With
Diabetes
37
Pneumococcal Vaccination Rates in VHA
--BRFSS 90th--
--BRFSS--

• Iowa Petersen, Med Care 199937502-9. gt65/ch dz
• HHS National Health Interview Survey, gt64

38
Performance has improved
39
(No Transcript)
40
Changed to include refusals as failures
41
Outcomes have improved

• Increased rates of pneumococcal vaccination over
  past 5 years has averted over 4000 deaths
  nationally in VA patients with lung disease
• Diabetic complications markedly decreased
  amputations, peripheral neuropathy, visual
  impairment and loss

42
The Chronic Disease Care Model
Health System
Community
Resources and Policies
Organization of Health Care
Self-Management Support
VistA
DeliverySystem Design
Decision Support
Productive Interactions
Patient- Centered Coordinated
Timely and Efficient Evidence-based and Safe
Informed, Empowered Patient and Family
Prepared, Proactive Practice Team
My HealtheVet
Improved Outcomes
43
Highest Quality of Care For Patients with
Diabetes in VA

• Diabetes processes of care and 2 of 3
  intermediate outcomes were better for patients in
  the VA system than for patients in commercial
  managed care.
• Annals of Internal Medicine, August 17, 2004

44
Highest Quality of Care For Patients in VA
Measured Broadly

• Patients from the VHA received higher-quality
  care according to a broad measure. Differences
  were greatest in areas where the VHA has
  established performance measures and actively
  monitors performance.
• Annals of Internal Medicine, December 21, 2004

45
Guideline-Based Decision Support for Hypertension
with ATHENA DSS

• Implementation
• Evaluation
• Mary K. Goldstein, MD

46
Developing a Model Program

• To Provide a Model Program that can be extended
  to other clinical areas
• They selected hypertension as a model for
  guideline implementation because
• Hypertension is highly prevalent in adult medical
  practice
• There are excellent evidence-based guidelines for
  management
• There is also evidence that the guidelines are
  not well-followed
• a big improvability gap in IOM terms
• Steinman, M.A., M.A. Fischer, M.G. Shlipak, H.B.
  Bosworth, E.Z. Oddone, B.B. Hoffman and M.K.
  Goldstein, Are Clinicians Aware of Their
  Adherence to Hypertension Guidelines? Amer J.
  Medicine 117747-54, 2004.

47
What the Clinician Sees
48
ATHENA Hypertension AdvisoryBP- Prescription
Graphs
Goldstein, M. K. and B. B. Hoffman (2003).
Graphical Displays to Improve Guideline-Based
Therapy of Hypertension. Hypertension Primer. J.
L. Izzo, Jr and H. R. Black. Baltimore, Williams
Wilkins.
49
ATHENA HTN Advisory
BP targets
Primary recommendation
Drug recommendation
50
ATHENA HTN Advisory More Info
51
What is ATHENA DSS?

• Automated decision support system (DSS)
• Knowledge-based system automating guidelines
• Built with EON technology
• For patients with primary hypertension who meet
  eligibility criteria
• Patient specific information and recommendations
  at the point of care
• Purpose is to improve hypertension control and
  prescription concordance with guidelines

• Athena in Greek mythology is a symbol of good
  counsel, prudent restraint, and practical insight
• Proc AMIA 2000

52
ATHENA Protégé top level
53
ATHENA Protégé GL managementdiagram
54
Building ATHENA System From EON Components
VISTA
EON Servers
SQL Patient Database
VA CPRS
ATHENA Clients
Temporal Mediator
ATHENA Clients
Event Monitor
Event Monitor
Pre- computed Advisories
Guideline Interpreter
Data Converter
Advisory Client
Advisory Client
nightly data extraction
ATHENA HTN Guideline Knowledge Base
Protégé
ATHENA GUI
55
Path to Guideline Adherence

• The theoretical model we use for the path to
  guideline adherence is the Awareness to
  Adherence model, in which the clinician must
• Awareness of guideline
• Acceptance of guideline
• Adoption of guideline
• Adherence to guideline
• Pathman, D. E., T. R. Konard, et al. (1996).
  "The Awareness-to-Adherence Model of the Steps to
  Clinical Guideline Compliance." Medical Care
  34873-889.

56
Informatics Support for Clinical Practice
Guideline Implementation
Step Facilitators Informatics Support
Awareness Priming Activities such as profiling of baseline performance Profiling from pharmacy and diagnosis database
Acceptance Active education such as Academic Detailing Clinical Opinion Leaders Present evidence relevant to patient allow opinion leaders to browse knowledge
Adoption Enabling strategies such as incorporation into clinic workflow Integration with existing EMR
Adherence Reinforcing Strategies such as reminders Point-of-care patient-specific advisories
57
Challenge of Using IT for Quality Improvement

• Technical challenges of using information
  technology for quality improvement (QI)
• Difficult to integrate new forms of decision
  support into legacy data systems and electronic
  record interfaces
• We had many design requirements in order to meet
  research goals and institutional goals
• A sociotechnical challenge to implement
• Goldstein, M., R. Coleman, S. Tu, et. Al.
  Translating Research Into Practice
  SocioTechnical Integration of Automated Decision
  Support for Hypertension in Three Medical
  Centers. JAMIA 11 368-76, 2004.
• Available in pubmedcentral

58
Decision Support for Common Chronic Diseases
The physician often seen as wondering about a
clinical question and then seeking out decision
support
X

• The Field of Dreams approach to
• medical informatics implementations
• If you build it, they will come

59
Some Technical Challenges

• Extracting clinical data from VistA
• Generating a popup window that appears in CPRS
• At the right time, in the right clinic settings,
  for the right clinician, about the right patient
• Logging data about activity in the system
• Security issues

60
Some of the Social Challenges

• Clinicians extremely time-pressured in clinic
• Strike balance between ease of access to system
  and ease of ignoring it
• Enormous variability in comfort with computers
• And virtually no training time available
• Disagreements about the guidelines
• some want VA GLs, some want JNC

61
Taking on the Sociotechnical Challenge

• Aligning with institutional goals
• Discuss with local stakeholders
• VA performance standards and guidelines
• Speaking the language(s)
• understanding that different computer worlds are
  worlds apart
• Identify a bridge person to span the gap between
  IRMS expertise and non-VA programmers
• Iterative Design
• With opportunity for re-design cycles after input
  from key clinical staff
• Dont test in clinic prematurely
• Do your offline testing first
• Test with typical users, not just early adopters
• Recognize need for continual adaptation to our
  evolving informatics infrastructure

62
Evaluation Flowchart
Martins SB et al Proc AMIA 2006 in press
63
Physician Testers in Clinical Setting

• Project-friendly physicians who test the system
  in early stages in clinic
• Understanding it is not yet complete
• Must be prepared to make changes in response to
  their comments
• Some of these physicians become champions for the
  system
• Include clinical managers in early testing

64
Consensus Conference Calls

• Knowledge updates required in light of newly
  published clinical trials or new guidelines
• Need a knowledge management process for vetting
  new material and deciding what will be
  incorporated
• Make this process known to the clinicians who are
  end-users (especially local opinion leaders)
• Invite local input to the discussion
• Encode with a system that allows for easy
  updating
• Goldstein, M.K., B.B. Hoffman, et al,
  Implementing clinical practice guidelines while
  taking account of changing evidence ATHENA DSS,
  An easily modifiable decision-support system for
  managing hypertension in primary care. AMIA
  Symp 300-4, 2000.

65
Ontologies in Clinical Decision Support
Applications

• Health IT has the potential to improve patient
  care by adherence to clinical practice guidelines
• EON and ATHENA projects demonstrate use of
  ontologies in clinical decision support
  applications

66
EON project

• NLM-funded project at Stanford (PI Dr. Musen)
• Develop methodology, ontologies, and software
  components for creating decision-support system
  for guideline-based care
• Use Protégé knowledge-acquisition methodology and
  tool for construction of
• Domain concept ontologies
• Patient information model
• Guideline knowledge bases
• Develop software components that assist
  clinicians in specific tasks

67
ATHENA project

• Funded by VA Research Service HSRD
• Hypothesized that guideline-based interventions
  in management of hypertension can
• Change physicians prescribing behavior
• Change patient outcome
• Deployed and evaluated at primary care VA clinics
  in 9 geographically diverse cities over a
  15-month clinical trial
• Results
• Expert clinicians maintain hypertension knowledge
  base using Protégé
• Clinicians interacted with the ATHENA
  Hypertension Advisory at 54 of all patient
  visits
• Impact on prescribing behavior and patient
  outcome being analyzed

68
Stages in Evaluating Clinical Decision Support
Systems 1

1. Elaborated from Miller RA JAMIA 1996
2. Use Cases

69
Stages in Evaluating Clinical Decision Support
Systems (CDSS)
Goldstein, M.K., et al., Patient Safety in
Guideline-Based Decision Support for Hypertension
Management ATHENA DSS. JAMIA, 2002. 9(6 Suppl)
p. S11-6.
70
Patient Safety in New Health IT

• New computer systems have potential to reduce
  errors
• But also potential to create new opportunities
  for error

71
Errors due to new Health IT

• Studies of accidents have shown that new computer
  systems can affect human problem solving in ways
  that contribute to errors
• data overload
• computer collects and displays information out
  of proportion to human ability to use it
  effectively
• automation surprises
• bar code administration unobservable actions
• Goldstein, M.K., et al., Patient safety in
  guideline-based decision support for hypertension
  management ATHENA DSS. J Am Med Inform Assoc,
  2002. 9(6 Suppl) p. S11-6.

72
Charles Friedman and Jeremy Wyatt
73
Safety Testing Clinical Decision Support Systems

• Before disseminating any biomedical information
  resourcedesigned to influence real-world
  practice decisionscheck that it is safe
• Drug testing in vivo and in vitro
• Information resource safety testing
• how often it furnishes incorrect advice
• Friedman and Wyatt Evaluation Methods
• in Biomedical Informatics 2006

74
Stages in Evaluating Clinical Decision Support
Systems 1

1. Elaborated from Miller RA JAMIA 1996

75
Stages in Evaluating Clinical Decision Support
Systems
Both initially and after updates
After Miller RA JAMIA 1996
76
CDSS to Evaluate ATHENA-HTN
Electronic Medical Record System Patient Data
ATHENA HTN Guideline Knowledge Base

• DSS developed using the EON architecture from
  Stanford Medical Informatics (Musen et al)

Guideline Interpreter/ Execution Engine
SQL Server relational database
77
Knowledge Base

• Protégé ontology editor
• Open source (http//protege.stanford.edu/)
• EON model for practice guidelines
• Focus for evaluation
• Eligibility criteria for including patients
• Drug reasoning for drug recommendations

Tu SW, Musen MA. A Flexible Approach to Guideline
Modeling. Proc AMIA Symp 1999. 420-424
78
Execution Engine

• Applies the guideline as encoded in the
  knowledge base to the patients data
• Generates set of recommendations

Tu SW, Musen MA. Proc AMIA Symp 2000. 863-867
79
Testing the software for accuracy
80
The Art of Software Testing

• False definition of testing
• E.g., Testing is the process of demonstrating
  that errors are not present
• Testing should add value to the program
• improve the quality
• Start with assumption program contains errors
• A valid assumption for almost any program
• Testing is the process of executing a program
  with the intent of finding errors.
• Purpose of testing to find as many errors as
  possible

Myers G, Sandler C, Badgett T, Thomas T. The Art
of Software Testing. 2nd Ed. John Wiley Sons
2004
81
Software Regression Testing

• Software updates and changes are particularly
  error-prone
• Changes may introduce errors into a previously
  well-functioning system
• regress the system
• Desirable to develop a set of test cases with
  known correct output to run in updated systems
  before deployment

82
Stages in Evaluating Clinical Decision Support
Systems
Both initially and after updates
83
Clinical Decision Support System Accuracy Testing
Phases
84
Objectives for this phase of testing

• Test the knowledge base and the execution engine
  after an update to the knowledge base and prior
  to clinical deployment of the updated system
• to detect errors and improve quality of system
• Establish correct output (answers) for set of
  test cases

85
Methods Overview
86
Physician Evaluator (MD)

• Internist with experience in treating
  hypertension in primary care setting
• No previous involvement with ATHENA project
• Studied Rules and clarified any issues
• Had Rules and original guidelines available
  during evaluation of test cases

87
Elements examined

• Patient eligibility
• Did patient meet ATHENA exclusion criteria?
• Drug recommendations
• List of all possible anti-hypertensive drug
  recommendations concordant with guidelines
• Drug dosage increases
• Addition of new drugs
• Drug substitutions
• Comments by MD

88
Comparison Method

• Comparing ATHENA vs MD ouput
• Automated comparison for discrepancies
• Manual review of all cases
• Reviewing discrepancies
• Meeting with physician evaluator
• Adjudication by third party when categorizing
  discrepancies

89
Successful Test

• A successful test is one that finds errors
• so that you can fix them

90
Set of Gold Standard Test Cases

• Iteration between clinician review and system
  output
• Same test cases for bug fixes and elaborations in
  areas that dont affect the answers to test cases
  
• Change gold standard answers to test cases when
  the GL changes
• i.e., when what you previously thought was
  correct is no longer correct

91
Important features of Offline Testing Method

• Challenging CDSS with real patient data
• Clinician not involved in project fresh view

92
Additional observation

• Difficulty of maintaining a separate Rules
  document that describes encoded knowledge

93
Benefits of the Offline Testing

• Offline testing method was successful in
  identifying errors in ATHENAs Knowledge base
• Program boundaries were better defined
• Updates made improving accuracy before deployment
• Gold standard answers to test cases
• Offline Testing of the ATHENA Hypertension
  Decision Support System Knowledge Base to Improve
  the Accuracy of Recommendations.Martins SB, Lai
  S, Tu SW, Shankar R, Hastings SN, Hoffman BB,
  Dipilla N, Goldstein MK. AMIA Annu Symp Proc.
  2006539-43

94
Stages in Evaluating Clinical Decision Support
Systems (CDSS)
After Miller RA JAMIA 1996