Title: Methodologies and Automated Applications for Post-Marketing Outcomes Surveillance of Medical Devices and Medications
1Methodologies and Automated Applications for
Post-Marketing Outcomes Surveillance of Medical
Devices and Medications
- Michael E. Matheny, MD, MS
- NLM Biomedical Informatics Fellow
- Decision Systems Group, Department of Radiology
Brigham Womens Hospital, Boston, MA
2Outline
- Post-Marketing Surveillance Background
- Statistical Methodology Development
- Computer Application Development
- Clinical Examples
- Future Directions
3BackgroundSurveillance Rationale
- Phase 3 Trials insufficient to ensure adequate
safety of medications and devices - Low frequency events are not detected
- Protected populations (pregnant women, children)
and more ill populations not represented - Complications delayed by a number of years cannot
be detected
4BackgroundFDA Medical Devices
- 1,700 types of devices
- 500,000 device models
- 23,000 manufacturers
5BackgroundFDA New Drug Applications (NDA)
6BackgroundCurrent Post-Marketing Surveillance
- Combination of mandatory and voluntary adverse
event reporting - Mandatory reporting by manufacturers and health
facilities - Voluntary MedWatch / MAUDE reports by providers
and patients
2004 Drug-Related Adverse Event Reports 2004 Drug-Related Adverse Event Reports
Total 422,889
Manufacturer Facility Reports 401,396
MedWatch 21,493
7BackgroundMAUDE Cypher Reporting Rate
2003
2004
8BackgroundCurrent Post-Marketing Surveillance
- Phase 4 Trials
- Poor Compliance
- As of March 2006 report, 797 of 1231 (65)
agreed-upon trials had yet to be started - Barriers
- Lack of manufacturer incentives
- Expensive
- Drug already on the market
- Lack of regulatory enforcement
9BackgroundMedical Device Recalls
- Boston Scientific cardiac stent (1998)
- Balloon rupture at low pressures
- Guidant cardio-defibrillator (2005)
- Malfunction due to electrical short
- Vioxx (2004)
- cardiovascular complications
- Tequin (2006)
- Hypoglycemia and hyperglycemia
10BackgroundFDA Response
- Increasing demands for Phase 4 trials
- Legislation to increase quality of adverse event
reporting - Emphasizing trial registries (clinicaltrials.gov)
as way to prevent omission of results - Commissioned IOM report The Future of Drug
Safety
11BackgroundAdverse Event Data Continuum
Phase 3 Trials MAUDE / MedWatch Voluntary Registry Mandatory Registry
Internal Validity --
Generalizibility /- /-
Breadth
Immediacy ---
Lack of Bias --- ---
12Statistical MethodsMedical Outcomes Monitoring
- Using registry data that tracks all patients
allows different types of analysis than used in
the FDAs adverse event reporting systems - No generally accepted methods for monitoring
registry data for adverse events - Lack of sufficient discrete electronic data
sources to construct registries - Some outcomes are challenging to track for an
entire population - Increased variability in subjective medical data
- Physician inter-observer disagreement
13Objective
- Develop methodologies and implement an automated
computer monitoring system to perform outcomes
surveillance of registry data for new medical
devices and medications
14Statistical MethodsStatistical Process Control
15Statistical MethodsBayesian Updating Statistics
16Statistical MethodsEstablishing Baseline Data
- Primary Data Sources
- Phase 3 trial data
- Post-Marketing data from a closely related
medication/device - Alternative Data Sources
17Statistical MethodsEstablishing Alerting
Thresholds
- Use number of events and sample size to calculate
proportion with confidence intervals - Typically, medical domains use 95 CI or 1.96 SD
from the point estimate
18Statistical Methods Establishing Alerting
Thresholds
SPC
BUS
19Statistical Methods Establishing Alerting
Thresholds
20Statistical Methods Establishing Alerting
Thresholds
- Wilsons method of comparison between two
proportions
21Statistical MethodsRisk Stratification
- Allows creating subgroups for separate analyses
- Single variable
- Logistic regression model with scoring thresholds
22Application DevelopmentDELTA
- Data Extraction and Longitudinal Time Analysis
(DELTA) - Design Goals
- Generic data import format
- Allow both prospective and retrospective analyses
- Modular framework to allow sequential addition of
statistical methodologies - Multiple alerting methods
- Any number of concurrent ongoing analyses
23Application DevelopmentDELTA
DELTA Database
Source Database
Clinical Data Entry
Data Dictionary
Web Server
Statistical Modules
Source IT Manager
DELTA Users
24Application Test DataCypher Drug-Eluting Stent
(DES)
- Setting
- Brigham Womens Hospital (07/2003 12/2004)
- Population
- All patients receiving a drug-eluting stent
(2270) - Outcome
- Post-procedural in-hospital mortality (27)
- Baseline
- University of Michigan Data (1997-1999)
25Application DevelopmentDELTA
26Application DevelopmentDELTA
27Application DevelopmentDELTA
28Application DevelopmentDELTA
29Application DevelopmentDELTA
30Application DevelopmentDELTA
31Application DevelopmentDELTA
32Application DevelopmentDELTA
33Risk StratificationPotential Solution
- Incorporate individual risk prediction models in
order to adjust for case mix and illness severity
34Possible Risk Prediction Methods
- Linear / Logistic Regression
- Artificial Neural Networks
- Bayesian Networks
- Support Vector Machines
35LR External Validation Models
Model Dates Location Sample
NY 1992 1991 NY 5827
NY 1997 1991 1994 NY 62670
CC 1997 1993 1994 Cleveland, OH 12985
NNE 1999 1994 1996 NH, ME, MA, VT 15331
MI 2001 1999 2000 Detroit, MI 10796
BWH 2001 1997 1999 Boston, MA 2804
ACC 2002 1998 2000 National 100253
36LR External Validation
- Setting
- Brigham Womens Hospital (01/2002 09/2004)
- Population
- All patients undergoing percutaneous coronary
intervention (5216) - Outcome
- Post-procedural in-hospital mortality (71)
37LR External Validation Results
Curve Deaths AUC HL ?2 HL (p)
NY 1992 96.7 0.82 31.1 lt0.001
NY 1997 61.6 0.88 32.2 lt0.001
CC 1997 78.8 0.88 27.8 lt0.001
NNE 1999 56.2 0.89 45.9 lt0.001
MI 2001 61.8 0.86 30.4 lt0.001
BWH 2001 136.1 0.89 39.7 lt0.001
ACC 2002 49.9 0.90 42.0 lt0.001
BWH 2004 70.5 0.93 7.61 0.473
38LR External Validation Conclusions
- Excellent discrimination across all models
- Calibration (Hosmer-Lemeshow) poor for all models
but recent local one - Addressed categorical risk stratification by
keeping all records in one stratum - Calibration problems over time limit application,
and require exploration of recalibration methods
39OPUS (TIMI-16)
- Setting
- 888 Hospitals in 27 Countries
- Intervention
- Oral IIb-IIIa Inhibitor vs Placebo
- Population
- Intervention Arm Both arms identical at 30 days
(6867) - Outcome
- 30 day mortality
- Trial stopped early due to elevation in
intervention arm - Baseline
- Control Arm (3421)
40OPUS (TIMI-16)30 Day Mortality
Control Control Control Intervention Intervention Intervention
Period Events Patients Event Rate () Events Patients Event Rate () p
1 0 0 0.0 0 5 0.0
2 0 17 0.0 0 30 0.0
3 0 48 0.0 1 95 1.1 1.000
4 0 135 0.0 7 249 2.8 0.102
5 1 268 0.4 11 529 2.1 0.070
6 4 463 0.9 18 951 1.9 0.173
7 5 764 0.7 33 1,528 2.2 0.008
8 8 1,089 0.7 46 2,161 2.1 0.003
9 11 1,412 0.8 54 2,815 1.9 0.004
10 16 1,805 0.9 78 3,610 2.2 lt0.001
11 21 2,173 1.0 92 4,410 2.1 lt0.001
12 33 2,701 1.2 109 5,447 2.0 0.011
13 46 3,360 1.4 134 6,756 2.0 0.031
14 46 3,421 1.3 134 6,867 2.0 0.031
41OPUS (TIMI-16) Alert Summary
42CLARITY (TIMI-28)
- Setting
- 313 Hospitals in 23 Countries
- Intervention
- Oral Anti-Platelet Agent vs Placebo
- Population
- Intervention Arm (1751)
- Outcome
- Major Bleeding
- DSMB concerned, but trial did not stop early
- Baseline
- Control Arm (1739)
43CLARITY (TIMI-28)Major Bleeding
Control Control Control Intervention Intervention Intervention
Period Events Patients Event Rate () Events Patients Event Rate () p
1 0 4 0.0 0 1 0.0
2 0 13 0.0 0 12 0.0
3 1 27 3.7 0 23 0.0 1.000
4 2 40 5.0 1 49 2.0 0.586
5 2 73 2.7 2 83 2.4 1.000
6 3 116 2.6 5 126 4.0 0.724
7 4 168 2.4 7 173 4.0 0.548
8 4 214 1.9 9 226 4.0 0.262
9 7 276 2.5 10 284 3.5 0.624
10 8 337 2.4 12 361 3.3 0.502
11 10 424 2.4 15 450 3.3 0.423
44CLARITY (TIMI-28)Major Bleeding
Control Control Control Intervention Intervention Intervention
Period Events Patients Event Rate () Events Patients Event Rate () p
12 11 536 2.1 16 554 2.9 0.438
13 13 639 2.0 18 649 2.8 0.468
14 17 776 2.2 22 780 2.8 0.517
15 17 892 1.9 23 926 2.5 0.427
16 18 1,041 1.7 28 1,058 2.6 0.180
17 20 1,192 1.7 31 1,195 2.6 0.156
18 24 1,314 1.8 31 1,327 2.3 0.414
19 25 1,457 1.7 31 1,459 2.1 0.500
20 26 1,584 1.6 32 1,606 2.0 0.509
21 30 1,739 1.7 34 1,751 1.9 0.706
45CLARITY (TIMI-28)Alert Summary
46OPUS /CLARITYConclusions
- SPC performed well in the positive study, but did
have some false positive alerts in the negative
study - LR stratified SPC failed to alert early in the
positive study, but performed well in the
negative study - BUS was more specific than SPC in both studies
47Sensitivity Analysis
- Setting
- Brigham Womens Hospital (01/2002 12/2004)
- Population
- All patients undergoing percutaneous coronary
intervention (6175) - Outcome
- Post-procedural major adverse cardiac events
(403) - Death
- Post-Procedural Myocardial Infarction
- Repeat Vascularization
- Baseline
- Arbitrarily set event rates and sample sizes
48Sensitivity AnalysisResults
49Clinical Alert
- Setting
- Brigham Womens Hospital (01/2002 12/2004)
- Population
- All patients receiving a vascular closure device
after percutaneous coronary intervention (3947) - Outcome
- Retroperitoneal Hemorrhage (25)
- Baseline
- Stanford University Data (2000 2004)
50Event Rate Elevation
51Manual Review
- Triggered root cause analysis
- Manual chart review and multivariable analsysis
- Final Result Not related, confounded by
indication
52Future WorkMethodology
- Address Calibration Concerns
- Recalibration of Logistic Regression models
- Development of Machine Learning Risk Prediction
Models - Address BUS Insensitivity
- Incorporate data weight decay over time
53Future Work Application
54Future Work
- New Medication Outcomes Surveillance
- Inpatient (versus Outpatient)
- More frequent monitoring
- Higher quality source data
- Outcomes easier to capture
55Future WorkDevelop Data Repository
Local Institution
56Future WorkInitial Framework
- New medication laboratory monitoring protocol
- Standard measures that are most commonly affected
in new medications - AST, ALT, Creatinine, WBC, Platelets
- Establish reasonable baselines
- Closely Related medication lab results
- Unrelated medication lab results
- Expert Panel Estimation
57Conclusions
- Developed continuous monitoring methodologies
- Implemented an automated monitoring tool
- Evaluated the system for a number of clinical
outcomes - Identified areas for methodological refinement
- Outlined future work
58Acknowledgements
- Mentors
- Lucila Ohno-Machado, MD, PhD
- Frederic S. Resnic, MD, MS
- Collaborators
- Nipun Arora, MD
- Sharon Lise-Normand, PhD
- Ewout Steyerberg, PhD
- Programming Team
- Richard Cope
- Barry Coflan
- Atul Tatke
- Funding
- NLM R01-LM-08142
- NLM T15-LM-07092
59 Michael Matheny, MD MS mmatheny_at_dsg.harvard.edu
Brigham Womens HospitalThorn 30975 Francis
StreetBoston, MA 02115
The End