Methodologies and Automated Applications for Post-Marketing Outcomes Surveillance of Medical Devices and Medications - PowerPoint PPT Presentation

1 / 59
About This Presentation
Title:

Methodologies and Automated Applications for Post-Marketing Outcomes Surveillance of Medical Devices and Medications

Description:

From www.powerpointbackgrounds.com Messages do not appear on the full product. – PowerPoint PPT presentation

Number of Views:5015
Avg rating:3.0/5.0

less

Transcript and Presenter's Notes

Title: Methodologies and Automated Applications for Post-Marketing Outcomes Surveillance of Medical Devices and Medications


1
Methodologies and Automated Applications for
Post-Marketing Outcomes Surveillance of Medical
Devices and Medications
  • Michael E. Matheny, MD, MS
  • NLM Biomedical Informatics Fellow
  • Decision Systems Group, Department of Radiology
    Brigham Womens Hospital, Boston, MA

2
Outline
  • Post-Marketing Surveillance Background
  • Statistical Methodology Development
  • Computer Application Development
  • Clinical Examples
  • Future Directions

3
BackgroundSurveillance Rationale
  • Phase 3 Trials insufficient to ensure adequate
    safety of medications and devices
  • Low frequency events are not detected
  • Protected populations (pregnant women, children)
    and more ill populations not represented
  • Complications delayed by a number of years cannot
    be detected

4
BackgroundFDA Medical Devices
  • 1,700 types of devices
  • 500,000 device models
  • 23,000 manufacturers

5
BackgroundFDA New Drug Applications (NDA)
6
BackgroundCurrent Post-Marketing Surveillance
  • Combination of mandatory and voluntary adverse
    event reporting
  • Mandatory reporting by manufacturers and health
    facilities
  • Voluntary MedWatch / MAUDE reports by providers
    and patients

2004 Drug-Related Adverse Event Reports 2004 Drug-Related Adverse Event Reports
Total 422,889
Manufacturer Facility Reports 401,396
MedWatch 21,493
7
BackgroundMAUDE Cypher Reporting Rate
2003
2004
8
BackgroundCurrent Post-Marketing Surveillance
  • Phase 4 Trials
  • Poor Compliance
  • As of March 2006 report, 797 of 1231 (65)
    agreed-upon trials had yet to be started
  • Barriers
  • Lack of manufacturer incentives
  • Expensive
  • Drug already on the market
  • Lack of regulatory enforcement

9
BackgroundMedical Device Recalls
  • Boston Scientific cardiac stent (1998)
  • Balloon rupture at low pressures
  • Guidant cardio-defibrillator (2005)
  • Malfunction due to electrical short
  • Vioxx (2004)
  • cardiovascular complications
  • Tequin (2006)
  • Hypoglycemia and hyperglycemia

10
BackgroundFDA Response
  • Increasing demands for Phase 4 trials
  • Legislation to increase quality of adverse event
    reporting
  • Emphasizing trial registries (clinicaltrials.gov)
    as way to prevent omission of results
  • Commissioned IOM report The Future of Drug
    Safety

11
BackgroundAdverse Event Data Continuum
Phase 3 Trials MAUDE / MedWatch Voluntary Registry Mandatory Registry
Internal Validity --
Generalizibility /- /-
Breadth
Immediacy ---
Lack of Bias --- ---
12
Statistical MethodsMedical Outcomes Monitoring
  • Using registry data that tracks all patients
    allows different types of analysis than used in
    the FDAs adverse event reporting systems
  • No generally accepted methods for monitoring
    registry data for adverse events
  • Lack of sufficient discrete electronic data
    sources to construct registries
  • Some outcomes are challenging to track for an
    entire population
  • Increased variability in subjective medical data
  • Physician inter-observer disagreement

13
Objective
  • Develop methodologies and implement an automated
    computer monitoring system to perform outcomes
    surveillance of registry data for new medical
    devices and medications

14
Statistical MethodsStatistical Process Control
15
Statistical MethodsBayesian Updating Statistics
16
Statistical MethodsEstablishing Baseline Data
  • Primary Data Sources
  • Phase 3 trial data
  • Post-Marketing data from a closely related
    medication/device
  • Alternative Data Sources

17
Statistical MethodsEstablishing Alerting
Thresholds
  • Use number of events and sample size to calculate
    proportion with confidence intervals
  • Typically, medical domains use 95 CI or 1.96 SD
    from the point estimate

18
Statistical Methods Establishing Alerting
Thresholds
SPC
BUS
19
Statistical Methods Establishing Alerting
Thresholds
20
Statistical Methods Establishing Alerting
Thresholds
  • Wilsons method of comparison between two
    proportions

21
Statistical MethodsRisk Stratification
  • Allows creating subgroups for separate analyses
  • Single variable
  • Logistic regression model with scoring thresholds

22
Application DevelopmentDELTA
  • Data Extraction and Longitudinal Time Analysis
    (DELTA)
  • Design Goals
  • Generic data import format
  • Allow both prospective and retrospective analyses
  • Modular framework to allow sequential addition of
    statistical methodologies
  • Multiple alerting methods
  • Any number of concurrent ongoing analyses

23
Application DevelopmentDELTA
DELTA Database
Source Database
Clinical Data Entry
Data Dictionary
Web Server
Statistical Modules
Source IT Manager
DELTA Users
24
Application Test DataCypher Drug-Eluting Stent
(DES)
  • Setting
  • Brigham Womens Hospital (07/2003 12/2004)
  • Population
  • All patients receiving a drug-eluting stent
    (2270)
  • Outcome
  • Post-procedural in-hospital mortality (27)
  • Baseline
  • University of Michigan Data (1997-1999)

25
Application DevelopmentDELTA
26
Application DevelopmentDELTA
27
Application DevelopmentDELTA
28
Application DevelopmentDELTA
29
Application DevelopmentDELTA
30
Application DevelopmentDELTA
31
Application DevelopmentDELTA
32
Application DevelopmentDELTA
33
Risk StratificationPotential Solution
  • Incorporate individual risk prediction models in
    order to adjust for case mix and illness severity

34
Possible Risk Prediction Methods
  • Linear / Logistic Regression
  • Artificial Neural Networks
  • Bayesian Networks
  • Support Vector Machines

35
LR External Validation Models
Model  Dates Location Sample
NY 1992 1991 NY 5827
NY 1997 1991 1994 NY 62670
CC 1997 1993 1994 Cleveland, OH 12985
NNE 1999 1994 1996 NH, ME, MA, VT 15331
MI 2001 1999 2000 Detroit, MI 10796
BWH 2001 1997 1999 Boston, MA  2804
ACC 2002 1998 2000 National 100253
36
LR External Validation
  • Setting
  • Brigham Womens Hospital (01/2002 09/2004)
  • Population
  • All patients undergoing percutaneous coronary
    intervention (5216)
  • Outcome
  • Post-procedural in-hospital mortality (71)

37
LR External Validation Results
Curve Deaths AUC HL ?2 HL (p)
NY 1992  96.7 0.82 31.1 lt0.001
NY 1997  61.6 0.88 32.2 lt0.001
CC 1997 78.8 0.88 27.8 lt0.001
NNE 1999  56.2 0.89 45.9 lt0.001
MI 2001  61.8 0.86 30.4 lt0.001
BWH 2001  136.1 0.89 39.7 lt0.001
ACC 2002  49.9 0.90 42.0 lt0.001
BWH 2004 70.5 0.93 7.61 0.473
38
LR External Validation Conclusions
  • Excellent discrimination across all models
  • Calibration (Hosmer-Lemeshow) poor for all models
    but recent local one
  • Addressed categorical risk stratification by
    keeping all records in one stratum
  • Calibration problems over time limit application,
    and require exploration of recalibration methods

39
OPUS (TIMI-16)
  • Setting
  • 888 Hospitals in 27 Countries
  • Intervention
  • Oral IIb-IIIa Inhibitor vs Placebo
  • Population
  • Intervention Arm Both arms identical at 30 days
    (6867)
  • Outcome
  • 30 day mortality
  • Trial stopped early due to elevation in
    intervention arm
  • Baseline
  • Control Arm (3421)

40
OPUS (TIMI-16)30 Day Mortality
   Control   Control   Control   Intervention    Intervention    Intervention    
Period Events Patients Event Rate () Events Patients Event Rate () p
1 0 0 0.0 0 5 0.0
2 0 17 0.0 0 30 0.0
3 0 48 0.0 1 95 1.1 1.000
4 0 135 0.0 7 249 2.8 0.102
5 1 268 0.4 11 529 2.1 0.070
6 4 463 0.9 18 951 1.9 0.173
7 5 764 0.7 33 1,528 2.2 0.008
8 8 1,089 0.7 46 2,161 2.1 0.003
9 11 1,412 0.8 54 2,815 1.9 0.004
10 16 1,805 0.9 78 3,610 2.2 lt0.001
11 21 2,173 1.0 92 4,410 2.1 lt0.001
12 33 2,701 1.2 109 5,447 2.0 0.011
13 46 3,360 1.4 134 6,756 2.0 0.031
14 46 3,421 1.3 134 6,867 2.0 0.031
41
OPUS (TIMI-16) Alert Summary
42
CLARITY (TIMI-28)
  • Setting
  • 313 Hospitals in 23 Countries
  • Intervention
  • Oral Anti-Platelet Agent vs Placebo
  • Population
  • Intervention Arm (1751)
  • Outcome
  • Major Bleeding
  • DSMB concerned, but trial did not stop early
  • Baseline
  • Control Arm (1739)

43
CLARITY (TIMI-28)Major Bleeding
   Control  Control  Control Intervention Intervention Intervention  
Period Events Patients Event Rate () Events Patients Event Rate () p
1 0 4 0.0 0 1 0.0
2 0 13 0.0 0 12 0.0
3 1 27 3.7 0 23 0.0 1.000
4 2 40 5.0 1 49 2.0 0.586
5 2 73 2.7 2 83 2.4 1.000
6 3 116 2.6 5 126 4.0 0.724
7 4 168 2.4 7 173 4.0 0.548
8 4 214 1.9 9 226 4.0 0.262
9 7 276 2.5 10 284 3.5 0.624
10 8 337 2.4 12 361 3.3 0.502
11 10 424 2.4 15 450 3.3 0.423
44
CLARITY (TIMI-28)Major Bleeding
   Control  Control  Control Intervention Intervention Intervention  
Period Events Patients Event Rate () Events Patients Event Rate () p
12 11 536 2.1 16 554 2.9 0.438
13 13 639 2.0 18 649 2.8 0.468
14 17 776 2.2 22 780 2.8 0.517
15 17 892 1.9 23 926 2.5 0.427
16 18 1,041 1.7 28 1,058 2.6 0.180
17 20 1,192 1.7 31 1,195 2.6 0.156
18 24 1,314 1.8 31 1,327 2.3 0.414
19 25 1,457 1.7 31 1,459 2.1 0.500
20 26 1,584 1.6 32 1,606 2.0 0.509
21 30 1,739 1.7 34 1,751 1.9 0.706
45
CLARITY (TIMI-28)Alert Summary
46
OPUS /CLARITYConclusions
  • SPC performed well in the positive study, but did
    have some false positive alerts in the negative
    study
  • LR stratified SPC failed to alert early in the
    positive study, but performed well in the
    negative study
  • BUS was more specific than SPC in both studies

47
Sensitivity Analysis
  • Setting
  • Brigham Womens Hospital (01/2002 12/2004)
  • Population
  • All patients undergoing percutaneous coronary
    intervention (6175)
  • Outcome
  • Post-procedural major adverse cardiac events
    (403)
  • Death
  • Post-Procedural Myocardial Infarction
  • Repeat Vascularization
  • Baseline
  • Arbitrarily set event rates and sample sizes

48
Sensitivity AnalysisResults
49
Clinical Alert
  • Setting
  • Brigham Womens Hospital (01/2002 12/2004)
  • Population
  • All patients receiving a vascular closure device
    after percutaneous coronary intervention (3947)
  • Outcome
  • Retroperitoneal Hemorrhage (25)
  • Baseline
  • Stanford University Data (2000 2004)

50
Event Rate Elevation
51
Manual Review
  • Triggered root cause analysis
  • Manual chart review and multivariable analsysis
  • Final Result Not related, confounded by
    indication

52
Future WorkMethodology
  • Address Calibration Concerns
  • Recalibration of Logistic Regression models
  • Development of Machine Learning Risk Prediction
    Models
  • Address BUS Insensitivity
  • Incorporate data weight decay over time

53
Future Work Application
54
Future Work
  • New Medication Outcomes Surveillance
  • Inpatient (versus Outpatient)
  • More frequent monitoring
  • Higher quality source data
  • Outcomes easier to capture

55
Future WorkDevelop Data Repository
Local Institution
56
Future WorkInitial Framework
  • New medication laboratory monitoring protocol
  • Standard measures that are most commonly affected
    in new medications
  • AST, ALT, Creatinine, WBC, Platelets
  • Establish reasonable baselines
  • Closely Related medication lab results
  • Unrelated medication lab results
  • Expert Panel Estimation

57
Conclusions
  • Developed continuous monitoring methodologies
  • Implemented an automated monitoring tool
  • Evaluated the system for a number of clinical
    outcomes
  • Identified areas for methodological refinement
  • Outlined future work

58
Acknowledgements
  • Mentors
  • Lucila Ohno-Machado, MD, PhD
  • Frederic S. Resnic, MD, MS
  • Collaborators
  • Nipun Arora, MD
  • Sharon Lise-Normand, PhD
  • Ewout Steyerberg, PhD
  • Programming Team
  • Richard Cope
  • Barry Coflan
  • Atul Tatke
  • Funding
  • NLM R01-LM-08142
  • NLM T15-LM-07092

59
Michael Matheny, MD MS mmatheny_at_dsg.harvard.edu
Brigham Womens HospitalThorn 30975 Francis
StreetBoston, MA 02115
The End
Write a Comment
User Comments (0)
About PowerShow.com