STATISTICAL CONSIDERATIONS IN POSTMARKETING SAFETY EVALUATION A. Lawrence Gould Merck Research Laboratories West Point, PA [[e-mail]] FDA/Industry Workshop 29 September 2006 Washington, DC

1
STATISTICAL CONSIDERATIONS IN POSTMARKETING
SAFETY EVALUATIONA. Lawrence GouldMerck
Research LaboratoriesWest Point, PA
goulda_at_merck.comFDA/Industry Workshop29
September 2006Washington, DC
2
OVERVIEW
3
Spontaneous AE Reports

• Clinical trial safety information is limited
  relatively short duration
• Safety data collection continues after drug
  approval
• Detect rare adverse events
• Obtain tolerability information in a broader
  population
• Large amount of low-quality data collected
• Not usable for trt comparisons or risk assessment
• Unknown sensitivity specificity
• Evaluation by skilled clinicians
  epidemiologists
• Long history of research on issue

4
Information Available Postmarketing

• Previously undetected adverse and beneficial
  effects that may be uncommon or delayed, i.e.,
  emerging only after extended treatment
• Patterns of drug utilization
• Effect of drug overdoses
• Clinical experience with study drugs in their
  natural environment

5
The Pharmacovigilance Process
Traditional Methods
Data Mining
Detect Signals
Generate Hypotheses
Insight from Outliers
Public Health Impact, Benefit/Risk
Refute/Verify
Type A (Mechanism-based)
Estimate Incidence
Act
Inform
Type B (Idiosyncratic)
Restrict use/ withdraw
Change Label
6
Considerations Issues (An Incomplete List!)

• Incomplete reports of events, not reactions
• Bias noise in system
• Difficult to estimate incidence because no. of
  pats at risk, pat-yrs of exposure seldom reliable
• Significant under reporting (esp. OTC)
• Synonyms for drugs events ? sensitivity loss
• Duplicate reporting
• No certainty that a drug caused the reaction
  reported
• Cannot use accumulated reports to calculate
  incidence, estimate drug risk, or compare drugs

7
DATA MINING
8
Data Mining is a Part of Pharmacovigilance

• Identify subtle associations (e.g.,
  drugdrugevent) and complex relationships not
  apparent by simple summary
• Identify potential toxicity early
• Finding real D-E associations similar to
  finding potential active compounds or expressed
  genes not exactly the same (no H0) more like
  model selection
• Still need initial case review
• respond to reports involving severe, potential
  life-threatening events eg., Stevens-Johnson
  syndrome, agranulocytosis, anaphylactic shock
• Clinical/biological/epidemiological verification
  of apparent associations is essential

9
Typical Data Display
No. Reports Target AE Other AE Total
Target Drug a b nTD
Other Drug c d nOD
Total nTA nOA n
Basic idea Flag when R a/E(a) is large

• Some possibilities
• Reporting Ratio E(a) nTD ? nTA/n
• Proportional Reporting Ratio E(a) nTD ?
  c/nOD
• Odds Ratio E(a) b ? c/d
• Need to accommodate uncertainty, especially if
  a is small
• Bayesian approaches provide a way to do this

10
Currently Used Bayesian Approaches

• Empirical Bayes (DuMouchel, 1998) WHO (Bate,
  1998)
• Both use ratio nij / Eij where
• nij no. of reports mentioning both drug i
  event j
• Eij expected no. of reports of drug i event j
• Both report features of posterior distn of
  information criterion
• ICij log2 nij / Eij PRRij
• Eij usually computed assuming drug i event j
  are mentioned independently
• Ratio gt 1 (IC gt 0) ? combination mentioned more
  often than expected if independent

11
Comparative Example (DuMouchel, 1998)

• No. Reports 4,864,480, Mentioning drug 85,304

Headache Headache Polyneuritis Polyneuritis
Reports AE Both AE Both
Mentioning 71,209 1,614 262 3
Reporting Ratio 1.23 1.23 2.83 2.83
WHO FDA WHO FDA
Expected RR 1.29 1.23 0.76 1.42
5 Quantile -- 1.18 -- 0.58
Excess n 300 225 0 0
12
DATA MINING EXAMPLES INCORPORATING STATISTICAL
REFINEMENTS
13
Result From 6 Years of Reports on Lisinopril
Events w/Lower 5 RR Bnd gt 2 (Bold ? N ? 100)
14
Persistence ( Reliability) of Early Signals
15
Accumulating Information over Time

• Lower 5 quantiles of RR stabilized fairly soon

16
Time-Sliced Evolution of Risk Ratios

• See how values of criteria change over time
  within time intervals of fixed length

Change in ICij for reports of selected events on
A2A from 1995 to 2000 tension
hypotension failure heart
failure kalemia hyperkalemia edema
angioedema
17
Masking of AE-Drug Relationships (1)

• Company databases smaller than regulatory
  databases, more loaded with similar drugs
• eg, Drug A is 2nd generation version of Drug B,
  similar mechanism of action, many reports with B
• Elevated reporting frequency on Drug B could mask
  effect of Drug A
• May be useful to provide results when reports
  mentioning Drug B are omitted

18
Masking of AE-Drug Relationships (2)
19
Example 2 Vaccine-Vaccine Interaction

• From FDA VAERS database, reports from 1990-2002
• Intussusception is a serious intestinal malady
  observed to affect infants vaccinated against
  rotavirus
• Look at reports of intussusception that mention
  rotavirus vaccine (RV) and DTAP vaccine
• DTAP is a benign combination vaccine commonly
  administered to infants
• Demonstration question Intussusception very
  commonly reported with RV but does the
  reporting rate depend on whether DTAP was
  co-administered?
• Not easy to address using standard
  pharmacovigilance procedures

20
Outline of Analysis

• Standard tools provide intussusception reporting
  rate for pairs of vaccines, and for vaccines
  singly
• Result is a 3-way count table (corresponding to
  RV or -, DTAP or -, and intussusception or
  -)
• Use log-linear model to see if intussusception is
  mentioned with the two vaccines together more
  often than the separate vaccine-intussusception
  reporting associations would predict
• Turns out that there is an association
  Likelihood ratio chi-square is 17.41, 1 df,
  highly significant

21
Observed and Expected Report Rates
22
Comments

• Intussusception seems to be reported more often
  than expected when RV and DTAP are given together
  than when RV is given without DTAP, after
  adjusting for individual vaccine-intussusception
  associations
• Reports of intussception without RV are very
  rare, about 4.5/10,000 reports if RV is not
  mentioned
• The joint effect of RV and DTAP on
  intussusception reporting is small, but does
  reach statistical significance
• Not clear that apparent association means
  anything -- actual synergy between RV and DTAP
  seems unlikely, but explanation requires clinical
  knowledge

23
A NEW BAYESIAN APPROACH(Gould, Biometrical
Journal 2006, to appear)
24
Model for Process Generating Observations

• ni no. of reports mentioning i-th drug-event
  pair Poisson (true for EB approach as well)
• f(ni Ei, ?i) fPois(ni ?iEi)
• ?i drawn from a gamma(a0, b0) distribution or
  from a gamma(a1, b1) distribution
• A model selection problem
• Distns reflect physician/epidemiologists
  judgment as to what range of ? values corresponds
  to signals, and what does not

Expected count under independence
Association measure
25
Prior/Model Density of ?

• Bayes approach starts with a random mixture of
  gamma densities,
• f?0(? ?, a0, b0, a1, b1)
• (1 - ?)fgam(? a0, b0) ?fgam(? a1, b1)
• Use value of Ppost(g 1) for inference
• EB approach starts with expectation wrt ? given p
  ? nonrandom mixture of gamma densities,
• f?0(? p, a0, b0, a1, b1)
• pfgam(? a0, b0) (1-p)fgam(? a1, b1)
• Use quantiles of posterior distn of ? for
  inference

Analyst specifies parameter values
Data determine parameter values
26
Comments

• Bayes and EB approaches both model strength of
  drug-event reporting assn as a gamma mixture
• Diagnostic properties of Bayes method can be
  determined analytically or by simulation
• Unknown separation of the true alternative
  distns for ? more important than prior distn
  used for analysis
• Methods described here can be applied to other
  models Scott Berger (2005) used normal
  distributions could also use binomial instead
  of Poisson, beta instead of gamma distributions
  to develop screening methods for AEs in clinical
  trials

27
DISCUSSION
28
Discussion

• Bayesian approaches may be useful for detecting
  possible emerging signals, especially with few
  events
• MCA (UK) currently uses PRR for monitoring
  emergence of drug-event associations
• Signal detection combines numerical data
  screening, statistical interpretation, and
  clinical judgement
• Most apparent associations represent known
  problems
• 25 may represent signals about previously
  unknown associations
• The actual false positive rate is unknown

29
What Next?

• PhRMA/FDA working group has published a white
  paper addressing many of these issues
• Drug Safety (2005) 28 981-1007
• Further refine methods, look for associations
  among combinations of drugs and events, timing of
  reports
• Data mining is like screening, need to evaluate
  diagnostic properties of various approaches
• Need good dictionaries many synonyms ? difficult
  signal detection
• Event names MedDRA may help
• Drug names Need a common dictionary of drug
  names to minimize dilution effect of synonyms

30
Data Used to Construct Plot
Intussception Intussception Intussception - Intussception -
Observed Expected Observed Expected
RV DTAP 85 74 1111 1122
DTAP - 29 40 608 597
RV - DTAP 4 15 33520 33509
DTAP - 293 282 610714 610725