Adverse Event Reporting at FDA, Data Base Evaluation and Signal Generation

1
Adverse Event Reporting at FDA,Data Base
Evaluation and Signal Generation

• Robert T. ONeill, Ph.D.
• Director, Office of Biostatistics, CDER, FDA

Presented at the DIMACS Working Group Disease and
Adverse Event Reporting, Surveillance, and
Analysis October 16, 17, 18, 2002 Piscataway,
New Jersey
2
Outline of Talk

• The ADR reporting regulations
• The information collected on a report form
• The data base, its structure and size
• The uses of the data base over the years
• Current signal generation approaches - the data
  mining application
• Concluding remarks

3
Overview

• Adverse Event Reporting System (AERS)
• Report Sources
• Data Entry Process
• AERS Electronic Submissions (Esub)
• Production Program
• E-sub Entry Process
• MedDRA Coding

4
Adverse Event Reporting System (AERS) Database

• Database Origin 1969
• SRS until 11/1/97 changed to AERS
• 3.0 million reports in database
• All SRS data migrated into AERS
• Contains Drug and "Therapeutic" Biologic Reports
• exception vaccines
  VAERS
  1-800-822-7967

5
Adverse Event Reporting System Source of Reports

• Health Professionals, Consumers / Patients
• Voluntary Direct to FDA and/or to
  Manufacturer
• Manufacturers Regulations for Postmarketing
  Reporting

6
(No Transcript)
7
Current Guidance on Postmarketing Safety
Reporting (Summary)

• 1992 Reporting Guideline
• 1997 Reporting Guidance Clarification of What to
  Report
• 1998 ANPR for e-sub
• 2001 Draft Reporting Guidance (3/12/2001)
• 2001 E-sub Reporting of Expedited and Periodic
  ICSRs (11/29/2001)

8
Adverse Events Reports to FDA 1989 to 2001
9
Despite limitations, it is our primary window on
the real world

• What happens in the real world very different
  from world of clinical trials
• Different populations
• Comorbidities
• Coprescribing
• Off-label use
• Rare events

10
AERS Functionality

• Data Entry
• MedDRA Coding
• Routing
• Safety Evaluation
• Inbox
• Searches
• Reports
• Interface with Third-Party Tools
• AutoCode (MedDRA)
• RetrievalWare (images)

11
AERS Esub ProgramHistory

• Over 4 years
• Pilot, then production.
• PhRMA Electronic Regulatory Submission (ERS)
  Working Group
• PhRMA eADR Task Force
• EPrompt Initiative
• Regular meetings between FDA and Industry held to
  review status, address issues, share lessons
  learned

12
Adverse Event Reporting System Processing
MEDWATCH forms

• Goal Electronically Receive Expedited and
  Periodic ISRs
• Docket 92S-0251
• As of 10/2000, able to receive electronic 15-day
  reports
• Paper Reports
• Scanned upon arrival
• Data entered
• Electronic and Paper Reports
• Coded in MedDRA

13
Electronic Submission of Postmarketing ADR Reports

• MedDRA coding 3500A
• Narrative searched with Autocoder
• MedDRA coding E-sub
• Narrative searched with Autocoder
• Enabled companies accept their terms

14
AERS Esub ProgramAdditional Information

• www.fda.gov/cder (CDER)
• www.fda.gov/cder/aers/regs.htm (AERS)
• Reporting regulations, guidances, and updates
• www.fda.gov/cder/aerssub (PILOT)
• aersesub_at_cder.fda.gov (EMAIL)
• www.fda.gov/cder/present (CDER PRESENTATIONS)

15
AERS Esub ProgramAdditional Information(contd)

• www.fda.gov (FDA)
• www.fda.gov/oc/electronicsubmissions/interfaq.htm
  (GATEWAY)
• Draft Trading Partner Agreement, Frequently Asked
  Questions (FAQs) for FDAs ESTRI gateway
• edigateway_at_oc.fda.gov (EMAIL)
• www.fda.gov/medwatch/report/mfg.htm (MEDWATCH)
• Reporting regulations, guidances, and updates

16
AERS Esub ProgramAdditional Information(contd)

• www.ich.org (ICH home page)
• www.fda.gov/cder/m2/default.htm(M2)
• ICH ICSR DTD 2.0
• www.meddramsso.com (MedDRA MSSO)
• http//www.ifpma.org/pdfifpma/M2step4.PDF
• ICH ICSR DTD 2.1
• http//www.ifpma.org/pdfifpma/e2bm.pdf
• New E2BM changes
• http//www.ifpma.org/pdfifpma/E2BErrata.pdf
• Feb 5, 2001 E2BM editorial changes

17
AERS Users
FDA Contractor
Compliance
AERS
FOIA
Safety Evaluators
18
Uses of AERS

• Safety Signal Detection
• Creation of Case Profiles
• who is getting the drug
• who is running into trouble
• Hypothesis Generation for Further Study
• Signals of Name Confusion

19
Other references

• C. Anello and R. ONeill. 1998, Postmarketing
  Surveillance of New Drugs and Assessment of Risk,
  p 3450-3457 Vol 4 ,Encylopedia of Biostatistics,
  Eds. Armitage and Colton, John Wiley and Sons
• Describes many of the approaches to spontaneous
  reporting over the last 30 years

20
Related work on signal generation and modeling

• Finney , 1971, WHO
• ONeill ,1988
• Anello and ONeill, 1997 -Overview
• Tsong, 1995 adjustments using external drug use
  data compared to other drugs
• Compared to previous time periods
• Norwood and Sampson, 1988
• Praus, Schindel, Fescharek, and Schwarz, 1993
• Bate et al. , 1998 Bayes,

21
References

• ONeill and Szarfman, 1999 The American
  Statistician , Vol 53, No 3 190-195 Discussion
  of W. DuMouchels article on Bayesian Data Mining
  in Large Frequency Tables, With an Application to
  the FDA Spontaneous Reporting System (same issue)

22
Recent Post-marketing signaling strategies
Estimating associations needing follow-up

• Bayesian data mining
• Visual graphics
• Pattern recognition

23
The structure and content of FDAs database some
known features impacting model development

• SRS began in late 1960s (over 1.6 million
  reports)
• Reports of suspected drug-adverse event
  associations submitted to FDA by health care
  providers (voluntary, regulations)
• Dynamic data base new drugs, reports being added
  continuously ( 250,000 per year)
• Early warning system of potential safety problems
• Content of each report
• Drugs (multiple)
• Adverse events (multiple)
• Demographics (gender,age, other covariates)

24
The structure and content of FDAs database some
known features impacting model development

• Quality and completeness of a report is variable,
  across reports and manufacturers
• Serious/non-serious - known/unknown
• Time sensitive - 15 days
• Coding of adverse events (COSTART) determines one
  dimension of table - about 1300 terms
• Accuracy of coding / interpretation

25
The DuMouchel Model and its Assumptions

• Large two-dimensional table of size M (drugs) x
  N (ADR events) containing cross classified
  frequency counts - sparse
• Baseline model assumes independence of rows and
  columns - yields expected counts
• Ratios of observed / expected counts are modeled
  as mixture of two, two parameter gammas with a
  mixing proportion P
• Bayesian estimation strategy shrinks estimates in
  some cells
• Scores associated with Bayes estimates used to
  identify those cells which deviate excessively
  from expectation under null model
• Confounding for gender and chronological time
  controlled by stratification

26
The Model and its Assumptions

• Model validation for signal generation
• Goodness of fit
• higher than expected counts informative of true
  drug-event concerns
• Evaluating Sensitivity and Specificity of signals
• Known drug-event associations appearing in a
  label or identified by previous analysis of the
  data base use of negative controls where no
  association is known to be present
• Earlier identification in time of known
  drug-event association

27
Finding Interestingly Large Cell Counts in a
Massive Frequency Table

• Large Two-Way Table with Possibly Millions of
  Cells
• Rows and Columns May Have Thousands of Categories
• Most Cells Are Empty, even though N.. Is very
  Large
• Bayesian Data Mining in Large Frequency Tables
• The American Statistician (1999) (with
  Discussion)
• Analyzed SRS Database with 1398 Drugs and 952 AE
  Codes
• Nij Count of Reports Containing Drug i and
  Event j
• Only 386K out of 1331K Cells Have Nij gt 0
• 174 Drug-Event Combinations Have Nij gt 1000
• Naïve Baseline Frequencies Eij Ni. N.j / N..
• Extension to Stratification Sum Independence
  Frequencies Defined Separately over Strata Based
  on Age, Sex, etc.

28
Associations of Items in Lists

• Market Basket Data from Transaction Databases
• Tabulating Sets of Items from a Universe of K
  Items
• Supermarket Scanner DataSets of Items Bought
• Medical ReportsDrug Exposures and Symptoms
• Sparse RepresentationRecord Items Present
• Pijk Prob(Xi 1, Xj 1, Xk 1), (i lt j lt k)
  
• Marginal Counts and Probabilities Ni , Nij ,
  Nijk , Pi , Pij , Pijk
• Conditional Probabilities Prob( Xi Xj , Xk)
  Pijk /Pjk , etc.
• Pi Small, but Si Pi ( Expected
  Items/Transaction) gtgt 1
• Search for Interestingly Frequent Item Sets
• Item Sets Consisting of One Drug and One Event
  Reduce to the GPS Modeling Problem

29
Definitions of Interesting Item Sets

• Data Mining Literature Find All (a, b)
  Associations
• E.g., Find all Sets (Xi , Xj , Xk) Having Prob(
  Xi Xj , Xk) gt a Prob(Xi , Xj , Xk) gt b
• Complete Search Based on Proportions in Dataset,
  with No Statistical Modeling
• Note that a Triple (Xi , Xj , Xk) Can Qualify
  even if Xi Is Independent of (Xj , Xk)!
• We Use Joint Ps, Not Conditional Ps, and
  Bayesian Model
• E.g., Find all (i, j, k) Prob(lijk Pijk/pijk
  gt l0 Data) gt d
• pijk are Baseline Values
• Based on Independence or some other Null
  Hypothesis

30
Empirical Bayes Shrinkage Estimates

• Compute Posterior Geometric Mean (L) and 5th
  Percentile (l.05) of Ratios
• lij Pij /pij , lijk Pijk /pijk , lijkl
  Pijkl /pijkl , etc.
• Baseline Probs p Based on Within-Strata
  Independence
• Prior Distributions of ls Are Mixtures of Two
  Conjugate Gamma Distributions
• Prior Hyperparameters Estimated by MLE from
  Observed Negative Binomial Regression
• EB Calculations Are Compute-Intensive, but merely
  Counting Itemsets Is More So
• Conditioning on Nijk gt n Eases Burden of Both
  Counting and EB Estimation
• We Choose Smaller n than in Market Basket
  Literature

31
The rationale for stratification on gender and
chronological time intervals

• New drugs added to data base over time
• Temporal trends in drug usage and exposure
• Temporal trends in reporting independent of drug
  publicity, Weber effect
• Some drugs associated with gender-specific
  exposure
• Some adverse events associated with gender
  independent of drug usage
• Primary data-mining objective are signals the
  same or different according to gender
  (confounding and effect modification)
• A concern number of strata, sparseness, balance
  between stratification and sensitivity/specificity
  of signals

32
The control group and the issue of compared to
what?

• Signal strategies compare
• a drug with itself from prior time periods
• with other drugs and events
• with external data sources of relative drug usage
  and exposure
• Total frequency count for a drug is used as a
  relative surrogate for external denominator of
  exposure for ease of use, quick and efficient
• Analogy to case-control design where cases are
  specific ADR term, controls are other terms, and
  outcomes are presence or absence of exposure to a
  specific drug.

33
Other metrics useful in identifying unusually
large cell deviations

• Relative rate
• P-value type metric- overly influenced by sample
  size
• Shrinkage estimates for rare events potentially
  problematic
• Incorporation of a prior distribution on some
  drugs and/or events for which previous
  information is available - e.g. Liver events or
  pre-market signals

34
Interpreting the empirical Bayes scores and their
rankings the Role of visual graphics(Ana
Szarfman)

• Four examples of spatial maps that reduce the
  scores to patterns and user friendly graphs and
  help to interpret many signals collectively
• All maps are produced with CrossGraphs and have
  drill down capability to get to the data behind
  the plots

35
Example 1A spatial map showing the signal
scores for the most frequently reported events
(rows) and drugs (columns) in the database by the
intensity of the empirical Bayes signal score
(blue color is a stronger signal than purple)
36
(No Transcript)
37
Example 2Spatial map showing fingerprints of
signal scores allowing one to visually compare
the complexity of patterns for different drugs
and events and to identify positive or negative
co-occurrences
38
(No Transcript)
39
Example 3Cumulative scores and numbers of
reports according to the year when the signal was
first detected for selected drugs
40
(No Transcript)
41
Example 4Differences in paired male-female
signal scores for a specific adverse event across
drugs with events reported (red means females
greater, green means males greater)
42
(No Transcript)
43
Why consider data mining approaches

• Screening a lot of data, with multiple exposures
  and multiple outcomes
• Soon becomes difficult to identify patterns
• The need for a systematic approach
• There is some structure to the FDA data base,
  even though data quality may be questionable

44
Two applications

• Special population analysis
• Pediatrics
• Two or more item associations
• Drug interactions
• Syndromes (combining ADR terms)

45
Pediatric stratifications (age 16 and younger)

• Neonates
• Infants
• Children
• Adolescents
• Gender

46
Item Association

• Outcomes
• Drug exposures - suspect and others
• Events
• Covariates
• Confounders
• Uncertainties of information in each field
• dosage, formulation, timing, acute/chronic
  exposure
• Multiplicities of dimensions

47
Why apply to pediatrics ?

• Vulnerable populations for which labeling is poor
  and directions for use is minimal - a set up for
  safety concerns
• Little comparative clinical trial experience to
  evaluate effects of
• Metabolic differences, use of drugs is different,
  less is known about dosing, use with food,
  formalations and interactions Gender differences
  of interest

48
Challenges in the future

• More real time data analysis
• More interactivity
• Linkage with other data bases
• Quality control strategies
• Apply to active rather than passive systems where
  non-reporting is not an issue