Evaluating Syndromic Surveillance: Is It Worth The Effort, And If So, For What

1
Evaluating Syndromic Surveillance Is It Worth
The Effort, And If So, For What?

• Michael A. Stoto
• Contributions from Arvind Jain, Ron Fricker, Matt
  Schonlau, Lou Mariano, John Davies-Cole and DC
  DOH staff
• ASA Expert Panel on Syndromic Surveillance
• June 2006, Denver CO

2
Outline

• Evaluating syndromic surveillance systems
• DC Department of Healths syndromic surveillance
  system
• Actual outbreaks
• Multivariate detection algorithms
• Syndromic surveillance and public health practice

3
Everyone Wants a Public Health "Early Warning
System"

• The sooner you know about a terrorist attack, the
  more effective the response
• Smallpox
• Isolate and quarantine to prevent spread
• Vaccinate unexposed
• Prepare hospitals for cases
• Help identify perpetrators
• Pandemic influenza
• Start laboratory work to identify virus strain

4
Syndromic Surveillance Offers the Possibility of
Early Detection

• Suggested by 93 Milwaukee Cryptosporidium
  outbreak
• Focus on symptoms rather than confirmed diagnoses
• Especially flu-like symptoms typical of initial
  states of many bioterrorist agents (anthrax,
  smallpox, etc.)
• Builds on existing data systems
• Health care, medication sales, absenteeism,
• Usually computerized, often massive
• Statistical analyses used to detect sudden changes

5
Why is This Important?

• Ability to detect bioterrorist events earlier
  than otherwise presumably can enable a timely and
  effective public health response
• Major investments of federal funds provided to
  state and local public health departments
• Vendors with systems to sell
• Surveillance for pandemic flu and other non-BT
  health outcomes
• Relationships between hospitals and health
  departments
• General availability of health data
• Note Situational awareness is something
  different altogether

6
Why is This so Hard?

• Obtaining relevant and accurate data quickly and
  from a variety of sources
• Determining when something unusual is going on
• In the presence of highly variable and possibly
  unstable background variation
• When there may be other reasons for changes in
  the data
• Cant wait for more data

7
(No Transcript)
8
(No Transcript)
9
False Positive Rate

• All alarm systems face tradeoffs among
• Sensitivity
• False positive rate
• Timeliness
• Even low false positive rates lead to many alarms
• If each of 3,000 U.S. counties looked at one
  indicator, a 0.1 FPR ? 3 false positives/day
• If each of 20 hospitals in a city looked at 10
  indicators every day, a 0.1 FPR ? 6 false
  positives/month, one every 5 days
• Any of the three can be improved
• but only at the expense of the other two

10
Evaluation Questions

• What kinds of attacks are reasonably detectable?
  Characterize
• sensitivity, false positives, timeliness
• agents, location, timing, etc.
• Can more sophisticated statistical methods do
  better than simple methods? Why?
• How should syndromic surveillance be integrated
  into public health practice?
• How can syndromic surveillance help with other
  health goals?

11
Evaluation Approaches

• Epidemiologic analysis
• Symptoms, timing, disease characteristics
• Process analysis
• Data acquisition
• Public health response
• Retrospective and comparative analyses of known
  natural outbreaks
• What is the gold standard?
• Simulation studies
• Real baseline, simulated outbreak

12
Infectious vs. non-infectious agent

• 90 people exposed to a non-infectious agent
• 24 people exposed to an infectious agent, with
  second wave

13
Simulation Approach
15
10
Number of cases
5
0
0
10
20
30
40
Day
14
Fast Attacks Can be Detected by Day 2
15
Slow Attack Cant be Detected Until Day 9
16
Can This Performance Be Improved?

• Choose a syndrome that is less common
• Pool data over multiple hospitals
• Analyze more indicators or hospitals
• Improve baseline model to reduce noise
• Look for geographic or other patterns

17
DC DOHs SS System

• Emergency Room S. S. System (ERSSS)
• Since 9/12/01, DC DOH has been collecting data on
  a daily basis from hospital ERs
• Part of a regional system system including
  suburban Maryland and Northern Virginia
• Data for this presentation through 5/8/04
• Number of patients with chief complaint coded at
  DOH (hierarchical system)
• Respiratory -- Neurological
• Gastro-intestinal -- Sepsis
• Unspecified infection -- Death
• Rash -- Other
• Focus on 7/9 hospitals with completeness gt 75

18
Detection Algorithms

• Shewhart
• Alarm if yt gt h
• CUSUM (CUmulative SUMmation)
• Ct max Ct-1 (yt - ?) k, 0 Alarm if Ct gt
  h
• Expo (mean-adjusted) CUSUM
• EWMA (Exponentially Weighted Moving Average)
• zt ??yt (1-?)zt-1
• Ct max Ct-1 (yt - zt) k, 0 Alarm if Ct gt
  h
• Thresholds set empirically so that alarm rate
  1 outside of flu season (May 1 Nov. 30)
• Empirical p-values calculated
• CDC EARS (3 tests with different sensitivity
  levels )

19
Possible 2003 GI Outbreak

• Hospital C
• CUSUM, EXPO and EARS flags late January early
  February
• Hospital I
• EXPO flags late March
• Hospital A
• CUSUM flags mid April
• Not seen in other hospitals or total hospitals
• Compare to larger gastro outbreak in 2004

20
(No Transcript)
21
Anthrax Attacks in Fall 2001

• Unspecified infection flags around October 22,
  when anthrax deaths were in the news
• EARS, EXPO, and CUSUM in hospital A
• EARS and EXPO in hospital D
• EARS, EXPO, and CUSUM in hospital H
• Not seen in other hospitals
• Rash flags in hospital H shortly beforehand
• Shows the signature of the worried well

22
(No Transcript)
23
2004 Flu Season

• Strong signals unspecified infection in some
  hospitals and total hospitals before Dec. 1
• when season began in Mid-Atlantic states
• not so clear this early from raw counts or
  analysis of individual hospitals
• respiratory flags in late December
• Different syndromes and detection algorithms flag
  in each hospital
• Or not at all
• CUSUM for unspecified infection provides most
  consistent signal
• Hospital I flags sooner than in 02, but still
  late

24
(No Transcript)
25
(No Transcript)
26
Conclusions about Detection Algorithms

• 03 gastrointestinal
• EXPO flags for Hospitals C and I
• CUSUM flags for Hospitals C and A
• Shewhart not consistent for any hospital
• 01 anthrax - no consistent pattern
• 02 and 04 flu
• EXPO about three weeks before CUSUM
• Shewhart almost as good as EXPO
• No consistent best performers

27
Conclusions about DCs Surveillance System

• If completeness and timeliness of data
  acquisition can be improved through automated
  reporting, syndromic surveillance offers
  potential for early detection of flu and other
  disease outbreaks
• More research is needed to
• Characterize normal patterns in the data
• Identify the more effective detection algorithms
  and symptom groups
• Characterize sensitivity and specificity when
  used prospectively in real time

28
Multivariate Detection Algorithms

• Motivation Gastrointestinal illness, 2003
• Simply looking at counts doesnt allow for
  pattern to emerge from noise
• Pooling across hospitals might help
• Not if concentrated in one hospital
• Not with missing data in 1 or 2 hospitals
• Analyzing each syndrome x hospital works
• But may lead to excess false positives
• Need to develop, test and characterize
  performance of univariate and multivariate
  detection algorithms
• Focus on pooling across hospitals vs.
  multivariate alternatives

29
Shewhart Methodology
Normal density
?
Threshold h
30
Picturing Shewhart
flag
threshold
Daily count (Xi)
time
31
(Univariate) CUSUM Methodology

• Sequential likelihood ratio test (Page, 1951)
• Using normal distributions in the derivation for
• Good distribution with mean ?
• Bad distribution with mean ? ?
• CUSUM (cumulative sum) recursion is
• where usually
• Flag at time i when
  

32
Picturing the CUSUM
flag
Shewhart threshold
CUSUM threshold
CUSUM (Si)
Daily count (Xi)
time
33
Multivariate generalization of Shewhart
Hotellings T 2

• Hotelling (1947) proposed generalization to
  Shewharts procedure
• Iteratively calculate
• Flag when T gt h (set empirically)
• Essentially, flag when an observed value is
  outside probability ellipsoid (for a multivariate
  normal distribution)

34
Tailoring Hotellings T 2 to syndromic
surveillance

• T 2 insensitive to direction of shift
• Negative shifts in mean generate a signal, as do
  increases
• Add in second criteria to stopping rule
• Stop at time i when T gt h and Xi is in a
  particular region of the space
• Allows for a smaller h for same false positive
  rate
• ? More sensitive to shifts in desired direction

35
Picturing the modified T 2 (2d)
Y
flag
X
36
Crosiers Multivariate CUSUMs

• Crosier (1988) proposed various ad hoc CUSUM
  methods. His preferred, define

37
Tailoring Crosiers MV-CUSUM to syndromic
surveillance

• Unlike univariate CUSUM, Crosiers MV-CUSUM not
  reflected at zero
• Designed to detect a shift in any direction
• Cum. sums in each dimension can get very negative
• In this problem, need to quickly detect positive
  shifts
• So bound S in each dimension by zero

38
Univariate detection algorithms

• a. Shewhart Alarm if yt gt h
• b. CUSUM (CUmulative SUMmation)
• Ct max Ct-1 (yt - ?) k, 0 Alarm if Ct gt
  h
• c. Expo (mean-adjusted) CUSUM
• EWMA (Exponentially Weighted Moving Average)
• zt ??yt (1-?)zt-1
• Ct max Ct-1 (yt - zt) k, 0 Alarm if Ct gt
  h
• Empirical p-values calculated

39
Pooling univariate detection algorithms

• Z-analyses (1a, 1b, 1c)
• Set threshold empirically so that daily alarm
  rate (probability gt1 series flag) is 1 outside
  flu season
• P-analyses (2a, 2b, 2c)
• Pool P-values for all series (-2?ln(pi))
• Set threshold so that alarm rate is 1
• C-analyses (3a, 3b, 3c)
• Combine data across hospitals for each syndrome
• Apply Z-analyses
• CP-analyses (4a, 4b, 4c)
• Combine data across hospitals for each syndrome
• Apply P-analyses

40
Multivariate detection algorithms

• Z-analyses Multivariate Shewhart (5)
• 5a. Standard
• 5b. With adjustment for lower left quadrant
• C-analyses Multivariate Shewhart (6)
• 6a. Standard
• 6b. With adjustment for lower left quadrant
• Z-analyses Crosiers Multivariate CUSUM (7)
• C-analyses Crosiers Multivariate CUSUM (8)

41
Simulation approach

• Seed 1, 2, 10 extra cases over 10 days
• Scenario A Add seed to all series
• 7 hospitals x 4 symptoms
• Scenario C Add same seed to "unspecified
  infection" only across all hospitals
• Scenario CA Add seed to "unspecified infection"
  only (Seed 4, 8, 40)
• Scenario D Add same seed to all syndromes in
  one mid-sized hospital
• Scenario DA Add seed to one mid-sized hospital
  only (Seed 7, 14, 70)

42
Index ?i pr(flag)i

• MV-CUSUM performs slightly better than Z-CUSUM in
  Scenario A and DA
• Z-CUSUM performs slightly better (1 day) than
  MV-CUSUM in C, equal in CA D.

43
Performance summary
Index ?i pr(flag)i

• P analyses do not perform well, except in
  Scenario A
• C analyses worse than corresponding Z analysis,
  especially CUSUM
• Z and MV analyses perform best for large
  outbreaks (A, CA, DA)
• MV-CUSUM (7) best overall, followed by Z-CUSUM
  (1b)

44

• In flu season Z-CUSUM has 50 false positive
  rate and MV-CUSUM has 15 false positive rate.
• Z and MV CUSUM both reach 80 probability of
  detection at roughly the same time, except for
  Scenario C.

45
Performance summary

• Outside of flu season
• MV-CUSUM slightly better or equivalent in all but
  Scenario C (and only slightly worse there)
• 50 sensitivity reached on
• Day 1 for both methods in Scenarios CA and DA
• Day 2 for both methods in Scenario A
• Day 4 for MV and day 5 for Z in Scenario C
• Day 5 for both methods in Scenario D
• In flu season
• false positive rate 50 for Z-CUSUM, 15 for
  MV-CUSUM
• both reach 80 probability of detection at
  roughly the same time, except for Scenario C

46
Conclusions

• No detection algorithm best in all situations,
  but two do reasonably well in all scenarios
  tested
• Multivariate CUSUM slightly preferable to
  univariate CUSUM (Z-analyses)
• Univariate slightly preferable to multivariate
  CUSUM in Scenario C (may be the most likely)
• Combining P-values generally doesnt work well
• Combining data across hospitals (C-analyses)
  doesnt help in Scenario A or for CUSUM
• Better for Scenarios A and D, and for Shewhart
  and EXPO

47
Integrating Syndromic and Public Health
Surveillance

• Syndromic surveillance cannot be expected to
  detect attacks with few cases
• e.g. anthrax in 2001
• Syndromic surveillance intended to alert public
  health officials to possible bioevent
• Must be followed with
• Epidemiologic investigation
• Policy decisions regarding intervention
• Syndromic surveillance must be linked to other
  surveillance systems in advance

48
Appropriate Physician Involvement Is Essential

• Syndromic surveillance efforts often minimize
  involvement of physicians
• May have consequences after an alert when
  physician communication and cooperation is needed
  for
• Active surveillance
• Epidemiologic investigation
• Mass prophylaxis
• Consider active syndromic surveillance by phone
  or computer

49
Discussion

• Much impressive work has been done
• Information technology Real-time integration of
  many disparate data streams
• Analysis Development of background models,
  detection algorithms, visualization
• Value of syndromic surveillance for bioterrorism
  has not yet been demonstrated
• Relatively small window between what can be
  detected in the first few days and what is
  obvious
• Better integration with public health systems
  needed
• Most important contribution may be for natural
  disease outbreaks, such as gastro, flu, and
  pandemic flu
• Design for this rather than focus only on
  timeliness