Title: Evaluating Syndromic Surveillance: Is It Worth The Effort, And If So, For What
1Evaluating Syndromic Surveillance Is It Worth
The Effort, And If So, For What?
- Michael A. Stoto
- Contributions from Arvind Jain, Ron Fricker, Matt
Schonlau, Lou Mariano, John Davies-Cole and DC
DOH staff - ASA Expert Panel on Syndromic Surveillance
- June 2006, Denver CO
2Outline
- Evaluating syndromic surveillance systems
- DC Department of Healths syndromic surveillance
system - Actual outbreaks
- Multivariate detection algorithms
- Syndromic surveillance and public health practice
3Everyone Wants a Public Health "Early Warning
System"
- The sooner you know about a terrorist attack, the
more effective the response - Smallpox
- Isolate and quarantine to prevent spread
- Vaccinate unexposed
- Prepare hospitals for cases
- Help identify perpetrators
- Pandemic influenza
- Start laboratory work to identify virus strain
4Syndromic Surveillance Offers the Possibility of
Early Detection
- Suggested by 93 Milwaukee Cryptosporidium
outbreak - Focus on symptoms rather than confirmed diagnoses
- Especially flu-like symptoms typical of initial
states of many bioterrorist agents (anthrax,
smallpox, etc.) - Builds on existing data systems
- Health care, medication sales, absenteeism,
- Usually computerized, often massive
- Statistical analyses used to detect sudden changes
5Why is This Important?
- Ability to detect bioterrorist events earlier
than otherwise presumably can enable a timely and
effective public health response - Major investments of federal funds provided to
state and local public health departments - Vendors with systems to sell
- Surveillance for pandemic flu and other non-BT
health outcomes - Relationships between hospitals and health
departments - General availability of health data
- Note Situational awareness is something
different altogether
6Why is This so Hard?
- Obtaining relevant and accurate data quickly and
from a variety of sources - Determining when something unusual is going on
- In the presence of highly variable and possibly
unstable background variation - When there may be other reasons for changes in
the data - Cant wait for more data
7(No Transcript)
8(No Transcript)
9False Positive Rate
- All alarm systems face tradeoffs among
- Sensitivity
- False positive rate
- Timeliness
- Even low false positive rates lead to many alarms
- If each of 3,000 U.S. counties looked at one
indicator, a 0.1 FPR ? 3 false positives/day - If each of 20 hospitals in a city looked at 10
indicators every day, a 0.1 FPR ? 6 false
positives/month, one every 5 days - Any of the three can be improved
- but only at the expense of the other two
10Evaluation Questions
- What kinds of attacks are reasonably detectable?
Characterize - sensitivity, false positives, timeliness
- agents, location, timing, etc.
- Can more sophisticated statistical methods do
better than simple methods? Why? - How should syndromic surveillance be integrated
into public health practice? - How can syndromic surveillance help with other
health goals?
11Evaluation Approaches
- Epidemiologic analysis
- Symptoms, timing, disease characteristics
- Process analysis
- Data acquisition
- Public health response
- Retrospective and comparative analyses of known
natural outbreaks - What is the gold standard?
- Simulation studies
- Real baseline, simulated outbreak
12Infectious vs. non-infectious agent
- 90 people exposed to a non-infectious agent
- 24 people exposed to an infectious agent, with
second wave
13Simulation Approach
15
10
Number of cases
5
0
0
10
20
30
40
Day
14Fast Attacks Can be Detected by Day 2
15Slow Attack Cant be Detected Until Day 9
16Can This Performance Be Improved?
- Choose a syndrome that is less common
- Pool data over multiple hospitals
- Analyze more indicators or hospitals
- Improve baseline model to reduce noise
- Look for geographic or other patterns
17DC DOHs SS System
- Emergency Room S. S. System (ERSSS)
- Since 9/12/01, DC DOH has been collecting data on
a daily basis from hospital ERs - Part of a regional system system including
suburban Maryland and Northern Virginia - Data for this presentation through 5/8/04
- Number of patients with chief complaint coded at
DOH (hierarchical system) - Respiratory -- Neurological
- Gastro-intestinal -- Sepsis
- Unspecified infection -- Death
- Rash -- Other
- Focus on 7/9 hospitals with completeness gt 75
18Detection Algorithms
- Shewhart
- Alarm if yt gt h
- CUSUM (CUmulative SUMmation)
- Ct max Ct-1 (yt - ?) k, 0 Alarm if Ct gt
h - Expo (mean-adjusted) CUSUM
- EWMA (Exponentially Weighted Moving Average)
- zt ??yt (1-?)zt-1
- Ct max Ct-1 (yt - zt) k, 0 Alarm if Ct gt
h - Thresholds set empirically so that alarm rate
1 outside of flu season (May 1 Nov. 30) - Empirical p-values calculated
- CDC EARS (3 tests with different sensitivity
levels )
19Possible 2003 GI Outbreak
- Hospital C
- CUSUM, EXPO and EARS flags late January early
February - Hospital I
- EXPO flags late March
- Hospital A
- CUSUM flags mid April
- Not seen in other hospitals or total hospitals
- Compare to larger gastro outbreak in 2004
20(No Transcript)
21Anthrax Attacks in Fall 2001
- Unspecified infection flags around October 22,
when anthrax deaths were in the news - EARS, EXPO, and CUSUM in hospital A
- EARS and EXPO in hospital D
- EARS, EXPO, and CUSUM in hospital H
- Not seen in other hospitals
- Rash flags in hospital H shortly beforehand
- Shows the signature of the worried well
22(No Transcript)
232004 Flu Season
- Strong signals unspecified infection in some
hospitals and total hospitals before Dec. 1 - when season began in Mid-Atlantic states
- not so clear this early from raw counts or
analysis of individual hospitals - respiratory flags in late December
- Different syndromes and detection algorithms flag
in each hospital - Or not at all
- CUSUM for unspecified infection provides most
consistent signal - Hospital I flags sooner than in 02, but still
late
24(No Transcript)
25(No Transcript)
26Conclusions about Detection Algorithms
- 03 gastrointestinal
- EXPO flags for Hospitals C and I
- CUSUM flags for Hospitals C and A
- Shewhart not consistent for any hospital
- 01 anthrax - no consistent pattern
- 02 and 04 flu
- EXPO about three weeks before CUSUM
- Shewhart almost as good as EXPO
- No consistent best performers
27Conclusions about DCs Surveillance System
- If completeness and timeliness of data
acquisition can be improved through automated
reporting, syndromic surveillance offers
potential for early detection of flu and other
disease outbreaks - More research is needed to
- Characterize normal patterns in the data
- Identify the more effective detection algorithms
and symptom groups - Characterize sensitivity and specificity when
used prospectively in real time
28Multivariate Detection Algorithms
- Motivation Gastrointestinal illness, 2003
- Simply looking at counts doesnt allow for
pattern to emerge from noise - Pooling across hospitals might help
- Not if concentrated in one hospital
- Not with missing data in 1 or 2 hospitals
- Analyzing each syndrome x hospital works
- But may lead to excess false positives
- Need to develop, test and characterize
performance of univariate and multivariate
detection algorithms - Focus on pooling across hospitals vs.
multivariate alternatives
29Shewhart Methodology
Normal density
?
Threshold h
30Picturing Shewhart
flag
threshold
Daily count (Xi)
time
31(Univariate) CUSUM Methodology
- Sequential likelihood ratio test (Page, 1951)
- Using normal distributions in the derivation for
- Good distribution with mean ?
- Bad distribution with mean ? ?
- CUSUM (cumulative sum) recursion is
- where usually
- Flag at time i when
32Picturing the CUSUM
flag
Shewhart threshold
CUSUM threshold
CUSUM (Si)
Daily count (Xi)
time
33Multivariate generalization of Shewhart
Hotellings T 2
- Hotelling (1947) proposed generalization to
Shewharts procedure - Iteratively calculate
- Flag when T gt h (set empirically)
- Essentially, flag when an observed value is
outside probability ellipsoid (for a multivariate
normal distribution)
34Tailoring Hotellings T 2 to syndromic
surveillance
- T 2 insensitive to direction of shift
- Negative shifts in mean generate a signal, as do
increases - Add in second criteria to stopping rule
- Stop at time i when T gt h and Xi is in a
particular region of the space - Allows for a smaller h for same false positive
rate - ? More sensitive to shifts in desired direction
35Picturing the modified T 2 (2d)
Y
flag
X
36Crosiers Multivariate CUSUMs
- Crosier (1988) proposed various ad hoc CUSUM
methods. His preferred, define
37Tailoring Crosiers MV-CUSUM to syndromic
surveillance
- Unlike univariate CUSUM, Crosiers MV-CUSUM not
reflected at zero - Designed to detect a shift in any direction
- Cum. sums in each dimension can get very negative
- In this problem, need to quickly detect positive
shifts - So bound S in each dimension by zero
38Univariate detection algorithms
- a. Shewhart Alarm if yt gt h
- b. CUSUM (CUmulative SUMmation)
- Ct max Ct-1 (yt - ?) k, 0 Alarm if Ct gt
h - c. Expo (mean-adjusted) CUSUM
- EWMA (Exponentially Weighted Moving Average)
- zt ??yt (1-?)zt-1
- Ct max Ct-1 (yt - zt) k, 0 Alarm if Ct gt
h - Empirical p-values calculated
39Pooling univariate detection algorithms
- Z-analyses (1a, 1b, 1c)
- Set threshold empirically so that daily alarm
rate (probability gt1 series flag) is 1 outside
flu season - P-analyses (2a, 2b, 2c)
- Pool P-values for all series (-2?ln(pi))
- Set threshold so that alarm rate is 1
- C-analyses (3a, 3b, 3c)
- Combine data across hospitals for each syndrome
- Apply Z-analyses
- CP-analyses (4a, 4b, 4c)
- Combine data across hospitals for each syndrome
- Apply P-analyses
40Multivariate detection algorithms
- Z-analyses Multivariate Shewhart (5)
- 5a. Standard
- 5b. With adjustment for lower left quadrant
- C-analyses Multivariate Shewhart (6)
- 6a. Standard
- 6b. With adjustment for lower left quadrant
- Z-analyses Crosiers Multivariate CUSUM (7)
- C-analyses Crosiers Multivariate CUSUM (8)
41Simulation approach
- Seed 1, 2, 10 extra cases over 10 days
- Scenario A Add seed to all series
- 7 hospitals x 4 symptoms
- Scenario C Add same seed to "unspecified
infection" only across all hospitals - Scenario CA Add seed to "unspecified infection"
only (Seed 4, 8, 40) - Scenario D Add same seed to all syndromes in
one mid-sized hospital - Scenario DA Add seed to one mid-sized hospital
only (Seed 7, 14, 70)
42Index ?i pr(flag)i
- MV-CUSUM performs slightly better than Z-CUSUM in
Scenario A and DA - Z-CUSUM performs slightly better (1 day) than
MV-CUSUM in C, equal in CA D.
43Performance summary
Index ?i pr(flag)i
- P analyses do not perform well, except in
Scenario A - C analyses worse than corresponding Z analysis,
especially CUSUM - Z and MV analyses perform best for large
outbreaks (A, CA, DA) - MV-CUSUM (7) best overall, followed by Z-CUSUM
(1b)
44- In flu season Z-CUSUM has 50 false positive
rate and MV-CUSUM has 15 false positive rate. - Z and MV CUSUM both reach 80 probability of
detection at roughly the same time, except for
Scenario C.
45Performance summary
- Outside of flu season
- MV-CUSUM slightly better or equivalent in all but
Scenario C (and only slightly worse there) - 50 sensitivity reached on
- Day 1 for both methods in Scenarios CA and DA
- Day 2 for both methods in Scenario A
- Day 4 for MV and day 5 for Z in Scenario C
- Day 5 for both methods in Scenario D
- In flu season
- false positive rate 50 for Z-CUSUM, 15 for
MV-CUSUM - both reach 80 probability of detection at
roughly the same time, except for Scenario C
46Conclusions
- No detection algorithm best in all situations,
but two do reasonably well in all scenarios
tested - Multivariate CUSUM slightly preferable to
univariate CUSUM (Z-analyses) - Univariate slightly preferable to multivariate
CUSUM in Scenario C (may be the most likely) - Combining P-values generally doesnt work well
- Combining data across hospitals (C-analyses)
doesnt help in Scenario A or for CUSUM - Better for Scenarios A and D, and for Shewhart
and EXPO
47Integrating Syndromic and Public Health
Surveillance
- Syndromic surveillance cannot be expected to
detect attacks with few cases - e.g. anthrax in 2001
- Syndromic surveillance intended to alert public
health officials to possible bioevent - Must be followed with
- Epidemiologic investigation
- Policy decisions regarding intervention
- Syndromic surveillance must be linked to other
surveillance systems in advance
48Appropriate Physician Involvement Is Essential
- Syndromic surveillance efforts often minimize
involvement of physicians - May have consequences after an alert when
physician communication and cooperation is needed
for - Active surveillance
- Epidemiologic investigation
- Mass prophylaxis
- Consider active syndromic surveillance by phone
or computer
49Discussion
- Much impressive work has been done
- Information technology Real-time integration of
many disparate data streams - Analysis Development of background models,
detection algorithms, visualization - Value of syndromic surveillance for bioterrorism
has not yet been demonstrated - Relatively small window between what can be
detected in the first few days and what is
obvious - Better integration with public health systems
needed - Most important contribution may be for natural
disease outbreaks, such as gastro, flu, and
pandemic flu - Design for this rather than focus only on
timeliness