A Martingale Framework for Concept Change Detection in Time-Varying Data Stream

1
A Martingale Framework for Concept Change
Detection in Time-Varying Data Stream Ho
Shen-Shyang sho_at_gmu.edu Department of Computer
Science George Mason University
2
Preview

• Problem In a data streaming setting, data
  points are
• observed one by one. The concepts to be learned
  from
• the data stream may change infinitely often.
• How do we detect the changes efficiently?
• Other Topics Concept Drift, Anamoly detection,
  ... ...
• Testing Exchangeability Online (Vovk et.al.,
  ICML 2003)

3
(No Transcript)
4
Outline

• Background Strangeness, Martingale,
• Exchangeability,
• Martingale Framework - Two Tests
• Theoretical Justifications
• Additional Theoretical Results
• Experimental Results

5
Strangeness Measure (Saunders et. al., IJCAI
1999)
scoring how a data point is different from the
rest.
a

• Support Vector Machine Value of Lagrange
  Multipler
• or Distance from the hyperplane
• (we use SVM/Lagrange Multiplier
• incremental SVM (Cauwenberghs and Poggio, NIPS
  2000))
• K-nearest-neighbor rule A/B where
• A Sum of the distance of a point from
• the k nearest points with the same label
• B Sum of the distance of a point from
• the k nearest points with different label

6
Testing Exchangeability Definitions
Let Zi 1 i lt 8 be a sequence of r.v. A
finite sequence of r.v. Z1,..., Zn is
exchangeable if the joint distribution p(Z1,...,
Zn) is invariant under any permutation of the
indices of the r.v. A martingale is a sequence
of r.v. Mi 0 i lt 8 such that Mn is a
measurable function of Z1,..., Zn for all n 0,
1, ... (M0 is a constant, say 1) and the
conditional expectation of Mn1 given M1,..., Mn
is equal to Mn, i.e. E(Mn1 M1,..., Mn )
Mn
7
Testing Exchangeability (Vovk et. al., ICML 2003)
pn V(Z U zn, ?n)
where e in 0,1 (say 0.92) and M0 1
8
Performing Kolmogorov-Smirnov Test on the
p-value distribution as data is observed one by
one.
Skewed p-value distribution small p-values
inflate the martingale values
9
Martingale Framework Test for Change Detection
Consider the simple null hypothesis H0 no
concept change in the data stream against the
alternative hypothesis H1 concept change occurs
in the data stream
10
Martingale Framework Test for Change Detection
Martingale Test 1 (MT1) 0 lt Mn(e)lt ? where ? is
a positive number. One rejects the null
hypothesis when Mn(e) ?. Martingale Test 2
(MT2) 0 lt Mn(e) - Mn-1(e) lt t where t is a
positive number. One rejects the null hypothesis
when Mn(e) - Mn-1(e) t.
11
Justification for Martingale Test 1 Doob's
Maximal Inequality
Assuming that Mi 0 i lt 8 is a nonnegative
martingale, the Doob's Maximal Inequality states
that for any ? gt 0 and 0 n lt 8,
Hence, if E(Mn) E(M0) 1, then
12
Justification for Martingale Test 2
Hoeffding-Azuma Inequality
Let c1, ..., cm be positive constants and let Y1,
..., Ym be a martingale difference sequence
with Yk ck for each k. Then for any t 0,
At each n, the martingale difference is maximum
and bounded when pn is 1/n for the deterministic
martingale (?n1 for all n)
13
Justification for Martingale Test 2
When m 1, the Hoeffding-Azuma Inequality becomes
Assuming that Mn-1(e) M0(e) 1,
14
Comparison
15
Some Theoretical Results for Martingale Test 1
(Ho Wechsler, UAI 2005)

• Martingale Test based on the Doob's Inequality
  is
• an approximaton of the sequential probability
  ratio test.

Where a is the desirable size (type I error) and
ß is the probability of the type II error

• The mean delay time from the true change point
  is

where
16
Experiments
Number of Correct Detections Number of Detections
Precision Recall
Number of Correct Detections Number of True
Changes
Precision Probability that a detection is
actually correct Recall Probability that the
system recognizes a true change Delay time (for a
detected change) the number of time units from a
true change point to the detected change point,
if any
17
Experimental Results Synthetic Data Stream with
noise (10-D Rotating Hyperplane) Precision and
Recall
18
Experimental Results Synthetic Data Stream
Mean and Median Delay Time
19
Experimental Results Numerical (WaveNorm
TwoNorm) and Categorical data streams (Nursery)
20
Experimental Results Multi-class data streams
(Modified USPS data-set)
Dataset 10 classes, 256 dimensions, 7291 data
points
Data stream 3 classes.
21
Experimental Results Multi-class data streams
(Modified USPS data-set)
22
Conclusions

• Our martingale approach is an efficient,
  one-pass
• incremental algorithm that
• Does not require a sliding window on the data
  stream
• Does not require monitoring the performance of a
• base classifier as data is streaming
• Works well for high dimensional, multiclass data
  stream
• Theoretically justified.

23
Conclusions/Future (Current) Work

• Previous works Kifer et. al. (VLDB 2004),
• Fan et. al.(SDM 2004), Wald (1947), Page (1957)
  ......
• Extension to Unlabeled and One-class data
  streams
• Application Keyframe Extraction, Anomaly
  Detection,
• Adaptive Classifier (Ho and Wechsler, IJCAI 2005)
• Comparison using different classifiers (i.e.
  Different
• strangeness measure, also weak classifiers)
• Comparison with other change detection
  algorithms.
• http//cs.gmu.edu/sho/research/change_detection.
  html
• Acknowledgement Vladimir Vovk, Harry Wechsler.