Robust Multi-Pedestrian Tracking in Thermal-Visible Surveillance Videos

1
Robust Multi-Pedestrian Tracking in
Thermal-Visible Surveillance Videos

• Alex Leykin and Riad Hammoud

2
Goal

• Create a pedestrian tracker that operates in
• Varying illumination conditions
• Crowded environment
• To achieve it we create a fusion pedestrian
  tracker that uses input from
• IR camera
• RGB camera
• Our approach consists of two stages

adaptive fusion background model
Bayesian tracker
blob array
3
Related Work

• Fusion background model
• Y.Owechko, S.Medasani, and N.Srinivasa
  Classifier swarms for human detection in
  infrared imagery, OTCBVS 2004
• M.Yasuno, N.Yasuda, andM.Aoki Pedestrian
  detection and tracking in far infrared images
  OTCBVS 2004
• C. Dai, Y. Zheng, X. Li Layered Representation
  for Pedestrian Detection and Tracking in Infrared
  Imagery OTCBVS 2005
• J.Davis, V.Sharma Fusion-based Background
  Subtraction Using Contour Saliency, OTCBVS 2005
• Bayesian formulation
• J. Deutscher, B. North, B. Bascle and A. Blake
  Tracking through singularities and
  discontinuities by random sampling, ICCV 1999
• A. Elgammal and L. S. Davis, Probabilistic
  Framework for Segmenting People Under Occlusion,
  ICCV 2001.
• M. Isard, J. MacCormick, BraMBLe a Bayesian
  multiple-blob tracker, ICCV 2001
• T. Zhao, R. Nevatia Tracking Multiple Humans in
  Crowded Environment, CVPR 2004

4
Background Model

• Two stacks of codeword values (codebooks)

• Color
• µRGB
• Ilow
• Ihi

• Thermal
• thigh
• tlow

5
Adaptive Background Update

• Match pixel p to the codebook b

I(p) gt Ilow I(p) lt Ihigh (RGB(p) µRGB) lt TRGB
t(p)/thigh gt Tt1 t(p)/tlow gt Tt2

• If there is no match create new codeword
• Else update the codeword with new pixel
  information
• If gt1 matches then merge matching codewords
• Remove the codeword if it had not appeared for a
  prolonged period of time
• Discard infrequent codewords
• Exclude p from update if it corresponds to a
  currently tracked body

6
Subtraction Results
Color model only
Combined color and thermal model
7
Tracking

• Location of each pedestrian is estimated
  probabilistically based on
• Current image
• Model of pedestrians
• Model of obstacles

The goal of our tracking system is to find the
candidate state x (a set of bodies along with
their parameters) which, given the last known
state x, will best fit the current observation z
P(x z, x) P(zx) P(xx)
8
Tracking Accepting the State
x and x ? candidate and current states
P(x) ? stationary distribution of Markov chain
mt ? proposal distribution
Candidate proposal state x is drawn with
probability mt(xx) and then accept it with the
probability a(x, x)
9
Tracking Priors
Constraints on the body parameters
N(hµ, hs2) and N(wµ,ws2) ? body width and height
Temporal continuity
d(wt, wt-1) and d(ht, ht-1) ? variation from the
previous size
N(µdoor, sdoor) ? distance to the closest door
(for new bodies)
10
Tracking Likelihoods Distance weight plane
Problem blob trackers ignore blob position in 3D
(see Zhao and Nevatia CVPR 2004)
Solution employ distance weight plane Dxy
Pxyz, Cxyz where P and C are world coordinates
of the camera and reference point correspondingly
and
11
Tracking Likelihoods Z-buffer
0 background, 1furthermost body, 2 next
closest body, etc
12
Tracking Likelihoods
Color observation likelihood is based on the
Bhattacharya distance between candidate and
observed color histograms
Implementation of z-buffer (Z) and distance
weight plane (D) allows to compute multiple-body
configuration with one computationally efficient
step. Let I - set of all blob pixels O - set
of body pixels Then
13
Tracking Jump-Diffuse Transitions

• Add a new body
• Delete a body
• Recover a recently deleted body
• Change body dimensions
• Change body position

14
Tracking Anisotropic Weighted Mean Shift
Classic Mean-Shift
Our Mean-Shift
15
Results
16
Results
17
Results
Sequence Frames People People missed Frames missed False hits Frames in false hits Identity switches
1 1054 15 3 20 1 1 3
2 0601 8 0 0 3 2 4
3 1700 16 5 10 14 5 15
4 1506 3 0 0 0 0 1
5 2031 2 0 0 0 0 2
6 1652 4 0 0 0 0 1
100 100 16 5 6.2 1.3 8.9
18
Conclusions

• A method to fuse visible and thermal inputs for
  background model creation
• robust to illumination changes
• adaptive
• computationally efficient (30fps)
• A novel formulation of priors in MCMC particle
  filter

19
Future Work

• Extend binary background mask with foreground
  probability values
• Incorporate these probabilities into
  appearance-based fitness equation for particle
  filter-based tracker
• Utilize tracklet stitching to decrease the number
  of broken paths

20
Thank you!