Background Modeling and Foreground Detection for Video Surveillance

1
Background Modeling and Foreground Detection for
Video SurveillanceRecent Advances and Future
Directions
Thierry BOUWMANS Associate Professor MIA Lab -
University of La Rochelle - France
2
Plan

• Introduction
• Fuzzy Background Subtraction
• Background Subtraction via a Discriminative
  Subspace Learning IMMC
• Foreground Detection via Robust Principal
  Component Analysis (RPCA)
• Conclusion - Perspectives

3
Goal

• Detection of moving objects in video sequence.
• Pixels are classified as

Background(B)
Foreground (F)
Séquence Pets 2006 Image298 (720 x 576 pixels)
4
Background Subtraction Process
Incremental Algorithm
Background Maintenance
t gt N
t N
tt1
Batch Algorithm
t N
N images
Foreground Detection
Background Initialization
F(t)
Video
I(t1)
Foreground Mask
N1
Classification task
5
Related Applications

• Video surveillance
• Optical Motion Capture
• Multimedia Applications

Projet ATON Université de Californie San Diego
Séquence Danse Mikic 2002
Séquence Jump Mikic 2002
Projet Aqu_at_theque Université de La Rochelle
6
On the importance of the background subtraction
7
Challenges

• Critical situations which generate false
  detections

• Shadows -

Illumination variations
Source Séquence Pets 2006 Image 0298 (720 x 576
pixels)
8

• Multimodal Backgrounds

Rippling Water
Water Surface
Camera Jitter
Waving Trees
Source http//perception.i2r.a-star.edu.sg/bk_mod
el/bk_index.html
8
9
Statistical Background Modeling

• Background Subtraction Web Site References
  (553), datasets (10) and codes (27).

Source http//sites.google.com/site/backgroundsub
traction/Home.html (6256 Visitors, Source Google
Analytics).
9
10
Plan

• Introduction
• Fuzzy Background Subtraction
• Background Subtraction via a Discriminative
  Subspace Learning IMMC
• Foreground Detection via Robust Principal
  Component Analysis (RPCA)
• Conclusion - Perspectives

11
Fuzzy Background Subtraction

• A survey in Handbook on Soft Computing for Video
  Surveillance, Taylor and Francis Group HSCVS
  2012
• Three approaches developed at the MIA Lab
• Background modeling by Type-2 Fuzzy Mixture of
  Gaussians Model ISVC 2008.
• Foreground Detection using the Choquet Integral
  WIAMIS 2008FUZZIEEE 2008
• Fuzzy Background Maintenance ICIP 2008

12
Weakness of the original MOG

1. False detections due to the matching test

13
Weakness of the original MOG

1. False detections due to the presence of outliers
   in the training step

Exact distribution
14
Mixture of Gaussians with uncertainty on
the mean and the variance Zeng
2006
(T2 FMOG-UM)
(T2 FMOG-UV)
15
Mixture of Gaussians with uncertainty on the
mean(T2 FMOG-UM)
Intensity vector in the RGB color space
16
Mixture of Gaussians with uncertainty on the
variance (T2 FMOG-UV)
Intensity vector in the RGB color space
17
Classification B/F by T2-FMOG

• Matching test
• Classification B/F as the MOG ?

18
Results on the SHAH dataset(160 x 128 pixels)
Camera Jitter
Video at http//sites.google.com/site/t2fmog/
Original sequence MOG
T2 FMOG-UM (km2) T2 FMOG-UV
(kv0.9)
18
19
Results on the SHAH dataset(160 x 128 pixels)
Camera Jitter
Method Error Type Image 271 Image 373 Image 410 Image 465 Total Error Variation in
MOG FN FP 0 2093 1120 4124 4818 2782 2050 1589 18576
T2-FMOG-UM FN FP 0 203 1414 153 6043 252 2520 46 10631 42,77
T2-FMOG-UV FN FP 0 3069 957 1081 2217 1119 1069 1158 10670 42.56
19
20
Results on the SHAH dataset(160 x 128 pixels)
Camera Jitter
Stauffer 1999
Bowden 2001 Initialization
Zivkovic 2004 K is variable
20
21
Results on the sequence CAMPUS (160 x 128
pixels) Waving Trees
Video at http//sites.google.com/site/t2fmog/
Original Sequence MOG
T2 FMOG-UM (km2) T2 FMOG-UV
(kv0.9)
21
22
Resultat on the sequence Water Surface (160 x
128 pixels) Water Surface
Video at http//sites.google.com/site/t2fmog/
Original Sequence MOG
T2 FMOG-UM (km2) T2 FMOG-UV
(kv0.9)
22
23
Fuzzy Foreground Detection

• Features color, edge, stereo features, motion
  features, texture.
• Multiple features
• More robustness in presence of illumination
  changes, shadows and multimodal backgrounds

24
Choice of the features

• Color (3 components)
• Texture (Local Binary Pattern Heikkila PAMI
  2006)
• For each feature, a similarity (S) is computed
  following its value in the background image and
  its value in the current image.

25
Aggregation of the Color and Texture features
with the Choquet Integral
BG(t)
I(t1)
Texture Features
Color Features
SC,1 SC,2 SC,3 ST
Similarity mesure for the Color
Similarity measure for the Texture
Fuzzy Integral
Classification B/F
Foreground Mask
26
How to compute S for the Color and the Texture?




0 T,C 255
Background Image
Current Image
0 S 1
For the Texture
For the Color
kone of the color components
26
27
Fuzzy operators

•  Sugeno Integral et Choquet Integral
• Uncertainty and imprecision
• Great flexibility
• Fast and simple operations

ordinal
cardinal
28
Data Fusion using the Choquet Integral
Mesures floues
Intégrale de Choquet
29
Fuzzy Foreground Detection

• Classification using the Choquet integral
• If then else
• where Th is constant threshold. is the value
  of the Choquet integral for the pixel (x,y)

30
Aggregation Color, Texture

• Aqu_at_thèque (384 x 288 pixels) - Ohta color space

Integral Color space Choquet Ohta Sugeno Ohta
S(A,B) 0.40 0.27
a) Current image
b) Ground truth
Comparison between the Sugeno and Choquet
Integrals
c) Choquet integral
d) Sugeno integral Zhang 2006
30
31
Aggregation Colors, Texture Ohta, YCrCb, HSV

• Aqu_at_thèque (384 x 288 pixels)

Texture
Color

0.6 0.5 0.5 0.5 0.53 0.3 0.4 0.3 0.39 0.34 0.1 0.1 0.2 0.11 0.13 0.9 0.9 0.8 0.89 0.87 0.7 0.6 0.7 0.61 0.66 0.4 0.5 0.5 0.5 0.47 1 1 1 1 1
Choquet - Ohta
Choquet - YCrCb
Choquet - HSV
Integral Color Space Ohta YCrCb HSV
S(A,B) 0.40 0.42 0.30
Values of the fuzzy measures
Evaluation of the Choquet integral for different
color spaces
31
32
Aggregation Color, Texture

• VS-Pets 2003 (720 x 576)

Current Image Choquet - YCrCb Sugeno
Ohta Zhang 2006
33
Aggregation Colors Pets 2006 (384 x 288 pixels)

Original sequence Ground truth
OR Sugeno Integral
Choquet Integral
YCrCb
Ohta
HSV
33
34
Fuzzy Background maintenance

• No-selective rule
• Selective rule
• Here, the idea is to adapt very quickly a pixel
  classified as
• background and very slowly a pixel classified as
  foreground.

35
Fuzzy adaptive rule
and

• Combination of the update rules of the selective
  scheme

36
Results on the Wallflower dataset
Sequence Time of Day
Original Image 1850
Ground Truth
Fuzzy adaptive rule
No selective rule
Selective rule
Similarity measure
No selective Selective Fuzzy adaptive
S(A,B) 58.40 57.08 58.96
36
37
Computation Time
Algorithm Frames/Second
T2-FMOG-UM 11
T2-FMOG-UV 12
MOG 20
Choquet integral 31
Sugeno integral 22
OR 40
Resolution 384288, RGB, Pentium 1,66GHz, RAM 1GB
38
Assessment
Perspectives

• Fuzzy Background Modeling by T2-FMOG
• Multimodal Backgrounds
• Fuzzy Foreground Detection using multi-features
• Fuzzy Background Maintenance

• Using fuzzy approaches in other statistical
  models.
• Using more than two features
• Fuzzy measures by learning

39
Plan

• Introduction
• Fuzzy Background Subtraction
• Background Subtraction via a Discriminative
  Subspace Learning IMMC
• Foreground Detection via Robust Principal
  Component Analysis (RPCA)
• Conclusion - Perspectives

40
Background Modeling and Foreground Detection via
a Discriminative Subspace Learning (MIA Lab)

• Reconstructive subspace learning models (PCA,
  ICA, IRT) RPCS 2009
• Assumption The main information contained in
  the training sequence is the background meaning
  that the foreground has a low contribution.
• However, this assumption is only verified when
  the moving objects are either small or far away
  from the camera.

41
Discriminative Subspace Learning

• Advantages
• More efficient and often give better
  classification results.
• Robust supervised initialization of the
  background
• Incremental update of the eigenvectors and
  eigenvalues.
• Approach developed at the MIA Lab
• Background initialization via MMC MVA 2012
• Background maintenance via Incremental Maximum
  Margin Criterion (IMMC) MVA 2012

42
Background Subtraction via Incremental Maximum
Margin Criterion

• Denote the training video sequences S I1,
  ...IN
• where It is the frame at time t
• N is the number of training frames.
• Let each pixel (x,y) be characterized by its
  intensity in the grey scale and asssume that we
  have the ground truth corresponding to this
  training video sequence, i.e we know for each
  pixel its class label that can be foreground or
  background.

43
Background Subtraction via Incremental Maximum
Margin Criterion

• Thus, we compute respectively the inter-class
  scatter matrix Sb and the intra-class scatter
  matrix Sw
• where c 2
• I is the mean of the intensity of the pixel
  (x,y) over the training video
• Ii is the mean of samples belonging to class i
• pi is the prior probability for a sample
  belonging to class i (Background, Foreground).

44
Background Subtraction via Incremental Maximum
Margin Criterion

• Batch Maximum Margin Criterion algorithm.
• Extract the first leading eigenvectors that
  correspond to the background. The corresponding
  eigenvalues are contained in the matrix LM and
  the leading eigenvectors in the matrix FM.
• The current image It can be approximated by the
  mean background and weighted sum of the leading
  eigenbackgrounds FM.

45
Background Subtraction via Incremental Maximum
Margin Criterion

• The coordinates in leading eigenbackground space
  of the current image It can be computed
• When wt is back projected onto the image space,
  the background image is created

46
Background Subtraction via Incremental Maximum
Margin Criterion

• Foreground detection
• Background maintenance via IMMC

47
Principle - Illustration
Current Image
IBackground
IForeground
Background image
Foreground mask
48
Results on the Wallflower dataset
Original image, ground truth , SG, MOG, KDE,
PCA, INMF, IRT, IMMC (30), IMMC (100)
49
Assessment
Perspectives

• Advantages
• Robust supervised initialization of the
  background.
• Incremental update of the eigenvectors and
  eigenvalues.
• Disadvantages
• Needs ground truth in the training step.
• Others Discriminative Subspace Learning methods
  such as LDA.

50
Plan

• Introduction
• Fuzzy Background Subtraction
• Background Subtraction via a Discriminative
  Subspace Learning IMMC
• Foreground Detection via Robust Principal
  Component Analysis (RPCA)
• Conclusion - Perspectives

51
Foreground Detection via Robust Principal
Component Analysis

• PCA (Oliver et al 1999) Not robust to outliers.
• Robust PCA (Candes et al. 2011) Decomposition
  into low-rank and sparse matrices
• Approach developed at the MIA Lab
• Validation ICIP 2012ICIAR 2012ISVC 2012
• RPCA via Iterative Reweighted Least Squares BMC
  2012

52
Robust Principal Component Analysis

• Candes et al. (ACM 2011) proposed a convex
  optimization to address the robust PCA problem.
  The observation matrix A is assumed represented
  as
• where L is a low-rank matrix and S must be sparse
  matrix with a small fraction of nonzero entries.

http//perception.csl.illinois.edu/matrix-rank/hom
e.html
53
Robust Principal Component Analysis

• This research seeks to solve for L with the
  following optimization problem
• where . and .1 are the nuclear norm
  (which is the l1-norm of singular value) and
  l1-norm, respectively, and ? gt 0 is an arbitrary
  balanced parameter.
• Under these minimal assumptions, this approach
  called Principal Component Pursuit (PCP) solution
  perfectly recovers the low-rank and the sparse
  matrices.

54
Algorithms for solving PCP
Time required to solve a 1000x1000106 RPCA
problem
Algorithms Accuracy Rank E_0 iterations time (sec)
IT 5.99e-006 50 101,268 8,550 119,370.3
DUAL 8.65e-006 50 100,024 822 1,855.4
APG 5.85e-006 50 100,347 134 1,468.9
APGP 5.91e-006 50 100,347 134 82.7
ALMP 2.07e-007 50 100,014 34 37.5
ADMP 3.83e-007 50 99,996 23 11.8
Source Z. Lin , Y. Ma The Pursuit of
Low-dimensional Structures in High-dimensional
(Visual) Data Fast and Scalable Algorithms
Time required is still acceptable for ADM but for
background modeling and foreground detection?
55
Application to Background Modeling and Foreground
Detection
n is the amount of pixels in a frame (106) m is
the number of frames considered (200) Computation
time is 200 12s 40 minutes!!!
Source http//perception.csl.illinois.edu/matrix-
rank/home.html
56
PCP and its application to Background Modeling
and Foreground Detection

• Only visual validations are provided!!!
• Limitations
• Spatio-temporal aspect None!
• Real Time Aspect PCP takes 40 minutes with the
  ADM!!!
• Incremental Aspect PCP is a batch algorithm. For
  example, (Candes et al. 2011) collected 200
  images.

57
PCP and its variants

• How to improve PCP?
• Algorithms for solving PCP (17 Algorithms)
• Incremental PCP (5 papers)
• Real-Time PCP (2 papers)
• Validation for background modeling and foreground
  detection (3 papers) ICIP 2012ICIAR 2012ISVC
  2012

Source T. Bouwmans, Foreground Detection using
Principal Component Pursuit A Survey, under
preparation.
58
PCP and its variants
Source T. Bouwmans, Foreground Detection using
Principal Component Pursuit A Survey, under
preparation.
59
Validation Background Modeling and Foreground
Detection Qualitative Evaluation
Source ICIP 2012, ICIAR 2012, ISVC 2012
60
Validation Background Modeling and Foreground
Detection Quantitative Evaluation
F-Measure
Block PCP gives the best performance!
Source ICIP 2012, ICIAR 2012, ISVC 2012
61
PCP and its application to Background Modeling
and Foreground Detection

• Recent improvements
• BPCP (Tang et Nehorai (2012)) Spatial but not
  incremental and not real time!
• Recursive Robust PCP (Qiu and Vaswani (2012) )
  Incremental but not real time!
• Real Time Implementation on GPU (Anderson et al.
  (2012) ) Real time but not incremental!
• What we can do?
• Research on real time incremental robust PCP!

62
Conclusion
Perspectives

• Fuzzy Background Subtraction
• Background Subtraction via a Discriminative
  Subspace Learning IMMC
• Foreground Detection via Robust Principal
  Component Analysis (RPCA)
• Fuzzy Learning Rate
• Other Discriminative Subspace Learning methods
  such as LDA
• Incremental and real time RPCA

63
Publications
Fuzzy Background Subtraction

• Chapter
• T. Bouwmans, Background Subtraction For Visual
  Surveillance A Fuzzy Approach, Handbook on Soft
  Computing for Video Surveillance, Taylor and
  Francis Group, Chapter 5, March 2012.
• International Conferences
• F. El Baf, T. Bouwmans, B. Vachon, Fuzzy
  Statistical Modeling of Dynamic Backgrounds for
  Moving Object Detection in Infrared Videos, CVPR
  2009 Workshop, pages 1-6, Miami, USA, 22 June
  2009.
• F. El Baf, T. Bouwmans, B. Vachon, Type-2 Fuzzy
  Mixture of Gaussians Model Application to
  Background Modeling, ISVC 2008, pages 772-781,
  Las Vegas, USA, December 2008
• F. El Baf, T. Bouwmans, B. Vachon, A Fuzzy
  Approach for Background Subtraction, ICIP 2008,
  San Diego, California, U.S.A, October 2008.
• F. El Baf, T. Bouwmans, B. Vachon. " Fuzzy
  Integral for Moving Object Detection ",
  IEEE-FUZZY 2008, Hong Kong, China, June 2008.
• F. El Baf, T. Bouwmans, B. Vachon, Fuzzy
  Foreground Detection for Infrared Videos, CVPR
  2008 Workshop, pages 1-6, Anchorage, Alaska, USA,
  27 June 2008.
• F. El Baf, T. Bouwmans, B. Vachon, Foreground
  Detection using the Choquet Integral,
  International Workshop on Image Analysis for
  Multimedia Interactive Services, WIAMIS 2008,
  pages 187-190, Klagenfurt, Austria, May 2008.

64
Publications
Background Subtraction via IMMC

• Journal
• D. Farcas, C. Marghes, T. Bouwmans, Background
  Subtraction via Incremental Maximum Margin
  Criterion A discriminative approach , Machine
  Vision and Applications, March 2012.
• International Conferences
• C. Marghes, T. Bouwmans, "Background Modeling
  via Incremental Maximum Margin Criterion",
  International Workshop on Subspace Methods, ACCV
  2010 Workshop Subspace 2010, Queenstown, New
  Zealand, November 2010.
• D. Farcas, T. Bouwmans, "Background Modeling via
  a Supervised Subspace Learning", International
  Conference on Image, Video Processing and
  Computer Vision, IVPCV 2010, pages 1-7, Orlando,
  USA , July 2010.

65
Publications
Foreground Detection via RPCA

• Chapter
• C. Guyon, T. Bouwmans, E. Zahzah, Robust
  Principal Component Analysis for Background
  Subtraction Systematic Evaluation and
  Comparative Analysis, INTECH, Principal
  Component Analysis, Book 1, Chapter 12, page
  223-238, March 2012.
• International Conferences
• C. Guyon, T. Bouwmans. E. Zahzah, Foreground
  Detection via Robust Low Rank Matrix
  Factorization including Spatial Constraint with
  Iterative Reweighted Regression, International
  Conference on Pattern Recognition, ICPR 2012,
  Tsukuba, Japan, November 2012.
• C. Guyon, T. Bouwmans. E. Zahzah, Moving Object
  Detection via Robust Low Rank Matrix
  Decomposition with IRLS scheme, International
  Symposium on Visual Computing, ISVC 2012,pages
  665674, Rethymnon, Crete, Greece, July 2012.
• C. Guyon, T. Bouwmans, E. Zahzah, Moving Object
  Detection by Robust PCA solved via a Linearized
  Symmetric Alternating Direction Method,
  International Symposium on Visual Computing, ISVC
  2012, pages 427-436, Rethymnon, Crete, Greece,
  July 2012.
• C. Guyon, T. Bouwmans, E. Zahzah, "Foreground
  Detection by Robust PCA solved via a Linearized
  Alternating Direction Method", International
  Conference on Image Analysis and Recognition,
  ICIAR 2012, pages 115-122, Aveiro, Portugal, June
  2012.
• C. Guyon, T. Bouwmans, E. Zahzah, "Foreground
  detection based on low-rank and block-sparse
  matrix decomposition", IEEE International
  Conference on Image Processing, ICIP 2012,
  Orlando, Florida, September 2012.