Background Modeling and Foreground Detection for Video Surveillance - PowerPoint PPT Presentation

View by Category
About This Presentation
Title:

Background Modeling and Foreground Detection for Video Surveillance

Description:

Background Modeling and Foreground Detection for Video Surveillance: Recent Advances and Future Directions – PowerPoint PPT presentation

Number of Views:352

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Background Modeling and Foreground Detection for Video Surveillance


1
Background Modeling and Foreground Detection for
Video SurveillanceRecent Advances and Future
Directions
Thierry BOUWMANS Associate Professor MIA Lab -
University of La Rochelle - France
2
Plan
  • Introduction
  • Fuzzy Background Subtraction
  • Background Subtraction via a Discriminative
    Subspace Learning IMMC
  • Foreground Detection via Robust Principal
    Component Analysis (RPCA)
  • Conclusion - Perspectives

3
Goal
  • Detection of moving objects in video sequence.
  • Pixels are classified as

Background(B)
Foreground (F)
Séquence Pets 2006 Image298 (720 x 576 pixels)
4
Background Subtraction Process
Incremental Algorithm
Background Maintenance
t gt N
t N
tt1
Batch Algorithm
t N
N images
Foreground Detection
Background Initialization
F(t)
Video
I(t1)
Foreground Mask
N1
Classification task
5
Related Applications
  • Video surveillance
  • Optical Motion Capture
  • Multimedia Applications

Projet ATON Université de Californie San Diego
Séquence Danse Mikic 2002
Séquence Jump Mikic 2002
Projet Aqu_at_theque Université de La Rochelle
6
On the importance of the background subtraction
7
Challenges
  • Critical situations which generate false
    detections
  • Shadows -

Illumination variations
Source Séquence Pets 2006 Image 0298 (720 x 576
pixels)
8
  • Multimodal Backgrounds

Rippling Water
Water Surface
Camera Jitter
Waving Trees
Source http//perception.i2r.a-star.edu.sg/bk_mod
el/bk_index.html
8
9
Statistical Background Modeling
  • Background Subtraction Web Site References
    (553), datasets (10) and codes (27).

Source http//sites.google.com/site/backgroundsub
traction/Home.html (6256 Visitors, Source Google
Analytics).
9
10
Plan
  • Introduction
  • Fuzzy Background Subtraction
  • Background Subtraction via a Discriminative
    Subspace Learning IMMC
  • Foreground Detection via Robust Principal
    Component Analysis (RPCA)
  • Conclusion - Perspectives

11
Fuzzy Background Subtraction
  • A survey in Handbook on Soft Computing for Video
    Surveillance, Taylor and Francis Group HSCVS
    2012
  • Three approaches developed at the MIA Lab
  • Background modeling by Type-2 Fuzzy Mixture of
    Gaussians Model ISVC 2008.
  • Foreground Detection using the Choquet Integral
    WIAMIS 2008FUZZIEEE 2008
  • Fuzzy Background Maintenance ICIP 2008

12
Weakness of the original MOG
  1. False detections due to the matching test

13
Weakness of the original MOG
  1. False detections due to the presence of outliers
    in the training step

Exact distribution
14
Mixture of Gaussians with uncertainty on
the mean and the variance Zeng
2006
(T2 FMOG-UM)
(T2 FMOG-UV)
15
Mixture of Gaussians with uncertainty on the
mean(T2 FMOG-UM)
Intensity vector in the RGB color space
16
Mixture of Gaussians with uncertainty on the
variance (T2 FMOG-UV)
Intensity vector in the RGB color space
17
Classification B/F by T2-FMOG
  • Matching test
  • Classification B/F as the MOG ?

18
Results on the SHAH dataset(160 x 128 pixels)
Camera Jitter
Video at http//sites.google.com/site/t2fmog/
Original sequence MOG
T2 FMOG-UM (km2) T2 FMOG-UV
(kv0.9)
18
19
Results on the SHAH dataset(160 x 128 pixels)
Camera Jitter
Method Error Type Image 271 Image 373 Image 410 Image 465 Total Error Variation in
MOG FN FP 0 2093 1120 4124 4818 2782 2050 1589 18576
T2-FMOG-UM FN FP 0 203 1414 153 6043 252 2520 46 10631 42,77
T2-FMOG-UV FN FP 0 3069 957 1081 2217 1119 1069 1158 10670 42.56
19
20
Results on the SHAH dataset(160 x 128 pixels)
Camera Jitter
Stauffer 1999
Bowden 2001 Initialization
Zivkovic 2004 K is variable
20
21
Results on the sequence CAMPUS (160 x 128
pixels) Waving Trees
Video at http//sites.google.com/site/t2fmog/
Original Sequence MOG
T2 FMOG-UM (km2) T2 FMOG-UV
(kv0.9)
21
22
Resultat on the sequence Water Surface (160 x
128 pixels) Water Surface
Video at http//sites.google.com/site/t2fmog/
Original Sequence MOG
T2 FMOG-UM (km2) T2 FMOG-UV
(kv0.9)
22
23
Fuzzy Foreground Detection
  • Features color, edge, stereo features, motion
    features, texture.
  • Multiple features
  • More robustness in presence of illumination
    changes, shadows and multimodal backgrounds

24
Choice of the features
  • Color (3 components)
  • Texture (Local Binary Pattern Heikkila PAMI
    2006)
  • For each feature, a similarity (S) is computed
    following its value in the background image and
    its value in the current image.

25
Aggregation of the Color and Texture features
with the Choquet Integral
BG(t)
I(t1)
Texture Features
Color Features
SC,1 SC,2 SC,3 ST
Similarity mesure for the Color
Similarity measure for the Texture
Fuzzy Integral
Classification B/F
Foreground Mask
26
How to compute S for the Color and the Texture?




0 T,C 255
Background Image
Current Image
0 S 1
For the Texture
For the Color
kone of the color components
26
27
Fuzzy operators
  •  Sugeno Integral et Choquet Integral
  • Uncertainty and imprecision
  • Great flexibility
  • Fast and simple operations

ordinal
cardinal
28
Data Fusion using the Choquet Integral
Mesures floues
Intégrale de Choquet
29
Fuzzy Foreground Detection
  • Classification using the Choquet integral
  • If then else
  • where Th is constant threshold. is the value
    of the Choquet integral for the pixel (x,y)

30
Aggregation Color, Texture
  • Aqu_at_thèque (384 x 288 pixels) - Ohta color space

Integral Color space Choquet Ohta Sugeno Ohta
S(A,B) 0.40 0.27
a) Current image
b) Ground truth
Comparison between the Sugeno and Choquet
Integrals
c) Choquet integral
d) Sugeno integral Zhang 2006
30
31
Aggregation Colors, Texture Ohta, YCrCb, HSV
  • Aqu_at_thèque (384 x 288 pixels)

Texture
Color

0.6 0.5 0.5 0.5 0.53 0.3 0.4 0.3 0.39 0.34 0.1 0.1 0.2 0.11 0.13 0.9 0.9 0.8 0.89 0.87 0.7 0.6 0.7 0.61 0.66 0.4 0.5 0.5 0.5 0.47 1 1 1 1 1
Choquet - Ohta
Choquet - YCrCb
Choquet - HSV
Integral Color Space Ohta YCrCb HSV
S(A,B) 0.40 0.42 0.30
Values of the fuzzy measures
Evaluation of the Choquet integral for different
color spaces
31
32
Aggregation Color, Texture
  • VS-Pets 2003 (720 x 576)

Current Image Choquet - YCrCb Sugeno
Ohta Zhang 2006
33
Aggregation Colors Pets 2006 (384 x 288 pixels)

Original sequence Ground truth
OR Sugeno Integral
Choquet Integral
YCrCb
Ohta
HSV
33
34
Fuzzy Background maintenance
  • No-selective rule
  • Selective rule
  • Here, the idea is to adapt very quickly a pixel
    classified as
  • background and very slowly a pixel classified as
    foreground.

35
Fuzzy adaptive rule
and
  • Combination of the update rules of the selective
    scheme

36
Results on the Wallflower dataset
Sequence Time of Day
Original Image 1850
Ground Truth
Fuzzy adaptive rule
No selective rule
Selective rule
Similarity measure
No selective Selective Fuzzy adaptive
S(A,B) 58.40 57.08 58.96
36
37
Computation Time
Algorithm Frames/Second
T2-FMOG-UM 11
T2-FMOG-UV 12
MOG 20
Choquet integral 31
Sugeno integral 22
OR 40
Resolution 384288, RGB, Pentium 1,66GHz, RAM 1GB
38
Assessment
Perspectives
  • Fuzzy Background Modeling by T2-FMOG
  • Multimodal Backgrounds
  • Fuzzy Foreground Detection using multi-features
  • Fuzzy Background Maintenance
  • Using fuzzy approaches in other statistical
    models.
  • Using more than two features
  • Fuzzy measures by learning

39
Plan
  • Introduction
  • Fuzzy Background Subtraction
  • Background Subtraction via a Discriminative
    Subspace Learning IMMC
  • Foreground Detection via Robust Principal
    Component Analysis (RPCA)
  • Conclusion - Perspectives

40
Background Modeling and Foreground Detection via
a Discriminative Subspace Learning (MIA Lab)
  • Reconstructive subspace learning models (PCA,
    ICA, IRT) RPCS 2009
  • Assumption The main information contained in
    the training sequence is the background meaning
    that the foreground has a low contribution.
  • However, this assumption is only verified when
    the moving objects are either small or far away
    from the camera.

41
Discriminative Subspace Learning
  • Advantages
  • More efficient and often give better
    classification results.
  • Robust supervised initialization of the
    background
  • Incremental update of the eigenvectors and
    eigenvalues.
  • Approach developed at the MIA Lab
  • Background initialization via MMC MVA 2012
  • Background maintenance via Incremental Maximum
    Margin Criterion (IMMC) MVA 2012

42
Background Subtraction via Incremental Maximum
Margin Criterion
  • Denote the training video sequences S I1,
    ...IN
  • where It is the frame at time t
  • N is the number of training frames.
  • Let each pixel (x,y) be characterized by its
    intensity in the grey scale and asssume that we
    have the ground truth corresponding to this
    training video sequence, i.e we know for each
    pixel its class label that can be foreground or
    background.

43
Background Subtraction via Incremental Maximum
Margin Criterion
  • Thus, we compute respectively the inter-class
    scatter matrix Sb and the intra-class scatter
    matrix Sw
  • where c 2
  • I is the mean of the intensity of the pixel
    (x,y) over the training video
  • Ii is the mean of samples belonging to class i
  • pi is the prior probability for a sample
    belonging to class i (Background, Foreground).

44
Background Subtraction via Incremental Maximum
Margin Criterion
  • Batch Maximum Margin Criterion algorithm.
  • Extract the first leading eigenvectors that
    correspond to the background. The corresponding
    eigenvalues are contained in the matrix LM and
    the leading eigenvectors in the matrix FM.
  • The current image It can be approximated by the
    mean background and weighted sum of the leading
    eigenbackgrounds FM.

45
Background Subtraction via Incremental Maximum
Margin Criterion
  • The coordinates in leading eigenbackground space
    of the current image It can be computed
  • When wt is back projected onto the image space,
    the background image is created

46
Background Subtraction via Incremental Maximum
Margin Criterion
  • Foreground detection
  • Background maintenance via IMMC

47
Principle - Illustration
Current Image
IBackground
IForeground
Background image
Foreground mask
48
Results on the Wallflower dataset
Original image, ground truth , SG, MOG, KDE,
PCA, INMF, IRT, IMMC (30), IMMC (100)
49
Assessment
Perspectives
  • Advantages
  • Robust supervised initialization of the
    background.
  • Incremental update of the eigenvectors and
    eigenvalues.
  • Disadvantages
  • Needs ground truth in the training step.
  • Others Discriminative Subspace Learning methods
    such as LDA.

50
Plan
  • Introduction
  • Fuzzy Background Subtraction
  • Background Subtraction via a Discriminative
    Subspace Learning IMMC
  • Foreground Detection via Robust Principal
    Component Analysis (RPCA)
  • Conclusion - Perspectives

51
Foreground Detection via Robust Principal
Component Analysis
  • PCA (Oliver et al 1999) Not robust to outliers.
  • Robust PCA (Candes et al. 2011) Decomposition
    into low-rank and sparse matrices
  • Approach developed at the MIA Lab
  • Validation ICIP 2012ICIAR 2012ISVC 2012
  • RPCA via Iterative Reweighted Least Squares BMC
    2012

52
Robust Principal Component Analysis
  • Candes et al. (ACM 2011) proposed a convex
    optimization to address the robust PCA problem.
    The observation matrix A is assumed represented
    as
  • where L is a low-rank matrix and S must be sparse
    matrix with a small fraction of nonzero entries.

http//perception.csl.illinois.edu/matrix-rank/hom
e.html
53
Robust Principal Component Analysis
  • This research seeks to solve for L with the
    following optimization problem
  • where . and .1 are the nuclear norm
    (which is the l1-norm of singular value) and
    l1-norm, respectively, and ? gt 0 is an arbitrary
    balanced parameter.
  • Under these minimal assumptions, this approach
    called Principal Component Pursuit (PCP) solution
    perfectly recovers the low-rank and the sparse
    matrices.

54
Algorithms for solving PCP
Time required to solve a 1000x1000106 RPCA
problem
Algorithms Accuracy Rank E_0 iterations time (sec)
IT 5.99e-006 50 101,268 8,550 119,370.3
DUAL 8.65e-006 50 100,024 822 1,855.4
APG 5.85e-006 50 100,347 134 1,468.9
APGP 5.91e-006 50 100,347 134 82.7
ALMP 2.07e-007 50 100,014 34 37.5
ADMP 3.83e-007 50 99,996 23 11.8
Source Z. Lin , Y. Ma The Pursuit of
Low-dimensional Structures in High-dimensional
(Visual) Data Fast and Scalable Algorithms
Time required is still acceptable for ADM but for
background modeling and foreground detection?
55
Application to Background Modeling and Foreground
Detection
n is the amount of pixels in a frame (106) m is
the number of frames considered (200) Computation
time is 200 12s 40 minutes!!!
Source http//perception.csl.illinois.edu/matrix-
rank/home.html
56
PCP and its application to Background Modeling
and Foreground Detection
  • Only visual validations are provided!!!
  • Limitations
  • Spatio-temporal aspect None!
  • Real Time Aspect PCP takes 40 minutes with the
    ADM!!!
  • Incremental Aspect PCP is a batch algorithm. For
    example, (Candes et al. 2011) collected 200
    images.

57
PCP and its variants
  • How to improve PCP?
  • Algorithms for solving PCP (17 Algorithms)
  • Incremental PCP (5 papers)
  • Real-Time PCP (2 papers)
  • Validation for background modeling and foreground
    detection (3 papers) ICIP 2012ICIAR 2012ISVC
    2012

Source T. Bouwmans, Foreground Detection using
Principal Component Pursuit A Survey, under
preparation.
58
PCP and its variants
Source T. Bouwmans, Foreground Detection using
Principal Component Pursuit A Survey, under
preparation.
59
Validation Background Modeling and Foreground
Detection Qualitative Evaluation
Source ICIP 2012, ICIAR 2012, ISVC 2012
60
Validation Background Modeling and Foreground
Detection Quantitative Evaluation
F-Measure
Block PCP gives the best performance!
Source ICIP 2012, ICIAR 2012, ISVC 2012
61
PCP and its application to Background Modeling
and Foreground Detection
  • Recent improvements
  • BPCP (Tang et Nehorai (2012)) Spatial but not
    incremental and not real time!
  • Recursive Robust PCP (Qiu and Vaswani (2012) )
    Incremental but not real time!
  • Real Time Implementation on GPU (Anderson et al.
    (2012) ) Real time but not incremental!
  • What we can do?
  • Research on real time incremental robust PCP!

62
Conclusion
Perspectives
  • Fuzzy Background Subtraction
  • Background Subtraction via a Discriminative
    Subspace Learning IMMC
  • Foreground Detection via Robust Principal
    Component Analysis (RPCA)
  • Fuzzy Learning Rate
  • Other Discriminative Subspace Learning methods
    such as LDA
  • Incremental and real time RPCA

63
Publications
Fuzzy Background Subtraction
  • Chapter
  • T. Bouwmans, Background Subtraction For Visual
    Surveillance A Fuzzy Approach, Handbook on Soft
    Computing for Video Surveillance, Taylor and
    Francis Group, Chapter 5, March 2012.
  • International Conferences
  • F. El Baf, T. Bouwmans, B. Vachon, Fuzzy
    Statistical Modeling of Dynamic Backgrounds for
    Moving Object Detection in Infrared Videos, CVPR
    2009 Workshop, pages 1-6, Miami, USA, 22 June
    2009.
  • F. El Baf, T. Bouwmans, B. Vachon, Type-2 Fuzzy
    Mixture of Gaussians Model Application to
    Background Modeling, ISVC 2008, pages 772-781,
    Las Vegas, USA, December 2008
  • F. El Baf, T. Bouwmans, B. Vachon, A Fuzzy
    Approach for Background Subtraction, ICIP 2008,
    San Diego, California, U.S.A, October 2008.
  • F. El Baf, T. Bouwmans, B. Vachon. " Fuzzy
    Integral for Moving Object Detection ",
    IEEE-FUZZY 2008, Hong Kong, China, June 2008.
  • F. El Baf, T. Bouwmans, B. Vachon, Fuzzy
    Foreground Detection for Infrared Videos, CVPR
    2008 Workshop, pages 1-6, Anchorage, Alaska, USA,
    27 June 2008.
  • F. El Baf, T. Bouwmans, B. Vachon, Foreground
    Detection using the Choquet Integral,
    International Workshop on Image Analysis for
    Multimedia Interactive Services, WIAMIS 2008,
    pages 187-190, Klagenfurt, Austria, May 2008.

64
Publications
Background Subtraction via IMMC
  • Journal
  • D. Farcas, C. Marghes, T. Bouwmans, Background
    Subtraction via Incremental Maximum Margin
    Criterion A discriminative approach , Machine
    Vision and Applications, March 2012.
  • International Conferences
  • C. Marghes, T. Bouwmans, "Background Modeling
    via Incremental Maximum Margin Criterion",
    International Workshop on Subspace Methods, ACCV
    2010 Workshop Subspace 2010, Queenstown, New
    Zealand, November 2010.
  • D. Farcas, T. Bouwmans, "Background Modeling via
    a Supervised Subspace Learning", International
    Conference on Image, Video Processing and
    Computer Vision, IVPCV 2010, pages 1-7, Orlando,
    USA , July 2010.

65
Publications
Foreground Detection via RPCA
  • Chapter
  • C. Guyon, T. Bouwmans, E. Zahzah, Robust
    Principal Component Analysis for Background
    Subtraction Systematic Evaluation and
    Comparative Analysis, INTECH, Principal
    Component Analysis, Book 1, Chapter 12, page
    223-238, March 2012.
  • International Conferences
  • C. Guyon, T. Bouwmans. E. Zahzah, Foreground
    Detection via Robust Low Rank Matrix
    Factorization including Spatial Constraint with
    Iterative Reweighted Regression, International
    Conference on Pattern Recognition, ICPR 2012,
    Tsukuba, Japan, November 2012.
  • C. Guyon, T. Bouwmans. E. Zahzah, Moving Object
    Detection via Robust Low Rank Matrix
    Decomposition with IRLS scheme, International
    Symposium on Visual Computing, ISVC 2012,pages
    665674, Rethymnon, Crete, Greece, July 2012.
  • C. Guyon, T. Bouwmans, E. Zahzah, Moving Object
    Detection by Robust PCA solved via a Linearized
    Symmetric Alternating Direction Method,
    International Symposium on Visual Computing, ISVC
    2012, pages 427-436, Rethymnon, Crete, Greece,
    July 2012.
  • C. Guyon, T. Bouwmans, E. Zahzah, "Foreground
    Detection by Robust PCA solved via a Linearized
    Alternating Direction Method", International
    Conference on Image Analysis and Recognition,
    ICIAR 2012, pages 115-122, Aveiro, Portugal, June
    2012.
  • C. Guyon, T. Bouwmans, E. Zahzah, "Foreground
    detection based on low-rank and block-sparse
    matrix decomposition", IEEE International
    Conference on Image Processing, ICIP 2012,
    Orlando, Florida, September 2012.
About PowerShow.com