Illumination Normalization with Time-Dependent Intrinsic Images for Video Surveillance

1
Illumination Normalization with Time-Dependent
Intrinsic Images for Video Surveillance

• Yasuyuki Matsushita, Member, IEEE, Ko Nishino,
  Member, IEEE,
• Katsushi Ikeuchi, Fellow, IEEE, and Masao
  Sakauchi, Member, IEEE

2
Outline

• Introduction
• Proposed method overview
• Intrinsic image estimation
• Shadow removal
• Illumination eigenspace for direct estimation of
  illumination
• Experimental results
• Conclusions

3
Introduction(1/2)

• VIDEO surveillance systems involving object
  detection and tracking require robustness against
  illumination changes
• Illumination change caused by
• Weather conditions
• Large cast shadows of surrounding structures
  (large buildings and trees)

4
Introduction(2/2)

• Goal
• Normalize the input image sequence in terms of
  the distribution of incident lighting to remove
  illumination effects including shadow effects.
• Proposed approach based on intrinsic images

5
Proposed method overview

• Our method is composed of two parts
• Estimation of intrinsic images
• Using this background image sequence, we then
  derive intrinsic images using our estimation
  method (extended from Weisss ML estimation
  method)
• Direct estimation of illumination images
• ? Using the preconstructed illumination
  eigenspace, we estimate an illumination image
  directly from an input image.

6
Proposed method overview
7
Intrinsic image estimation

• Goal
• Estimate intrinsic images under varying
  illumination (inspired by ML estimation
  method21)
• ML is effective to extract the scene texture
  under Lambertian assumption
• ? In real scene ,its often difficult to
  expect the Lambertian assumption to be hold

Lambertian model????? ???????? ? ??????
8
Intrinsic image estimation

• This paper propose a set of time-varying
  reflectance images R(x y t) instead of a time
  invariant reflectance image R(x y)
• Start with ML estimation method21
• Applying ML method, a single reflectance image
  Rw(x,y) and a set of illumination images Lw(x,y)
  are estimated
• Scene texture image reflectance image
• Our Goal I(x y)R(x y t)L(x y t)
• ? derive R(x y t) and L(x y t)

9
Intrinsic image estimation

• With nth derivative filters fn, a filtered
  reflectance image is computed by taking median
  along the time axis
• With those filters , input images are decomposed
  into intrinsic images

The filtered illumination images are then
computed by using estimated filtered reflectance
image
We take a straightforward approach to remove
texture edges from lw and derive illumination
images l(x y t)
10
Intrinsic image estimation
11
Shadow removal

• Using the obtained scene illumination images by
  our method, the input image sequence can be
  normalized in terms of illumination.
• Create background images in each short time range
  (?T)
• Illumination doesnt vary in ?T
• moving objects in the scene are not observed at
  the same point longer than the background in ?T.
• Using the estimation method to decompose each
  image in the background image sequence into
  corresponding reflectance images R(x y t) and
  illumination images L(x y t).

12
Shadow removal
Resulting illuminance-invariant image N(x y
t) can be derived by the following equation
13
Illumination eigenspace for direct estimation of
illumination

• We propose illumination eigenspace to model
  variations of illumination images of the scene.
• Use principal component analysis (PCA) to
  construct the illumination eigenspace of a target
  scene
• The basic idea in PCA is to find the basic
  components s1 s2 . . . sn that explain the
  maximum amount of variance possible by n linearly
  transformed components.

14
Illumination eigenspace for direct estimation of
illumination

• we mapped L(x y t) into the illumination
  eigenspace
• an illumination space matrix is constructed by
• subtracting Lw, which is the average of all Lw,

P is an NM matrix, where N is the number of
pixels in the illumination image and M is the
number of illumination images Lw.
We made the covariance matrix Q of P as follows
Finally, the eigenvectors ei and the
corresponding eigenvalues i of Q are determined
by solving
15
Illumination eigenspace for direct estimation of
illumination

• Using the illumination eigenspace, direct
  estimation of an illumination image can be done
  given an input image which contains moving
  objects.
• We first divide the input image by a reflectance
  image to get a pseudoillumination image L
• Using this pseudoillumination image as a query,
  best approximation of the corresponding
  illumination image is estimated

16
Illumination eigenspace for direct estimation of
illumination

• The number of stored images for this experiment
  was 2,048 and the contribution ratio was 84.5
  percent at 13 dimensions, 90.0 percent at 23
  dimensions, and 99.0 percent at 120 dimensions.
• The disk space needed to store the
• subspace was about 32 MBytes when the image size
  is 320243

17
Illumination eigenspace for direct estimation of
illumination
18
Illumination eigenspace for direct estimation of
illumination
The average time of the NN search is shown in
Table 1 with MIPS R12000 300MHz, when the number
of stored illumination images is 2,048, the image
size is 360 243 ?The estimation time is fast
enough for real time processing.
19
Experimental results

• Evaluated our shadow elimination method by object
  tracking based on block matching
• sum of squared differences(SSD)
• normalized correlation function (NCF)

20
Experimental results
21
Experimental results
22
Conclusions

• We have described a framework for normalizing
  illumination effects of real world scenes
• Extend current method to properly handle surfaces
  with nonrigid reflectance properties
• Utilize illumination eigenspace, a preconstructed
  database which captures the illumination
  variation of the target scene
• Effectively handle the appearance variation
  caused by illumination
• Disadvantage
• Current implementation in research code is not
  fast enough for realtime processing