Image Sequence Based Particle Filter for Point Tracking

1
Image Sequence Based Particle Filter for Point
Tracking
Journées Thématiques Filtrage Particulaire

• Elise Arnaud - IRISA
• Etienne Mémin - IRISA
• Bruno Cernushi-Frias - Universitad de Buenos Aires

2
Introduction

• Objective

Point tracking in computer vision
Reconstruction of a point trajectory along a
given image sequence
Particular framework with no a priori knowledge

• Problem on which many high level tasks depend
  motion estimation, 3D reconstruction, dynamic
  vision, etc.

• Applications robotics, medical imaging,
  meteorological imaging, surveillance, etc.

3
Presentation
1. Introduction 2. Related works on point
tracking 3. Why an image sequence based filter
? 4. Image sequence based particle filter 5.
Application to point tracking 6. Results and
comparison 7. Conclusion 8. Perspectives
4
Related works on point tracking
2.1 Assumptions 2.2 Matching approaches 2.3
Differential approaches 2.4 Use of filtering
methods
5
Assumptions
Related works on point tracking
(1) Motion hypotheses drifting point, constant
velocity, constant acceleration, periodic
motion

• too difficult to model without any a priori
  knowledge
• too restrictive in the case of abrupt changes of
  the trajectory

(2) Relative position conservation within a rigid
geometric structure of the scene

• Problem in the case of points moving
  independently of the scene

(3) Luminance pattern conservation along the
trajectory
6
Matching Approaches
Related works on point tracking
Luminance pattern conservation

• Maximization of a similarity criterion between
  the target point and the candidate point

• Similarity criteria based on a description of
  the luminance pattern

• Necessity of exhaustive research ? time
  consuming

• Most similarity criteria are not invariant to
  affine changes

• Comparative study of the most used criteria
  Aschwanden92

7
Differential approaches
Related works on point tracking
Luminance pattern conservation

• Differential formulation of a similarity
  criterion

• Point intensity conservation ? optical flow
  constraint

• Sum of square difference ? Shi-Tomasi-Kanade
  tracker Shi94

8
Use of filtering methods
Related work on point tracking

• To design a tracker more robust to outliers and
  occlusions

• State of the filter feature position (
  intensity velocity )

• Kalman filter for tracking in an image sequence
  Nguyen01 Meyer94 Ricquebourg00

9
Why an image sequence based filter ?
3.1 Notations 3.2 Which type of available
measures ? 3.3 Which type of dynamic ? 3.4
Objectives and assumptions
10
Notations
Why an image sequence based filter ?
state of the system at time k
trajectory from time 0 to time k
measure at time k
measures from time 1 to time k
random vector corresponding to an image at time k
image sequence from time 0 to time k
11
Which type of available measures ?
Why an image sequence based filter ?

• All available information contained in the image
  sequence

Ideal case measure image

• Often impossible to specify the relationship
  between state and image ( too complex
  structure and large size )

Use of a condensed information obtained from the
sequence

• Highly nonlinear form with respect to the images

• Simple form with respect to the state

Measure equation
12
Which type of dynamic ?
Why an image sequence based filter ?

• Dynamic a priori (constant velocity, periodic
  motion etc. )

• Dynamic captured from learning Blake98

• If no a priori knowledge

Use of a dynamic obtained from the image sequence
Dynamic equation
13
Objectives and assumptions
Why an image sequence based filter ?

• Objective estimation of the trajectory given
  all available data, i.e. measures and image
  sequence

• Assumptions

14
Objectives and assumptions
Why an image sequence based filter ?

• Dynamic equation and measures depend on the
  sequence

• Such a dependency has to be taken into account

• Conditioning with respect to the image sequence
  in the equations of the filter

Definition of an image sequence based filter

• Case of nonlinear dynamic or nonlinear measure
  equations

Definition of an image sequence based particle
filter
15
Image sequence based particle filter

• Objective approximation of

• Knowledge of - N samples according to the
  importance function - N associated normalized
  weights

• Non-normalized weights given by

16
Image sequence based particle filter

• Recursive equation of the importance function
  assumed

• Recursive formulation of the weights

• Increase over time of the weights variance

• Optimal importance function minimizes the
  weights variance conditioned upon

• New recursive formulation of the weights

17
Image sequence based particle filter

• Initialisation

for k 1 n

• Measures Evaluate the measure and the dynamic
  from the image sequence

• Prediction

• Weights update

• Resampling

• Trajectory estimation

18
Application to point tracking
Proposed tracker combines a dynamic and some
measures all depending on the image data
State of the filter point position
4.1 Conditional observation equation 4.2
Conditional dynamic equation 4.3 Image sequence
based particle filter for point tracking
19
Conditional observation equation
Application to point tracking

• Restriction linear observation equation

• result of an estimation process on the
  image sequence

• Gaussian white noise conditionally to

is a Gaussian function
20
Conditional observation equation
Application to point tracking

• most similar point to in image

• Several matching criteria can be used to
  quantify the similarity

• Criterion used invariant to affine
  transformations, originally defined for image
  matching applications Schmid97

• Measure carries enough information to write

21
Conditional observation equation
Application to point tracking
initial point? initial vector of
characteristics - characterization of the
luminance pattern in the neighborhood -
invariant to affine transformations
selected measure ? associated to the vector the
most similar to the initial vector
22
Conditional dynamic equation
Application to point tracking

• Use of an instantaneous motion measure from
  image data

• Gaussian white noise conditionally to

• motion vector of

• motion parameters vector, result of an
  estimation process between images and

23
Conditional dynamic equation
Application to point tracking

• Point belonging to the background

- motion parameters vector corresponds to a
unique global linear motion
- linear dynamic equation
Image sequence based Kalman filter

• Point with a motion different from the global
  motion

- motion parameters vector corresponds to a local
linear motion
- nonlinear dynamic equation
Image sequence based particle filter
24
Image sequence based particle filter for point
tracking
Application to point tracking
and
Gaussians

• Knowledge of the optimal importance function,
  which is Gaussian.

• Knowledge of the distribution involved in the
  weights recursion, which is also Gaussian

25
Image sequence based particle filter for point
tracking
Application to point tracking
26
Results and comparisons
Sequence Caltra - 40 frames (190 ? 180 pixels)
Shi-Tomasi-Kanade tracker
Image sequence based particle filter for tracking
27
Results and comparisons
Sequence Meteo - 14 frames (256 ? 512 pixels)
Image sequence based particle filter for tracking
28
Conclusion

• Definition of an image sequence based particle
  filter

• Application to point tracking

• Proposed tracker combines a dynamic and some
  measures all depending on the image data

• No a priori knowledge

• Trajectories undergoing abrupt changes

• Sequence with a cluttered background

29
Perspectives

• Consider a confidence measure of the observation

• Include occlusion rules in the tracker

• Test other dynamic calculated from image
  sequence use of dense motion field

• Application to fluid imagery ( meteorological
  sequence )