KLT tracker

1
KLT tracker triangulationClass 6

• Read Shi and Tomasis paper on good features to
  track
• http//www.unc.edu/courses/2004fall/comp/290/089/p
  apers/shi-tomasi-good-features-cvpr1994.pdf
• Optional Lucas-Kanade 20 Years On
  http//www.ri.cmu.edu/projects/project_515.html

2
Feature matching vs. tracking
Image-to-image correspondences are key to passive
triangulation-based 3D reconstruction
Extract features independently and then match by
comparing descriptors
Extract features in first images and then try to
find same feature back in next view
What is a good feature?
3
Feature point extraction

• Approximate SSD for small displacement ?

• Find points for which the following is maximum
• maximize smallest eigenvalue of M

4
SIFT features

• Scale-space DoG maxima
• Verify minimum contrast and cornerness
• Orientation from dominant gradient
• Descriptor based on gradient distributions

5
Feature tracking

• Identify features and track them over video
• Small difference between frames
• potential large difference overall
• Standard approach
• KLT (Kanade-Lukas-Tomasi)

6
Intermezzo optical flow

• Brightness constancy assumption

(small motion)

• 1D example

possibility for iterative refinement
7
Intermezzo optical flow

• Brightness constancy assumption

(small motion)

• 2D example

the aperture problem
(1 constraint)
?
(2 unknowns)
isophote I(t1)I
isophote I(t)I
8
Intermezzo optical flow

• How to deal with aperture problem?

(3 constraints if color gradients are different)
Assume neighbors have same displacement
9
Lucas-Kanade
Assume neighbors have same displacement
least-squares
10
Alternative derivation

• Compute translation assuming it is small

differentiate
Affine is also possible, but a bit harder (6x6 in
stead of 2x2)
11
Revisiting the small motion assumption

• Is this motion small enough?
• Probably notits much larger than one pixel (2nd
  order terms dominate)
• How might we solve this problem?

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
12
Reduce the resolution!
From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
13
Coarse-to-fine optical flow estimation
slides from Bradsky and Thrun
14
Coarse-to-fine optical flow estimation
slides from Bradsky and Thrun
run iterative L-K
15
Good feature to track

• Tracking
• Use same window in feature selection as for
  tracking itself

• maximize minimal eigenvalue of M
• Strategy
• Look for strong well distributed features,
  typically few hundreds
• initialize and then track, renew feature when too
  many are lost

16
Example
Simple displacement is sufficient between
consecutive frames, but not to compare to
reference template
17
Example
18
Synthetic example
19
Good features to keep tracking

• Perform affine alignment between first and last
  frame
• Stop tracking features with too large errors

20
Live demo

• OpenCV (try it out!)

LKdemo
21
Triangulation
m1
C1
L1
Triangulation

• calibration
• correspondences

22
Triangulation

• Backprojection

• Triangulation

Iterative least-squares

• Maximum Likelihood Triangulation

23
Backprojection

• Represent point as intersection of row and column

• Condition for solution?

Useful presentation for deriving and
understanding multiple view geometry (notice 3D
planes are linear in 2D point coordinates)
24
Next class epipolar geometry