Computer Vision

1
Computer Vision

• Optical Flow

Some slides from K. H. Shafique
http//www.cs.ucf.edu/courses/cap6411/cap5415/
and T. Darrell
2
Correspondence

• Which pixel went where?

Time t
Time t dt
3
Motion Field vs. Optical Flow

• Scene flow 3D velocities of scene points.
• Motion field 2D projection of scene flow.
• Optical flow Approximation of motion field
  derived from apparent motion of brightness
  patterns in image.

4
Motion Field vs. Optical Flow

• Consider perfectly uniform sphere rotating in
  front of camera.
• Motion field follows surface points.
• Optical flow is zero since motion is not visible.
• Now consider stationary sphere with moving light
  source.
• Motion field is zero.
• But optical flow results from changing shading.

But, in general, optical flow is a reliable
indicator of motion field.
5
Applications

• Object tracking
• Video compression
• Structure from motion
• Segmentation
• Correct for camera jitter (stabilization)
• Combining overlapping images (panoramic image
  construction)

6
Optical Flow Problem

• How to estimate pixel motion from one image to
  another?

7
Computing Optical Flow

• Assumption 1 Brightness is constant.
• Assumption 2 Motion is small.

(from Taylor series expansion)
8
Computing Optical Flow

• Combine

In the limit as u and v goes to zero, the
equation becomes exact
(optical flow equation)
9
Computing Optical Flow

• At each pixel, we have one equation, two
  unknowns.
• This means that only the flow component in the
  gradient direction can be determined.

(optical flow equation)
The motion is parallel to the edge, and it cannot
be determined.
This is called the aperture problem.
10
Computing Optical Flow

• We need more constraints.
• The most commonly used assumption is that optical
  flow changes smoothly locally.

11
Computing Optical Flow

• One method The (u,v) is constant within a small
  neighborhood of a pixel.

Optical flow equation
Use a 5x5 window
Two unknowns, 25 equation !
12
Computing Optical Flow

• Find minimum least squares solution

Lucas Kanade method
13
Computing Optical Flow

• Lucas Kanade

• When is This Solvable?
• ATA should be invertible
• ATA should not be too small. Other wise noise
  will be amplified when inverted.)

14
Computing Optical Flow

• What are the potential causes of errors in this
  procedure?
• Brightness constancy is not satisfied
• The motion is not small
• A point does not move like its neighbors
• window size is too large

15
Improving accuracy

• Recall our small motion assumption

• This is not exact
• To do better, we need to add higher order terms
  back in

• This is a polynomial root finding problem
• Can solve using Newtons method
• Also known as Newton-Raphson method
• Lucas-Kanade method does one iteration of
  Newtons method
• Better results are obtained via more iterations

16
Iterative Refinement

• Iterative Lucas-Kanade Algorithm
• Estimate velocity at each pixel by solving
  Lucas-Kanade equations
• Warp H towards I using the estimated flow field
• - use image warping techniques
• Repeat until convergence

17
Revisiting the small motion assumption

• What can we do when the motion is not small?

18
Reduce the resolution!
19
Coarse-to-fine optical flow estimation
20
Coarse-to-fine optical flow estimation
run iterative L-K
21
Multi-resolution Lucas-Kanade Algorithm
22
Block-Based Motion Estimation
23
Block-Based Motion Estimation
24
Block-Based Motion Estimation
Hierarchical search
25
Parametric (Global) Motion

• Sometimes few parameters are enough to represent
  the motion of whole image

translation
rotation
scale
26
Parametric (Global) Motion

• Affine Flow

27
Parametric (Global) Motion

• Affine Flow

28
Parametric (Global) Motion

• Affine Flow Approach

29
Iterative Refinement

• Iterative Estimation
• Estimate parameters by solving the linear system.
  
• Warp H towards I using the estimated flow field
• - use image warping techniques
• Repeat until convergence or a fixed number of
  iterations

30
Coarse-to-fine global flow estimation
Compute Flow Iteratively
31
Global Flow
Find features Match features Fit parametric model
Application Mosaic construction
32
Image Warping
33
Image Warping
Interpolate to find the intensity at (x,y)
Pixel values are known in this image
Pixel values are to found in this image
34
Other Parametric Motion Models
Perspective
and
Pseudo-Perspective Approximation to perspective.
(Planar scene Perspective projection)