Image Alignment and Mosaicing

1
Image Alignment and Mosaicing

2
Image Alignment Applications

• Local alignment
• Tracking
• Stereo
• Global alignment
• Camera jitter elimination
• Image enhancement
• Panoramic mosaicing

3
Image Stitching Problems

• Motion model
• Parameterization
• Correspondence
• Warping
• Blending

4
Motion Model

• How do images relate to each other?
• Possibilities
• Pure translation
• Rigid body (translation rotation)
• Affine
• Parallax
• General motion

5
Motion Model

• Common case images taken with perspective camera
• Motion around center of projection

6
Parameterization

• Single parameterization for all images
• Cylindrical parameterization often used for
  panoramic mosaics

7
Cylindrical Parameterization

• After image warping, motion model ispure
  translation

8
Correspondence Approaches

• Optical flow
• Correlation
• Correlation optical flow
• Any of the above, iterated (e.g. Lucas-Kanade)
• Any of the above, coarse-to-fine

9
Correspondence Approaches

• Optical flow
• Correlation
• Correlation optical flow
• Any of the above, iterated (e.g. Lucas-Kanade)
• Any of the above, coarse-to-fine

10
Optical Flow for Image Registration

• Compute local matches
• Least-squares fit to motion model
• Problem outliers

11
Outlier Rejection

• Least squares very sensitive to outliers

12
Outlier Rejection

• Robust estimation tolerant of outliers
• In general, methods based on absolute value
  rather than square minimize Sxi-f, not
  S(xi-f )2

13
Robust Fitting

• In general, hard to compute
• Analytic solutions in a few cases
• Mean ? Median
• Solution for fitting lines

14
Approximation to Robust Fitting

• Compute least squares
• Assign weights based on average deviation
• Simplest weight 1 if deviation lt 2.5s,
  0 otherwise
• Smoother version weight Gs (deviation)
• Weighted least squares
• Iterate

15
Alternative Approximation toRobust Fitting

• Least median of squares
• Select n random subsets of data points
• Perform least-squares fit on each
• Compute average deviation for each
• Select fit with median deviation

16
Correspondence Approaches

• Optical flow
• Correlation
• Correlation optical flow
• Any of the above, iterated (e.g. Lucas-Kanade)
• Any of the above, coarse-to-fine

17
Correlation / Search Methods

• Assume translation only
• Given images I1, I2
• For each translation (tx, ty) compute
• Select translation that maximizes c
• Depending on window size, local or global

18
Cross-Correlation

• Statistical definition of correlation
• Disadvantage sensitive to local variations in
  image brightness

19
Normalized Cross-Correlation

• Normalize to eliminate brightness
  sensitivitywhere

20
Sum of Squared Differences

• More intuitive measure
• Negative sign so that higher values mean greater
  similarity
• Expand

21
Correlation vs. Optical Flow

• Intuitively similar to functional minimization
  search vs. Newtons method

22
Local vs. Global

• Correlation with local windows not too expensive
• High cost if window size whole image
• But computation looks like convolution
• FFT to the rescue!

23
Correlation
24
Fourier Transform with Translation
25
Fourier Transform with Translation

• Therefore, if I1 and I2 differ by translation,
• So, F-1(F1/F2) will have a peak at (Dx,Dy)

26
Phase Correlation

• In practice, use cross power spectrum
• Compute inverse FFT, look for peaks
• Kuglin Hines 1975

27
Phase Correlation

• Advantages
• Fast computation
• Low sensitivity to global brightness
  changes(since equally sensitive to all
  frequencies)

28
Phase Correlation

• Disadvantages
• Sensitive to white noise
• No robust version
• Translation only
• Extensions to rotation, scale
• But not local motion
• Not too bad in practice with small local motions

29
Correspondence Approaches

• Optical flow
• Correlation
• Correlation optical flow
• Any of the above, iterated (e.g. Lucas-Kanade)
• Any of the above, coarse-to-fine

30
Correlation plus Optical Flow

• Use e.g. phase correlation to find average
  translation (may be large)
• Use optical flow to find local motions

31
Correspondence Approaches

• Optical flow
• Correlation
• Correlation optical flow
• Any of the above, iterated (e.g. Lucas-Kanade)
• Any of the above, coarse-to-fine

32
Iteration

• Local refinement of optical flow estimate
• Sort of equivalent to multiple iterations of
  Newtons method

33
Correspondence Approaches

• Optical flow
• Correlation
• Correlation optical flow
• Any of the above, iterated (e.g. Lucas-Kanade)
• Any of the above, coarse-to-fine

34
Image Pyramids

• Pre-filter images to collect information at
  different scales
• More efficient computation, allowslarger motions

35
Image Pyramids
Szeliski
36
Pyramid Creation

• Gaussian Pyramid
• Laplacian Pyramid
• Created from Gaussianpyramid by subtractionLi
  Gi expand(Gi1)

Szeliski
37
Octaves in the Spatial Domain
Lowpass Images

• Bandpass Images

Szeliski
38
Image Warping

• Given a coordinate transform x h(x) and a
  source image f(x), how do we compute a
  transformed image g(x) f(h(x))?

h(x)
x
x
f(x)
g(x)
Szeliski
39
Forward Warping

• Send each pixel f(x) to its corresponding
  location x h(x) in g(x)
• What if pixel lands between two pixels?

h(x)
x
x
f(x)
g(x)
Szeliski
40
Forward Warping

• Send each pixel f(x) to its corresponding
  location x h(x) in g(x)
• What if pixel lands between two pixels?
• Answer add contribution to several pixels,
  normalize later (splatting)

h(x)
x
x
f(x)
g(x)
Szeliski
41
Inverse Warping

• Get each pixel g(x) from its corresponding
  location x h-1(x) in f(x)
• What if pixel comes from between two pixels?

h-1(x)
x
x
f(x)
g(x)
Szeliski
42
Inverse Warping

• Get each pixel g(x) from its corresponding
  location x h-1(x) in f(x)
• What if pixel comes from between two pixels?
• Answer resample color value from interpolated
  (prefiltered) source image

Szeliski
43
Blending

• Blend over too small a region seams
• Blend over too large a region ghosting

44
Multiresolution Blending

• Different blending regions for different levels
  in a pyramid Burt Adelson
• Blend low frequencies over large regions
  (minimize seams due to brightness variations)
• Blend high frequencies over small regions
  (minimize ghosting)

45
Pyramid Blending
Szeliski
46
Minimum-Cost Cuts

• Instead of blending high frequencies along a
  straight line, blend along line of minimum
  differences in image intensities

47
Minimum-Cost Cuts
Moving object, simple blending ? blur
Davis 98
48
Minimum-Cost Cuts
Minimum-cost cut ? no blur
Davis 98