Features

1
Features?
Local versus global
2
Vanishing Points
3
Vanishing Points
A. Canaletto 1740, Arrival of the French
Ambassador in Venice
4
From Edges to Lines
5
Hough Transform
6
Hough Transform Quantization
m
Detecting Lines by finding maxima / clustering in
parameter space
7
Hough Transform Results
Hough Transform
Image
Edge detection
8
Houghing for other things

• Circles?
• Complexity of measurement vs.
• Number of votes.

9
The Fourier Transform

• Represent function on a new basis
• Like PCA, but someone else gives you the
  principle components.
• Instead of the 1st, 2nd, 3rd, component, and so
  on, the fourier components have 2 coordinates,
  u,v.
• We now apply a linear transformation to transform
  the basis
• dot product with each basis element
• In the expression, u and v select the basis
  element, so a function of x and y becomes a
  function of u and v
• basis elements have the form

10

• Fourier basis element
• example, real part
• Fu,v(x,y)
• Fu,v(x,y)const. for (uxvy)const.
• Vector (u,v)
• Magnitude gives frequency
• Direction gives orientation.

11
Here u and v are larger than in the previous
slide.
12
And larger still...
13
Phase and Magnitude

• Fourier transform of a real function is complex
• difficult to plot, visualize
• instead, we can think of the phase and magnitude
  of the transform
• Phase is the phase of the complex transform
• Magnitude is the magnitude of the complex
  transform
• Curious fact
• all natural images have about the same magnitude
  transform
• hence, phase seems to matter, but magnitude
  largely doesnt
• Demonstration
• Take two pictures, swap the phase transforms,
  compute the inverse - what does the result look
  like?

14
(No Transcript)
15
This is the magnitude transform of the cheetah
picture
16
This is the phase transform of the cheetah picture
17
(No Transcript)
18
This is the magnitude transform of the zebra
picture
19
This is the phase transform of the zebra picture
20
Reconstruction with zebra phase, cheetah magnitude
21
Reconstruction with cheetah phase, zebra magnitude
22
SIFT Reference

• Distinctive image features from scale-invariant
  keypoints. David G. Lowe, International Journal
  of Computer Vision, 60, 2 (2004), pp. 91-110.
• SIFT Scale Invariant Feature Transform

23
Invariant Local Features

• Image content is transformed into local feature
  coordinates that are invariant to translation,
  rotation, scale, and other imaging parameters

SIFT Features
24
Advantages of invariant local features

• Locality features are local, so robust to
  occlusion and clutter (no prior segmentation)
• Distinctiveness individual features can be
  matched to a large database of objects
• Quantity many features can be generated for even
  small objects
• Efficiency close to real-time performance
• Extensibility can easily be extended to wide
  range of differing feature types, with each
  adding robustness

25
SIFT On-A-Slide

• Enforce invariance to scale Compute Gaussian
  difference max, for may different scales
  non-maximum suppression, find local maxima
  keypoint candidates
• Localizable corner For each maximum fit
  quadratic function. Compute center with sub-pixel
  accuracy by setting first derivative to zero.
• Eliminate edges Compute ratio of eigenvalues,
  drop keypoints for which this ratio is larger
  than a threshold.
• Enforce invariance to orientation Compute
  orientation, to achieve scale invariance, by
  finding the strongest second derivative direction
  in the smoothed image (possibly multiple
  orientations). Rotate patch so that orientation
  points up.
• Compute feature signature Compute a "gradient
  histogram" of the local image region in a 4x4
  pixel region. Do this for 4x4 regions of that
  size. Orient so that largest gradient points up
  (possibly multiple solutions). Result feature
  vector with 128 values (15 fields, 8 gradients).
• Enforce invariance to illumination change and
  camera saturation Normalize to unit length to
  increase invariance to illumination. Then
  threshold all gradients, to become invariant to
  camera saturation.

26
SIFT On-A-Slide

• Enforce invariance to scale Compute Gaussian
  difference max, for may different scales
  non-maximum suppression, find local maxima
  keypoint candidates
• Localizable corner For each maximum fit
  quadratic function. Compute center with sub-pixel
  accuracy by setting first derivative to zero.
• Eliminate edges Compute ratio of eigenvalues,
  drop keypoints for which this ratio is larger
  than a threshold.
• Enforce invariance to orientation Compute
  orientation, to achieve scale invariance, by
  finding the strongest second derivative direction
  in the smoothed image (possibly multiple
  orientations). Rotate patch so that orientation
  points up.
• Compute feature signature Compute a "gradient
  histogram" of the local image region in a 4x4
  pixel region. Do this for 4x4 regions of that
  size. Orient so that largest gradient points up
  (possibly multiple solutions). Result feature
  vector with 128 values (15 fields, 8 gradients).
• Enforce invariance to illumination change and
  camera saturation Normalize to unit length to
  increase invariance to illumination. Then
  threshold all gradients, to become invariant to
  camera saturation.

27
Finding Keypoints (Corners)

• Idea Find Corners, but scale invariance
• Approach
• Run linear filter (diff of Gaussians)
• At different resolutions of image pyramid

28
Difference of Gaussians
Minus
Equals
29
Difference of Gaussians

• surf(fspecial('gaussian',40,4))
• surf(fspecial('gaussian',40,8))
• surf(fspecial('gaussian',40,8) -
  fspecial('gaussian',40,4))

30
Find Corners with DiffOfGauss

• im imread('bridge.jpg')
• bw double(im(,,1)) / 256
• for i 1 10
• gaussD fspecial('gaussian',40,2i) -
  fspecial('gaussian',40,i)
• mesh(gaussD) drawnow
• res abs(conv2(bw, gaussD, 'same'))
• res res / max(max(res))
• imshow(res) title('\bf i ' num2str(i))
  drawnow
• end

31
Build Scale-Space Pyramid

• All scales must be examined to identify
  scale-invariant features
• An efficient function is to compute the
  Difference of Gaussian (DOG) pyramid (Burt
  Adelson, 1983)

32
Key point localization

• Detect maxima and minima of difference-of-Gaussian
  in scale space

33
Example of keypoint detection
(a) 233x189 image (b) 832 DOG extrema
34
SIFT On-A-Slide

• Enforce invariance to scale Compute Gaussian
  difference max, for may different scales
  non-maximum suppression, find local maxima
  keypoint candidates
• Localizable corner For each maximum fit
  quadratic function. Compute center with sub-pixel
  accuracy by setting first derivative to zero.
• Eliminate edges Compute ratio of eigenvalues,
  drop keypoints for which this ratio is larger
  than a threshold.
• Enforce invariance to orientation Compute
  orientation, to achieve scale invariance, by
  finding the strongest second derivative direction
  in the smoothed image (possibly multiple
  orientations). Rotate patch so that orientation
  points up.
• Compute feature signature Compute a "gradient
  histogram" of the local image region in a 4x4
  pixel region. Do this for 4x4 regions of that
  size. Orient so that largest gradient points up
  (possibly multiple solutions). Result feature
  vector with 128 values (15 fields, 8 gradients).
• Enforce invariance to illumination change and
  camera saturation Normalize to unit length to
  increase invariance to illumination. Then
  threshold all gradients, to become invariant to
  camera saturation.

35
Example of keypoint detection
Threshold on value at DOG peak and on ratio of
principle curvatures (Harris approach)

• (c) 729 left after peak value threshold
• (d) 536 left after testing ratio of principle
  curvatures

36
SIFT On-A-Slide

• Enforce invariance to scale Compute Gaussian
  difference max, for may different scales
  non-maximum suppression, find local maxima
  keypoint candidates
• Localizable corner For each maximum fit
  quadratic function. Compute center with sub-pixel
  accuracy by setting first derivative to zero.
• Eliminate edges Compute ratio of eigenvalues,
  drop keypoints for which this ratio is larger
  than a threshold.
• Enforce invariance to orientation Compute
  orientation, to achieve scale invariance, by
  finding the strongest second derivative direction
  in the smoothed image (possibly multiple
  orientations). Rotate patch so that orientation
  points up.
• Compute feature signature Compute a "gradient
  histogram" of the local image region in a 4x4
  pixel region. Do this for 4x4 regions of that
  size. Orient so that largest gradient points up
  (possibly multiple solutions). Result feature
  vector with 128 values (15 fields, 8 gradients).
• Enforce invariance to illumination change and
  camera saturation Normalize to unit length to
  increase invariance to illumination. Then
  threshold all gradients, to become invariant to
  camera saturation.

37
Select canonical orientation

• Create histogram of local gradient directions
  computed at selected scale
• Assign canonical orientation at peak of smoothed
  histogram
• Each key specifies stable 2D coordinates (x, y,
  scale, orientation)

38
SIFT On-A-Slide

• Enforce invariance to scale Compute Gaussian
  difference max, for may different scales
  non-maximum suppression, find local maxima
  keypoint candidates
• Localizable corner For each maximum fit
  quadratic function. Compute center with sub-pixel
  accuracy by setting first derivative to zero.
• Eliminate edges Compute ratio of eigenvalues,
  drop keypoints for which this ratio is larger
  than a threshold.
• Enforce invariance to orientation Compute
  orientation, to achieve scale invariance, by
  finding the strongest second derivative direction
  in the smoothed image (possibly multiple
  orientations). Rotate patch so that orientation
  points up.
• Compute feature signature Compute a "gradient
  histogram" of the local image region in a 4x4
  pixel region. Do this for 4x4 regions of that
  size. Orient so that largest gradient points up
  (possibly multiple solutions). Result feature
  vector with 128 values (15 fields, 8 gradients).
• Enforce invariance to illumination change and
  camera saturation Normalize to unit length to
  increase invariance to illumination. Then
  threshold all gradients, to become invariant to
  camera saturation.

39
SIFT vector formation

• Thresholded image gradients are sampled over
  16x16 array of locations in scale space
• Create array of orientation histograms
• 8 orientations x 4x4 histogram array 128
  dimensions

40
Nearest-neighbor matching to feature database

• Hypotheses are generated by approximate nearest
  neighbor matching of each feature to vectors in
  the database
• SIFT use best-bin-first (Beis Lowe, 97)
  modification to k-d tree algorithm
• Use heap data structure to identify bins in order
  by their distance from query point
• Result Can give speedup by factor of 1000 while
  finding nearest neighbor (of interest) 95 of the
  time

41
Test of illumination invariance

• Same image under differing illumination

273 keys verified in final match
42
Examples of view interpolation
43
Location recognition
44

• Sony Aibo
• (Evolution Robotics)
• SIFT usage
• Recognize
• charging
• station
• Communicate
• with visual
• cards