Shape Matching and Object Recognition using Low Distortion Correspondence

1
Shape Matching and Object Recognition using Low
Distortion Correspondence

• Alexander C. Berg, Tamara L. Berg, Jitendra
  Malik
• U.C. Berkeley

2
Object Category Recognition
3
Deformable Template Matching with Exemplars for
Recognition

• Use exemplars as deformable templates
• Find a correspondence between the query image and
  each template

Database of Templates
Query Image
4
Deformable Template Matching with Exemplars for
Recognition

• Use exemplars as deformable templates
• Find a correspondence between the query image and
  each template

Database of Templates
Query Image
Best matching template is a helicopter
5
Correspondence for Deformable Template Matching
Query
Template

• Evaluate correspondence based on
• Similarity of appearance near feature points
• Similarity in configuration of the feature points

6
Correspondence for Deformable Template Matching
Query
Template

• Evaluate correspondence based on
• Similarity of appearance near feature points
• Similarity in configuration of the feature points

7
Correspondence for Deformable Template Matching
Query
Template

• Evaluate correspondence based on
• Similarity of appearance near feature points
• Similarity in configuration of the feature points

8
Correspondence for Deformable Template Matching
Query
Template

• Evaluate correspondence based on
• Similarity of appearance near feature points
• Similarity in configuration of the feature points

9
Correspondence Result
Query
Template
10
Interpolated CorrespondenceUsing Thin Plate
Splines
Query
Template
11
Correspondence for Deformable Template Matching
Query
Template
ri'j'
rij
12
Geometric Blur(Local Appearance Descriptor)
Berg Malik '01
Compute sparse channels from image
Extract a patch in each channel
Apply spatially varying blur and sub-sample

Descriptor is robust to small affine distortions
Geometric Blur Descriptor
(Idealized signal)
13
Geometric Blur(Local Appearance Similarity)

Geometric Blur Descriptor
Geometric Blur Descriptor
14
Are Features Enough?
Color indicates similarity using Geometric Blur
Descriptor
Not Quite...
15
Measuring Distortion(Similarity in Configuration)
Query
Template
Rij
Si'j'
Measure distortion in vectors between pairs of
feature points - R and S same length for
rotations - R and S same direction for scalings
16
Cost Function as IQP
cf. Maciel Costeira '03
Appearance cost if i -gt j
Distortion cost if i -gt j and k -gt l
Integer Quadratic Programming Problem...
17
Optimization

• Integer Quadratic Programming is NP hard
• The instances we generate seem easy
• Using a linear bound to initialize gradient
  descent provides good results
• (In fact better than the guarantee of Goemans
  Williamson's randomized algorithm)
• Varying the linear constraints on x allows
• one-one, one-many, or fixed number of outliers,
  etc.

18
Correspondence Result
19
Interpolated CorrespondenceUsing Thin Plate
Splines
20
Quadratic Assignment(Using IQP)
21
Linear Assignment(e.g. Hungarian)
22
Correspondence Examples (Shape Matching)
23
Correspondence Examples(Shape Matching)
24
Correspondence Examples(Shape Matching)
25
Correspondence Examples(Shape Matching)
26
Correspondence Examples(Shape Matching)
27
Correspondence Examples(Shape Matching)
28
Correspondence Examples(Shape Matching)
29
Correspondence Examples(Shape Matching)
30
Correspondence Examples(Shape Matching)
31
Correspondence Examples(Shape Matching)
32
Correspondence Examples(Shape Matching)
33
Application to RecognitionCaltech 101

• 101 classes of objects background
• Large Scale
• Roughly aligned
• Large intra-class variation
• Fei-Fei, Fergus, Perona '04

34
Caltech 101 Recognition Results
102 way Alternative Forced Choice test (15
training examples per class)
Chance 1 N.N. whole image 16 Discrimi
native version of Constellation Model 27 N.N.
Geometric Blur Descriptors 38 Low
Distortion Correspondence (GBIQP) 45
102 way confusion matrix
100
0
35
Model Building for Segmentation
Average quality of alignment
Rough correspondence to each example image
Threshhold
36
Automatic vs Hand Segmentation
37
Application to RecognitionFaces

• Face dataset from Berg et al '03
• Medium to large scale faces
• AP News photos
• 20 face exemplars
• Same methodology as Caltech 101, but multiple
  objects / image
• After one face is identified its features are
  removed and the search continues
• Compared to a detector from Mikolajczyk based on
  Schneiderman Kanade, that is quite successful
  on this dataset

38
Application to RecognitionFaces
39
Conclusion

• Use rich descriptors that are insensitive to
  typical transformations
• Geometric Blur
• Enforce relationship constraints among
  corresponding features
• Integer Quadratic Programming
• Estimate smooth transform
• Thin Plate Splines

40
Thank You
Acknowledgements Charless Fowlkes Xiaofeng
Ren David Forsyth