Segmentation of Salient Regions in Outdoor Scenes Using Imagery and 3-D Data

1
Unsupervised Modeling of Object Categories
Using Link Analysis Techniques
Gunhee KimChristos FaloutsosMartial
HebertComputer ScienceCarnegie Mellon
University
Presenter Ramin Mehran
2
Outline

• Problem Statement the Approach
• Network Construction
• Link Analysis Techniques
• Ranking of features wrt an image/object
• Structural Similarity
• Unsupervised Modeling
• Category Discovery
• Localization
• Complexity
• Conclusion

3
Unsupervised Modeling1-5

• Category discovery Localization

• Category discovery Localization

• Category discovery Localization

1 Sivic et al, ICCV 20052 FritzSchiele,
DAGM 20063 GraumanDarrell, CVPR 20064
TodorovicAhuja, CVPR 20065 CaoFei-Fei, ICCV
2007
4
Previous Work

• Topic models based on bag of words125

• Tree matching4

• Clustering with partial matching3

1 Sivic et al, ICCV 20052 FritzSchiele,
DAGM 20063 GraumanDarrell, CVPR 20064
TodorovicAhuja, CVPR 20065 CaoFei-Fei, ICCV
2007
5
Intuition
Visual Information
A large-scale Network
Link Analysis Techniques

Solve Visual tasks
6
Statistics of Link Structure
7
Large-Scale Networks
(1) http//www.opte.org, (2) The Academy of
Motion Picture Arts and Sciences, (4)
http//www.willamette.edu/gorr/classes/cs449/brai
n.html (5) MarkNewman
WWW1
Oscars social network2
Metabolic network3
Neural network4
A food web5
8
Outline

• Problem Statement Our Approach
• Network Construction
• Link Analysis Techniques
• Ranking of features wrt an image/object
• Structural Similarity
• Unsupervised Modeling
• Category Discovery
• Localization
• Complexity
• Conclusion

9
Visual Similarity Network Vertices

• Vertices Any Local Features
• Harris Affine SIFT

I1
Im
10
Visual Similarity Network Edges Weights

• Edges Correspondences by image matching
• Spectral Matching1-2 Appearance affinity
  Geometric Consistency
• Weights Stronger geometric consistency, higher
  values

Ib
n?n
Ia
Ia
Ib
M
12 Leordeanu Hebert, ICCV05, ICML06.
11
Spectral Matching1-2
Pairs of wrong correspondences are unlikely to
preserve geometry
OK
Pairs of correct correspondences are very likely
to preserve geometry
1 M. Leordeanu and M. Hebert. A spectral
technique for correspondence problems using
pairwise constraints, 2005. ICCV. 2 M.
Leordeanu and M. Hebert. Efficient map
approximation for dense energy functions, 2006.
ICML.
12
Geometric Consistency
1 M. Leordeanu and M. Hebert. A spectral
technique for correspondence problems using
pairwise constraints, 2005. ICCV. 2 M.
Leordeanu and M. Hebert. Efficient map
approximation for dense energy functions, 2006.
ICML.
13
Outline

• Problem Statement Our Approach
• Network Construction
• Link Analysis Techniques
• Ranking of features wrt an image/object
• Structural Similarity
• Unsupervised Modeling
• Category Discovery
• Localization
• Complexity
• Conclusion

14
1. Ranking of the Features
gtgt
Fergus et al1 models capture the essence of
categories
?models capture the hubs in the visual network
1 Fergus, Perona, Zisserman, IJCV 2007.
15
Ranking Removes Noisy Matching
16
How to Rank the Features

• PageRank1

1 Brin and Page. WWW 1998

• Recursive Definition

Vote
Vote
17
Rationale Why Ranking Works?
Outlier
Hub
Consistent Matching
Highly varient Matching
18
2. Structural Similarity

• Similar vertices ? Similar Link structures

(1) Appearance Similarity (2) Geometric
consistency
(3) similarity of matching behaviors
19
How to Mine Structural Similarity

• Automatic Extraction of Synonyms1

1 Blondel et al. SIAM review 2004.
u
v
If v appears in the definition of u
n?n
A vertex structural similarity matrix Z
20
Outline

• Problem Statement Our Approach
• Network Construction
• Link Analysis Techniques
• Ranking of features wrt an image/object
• Structural Similarity
• Unsupervised Modeling
• Category Discovery
• Localization
• Complexity
• Conclusion

21
Compute Ranking wrt Each Image
Ia
I1
I2
Im
22
Meaning of Ranking
n?1
Ib
Ranked importance of features in image a
Highly correlated
Ia
Ia
Ranked importance of the other features wrt
image a
Ic
Less correlated
Valuable for Category discovery
P-vector for Ia
23
Image Affinity Matrix
m PageRank vectors
n?1
I1
I2
Im
Vertex structural similarity matrix Z
Image affinity matrix A
m?m
n?n
n gtgt m
(Ex. 1mil gtgt 600)
24
Category Discovery by Clustering
k-NN graph 1
k 10log(m)
Normalized spectral Clustering 2
600 images of 6 Object Classes of Catech-101
1 Luxburg. Statistics and Computing, 2007
2 Shi Malik, PAMI 2000
25
Results of Category Discovery

• TUD/ETHZ dataset
• Experimental Setup follows 1
• 75 images per object, 10 repetition

95.47
Motorbikes Cars Giraffes
Motorbikes 93.3 ? 2.7 0.0 6.7 ? 2.7
Cars 4.8 ? 2.6 95.2 ? 2.6 0.0
Giraffes 2.0 ? 1.1 0.1 ? 0.4 97.9 ? 1.4
1 Grauman Darrell, CVPR 2006
26
Results of Category Discovery

• Caltech-101 Object classes (100 per object)

A C F M
A 98.4 1.0 0.1 0.5
C 0.2 99.8 0.0 0.0
F 1.9 0.1 98.0 0.0
M 1.4 0.6 0.0 98.0
4 obj 98.55
gt 2 98, 1 86
A C F M W
A 98.2 0.7 0.1 0.8 0.2
C 0.6 99.3 0.0 0.0 0.1
F 2.2 0.1 96.2 0.0 1.5
M 1.3 0.9 0.0 97.5 0.3
W 2.7 0.8 0.0 1.2 95.3
A
M
5 obj 97.30
W
C
A C F M W K
A 94.5 0.5 0.0 0.5 0.3 4.2
C 1.1 97.1 0.0 0.0 0.0 1.8
F 1.5 0.0 95.6 0.0 1.8 1.1
M 1.4 0.4 0.0 93.5 0.1 4.6
W 2.2 0.3 0.0 0.3 93.4 3.8
K 1.5 0.0 0.1 0.0 0.0 98.4
F
K
6 obj 95.42
1 Grauman Darrell, CVPR 2006 2 Sivic et al,
ICCV 2005
27
Compute Ranking wrt a Category
nc1?1
P-vector for category C1
nc2?1
nc3?1
P-vector for category C2
P-vector for category C3
PageRank
28
Meaning of Ranking
nc?1
Ranked importance of each feature wrt its category
Valuable for Localization
Ia
P-vector for a giraffe class
29
Localization Confidence Values
P-vectors and Vertex similarity matrix for each
category
nc1?1
nc1?nc1
nc3?1
nc3?nc3
nc2?1
nc2?nc2
30
Examples of Localization
31
Quantitative Results of Localization
False Positives
1 Quack et al. ICCV 2007
32
Outline

• Problem Statement Our Approach
• Network Construction
• Link Analysis Techniques
• Ranking of features wrt an image/object
• Structural Similarity
• Unsupervised Modeling
• Category Discovery
• Localization
• Complexity
• Conclusion

33
Complexity Issues

• The VSN representation is
• Sparsity of the network
• Power iterations for sparse matrices
• Scale-free network

Percentage of vertices
Ex. 6 objects of Caltech-101 ? 900K nodes
Degrees of Vertices
34
Outline

• Problem Statement Our Approach
• Network Construction
• Link Analysis Techniques
• Ranking of features wrt an image/object
• Structural Similarity
• Unsupervised Modeling
• Category Discovery
• Localization
• Complexity
• Conclusion

35
Conclusion

• A new formulation of unsupervised modeling
• Statistics of the link structure
• Finding communities (categories) and hubs (class
  representative visual information)
• Link analysis techniques
• Competitive performance
• Future directions
• Statistical framework
• Scalability

36
Comments?
Thank You gunhee_at_cs.cmu.edu
37
Supplementary Material
If any questions and comments, please send me an
email at gunhee_at_cs.cmu.edu http//www.cs.cmu.edu/
gunhee
38
Weights of Edges

• 1. Stronger geometric consistency, higher weights

Cij gt 0.8
Cij gt 0.7
Cij gt 0.6
Cij gt 0.5
Cij gt 0.4
wij 0.0284
wij 0.0144
wij 0.0071
wij 0.0040
wij 0.0018

• 2. 10 matches / 50 features gt 10 matches / 100
  features

39
Example of Vertex Similarity1

• Similarity between Vertices of Directed Graphs
• Only based on link structures

1 Blondel et al. SIAM review 2004.
40
Ranking for Category Discovery

• Relative Importance wrt each image

Ia
O
O
Ia
O
O
x
x
Ma
M
Ia
PageRank
- Only consider the relations between Ia and the
others
P-vector for Ia
- Why? To avoid Topic Drift
41
Computation of Relative Importance

• Before Category Discovery

Ia
O
O
PageRank for Ia
Ia
Why? Topic Drift
O
O
M
42
Relative Importance for Modeling

• Before Category Discovery

Wait ! In General, majority of images are from
different object classes. What if the result is
distracted by them?
Ia
O
O
PageRank Recursive Definition!!
Ia
Ic
Ia
Ib
O
O
The vote by the same object will be much more
appreciated !
M
43
Localization

• Relative Importance wrt each category

Mc1
Mc2
Mc3
M
PageRank
P-vector for each category
44
Computation of Relative Importance

• After Category Discovery

Ranked importance of features wrt each object
category
O
Valuable for Localization
O
object category k
M
P-vector
45
Toy Example Relative Importance

• Matching behavior
• Consistent between important features in the same
  class, Highly variant between backgrounds and
  different objects

Matching
46
Toy Example Relative Importance

• Consistent between important features in the same
  class, Highly variant between backgrounds and
  different objects

Image 2
Image 1
47
Toy Example Relative Importance

• Consistent between important features in the same
  class, Highly variant between backgrounds and
  different objects

Image 3
Image 1
48
Toy Example Relative Importance

• Consistent between important features in the same
  class, Highly variant between backgrounds and
  different objects

Highly variant
Consistent
Image 4
Image 1
49
Unsupervised Modeling

• Category Discovery Localization

I1
Ranked importance of the features in I1 wrt Ia
??
Affinity of I1 to Ia
?
A(a,1)
I1
Ia
This value is not used !
??
Im
Ia
P-vector Pa
A vertex similarity matrix Z
50
Localization

• Category Discovery Localization

O
object category 1
O
O
O
object category k
P-vector
M
Z
ai
?? 0.8
?
??
ai
P-vector Pc
Zc
P-vector Pc