What Do those Images Have in Common Sinisa Todorovic joint work with Prof. Narendra Ahuja

1
What Do those Images Have in Common?Sinisa
Todorovicjoint work with Prof. Narendra Ahuja
2
Recurring Similar Patterns Objects in 2D
3
Outline

• Object category recognition -- Review
• Unsupervised SINGLE category recognition
• Generalization to texture analysis
• Unsupervised MULTIPLE category recognition
• Supervised image categorization

4
Object Category -- Word Dictionary Definition
gt 30,000 categories
5
Object Category Recognition System
6
Prior Work Training
Uncertainty 1) Natural variations of a category
2) Occlusion, clutter, varying illumination, etc.

• Categories defined by the user
• Category must occur in every training image
• Supervised training Manually segmented objects
• FischlerElschlager 73, Winston 75, Leibe et al.
  04, Winn et al. 05, Opelt et al. 06
• Weakly supervised training Labeled images
  Background
• Weber et al. 00, Forsyth et al. 02 , Fergus et
  al.03, Fei-Fei et al. 04, Lowe et al. 04, Schmid
  et al. 04, Sivic et al. 05, Lazebnik et al. 06,
  Grauman et al. 06, etc.

7
Prior Work Feature Extraction

• Keypoints (e.g., Harris-Laplacian corners)
• Fergus et al. 03, Lowe 04, Fei-Fei et al. 04,
  Torralba et al. 04, GraumanDarrell 05,
  MokolajczykSchmid 05, Sivic et al. 05, Sudderth
  et al. 05, Lazebnik et al. 06, etc.
• Edges (e.g., Canny)
• Rosenfeld 72, Shotton et al. 05, Fergus et al.
  05, Ren et al. 05, Opelt et al. 06, Leordeanu et
  al. 07, etc.
• Regions (e.g., Mean-shift, N-cuts, Scale-space)
• HansonRiseman 78, Nevatia 89, BasriJacobs 97,
  KeselmanDickinson 05, WeissRay 05, Shokoufandeh
  et al. 06, Russell et al. 06, PantofaruHerbert
  07, etc.

8
Prior Work Object Representation

• Planar graph
• FischlerElschlager 73, Fergus et al. 03,
  FelzenszwalbHuttenlocher 05,
• Groups Forsyth, Lowe, Torr, Triggs, Zisserman
  04-07
• Hierarchical graph
• CrowelySanderson 87, Ettinger 88, Utans 92,
  NishidaMori 93, BoumanShapiro 94, PerrinAhuja
  98, BretznerLindenberg 99, Shokoufandeh et al.
  99, StorkeyWilliams 03, KeselmanDickinson 05,
  TodorovicNechyba 05
• Groups Buhmann, Geman, Leonardis, S-C. Zhu,
  Ullman, Yuille 00-07

9
Prior Work Bayesian Tree Representation
Iterative estimation of 1) PDF of model
structure 2) PDF of random variables
Forest of Bayesian DAGs ? Object model
TodorovicNechyba 05, 07
10
Deficiencies of Prior Work

• Aimed at limited goals
• Often not scalable, not generalizable
• Exact learning often infeasible
• Approximate inference
• Variational
• MCMC
• User-specified model structure
• Number of nodes
• Hierarchy depth
• Branching factor
• Large training sets Background

11
Outline

• Object category recognition -- Review
• Unsupervised SINGLE category recognition
• Generalization to texture analysis
• Unsupervised MULTIPLE category recognition
• Supervised image categorization

12
Problem Statement
GIVEN
Arbitrary images each containing 0 category
instances
DETERMINE
Training
If a category is present
AND IF YES LEARN
Model of the category
GIVEN
A new image
RECOGNIZE and SEGMENT
All occurrences of the learned category
13
Unsupervised Training

• Category not defined by the user
• Each image contains 0 category instances
• No background images
• Small training sets
• Sivic et al. 05 Russell et al. 06
  TodorovicAhuja 06, 07

14
WHAT IS A CATEGORY?
15
Any similar 2D objects?
Category Set of Similar 2D Objects
Category Set of Recurring Similar 2D Objects
(1) Photometric (e.g., contrast) (2)
Geometric (e.g., area, shape) (3) Topological
spatial layout of subcategories
containment of subcategories
arbitrary images
16
Features Image Regions

• Advantages of regions over keypoints and edges
• Facilitate modeling of Cohesiveness,
  Containment, Contiguity, etc.
• Higher-dimensional ? Richer descriptors, more
  discriminative
• Region boundaries coincide with object(-part)
  boundaries

17
Instability of Segmentation
splitting and merging of ADJACENT regions
18
Image Tree ? Object Subtree
multiscale segmentation
segmentation tree
TodorovicAhuja 06
19
Region Properties Associated with Each Node

• Contrast
• Area
• Central moments
• Displacement of centroids
• Orientation
• Perimeter

...
Relative wrt parent properties ? Rotation and
scale invariance
20
Region Neighbor Relationships
Generalized Voronoi Diagram
TodorovicAhuja CVPR08
21
From Trees to DAGs
Hierarchical Neighbor Links
22
How to Discover a Category?
Category present Many similar subgraphs
Discovering category instances Graph matching
23
Prior Work Graph Matching

• Spectral Siddiqi et al. 99, Shokoufandeh et al.
  05
• Edit-distance EsheraFu 86, BunkeAllermann 83,
  SebastianKimia 05
• Max-clique Pelillo et al. 99, TorselloHancock
  03, TodorovicAhuja 07

24
Graph Matching Subgraph Isomorphism
Max common subgraph

• Match regions if their intrinsic properties are
  similar,
• AND the same holds for their subregions,
• AND the same holds for their neighbor regions
• Preserve original hierarchical and neighbor
  relations

25
Addressing Instability of Low-Level Segmentation

• Many-to-many matching Augmenting trees with
  mergers
• Matching all descendants under a node
  Transitive closure

26
Graph Matching Formulation
which MAXIMIZES the similarity measure
function of region properties
27
Bottom-Up Computation
28
Solution Max Clique of Association Graph

• Theorem TodorovicAhuja IJCV07
• Structure preserving subgraph isomorphism
  Max-weight clique
• Complexity O(N2), N - number of nodes

29
Theoretical Result
Theorem Matching-IJCV07 Minimum-cost sequence
of node removes on the transitive closures of two
segmentation trees, T1 and T2, augmented with
merger nodes, yields the maximum consistent
subtree isomorphism between T1 and T2.
Theorem Representation-CVPR08 The maximum
consistent subgraph isomorphism between two
graphs with weighted edges and nodes, G1 and G2,
is equal to the maximum weight clique of the
association graph AG1xG2
30
Example of Matching
31
How to Extract Category Occurrences?
Modes exist ? Categories are present
frequency of subtree pairs
similarity measure
training images
32
How to Extract Category Occurrences?
similarity measure
discovered category occurrences
training images
33
Model of Structured Data?
34
Aligning and Registering into Graph Union
discovered occurrences
category model graph union
35
Category Model Graph Union Bayesian Net
object part (hidden)
region properties
number of children
Markovian dependencies 1) Hierarchical
2) Neighbor
structure parameters
36
Simultaneous Recognition and Segmentation
Matching image tree against the learned
graph-union
37
Results Weizmann Horses
training images
category model
38
Results Weizmann Horses

• Object segmentation is good on contours that are
• Jagged
• Blurred
• Form complex patterns
• Low-contrast regions merge with background

39
ST vs. CST
Segmentation Tree
CST
input images
UIUC Hoofed Animals
LabelMe
CSTs outperform STs, especially for objects
without shallow hierarchical structure of regions
40
ST vs. CST
Real-valued strength of neighbor relationships
Binary strength of neighbor relationships
Degree of occlusion artificially made in the image
41
Outline

• Object category recognition -- Review
• Unsupervised SINGLE category recognition
• Generalization to texture analysis
• Unsupervised MULTIPLE category recognition
• Supervised image categorization

42
What is image texture?
...Repeated occurrence of image texture elements
(or texels)... Beck 82
43
Prior Work

• Hardly any work on TEXEL modeling and
  segmentation
• Closest work Locating points or blobs
  representing texels
• NevatiaPrice 82, VoorheesPoggio 88,
  BlosteinAhuja 89, TuceryanJain 90, TomitaTsuji
  90, LeungMalik 96, Seyda-Mahmood 99,
  SchaffalitzkyZisserman 99, TuytelaarsGool 01,
  LobayForsyth 06, LinLiu 07

44
Problem Statement
GIVEN an image of frontally viewed 2.1D
texture,
IDENTIFY the texels, and
LEARN the texel model
45
Texel Extraction and Learning

• Identify subimages representing (partial) texels
• Register the subimages ? Many overlaying texel
  samples
• Find their union ? Model structure
• Estimate PDF of subimage properties ? Model
  parameters

46
Theorem Texels-ICCV07 Edit-based tree
matching is equivalent to minimizing the MDL of
the maximum subtree isomorphism
47
Evaluation
Results Texel Segmentation
original image
extracted texels
Extracted texel boundaries approximate well
perceptual texel boundaries
48
Evaluation
Results Texel Segmentation
original image
extracted texels
Extracted texel boundaries approximate well
perceptual texel boundaries
49
Outline

• Object category recognition -- Review
• Unsupervised SINGLE category recognition
• Generalization to texture analysis
• Unsupervised MULTIPLE category recognition
• Supervised image categorization

50
Unsupervised Training
articulation (self-)occlusion clutter
zero occurrences
multiple occurrences of multiple categories
scale viewpoint illumination

• Categories not defined by the user -- Unlabeled
  images
• Small inter-category differences

51
Problem Statement
52
HOW TO EFFICIENTLY MODEL MULTIPLE CATEGORIES?
53
Prior Work Dendogram Taxonomy

• Learn sharing of local features, but not sharing
  of parts
• Taxonomy defined wrt number of shared features
• Torralba et al. 04 Opelt et al. 06 Fei-Fei et
  al. 05, 06, 07

54
Multi-Category Representation Grammar

• Categories Configurations of subcategories
• Sharing of subcategories by parent categories
• Efficient because
• Subcategories have smaller variations and occur
  more frequently
• Sharing of parts among objects Sublinear
  complexity

55
Multi-Category Grammar

• Modeling arbitrarily structured categories
• No fixed number of nodes, hierarchy depth,
  branching factor
• Exact learning -- no need for approximate
  inference

56
Overview of Multi-Category Recognition
1. TREE MATCHING
57
Agglomerative Clustering
58
From Clusters to a Particular Categorization
KS-Test
a 5
59
From Clusters to a Particular Categorization
KS-Test
a 5
60
From Clusters to a Particular Categorization
KS-Test
a 5
61
Training Set UIUC Hoofed Animals
62
Simultaneous Recognition and Segmentation
63
Results Animals
Simultaneous Detection, Recognition, Segmentation
Simultaneous Recognition and Segmentation
64
Learned Unshared Parts
65
Quantitative Evaluation Detection,
Segmentation, Recognition
Table 1 Average recall, precision, segmentation,
and recognition errors (in )
66
Outline

• Object category recognition -- Review
• Unsupervised SINGLE category recognition
• Generalization to texture analysis
• Unsupervised MULTIPLE category recognition
• Supervised image categorization

67
Supervised Image Categorization
Two image classes sharing multiple subcategories

• Subcategory wagon-top is not found in image x

TodorovicAhuja CVPR08
68
Image Categorization Caltech 256
baseball bat
people
wagon
horses
69
Shared Subcategories Less Relevant for
Recognition
baseball bat
people
wagon
horses
70
Unshared Subcategories More Relevant for
Recognition
baseball bat
people
wagon
horses
71
Learning Subcategory Relevances
linear classifier
horse
two classes of image points
P(wagon horse)P(horse)
face
car wheel
wagon wheel
baseball bat
Zero relevance for wagon
wagon top
Max relevance for wagon
Each axis measures the confidence of subcategory
detection in the image
Subcategory relevance P(categorysubcategory)
P(subcategory)
72
Lemma Categorization-CVPR08 The proposed
EM-based estimation of the subcategory relevances
has a closed form solution.
Theorem Categorization-CVPR08 The proposed
algorithm for estimating the subcategory
relevances converges to a unique, global solution
regardless of the initialization point.
73
Results Caltech-256
74
Contributions

• Operative definition of a visual category for
  unsupervised settings
• Multi-category representation Taxonomy
• Unsupervised learning of multiple categories and
  their relationships
• Combining graph-theoretic algorithms with
  Bayesian inference
• Simultaneous recognition and segmentation
• Providing a semantic basis of recognition
• Never done before Texel segmentation
• The learning algorithm for estimating subcategory
  relevances

75
Acknowledgment
Prof. Narendra Ahuja
Dr. Michael Nechyba