From%20Hierarchies%20of%20Regions%20to%20Image%20Understanding%20and%20Manipulation%20Prof.%20Sinisa%20Todorovic

1
From Hierarchies of Regions to Image
Understanding and ManipulationProf. Sinisa
Todorovic
2
Acknowledgment
UIUC Prof. Narendra Ahuja Himanshu Arora Varsha
Hedau Oregon State University William
Brendel Nadia Payet Muhamed Amer Prof. Eugene
Zhang
3
Goals Object Recognition
input image set
new image
detect segment explain all occurrences of the
learned objects
discover and learn all objects present
4
Goal Video Painterly Rendering
video sequence enhanced with multiple painting
styles -- one per each object
...
...
flower petals van Gogh stamens
expressionism background pointilism
5
Goals Texel-based Texture Segmentation
Many applications require unsupervised
partitioning of the image into textured and
non-textured subimages
6
Problem Statement

• Given a set of images
• Discover frequently occurring 2D objects
• Under illumination and scale changes
• Amidst background clutter
• Under partial occlusion
• Learn their generative, statistical models
• Use the models for
• Object recognition
• Object-based painterly rendering and synthesis
• Texel-based texture segmentation

7
Prior Work Object Recognition
PRIOR WORK
OUR EXTENSIONS
high degree of supervision relaxing supervision requirements
predominance of keypoint features using richer features regions
ignoring the spatial info accounting for multiscale spatial info
limited goals unified framework for many goals
8
Prior Work Painterly Rendering
PRIOR WORK
OUR EXTENSIONS
Uses only a single style Object-based multiple styles
Unrealistic, poor artistic expression Rich artistic expression
9
Prior Work Texture Segmentation
PRIOR WORK
OUR EXTENSIONS
Uses a pre-specified bank of filters Extraction of texels
Assumptions smoothness, scale Relaxing the assumptions
meanshift
active contours
our results
10
WHAT IS AN OBJECT?
11
Properties of Objects
3D objects in the scene
2D objects in the image
cohesive occupy regions
form characteristic spatial configurations with other objects context
have parts subregions
parts have characteristic spatial layout spatial layout of subregions
12
(No Transcript)
13
Physical Objects in 3D World vs. 2D Objects
Rationale -- Like a Small Child
Rationale for Learning -- Like a Small Child
input images

• It is likely to be meaningful
• If some parts repeat in the set of images
• If some configurations of the learned parts
  repeat in the set

14
Physical Objects in 3D World vs. 2D Objects
Rationale -- Like a Small Child
Rationale for Learning -- Like a Small Child
input images

• It is likely to be meaningful
• If some parts repeat in the set of images
• If some configurations of the learned parts
  repeat in the set

15
Any similar 2D objects?
Category Set of Similar 2D Objects
Category Set of Recurring Similar 2D Objects
(1) Photometric (e.g., color) (2) Geometric
(e.g., area, shape) (3) Structural
spatial layout of subcategories
containment of subcategories
input images
16
Object Grammar
Advantages

• Regions as image features
• Compositionality and Reusability
• Object Configuration of recursively simpler
  parts
• Sharing of parts by parent objects
• Efficient because
• Parts have smaller variations and occur more
  frequently
• Sharing of parts among objects Sublinear
  complexity

input images
17
Object Grammar
Caveat

• Many semantic categories are not visual
• Our definition addresses only those categories
  that can be defined via appearance, structure,
  and context

input images
18
Rest of the Talk

1. Image representation Hierarchy of regions
2. Region matching under unstable segmentations
3. Applications and results

19
Rest of the Talk

1. Image representation Hierarchy of regions
2. Region matching under unstable segmentations
3. Applications and results

20
Image Tree ? Object Subtree
multiscale segmentation
segmentation tree
Ahuja PAMI96, Tobb Ahuja TIP97, AroraAhuja
ICPR06
21
Connected Segmentation Trees
Lateral links Region neighbor relations
Hierarchical links Region embedding
AhujaTodorovic CVPR08
22
Region Properties Associated with Each Node

• Gray-level contrast with surround
• Boundary shape
• Displacement of centroids
• Orientation

...
Properties relative wrt parent ? Scale and
in-plane rotation invariance
23
Rest of the Talk

1. Image representation Hierarchy of regions
2. Region matching under unstable segmentations
3. Applications and results

24
How to Discover Repeating Image Parts?
Object category is present Many similar
subgraphs
Discovering objects Graph matching
25
Graph Matching Subgraph Isomorphism

• Match two regions
• If their immediate properties are similar
• AND the same holds for their subregions
• AND the same holds for their neighbors

26
Graph Matching Formulation
Find the mapping
which minimizes their cost of matching
27
Graph Matching Formulation
Linearization by introducing an indicator vector
matched pair
unmatched pair
28
Graph Matching Formulation
Relaxation of the discrete problem
TodorovicAhuja IJCV08, PAMI08, CVPR06-08,
ICCV07, ICPR06-08
29
Rest of the Talk

1. Image representation Hierarchy of regions
2. Region matching under unstable segmentations
3. Applications and results

30
Main Problem Splitting and Merging of Regions
region i
region j
image 1
image 2
candidate matches
candidate matches
31
Addressing Segmentation Irrepeatability
PayetTodorovic GbR09, BrendelTodorovic ICCV09
32
Unary Potential Cost of Matching Shape Parts
33
Unary Potential ? Circular Dynamic Time Warping
BrendelTodorovic ICCV09
34
Addressing Segmentation Irrepeatability
PayetTodorovic GbR09, BrendelTodorovic ICCV09
35
Pairwise Potentials Cost of Matching
Relationships
Consider all relationships in the image trees
original image trees
transitive closure
PayetTodorovic GbR09
36
Rest of the Talk

• Image representation Hierarchy of regions
• Region matching under unstable segmentations
• Applications and results
• Object recognition
• Video object segmentation
• Painterly rendering
• Texture segmentation

37
Discovering Objects Matching Clustering
training images
discovered category occurrences
38
Learning Repeating Image Parts Matching
Cluster
Discovering Objects Matching Clustering
training images
Each cluster Distinct Object
39
Learning a Model of Each Cluster Structural EM
matched subgraphs
hierarchical object model
model structure ?
model parameters ?
40
Structural EM Learning Grammar
Learning a Model Structural EM
TodorovicAhuja ICCV07
41
Category Model Bayesian Net
object part (hidden)
region properties
number of children
42
Rissanens Minimum Description Length
43
Theoretical Result

• Theorem ICCV 2007
• Finding maximum subgraph isomorphism between
  pairs of graphs is equivalent to minimizing the
  MDL of the graphs and their model.

44
Results Weizmann Horses
training images
category model
TodorovicAhuja PAMI08
45
Results Weizmann Horses

• Object segmentation is good on contours that are
• Jagged
• Blurred
• Form complex patterns
• Low-contrast regions merge with background

TodorovicAhuja PAMI09
46
UIUC Hoofed Animals Dataset
http//vision.ai.uiuc.edu/sintod/HoofedAnimalsDat
aset.html
training images
47
Multi-Object Recognition
1. TREE MATCHING
48
Overview of Multi-Category Recognition
1. TREE MATCHING
2. CLUSTERING
49
Overview of Multi-Category Recognition
1. TREE MATCHING
2. CLUSTERING
3. TAXONOMY OF ALL DISCOVERED CATEGORIES WITH
DIFFERENT COMPLEXITIES
4. RECOGNIZE SEGMENT EXPLAIN
50
Simultaneous Recognition and Segmentation
51
Results Animals
Simultaneous Detection, Recognition, Segmentation
Simultaneous Recognition and Segmentation
52
Learned Unshared Parts
53
Quantitative Evaluation Detection,
Segmentation, Recognition
Table 1 Average recall, precision, segmentation,
and recognition errors (in )
54
Discriminative Learning of Object Parts
TodorovicAhuja CVPR08
55
CVPR 2008 Results on Caltech-256
56
Rest of the Talk

• Image representation Hierarchy of regions
• Region matching under unstable segmentations
• Applications and results
• Object recognition
• Video object segmentation
• Painterly rendering
• Texture segmentation

57
Video Object Segmentation

• Objects occupy 2D regions in each video frame
• Moving objects form 2Dt subvolumes in space-time
  volume
• Goal Extract 2Dt subvolumes that are coherent
  in time and space

58
Video Object Segmentation
BrendelTodorovic ICCV09
59
Results Video Object Segmentation
input video
meanshift
our segmentation
BrendelTodorovic ICCV09
60
Results Video Object Segmentation
BrendelTodorovic ICCV09
61
Rest of the Talk

• Image representation Hierarchy of regions
• Region matching under unstable segmentations
• Applications and results
• Object recognition
• Video object segmentation
• Painterly rendering
• Texture segmentation

62
Multi-style Painterly Rendering
Collaboration with Prof. Eugene Zhang at Oregon
State University
63
Results Multi-style Painterly Rendering
Collaboration with Prof. Eugene Zhang at Oregon
State University
64
Results Multi-style Painterly Rendering
Collaboration with Prof. Eugene Zhang at Oregon
State University
65
Results Multi-style Painterly Rendering
Collaboration with Prof. Eugene Zhang at Oregon
State University
66
Rest of the Talk

• Image representation Hierarchy of regions
• Region matching under unstable segmentations
• Applications and results
• Object recognition
• Video object segmentation
• Painterly rendering
• Texture segmentation

67
What is image texture?
...Repeated occurrence of image texture elements
(or texels)... Beck 82
68
Texture Spatial Repetition of Texels

• Image texels Images of physical texture
  elements
• Texels are not identical, only statistically
  similar
• Texel placement is not regular

69
Problem Statement
DISCOVER and LEARN a model of unoccluded
texel from only partially visible texels
DETECT and SEGMENT all texels in a new image
by using the learned model
70
Unsupervised Learning of Texels
71
Evaluation
Results Texel Segmentation
Results Unsupervised Texel Extraction
Results UIUC Texture Dataset
original image
extracted texels
AhujaTodorovic ICCV07
72
Evaluation
Results UIUC Texture Dataset
original image
texel segmentation
AhujaTodorovic ICCV07
73
Results Texture Segmentation
TodorovicAhuja ICCV09
74
Results Texture Segmentation
original image
filter-based Galun et al ICCV03
texel-based TodorovicAhuja ICCV09
75
Results Texture Segmentation
original image
color-based DonserBischof CVPR09
texel-based TodorovicAhuja ICCV09
76
Summary

• Hierarchical region-based image representation
• Robust matching of regions
• Operative definition of an object category
• Hierarchical taxonomy of shared categories
• The framework allows
• Simultaneous recognition and segmentation
• Semantic basis of recognition
• Space-time coherent video object segmentation
• Texel-based texture analysis

77
Thank you!