Part 4: Combined segmentation and recognition

1
Part 4 Combined segmentation and recognition
by Rob Fergus (MIT)
2
Aim

• Given an image and object category, to segment
  the object

Object Category Model
Segmentation
Cow Image
Segmented Cow

• Segmentation should (ideally) be
• shaped like the object e.g. cow-like
• obtained efficiently in an unsupervised manner
• able to handle self-occlusion

Slide from Kumar 05
3
Feature-detector view
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
Examples of bottom-up segmentation

• Using Normalized Cuts, Shi Malik, 1997

Borenstein and Ullman, ECCV 2002
8
Jigsaw approach Borenstein and Ullman, 2002
9
Implicit Shape Model - Liebe and Schiele, 2003
Liebe and Schiele, 2003, 2005
10
Random Fields for segmentation
I Image pixels (observed) h
foreground/background labels (hidden) one label
per pixel ? Parameters
Likelihood
Posterior
Joint
Prior

• Generative approach models joint
• ? Markov random field (MRF)
• 2. Discriminative approach models posterior
  directly
• ? Conditional random field (CRF)

11
Generative Markov Random Field
i
Prior has no dependency on I
j
I (pixels)
Image Plane
12
Conditional Random Field
Lafferty, McCallum and Pereira 2001
Discriminative approach
Pairwise
Unary

• Dependency on I allows introduction of pairwise
  terms that make use of image.
• For example, neighboring labels should be
  similar only if pixel colors are similar ?
  Contrast term

e.g Kumar and Hebert 2003
13
OBJCUT
Kumar, Torr Zisserman 2005
Pairwise
Unary
Label smoothness
Distance from O
Color Likelihood
Contrast
O (shape parameter)

• O is a shape prior on the labels from a Layered
  Pictorial Structure (LPS) model
• Segmentation by
• - Match LPS model to image (get number of
  samples, each with a different pose
• Marginalize over the samples using a single
  graph cut
• Boykov Jolly, 2001

14
OBJCUTShape prior - O - Layered Pictorial
Structures (LPS)

• Generative model
• Composition of parts spatial layout

Layer 2
Spatial Layout (Pairwise Configuration)
Layer 1
Parts in Layer 2 can occlude parts in Layer 1
Kumar, et al. 2004, 2005
15
OBJCUT Results
Using LPS Model for Cow
In the absence of a clear boundary between object
and background
Segmentation
Image
16
Levin Weiss ECCV 2006
Segmentation alignment with image edges
Consistency with fragments segmentation
17
Layout Consistent Random Field
Winn and Shotton 2006
18
Layout consistency
Winn and Shotton 2006
Neighboring pixels
(p,q)
?
(p,q1)
(p,q)
(p1,q1)
(p-1,q1)
Layoutconsistent
19
Layout Consistent Random Field
Winn and Shotton 2006
20
Stability of part labelling
Part color key
21
Object-Specific Figure-Ground Segregation
Stella X. Yu and Jianbo Shi, 2002
22
Image parsing Tu, Zhu and Yuille 2003
23
Image parsing Tu, Zhu and Yuille 2003
24
Todorovic and Ahuja, CVPR 2006
.
Multiscale Seg.
Segmentation Trees
Overview
fused tree model for cars
Training images
Segment out all the cars
Unseen image
Segmented Cars
Slide from T. Wu
25
LOCUS model
Kannan, Jojic and Frey 2004 Winn and Jojic, 2005
Shared between images
Class shape p
Class edge sprite µo,so
Deformation field D
Position size T
Different for each image
Mask m
Edge image e
Object appearance ?1
Background appearance ?0
Image
26
In this section brief paper reviews

• Jigsaw approach Borenstein Ullman, 2001, 2002
• Concurrent recognition and segmentation Yu and
  Shi, 2002
• Image parsing Tu, Zhu Yuille 2003
• Interleaved segmentation Liebe Schiele, 2004,
  2005
• OBJCUT Kumar, Torr, Zisserman 2005
• LOCUS Winn and Jojic, 2005
• LayoutCRF Winn and Shotton, 2006
• Levin and Weiss, 2006
• Todorovic and Ahuja, 2006

27
Summary

• Strength
• Explains every pixel of the image
• Useful for image editing, layering, etc.
• Issues
• Invariance issues
• (especially) scale, view-point variations
• Inference difficulties

28
(No Transcript)
29
(No Transcript)
30
Conditional Random Fields for Segmentation

• Segmentation map x
• Image I

Low-level pairwise term
High-level local term
Pixel-wise similarity
31
Object-Specific Figure-Ground Segregation
Some segmentation/detection results
Yu and Shi, 2002
32

• Multiscale Conditional Random Fields for Image
  Labeling
• Xuming He Richard S. Zemel Miguel A .
  Carreira-Perpinan
• Conditional Random Fields for Object
• Recognition
• Ariadna Quattoni Michael Collins Trevor Darrell

33
OBJCUT

• Probability of labelling in addition has
• Unary potential which depend on distance from T
  (shape parameter)

T (shape parameter)
Unary Potential Fx(mxT)
mx
m (labels)
my
Object Category Specific MRF
x
y
D (pixels)
Image Plane
Kumar, et al. 2004, 2005
34
Localization using features
35
Levin and Weiss 2006
Levin and Weiss, ECCV 2006
36
Results horses
37
Results horses
38
Cows Results

• Segmentations from interest points
• Single-frame recognition - No temporal continuity
  used!

Liebe and Schiele, 2003, 2005
39
(No Transcript)
40
Examples of low-level image segmentation

• Normalized Cuts, Shi Malik, 1997

Borenstein Ullman, ECCV 2002
41
(No Transcript)
42
Jigsaw approach

• Each patch has foreground/background mask

43
LayoutCRF
44
(No Transcript)
45
Segmentation

• Interpretation of p(figure) map
• per-pixel confidence in object hypothesis
• Use for hypothesis verification

Liebe and Schiele, 2003, 2005