Recovering Occlusion Boundaries from a Single Image

1
Recovering Occlusion Boundaries from a Single
Image

• Derek Hoiem Andrew Stein
• Alexei Efros Martial Hebert

Carnegie Mellon University Robotics Institute
Now at Beckman Institute, University of Illinois
2
Scene Understanding Requires Occlusion Reasoning
photo credit clare_and_ben (flickr)
3
Object Detection Requires Occlusion Reasoning
4
Spatial Layout Requires Occlusion Reasoning
5
Related Work Single Image ? 3D
Hoiem Efros Hebert 05
Saxena Chung Ng 05,07
Saxena Sun Ng (3dRR Saturday)
6
Related Work Line Drawing Interpretation

• Polygonal
• Curved
• Algebraic/MRF Reformulations

Guzman68 Clowes71 Huffman71 Waltz75
Huffman77 Kanade80 Draper 81
Guzman 1968
JainAggarwal79 Malik87
Malik 1987
Sugihara84 Saund05
Saund 2005
7
Related Work
Figure/Ground in Natural Scenes (Ren Fowlkes
Malik 06)
Ground Truth
Manual Segmentation
or
Ground Truth
Pb Boundaries
Pb Martin Fowlkes Malik 02
8
Our Goal

• Recover occlusion boundaries of freestanding
  objects

9
Our Goal

• Recover occlusion boundaries of freestanding
  objects
• Assign figure/ground labels to boundaries

10
Our Goal
Automatically Annotated Image
Corresponding Depth Image (Scaled)
Sky
Ground

• Recover occlusion boundaries of freestanding
  objects
• Assign figure/ground labels to boundaries
• Get a good sense of depth

11
Geometric Context Dataset

• Train 50 images
• Test 250 images (50 quantitative)

Hoiem Efros Hebert IJCV 07
12
The Challenge

• How can we reason about occlusion from a single
  2D image?

13
Occlusion Reasoning as Classification
region1
boundary
region3
region2
junction
non-occlusion
region1 occludes
region2 occludes
14
Gradual Inference of Scene Structure
Oversegmentation
Learned Models CRF Inference
Agglomerative Clustering
P(occlusion)
Next Segmentation
15
Gradual Inference of Scene Structure
Oversegmentation
Occlusion Cues
Learned Models CRF Inference
Agglomerative Clustering
P(occlusion)
Next Segmentation
16
Getting the Initial Boundaries
Pb Watershed
Pb Martin Fowlkes Malik 02 Arbelaez 06
17
Occlusion Cues
Region color, position, shape
Boundary strength, length, continuity
Surface Layout
Depth differences in depth range
Pb Martin Fowlkes Malik 02
Surface Layout Hoiem Efros Hebert IJCV 07
18
Occlusion Cues Surfaces
Surface Confidence Gradient
Online code from Hoiem Efros Hebert 07
19
Occlusion Cues Depth
20
Occlusion Cues Depth
Minimum Depth Interpretation
Current Boundary Estimate

ground/sky labels figure/ground labels ground
contact points
Maximum Depth Interpretation
Ground contact estimation Lalonde et al. 2007
21
Learn to Identify Occlusions
Labeled Training Set (50 Images)

Boundary Cues
Boosted Decision Trees
LogitBoost Collins Schapire Singer 02
22
Learn to Identify Occlusions
Labeled Training Set (50 Images)

Boundary Cues
Boosted Decision Trees
Boundary Cues
Input Boundaries
P(occlusion) Boundary Map
LogitBoost Collins Schapire Singer 02
23
Local Estimates May Be Inconsistent
24
Global Consistency via CRF

• Boundary consistency
• Inference over junctions
• Enforce closure by valid junctions
• Encourage smooth contours by conditional
  likelihood

25
Global Consistency via CRF

• Surface consistency
• Inference over surface labels and boundaries
• Enforce boundary between different surface types
• Penalize floating objects

Object should occlude sky
Ground should occlude sky
Sky
e2
e4
e1
Object should occlude ground
Ground
e3
26
CRF Inference

• Max-product intractable due to high potentials
• Soft-max via sum-product with mean-field
  approximation

Image
Pre-CRF (Unary)
Post-CRF
Heskes Albers Kappen 03 (sum-product
inference) Yuille 02 (mean field
approximation) Potetz 07 (use of two for
max-prod approx)
27
Getting the Next Boundaries

• Agglomerative clustering with complete linkage
  (region distance value of strongest boundary
  between them)
• Threshold set by validation to get next
  segmentation

Boundary Confidence Map
Segmentation
28
Gradual Inference of Scene Structure
Boundaries

Depth Underestimate
Depth Overestimate
2nd Iteration
3rd Iteration
1st Iteration
29
Final Estimate
Depth (Min)
Boundaries, Foreground/Background, Contact
Depth (Max)
30
3D Cues Critical for Finding Occlusions
Occlusion vs. Non-Occlusion Classification
31
Iterative Approach, 3D/CRF Important
Figure/Ground Labeling
Edge/Region Cues 3D Cues With CRF
Iter 1 58.7 71.7 Not Used
Iter 2 65.4 75.6 77.3
Final 68.2 77.1 79.9
32
Segmentation Comparison
Conservation Efficiency (log2)
Our Algorithm 83.7 -0.8
Surface Labels 82.4 -1.4
Ncuts (CourBenezitShi05) 81.7 -1.2
(1, 0)
(0.90, -1.22)
(0.93, -1.26)
(0.85, -1.83)
(0.63, 1.26)
(0.68, -2.26)
(0.80, -1.32)
(1, 0)
33
Occlusion Result
Depth (Min)
Boundaries, Figure/Ground, Contact
Depth (Max)
34
Occlusion Result
Depth (Min)
Boundaries, Figure/Ground, Contact
Depth (Max)
35
Occlusion Result
Depth (Min)
Depth (Max)
Boundaries, Figure/Ground, Contact
36
Conclusions

• Occlusion provides new perspective on
  segmentation
• Much occlusion reasoning can be done from one
  image without identifying objects
• Look beyond the image into the scene

37
Thank you!