Stereo Computation using Iterative GraphCuts

1
Stereo ComputationusingIterative Graph-Cuts
Vision Course - Weizmann Institute
2
The Binocular Stereo problem
Right Camera
Left Camera
Two views of the same scene from a slightly
different point of view
3
The stereo problem

• Both images are very similar (like images that
  you see with your two eyes)
• Most of the pixels in the left image are
  present in the right image (except for few
  occlusions)

4
Rectification

• Image Reprojection
• reproject image planes onto common plane parallel
  to baseline

5
The stereo problem
After rectification all correspondences are
along the same horizontal scan lines
(pixels in one image simply shift horizontally in
the other image)
6
The relation between depth and disparities

• Origin at midpoint between camera centers
• Axes parallel to those of the two (rectified)
  cameras

Depth and Disparity are inverse proportional
7
The stereo problem

• The horizontal shifts between the images are
  sometimes called disparities
• The Disparities are related to depth Closer
  objects have larger disparities

8
The stereo problem compute the disparity map
between two images
9
Traditional Approaches Matching small windows
around each pixel Each window is matched
independently Modern approaches Finding
coherent correspondences for all pixels - Graph
cuts - Belief Propagation
10
Window-Based Approach

• Compute a cost for each location
• Location with the lowest cost wins

11
General Problem Ambiguity
Left
Right
scanline
12
Window-Based Approach
Small Window
Large Window
blurred boundaries
noisy in low texture areas
13
Results with best window size(still not good
enough)
Window-based matching (best window size)
Ground truth
14
Graph Cuts
Ground truth
Graph cuts
15
Maximum flow problem

• Max flow problem
• Each edge is a pipe
• Find the largest flow F of water that can be
  sent from the source to the sink along the
  pipes
• Edge weights give the pipes capacity

16
Minimum cut problem

• Min cut problem
• Find the cheapest way to cut the edges so that
  the source is completely separated from the
  sink
• Edge weights now represent cutting costs

17
Max flow/Min cut theorem

• Max Flow Min Cut
• Maximum flow saturates the edges along the
  minimum cut.
• Ford and Fulkerson, 1962
• Problem reduction!
• Ford and Fulkerson gave first polynomial time
  algorithm for globally optimal solution

18
The basic Ford-Fulkerson algorithm
for each edge
do
while there exists a path P from s to t in the
residual network Gf do cf (P) ? mincf (u, v )
(u, v) is on P for each edge (u, v) in P do
19
Min-Cut Important Rule
No subset of the cut can also be a cut
This is not a minimal cut
20
Energy Minimization Using Iterative Graph cuts
Fast Approximate Energy Minimization via Graph
Cuts Yuri Boykov, Olga Veksler and Ramin
ZabihPami 2001
More papers, code http//www.cs.cornell.edu/rdz/
graphcuts.html
21
To do better we need a better model of images

• We can make reasonable assumptions about the
  surfaces in the world
• Usually assume that the surfaces are smooth
• Can pose the problem of finding the corresponding
  points as an energy (or cost) minimization

f - assignment
neighboring pixels have similar disparities
how well the pixels match up for different
disparities
22
To do better we need a better model of images

• We can make reasonable assumptions about the
  surfaces in the world
• Usually assume that the surfaces are smooth
• Can pose the problem of finding the corresponding
  points as an energy (or cost) minimization

f - assignment p,q - pixels
Data term is calculated for each pixels
Smoothness is calculated on neighbor pixels
23
Example for Smoothness terms

• Quadratic
• L1
• Truncated L1
• Potts model

24
Constructing a Graph to Solve the Stereo Problem
25
Constructing a Graph to Solve the Stereo Problem
26
Constructing a Graph to Solve the Stereo Problem
The labels of each pixel are the possible
disparity values
27
Constructing a Graph to Solve the Stereo Problem
The labels of each pixel are the possible
disparity values
28
Relation between the Energy and the Graph
labeling problem
fp10
Data term
10
1
fq2
q
p
29
Relation between the Energy and the Graph
labeling problem
Dp(10)
Data term
10
V(p,q)(1, 10)
1
q
p
30
Iterative graph-cuts

• Use an iterative scheme to find a good local
  optimum of the energy function.
• In each iteration convert the original
  multi-label problem to a binary one, and solve it
  by finding a minimal graph-cut (max-flow).
• The most popular scheme is the expansion move.
• -expansion set the label of each pixel to be
  either or the current label.

31
Types of Moves
A Single Pixel Move
Problem A lot of local minima
32
Types of Moves
Expansion Move
Any pixel can change its label to alpha
33
Types of Moves
Expansion Move
Claim (without proof) The difference between the
optimal solution and the solution from the
iterative expansion moves is bounded
34
Energy Minimization Algorithm

• Start with arbitrary labeling f
• Set success 0
• For each label
• Find
• If set and success 1
  
• If success 1 goto (2)
• Return f

35
Conditions on the Smoothness for using expansion
moves
In other words V should be a metric
Note The Quadratic smoothness is not a metric
36
For each pair of vertices
such that we add a dummy
vertex (together with the respective edges as
shown in the table).
37
The Relation between the cut and the Energy

• Given a cut C, we define a labeling fc by
• The cost of a cut C is C E(fC) (plus a
  constant)

If the cut C separates p and
If the cut C separates p and
38
The Relation between the cut and the Energy
The case
39
The Relation between the cut and the Energy
The case
40
Conditions on the Smoothness for using expansion
moves
In other words V should be a metric
41
The image segmentation problem given an image,
group it to several regions containing pixels
with similar intensities / colors.
an image
a labeling
42
Supervised Image Segmentation

• We assume that for each segment, users scribbles
  are given pixels that are known in be inside
  this segment

examples with two segments
43
Again, we can describe the problem using a graph
Data term
fp5 means p belongs to the segment 5.
V(p,q)(1, 10) The penalty for assigning p and q
to different segments (1 and 10 respectively).
D(p)(5) The penalty for assigning p to segment
5.
44
The Data Smoothness terms
The penalty for assigning p and q to different
segments should be high if the colors of pixels p
and q are similar. For example
As a data term, we have only one constraint That
the user scribbles will be assigned with the
correct label
For pixels inside user scribbles
for the rest of the pixels
45
Graph Cuts can be used for problems other than
Stereo ! (segmentation, noise removal, image
stitching, etc).