Measuring Uncertainty in Graph Cut Solutions

1
Measuring Uncertainty in Graph Cut Solutions

• Pushmeet Kohli Philip H.S. Torr
• Department of Computing
• Oxford Brookes University

2
Objective
Graph Cut
No uncertainty measure associated with the
solution
Belief or Confidence
3
Outline

• Inference in Graphical Models
• Inference using Graph Cuts
• Computing Min-marginals using Graph Cuts
• Flow Potentials and Min-marginals
• Results

4
Outline

• Inference in Graphical Models
• Inference using Graph Cuts
• Computing Min-marginals using Graph Cuts
• Flow Potentials and Min-marginals
• Results

5
Inference in Graphical Models
x Set of latent variables D Observed Data
x Most Probable (MAP) Solution Pr Joint
Posterior Probability
x arg max Pr(xD)
x
E(xD) -log Pr(xD) constant
x arg min E(xD)
x
6
Inference in Graphical Models
MAP Solution arg max Pr(A,B) A1, B0
B
0
1
2
0
Joint Distribution
A
1
Pr(A,B)
2
7
Inference in Graphical Models

• Marginal
• Sum the joint probability over all other
  variables.

B
0
1
2
0
A
P(A1) 0.6
1
2
8
Inference in Graphical Models

• Max- Marginals (µ)
• Maximum joint probability over all other
  variables.

B
0
1
2
0
A
µA,1 0.35
1
2
9
Inference in Graphical Models

• Confidence or Belief (s)
• Normalized max-marginals

sA,1 µA,1 / Sx µA,x 0.35/ 0.58 0.603
B
0
1
2
µA,0 0.2
0
A
µA,1 0.35
1
2
µA,2 0.03
10
Inference in Graphical Models

• Min-Marginals Energies(?)
• Minimize joint energy over all other variables.
• Related to max-marginals as

µj (1/z)exp(-?j)
11
Inference in Graphical Models

• Min-Marginals Energies(?)
• Minimize joint energy over all other variables.
• Related to max-marginals as

µj (1/z)exp(-?j)
- Can be used to compute confidence as
sj µj / Sa µa exp(-?i) / Sa exp(-?a)
12
Inference in Graphical Models

• Min-Marginals Energies(?)
• Minimize joint energy over all other variables.
• Related to max-marginals as

µj (1/z)exp(-?j)
- Can be used to compute confidence as
sj µj / Sa µa exp(-?i) / Sa exp(-?a)
How to compute min-marginal energies using Graph
Cuts?
13
Inference in Graphical Models
14
Inference in Graphical Models
15
Outline

• Inference in Graphical Models
• Inference using Graph Cuts
• Computing Min-marginals using Graph Cuts
• Flow Potentials and Min-marginals
• Results

16
Inference using Graph cuts

• Given energy functions E(xD),
• Compute arg min E(xD)

x
17
Inference using Graph cuts

• Given energy functions E(xD),
• Compute arg min E(xD)

x

• Certain E(xD) can be minimized using graph cuts
  exactly.

18
Inference using Graph cuts

• Given energy functions E(xD),
• Compute arg min E(xD)

x

• Certain E(xD) can be minimized using graph cuts
  exactly.

• Class of energy function and graph contruction
• - Binary random variables
• - Submodular functions (Kolmogorov Zabih, ECCV
  2002)
• - Multi-valued variables
• Convex Pair-wise Terms (Ishikawa, PAMI 2003)

19
Inference using Graph cuts
Graph Construction for Binary Random Variables
EMRF(a1,a2)
Source (1)
a1
a2
Sink (0)
20
Inference using Graph cuts
EMRF(a1,a2) 2a1
Source (1)
2
t-edges (unary terms)
a1
a2
Sink (0)
21
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1
Source (1)
2
a1
a2
5
Sink (0)
22
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1 9a2 4a2
Source (1)
2
9
a1
a2
5
4
Sink (0)
23
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2
Source (1)
2
9
a1
a2
2
5
4
n-edges (pair-wise term)
Sink (0)
24
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (1)
2
9
1
a1
a2
2
5
4
Sink (0)
25
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (1)
2
9
1
a1
a2
2
5
4
Sink (0)
26
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (1)
2
9
Cost of st-cut 11
1
a1
a2
2
a1 1 a2 1
5
4
EMRF(1,1) 11
Sink (0)
27
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (1)
2
9
Cost of st-cut 8
1
a1
a2
2
a1 1 a2 0
5
4
EMRF(1,0) 8
Sink (0)
28
Inference using Graph cuts
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (1)
2
9
Cost of st-cut 8
1
a1
a2
2
a1 1 a2 0
5
4
MAP Solution
EMRF(1,0) 8
a1,map 1 a2,map 0
Sink (0)
29
Outline

• Inference in Graphical Models
• Inference using Graph Cuts
• Computing Min-marginals using Graph Cuts
• Flow Potentials and Min-marginals
• Results

30
Computing Min-marginals using Graph Cuts
31
Computing Min-marginals using Graph Cuts
Instead of minimizing E(.), minimize a projection
of E(.) where the value of latent variable xv is
fixed to label j.
All projections of a sub-modular function
are sub-modular Kolmogorov and Zabih, ECCV 2002
32
Computing Min-marginals using Graph Cuts
Instead of minimizing E(.), minimize a projection
of E(.) where the value of latent variable xv is
fixed to label j.
Problem Solved? Not Really!
33
Computing Min-marginals using Graph Cuts

• Computing a min-marginal requires computation of
  a st-cut.
• Typical Segmentation problem
• 640x480 image, 2 labels
• Variables 640x480 307200
• Number of Min-marginals 307200x2 614400
• Time taken for 1 graph cut .3 seconds
• Total computation time 614400x0.3 184320
  sec 51.2 hours!!!

34
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Source (1)
2
9
1
a1
a2
2
5
4
Sink (0)
35
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
a2 1
Source (1)
EMRF(a1,1) 2a1 5a1 9 a1
2
9
1
a1
a2
2
5
4
Sink (0)
36
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
a2 1
Source (1)
EMRF(a1,1) 2a1 5a1 9 a1
2
EMRF(a1,1) 2a1 6a1 9
9
a1
a2
6
Sink (0)
37
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Ka2
Alternative Construction
Source (1)
A high unary term (t-edge) can be used to
constrain the solution of the energy to be the
solution of the energy projection.
2
9
1
a1
a2
2
5
4
K
8
Sink (0)
38
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Ka2
Alternative Construction
Source (1)
A high unary term (t-edge) can be used to
constrain the solution of the energy to be the
solution of the energy projection.
2
9
1
a1
a2
2

• The minimum value of the energy projection can
  be calculated by using the same graph.

5
4
K
8
Sink (0)
39
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Ka2
Alternative Construction
Source (1)
A high unary term (t-edge) can be used to
constrain the solution of the energy to be the
solution of the energy projection.
2
9
1
a1
a2
2

• The minimum value of the energy projection can
  be calculated by using the same graph.

5
4
K
8
Sink (0)
Kgt Sum of Outgoing/Incoming edges
40
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Ka2
Alternative Construction
Source (1)
A high unary term (t-edge) can be used to
constrain the solution of the energy to be the
solution of the energy projection.
2
9
1
a1
a2
2

• The minimum value of the energy projection can
  be calculated by using the same graph.

5
4
8
K

• Results in a small change in the graph.

Sink (0)
41
Energy Projections and Graph Construction
EMRF(a1,a2) 2a1 5a1 9a2 4a2 2a1a2 a1a2
Ka2
Alternative Construction
Source (1)
A high unary term (t-edge) can be used to
constrain the solution of the energy to be the
solution of the energy projection.
2
9
1
a1
a2
2

• The minimum value of the energy projection can
  be calculated by using the same graph.

5
4
8
K

• Results in a small change in the graph.

Sink (0)
Solve using Dynamic Graph Cuts ?
42
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Energy function
Graph 2
Projection of Energy function
43
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Energy function
Graph 1 and 2 are similar
Graph 2
Projection of Energy function
44
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Graph 2
45
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Residual Graph 1
st-cut 1
Compute Max-flow
Computationally Expensive Procedure
Graph 2
Residual Graph 2
st-cut 2
Compute Max-flow
46
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Residual Graph 1
st-cut 1
Compute Max-flow
Re-parameterized Graph 2
Graph 2
Residual Graph 2
st-cut 2
47
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Residual Graph 1
st-cut 1
same solution
Re-parameterized Graph 2
Graph 2
Residual Graph 2
st-cut 2
48
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Residual Graph 1
st-cut 1
similar edge weights
Re-parameterized Graph 2
Graph 2
Residual Graph 2
st-cut 2
49
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Residual Graph 1
st-cut 1
Compute Max-flow
Re-parameterized Graph 2
Compute Max-flow
Extremely Fast Operation
Graph 2
Residual Graph 2
st-cut 2
50
Dynamic Graph Cuts
- Kohli and Torr ICCV 2005
Graph 1
Residual Graph 1
st-cut 1
300 msec
Re-parameterized Graph 2
0.002 msec
Extremely Fast Operation
Graph 2
Residual Graph 2
st-cut 2
51
Summary of the Algorithm

• Construct graph G for minimizing energy E
• For computing min-marginals do
• Obtain graph G energy function projection E
• (By adding constraining edges)
• Find the maximum flow in G using dynamic graph
  cut algorithm Kohli and Torr, ICCV 2005

52
Extension to Multiple Labels
Graph construction for multi-label random
variables (Ishikawa PAMI 2003)
Labels l1 . ln
Latent variables x1 . xn
MAP Labels
x1 l3
x2 l2
x3 l2
x4 l1
53
Extension to Multiple Labels
Graph construction for multi-label random
variables (Ishikawa PAMI 2003)
Labels l1 . ln
Latent variables x1 . xn
Graph for Projection
x4 l3
54
Outline

• Inference in Graphical Models
• Inference using Graph Cuts
• Computing Min-marginals using Graph Cuts
• Flow Potentials and Min-marginals
• Results

55
Min-Marginals and Flow Potentials

• Flow Potentials
• maximum amount of flow that can be passed from
  the node to a particular terminal.

56
Min-Marginals and Flow Potentials

• Flow Potentials
• maximum amount of flow that can be passed from
  the node to a particular terminal.

s
1
1
4
1
3
1
2
3
7
1
1
2
3
4
5
2
9
t
Graph
57
Min-Marginals and Flow Potentials

• Flow Potentials
• maximum amount of flow that can be passed from
  the node to a particular terminal.

s
1
1
4
1
3
1
2
3
7
1
1
2
3
4
5
2
9
t
Graph
Source (s) flow potential of node 4
58
Min-Marginals and Flow Potentials

• Flow Potentials
• maximum amount of flow that can be passed from
  the node to a particular terminal.

s
1
1
4
1
3
1
2
3
7
1
1
2
3
4
5
2
9
t
Graph
Source (s) flow potential of node 4
59
Min-Marginals and Flow Potentials

• Flow Potentials
• maximum amount of flow that can be passed from
  the node to a particular terminal.

s
s
1
1
4
1
1
1
3
1
2
3
1
2
3
7
1
1
1
2
1
3
4
5
4
5
2
9
t
Graph
t
Source (s) flow potential of node 4
60
Min-Marginals and Flow Potentials

• Flow Potentials
• maximum amount of flow that can be passed from
  the node to a particular terminal.

s
1
1
4
1
3
1
2
3
7
1
1
2
3
4
5
2
9
t
Graph
Sink (t) flow potential of node 4
61
Min-Marginals and Flow Potentials

• Flow Potentials
• maximum amount of flow that can be passed from
  the node to a particular terminal.

s
s
1
1
4
1
3
1
2
3
1
2
3
7
1
1
2
3
2
4
5
4
5
2
9
2
9
t
t
Graph
Sink (t) flow potential of node 4
62
Min-Marginals and Flow Potentials
Relationship between Min-marginal of binary
latent variables and flow potential.
MAP Solution energy
Flow potential in residual graph

Min-marginal
For details See theorem 1.
Implications Any algorithm for computing flow
potentials can be used to compute min-marginals.
63
Outline

• Inference in Graphical Models
• Inference using Graph Cuts
• Computing Min-marginals using Graph Cuts
• Flow Potentials and Min-marginals
• Results

64
Experimental Evaluation

• Typical Segmentation problem
• 640x480 image, 2 labels
• Computation Times
• Naïve Approach 51.2 hours!!!
• Our Algorithm (Dynamic Graph Cuts) 1.2 seconds

65
Experimental Evaluation
Computation Times (in seconds) for binary
variables
66
Min-Marginals in Image segmentation
Image Segmentation Energy Boykov and Jolly ICCV
2001, Blake et al. ECCV 2004
Unary likelihood
Contrast Term
Uniform Prior (Potts Model)
xi binary variable representing label (fg or
bg) of pixel i
67
Min-Marginals in Image segmentation
MAP Solution
Belief - Foreground
Image
MSR
Low smoothness
High smoothness
1
Colour Scale
0.5
Moderate smoothness
0
68
Min-Marginals in Image segmentation
Image Segmentation Energy Boykov and Jolly ICCV
2001, Blake et al. ECCV 2004
Unary likelihood
Contrast Term
Uniform Prior (Potts Model)
encourage smoothness
69
Min-Marginals in Image segmentation
Image Segmentation Energy Boykov and Jolly ICCV
2001, Blake et al. ECCV 2004
Unary likelihood
Contrast Term
Uniform Prior (Potts Model)
encourage smoothness
How smoothness effects solutions?
70
Min-Marginals in Image segmentation
Effect of increasing pair-wise terms of the
energy function
Image
71
Min-Marginals in Image segmentation
Effect of increasing pair-wise terms of the
energy function
MAP Segmentation Foreground Confidence Map
72
Concluding Remarks

• Efficient method for computing exact
  min-marginals for certain labelling problems
• Relationship between Flow-potentials and
  min-marginals
• Applications
• Computing M Most Probable Solutions
• Min-marginals for parameter learning
• Hierarchical Segmentation
• Future Work
• Efficient min-marginal computation for general
  problems

73
Thank You