An Analysis of Convex Relaxations (PART I) Minimizing Higher Order Energy Functions (PART 2)

1
An Analysis of Convex Relaxations (PART
I)Minimizing Higher Order Energy Functions
(PART 2)

• Philip Torr

Work in collaboration with Pushmeet Kohli,
Srikumar Ramalingam M. Pawan Kumar, Lubor
Ladický, Vladimir Kolmogorov
2
An Analysis of Convex Relaxations
for MAP Estimation

• M. Pawan Kumar
• Vladimir Kolmogorov
• Philip Torr

3
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Random Variables V V1, ... ,V4
Label Set L 0, 1
Labelling m 1, 0, 0, 1
4
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2 1 3 1 3 13
Minimum Cost Labelling? NP-hard problem
Which approximate algorithm is the best?
5
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Objectives

• Compare existing convex relaxations LP, QP and
  SOCP
• Develop new relaxations based on the comparison

6
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

7
Integer Programming Formulation
2
4
0
Unary Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
2 4
2
Unary Cost Vector u 5
8
Integer Programming Formulation
2
4
0
Unary Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
2 4 T
2
Unary Cost Vector u 5
Label vector x -1
1
1 -1 T
Recall that the aim is to find the optimal x
9
Integer Programming Formulation
2
4
0
Unary Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
2 4 T
2
Unary Cost Vector u 5
Label vector x -1
1
1 -1 T
1
Sum of Unary Costs
?i ui (1 xi)
2
10
Integer Programming Formulation
2
4
0
Pairwise Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
0
3
0
11
Integer Programming Formulation
2
4
0
Pairwise Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
Sum of Pairwise Costs
1
?ij Pij (1 xi)(1xj)
0
3
0
4
12
Integer Programming Formulation
2
4
0
Pairwise Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
Sum of Pairwise Costs
1
?ij Pij (1 xi xj xixj)
0
3
0
4
X x xT
Xij xi xj
13
Integer Programming Formulation
Constraints

• Integer Constraints

xi ?-1,1
X x xT
14
Integer Programming Formulation
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
Convex
xi ?-1,1
X x xT
15
Outline

• Integer Programming Formulation
• Existing Relaxations
• Linear Programming (LP-S)
• Semidefinite Programming (SDP-L)
• Second Order Cone Programming (SOCP-MS)
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

16
LP-S
Schlesinger, 1976
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
Relax Non-Convex Constraint
xi ?-1,1
X x xT
17
LP-S
Schlesinger, 1976
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Relax Non-Convex Constraint
X x xT
18
LP-S
Schlesinger, 1976
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
LP-S
Xij ?-1,1
1 xi xj Xij 0
19
Outline

• Integer Programming Formulation
• Existing Relaxations
• Linear Programming (LP-S)
• Semidefinite Programming (SDP-L)
• Second Order Cone Programming (SOCP-MS)
• Comparison
• Generalization of Results
• Two New SOCP Relaxations
• Experiments

20
SDP-L
Lasserre, 2000
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
Relax Non-Convex Constraint
xi ?-1,1
X x xT
21
SDP-L
Lasserre, 2000
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Relax Non-Convex Constraint
X x xT
22
SDP-L
Lasserre, 2000
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Xii 1
Inefficient
Accurate
23
Outline

• Integer Programming Formulation
• Existing Relaxations
• Linear Programming (LP-S)
• Semidefinite Programming (SDP-L)
• Second Order Cone Programming (SOCP-MS)
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

24
SOCP Relaxation
Derive SOCP relaxation from the SDP relaxation
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Xii 1
Further Relaxation
25
2-D Example
X11
X12
X
X21
X22
26
2-D Example
(X - xxT)
1 - x12
X12-x1x2
X12-x1x2
1 - x22
(x1 x2)2 ? 2 2X12
SOC of the form v 2 ? st
27
2-D Example
(X - xxT)
1 - x12
X12-x1x2
X12-x1x2
1 - x22
(x1 - x2)2 ? 2 - 2X12
SOC of the form v 2 ? st
28
SOCP Relaxation
C1 . ? 0
Kim and Kojima, 2000
UTx 2 ? X . C1
(X - xxT)
SOC of the form v 2 ? st
Continue for C2, C3, , Cn
29
SOCP Relaxation
How many constraints for SOCP SDP ?
Infinite.
Specify constraints similar to the 2-D example
Xij
xi
xj
(xi xj)2 ? 2 2Xij
(xi xj)2 ? 2 - 2Xij
30
SOCP-MS
Muramatsu and Suzuki, 2003
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Xii 1
31
SOCP-MS
Muramatsu and Suzuki, 2003
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
(xi xj)2 ? 2 2Xij
(xi - xj)2 ? 2 - 2Xij
32
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

33
Dominating Relaxation

A
B
For all MAP Estimation problem (u, P)
A dominates B
Dominating relaxations are better
34
Equivalent Relaxations
Strictly Dominating Relaxation
35
SOCP-MS
min ?ij Pij Xij
(xi xj)2 ? 2 2Xij
(xi - xj)2 ? 2 - 2Xij

• Pij 0

• Pij lt 0

SOCP-MS is a QP
Same as QP by Ravikumar and Lafferty, 2006
SOCP-MS QP-RL
36
LP-S vs. SOCP-MS
Differ in the way they relax X xxT
37
LP-S vs. SOCP-MS

• LP-S strictly dominates SOCP-MS

• LP-S strictly dominates QP-RL

• Where have we gone wrong?

• A Quick Recap !

38
Recap of SOCP-MS
Can we use a different subgraph ??
Can we use different C matrices ??
39
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• SOCP Relaxations on Trees
• SOCP Relaxations on Cycles
• Two New SOCP Relaxations

40
SOCP Relaxations on Trees
Choose any arbitrary tree
41
SOCP Relaxations on Trees
Choose any arbitrary C 0
Repeat over trees to get relaxation SOCP-T
LP-S strictly dominates SOCP-T
LP-S strictly dominates QP-T
42
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• SOCP Relaxations on Trees
• SOCP Relaxations on Cycles
• Two New SOCP Relaxations

43
SOCP Relaxations on Cycles
Choose an arbitrary even cycle
Pij 0
OR
Pij 0
44
SOCP Relaxations on Cycles
Choose any arbitrary C 0
Repeat over even cycles to get relaxation SOCP-E
LP-S strictly dominates SOCP-E
LP-S strictly dominates QP-E
45
SOCP Relaxations on Cycles

• True for odd cycles with Pij 0

• True for odd cycles with Pij 0 for only one
  edge

• True for odd cycles with Pij 0 for only one
  edge

• True for all combinations of above cases

46
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations
• The SOCP-C Relaxation
• The SOCP-Q Relaxation

47
The SOCP-C Relaxation
Include all LP-S constraints
True SOCP
a
b
Cycle of size 4
c
d
Define SOCP Constraint using appropriate C
SOCP-C strictly dominates LP-S
Which SOCP is strictly dominated by cycle
inequalities?
Open Question !!!
48
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations
• The SOCP-C Relaxation
• The SOCP-Q Relaxation

49
The SOCP-Q Relaxation
Include all cycle inequalities
True SOCP
a
b
Clique of size n
c
d
Define an SOCP Constraint using C 1
SOCP-Q strictly dominates LP-S
SOCP-Q strictly dominates cycle inequalities
50
Conclusions

• Large class of SOCP/QP dominated by LP-S
• New SOCP relaxations dominate LP-S
• Preliminary experiments conform with analysis

51
Future Work

• Comparison with cycle inequalities
• Determine best SOC constraints
• Develop efficient algorithms for new relaxations

52
Part 2
53
4-Neighbourhood MRF
Test SOCP-C
50 binary MRFs of size 30x30
u N (0,1)
P N (0,s2)
54
4-Neighbourhood MRF
s 2.5
55
8-Neighbourhood MRF
Test SOCP-Q
50 binary MRFs of size 30x30
u N (0,1)
P N (0,s2)
56
8-Neighbourhood MRF
s 1.125
57
Equivalent Relaxations

A
B
For all MAP Estimation problem (u, P)
A dominates B
B dominates A
58
Strictly Dominating Relaxation
gt
A
B
For at least one MAP Estimation problem (u, P)
A dominates B
B does not dominate A