An Analysis of Convex Relaxations

1
An Analysis of Convex Relaxations
for MAP Estimation

• M. Pawan Kumar
• Vladimir Kolmogorov
• Philip Torr

2
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Random Variables V V1, ... ,V4
Label Set L 0, 1
Labelling m 1, 0, 0, 1
3
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2
4
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1
5
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2
6
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2 1
7
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2 1 3
8
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2 1 3 1
9
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2 1 3 1 3
10
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2 1 3 1 3 13
Pr(m) ? exp(-Cost(m))
Minimum Cost Labelling MAP estimate
11
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Cost(m) 2 1 2 1 3 1 3 13
NP-hard problem
Which approximate algorithm is the best?
12
Aim

• To analyze convex relaxations for MAP estimation

6
3
0
1
2
4
0
Label 1
1
2
4
1
1
3
Label 0
1
0
5
3
7
0
2
V2
V3
V4
V1
Objectives

• Compare existing convex relaxations LP, QP and
  SOCP
• Develop new relaxations based on the comparison

13
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

14
Integer Programming Formulation
2
4
0
Unary Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
2 4
2
Unary Cost Vector u 5
15
Integer Programming Formulation
2
4
0
Unary Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
2 4 T
2
Unary Cost Vector u 5
Label vector x -1
1
1 -1 T
Recall that the aim is to find the optimal x
16
Integer Programming Formulation
2
4
0
Unary Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
2 4 T
2
Unary Cost Vector u 5
Label vector x -1
1
1 -1 T
1
Sum of Unary Costs
?i ui (1 xi)
2
17
Integer Programming Formulation
2
4
0
Pairwise Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
0
3
0
18
Integer Programming Formulation
2
4
0
Pairwise Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
Sum of Pairwise Costs
1
?ij Pij (1 xi)(1xj)
0
3
0
4
19
Integer Programming Formulation
2
4
0
Pairwise Cost
Label 1
1
3
Label 0
5
0
2
V2
V1
Labelling m 1 , 0
Sum of Pairwise Costs
1
?ij Pij (1 xi xj xixj)
0
3
0
4
X x xT
Xij xi xj
20
Integer Programming Formulation
Constraints

• Integer Constraints

xi ?-1,1
X x xT
21
Integer Programming Formulation
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
Convex
xi ?-1,1
X x xT
22
Outline

• Integer Programming Formulation
• Existing Relaxations
• Linear Programming (LP-S)
• Semidefinite Programming (SDP-L)
• Second Order Cone Programming (SOCP-MS)
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

23
LP-S
Schlesinger, 1976
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
Relax Non-Convex Constraint
xi ?-1,1
X x xT
24
LP-S
Schlesinger, 1976
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Relax Non-Convex Constraint
X x xT
25
LP-S
Schlesinger, 1976
X x xT
Xij ?-1,1
1 xi xj Xij 0
26
LP-S
Schlesinger, 1976
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Relax Non-Convex Constraint
X x xT
27
LP-S
Schlesinger, 1976
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1,
Xij ?-1,1
LP-S
1 xi xj Xij 0
28
Outline

• Integer Programming Formulation
• Existing Relaxations
• Linear Programming (LP-S)
• Semidefinite Programming (SDP-L)
• Second Order Cone Programming (SOCP-MS)
• Comparison
• Generalization of Results
• Two New SOCP Relaxations
• Experiments

29
SDP-L
Lasserre, 2000
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
Relax Non-Convex Constraint
xi ?-1,1
X x xT
30
SDP-L
Lasserre, 2000
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Relax Non-Convex Constraint
X x xT
31
SDP-L
.
.
.
1
1
x1
x2
xn
x1
x2
.
.
.
xn
Xii 1
Positive Semidefinite
Rank 1
32
SDP-L
.
.
.
1
1
x1
x2
xn
x1
x2
.
.
.
xn
Xii 1
Positive Semidefinite
33
Schurs Complement
A
B
BT
C
34
SDP-L
1
xT
x
X
35
SDP-L
Lasserre, 2000
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Relax Non-Convex Constraint
X x xT
36
SDP-L
Lasserre, 2000
Retain Convex Part
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
SDP-L
Xii 1
Inefficient
Accurate
37
Outline

• Integer Programming Formulation
• Existing Relaxations
• Linear Programming (LP-S)
• Semidefinite Programming (SDP-L)
• Second Order Cone Programming (SOCP-MS)
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

38
SOCP Relaxation
Derive SOCP relaxation from the SDP relaxation
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Xii 1
Further Relaxation
39
1-D Example
For two semidefinite matrices, Frobenius inner
product is non-negative
X - x2 0
x2 ? X
1
SOC of the form v 2 ? st
40
2-D Example
X11
X12
X
X21
X22
41
2-D Example
(X - xxT)
1 - x12
X12-x1x2
X12-x1x2
1 - x22
x12 ? 1
-1 ? x1 ? 1
42
2-D Example
C2. ? 0
C2 0
(X - xxT)
1 - x12
X12-x1x2
X12-x1x2
1 - x22
x22 ? 1
-1 ? x2 ? 1
43
2-D Example
C3. ? 0
C3 0
(X - xxT)
1 - x12
X12-x1x2
X12-x1x2
1 - x22
(x1 x2)2 ? 2 2X12
SOC of the form v 2 ? st
44
2-D Example
C4. ? 0
C4 0
(X - xxT)
1 - x12
X12-x1x2
X12-x1x2
1 - x22
(x1 - x2)2 ? 2 - 2X12
SOC of the form v 2 ? st
45
SOCP Relaxation
C1 . ? 0
Kim and Kojima, 2000
UTx 2 ? X . C1
(X - xxT)
SOC of the form v 2 ? st
Continue for C2, C3, , Cn
46
SOCP Relaxation
How many constraints for SOCP SDP ?
Infinite. For all C 0
Specify constraints similar to the 2-D example
Xij
xi
xj
(xi xj)2 ? 2 2Xij
(xi xj)2 ? 2 - 2Xij
47
SOCP-MS
Muramatsu and Suzuki, 2003
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
Xii 1
48
SOCP-MS
Muramatsu and Suzuki, 2003
1
1
? Pij (1 xi xj Xij)
x argmin
? ui (1 xi)

4
2
xi ?-1,1
(xi xj)2 ? 2 2Xij
(xi - xj)2 ? 2 - 2Xij
49
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations

50
Dominating Relaxation

A
B
For all MAP Estimation problem (u, P)
A dominates B
Dominating relaxations are better
51
Equivalent Relaxations

A
B
For all MAP Estimation problem (u, P)
A dominates B
B dominates A
52
Strictly Dominating Relaxation
gt
A
B
For at least one MAP Estimation problem (u, P)
A dominates B
B does not dominate A
53
SOCP-MS
Muramatsu and Suzuki, 2003
(xi xj)2 ? 2 2Xij
(xi - xj)2 ? 2 - 2Xij

• Pij 0

• Pij lt 0

SOCP-MS is a QP
Same as QP by Ravikumar and Lafferty, 2005
SOCP-MS QP-RL
54
LP-S vs. SOCP-MS
Differ in the way they relax X xxT
55
LP-S vs. SOCP-MS

• LP-S strictly dominates SOCP-MS

• LP-S strictly dominates QP-RL

• Where have we gone wrong?

• A Quick Recap !

56
Recap of SOCP-MS
Xij
xi
xj
(xi xj)2 ? 2 2Xij
57
Recap of SOCP-MS
Xij
xi
xj
(xi - xj)2 ? 2 - 2Xij
Can we use different C matrices ??
Can we use a different subgraph ??
58
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• SOCP Relaxations on Trees
• SOCP Relaxations on Cycles
• Two New SOCP Relaxations

59
SOCP Relaxations on Trees
Choose any arbitrary tree
60
SOCP Relaxations on Trees
Choose any arbitrary C 0
Repeat over trees to get relaxation SOCP-T
LP-S strictly dominates SOCP-T
LP-S strictly dominates QP-T
61
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• SOCP Relaxations on Trees
• SOCP Relaxations on Cycles
• Two New SOCP Relaxations

62
SOCP Relaxations on Cycles
Choose an arbitrary even cycle
Pij 0
OR
Pij 0
63
SOCP Relaxations on Cycles
Choose any arbitrary C 0
Repeat over even cycles to get relaxation SOCP-E
LP-S strictly dominates SOCP-E
LP-S strictly dominates QP-E
64
SOCP Relaxations on Cycles

• True for odd cycles with Pij 0

• True for odd cycles with Pij 0 for only one
  edge

• True for odd cycles with Pij 0 for only one
  edge

• True for all combinations of above cases

65
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations
• The SOCP-C Relaxation
• The SOCP-Q Relaxation

66
The SOCP-C Relaxation
Include all LP-S constraints
True SOCP
Non-submodular
Submodular
Submodular
67
The SOCP-C Relaxation
Include all LP-S constraints
True SOCP
0
0
1
0
0
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
1
0
c
a
a
b
b
c
Frustrated Cycle
68
The SOCP-C Relaxation
Include all LP-S constraints
True SOCP
0
0
1
0
0
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
1
0
c
a
a
b
b
c
LP-S Solution
Objective Function 0
69
The SOCP-C Relaxation
Include all LP-S constraints
True SOCP
0
0
1
0
0
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
1
0
c
a
a
b
b
c
LP-S Solution
1
1
-1
0
0
0
0
0
0
-1
-1
-1
-1
1
1
0
0
0
0
0
0
1
-1
1
c
a
a
b
b
c
Define an SOC Constraint using C 1
70
The SOCP-C Relaxation
Include all LP-S constraints
True SOCP
0
0
1
0
0
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
1
0
c
a
a
b
b
c
LP-S Solution
1
1
-1
0
0
0
0
0
0
-1
-1
-1
-1
1
1
0
0
0
0
0
0
1
-1
1
c
a
a
b
b
c
(xi xj xk)2 ? 3 2 (Xij Xjk Xki)
71
The SOCP-C Relaxation
Include all LP-S constraints
True SOCP
0
0
1
0
0
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
1
0
c
a
a
b
b
c
SOCP-C Solution
Objective Function 0.75
SOCP-C strictly dominates LP-S
72
Outline

• Integer Programming Formulation
• Existing Relaxations
• Comparison
• Generalization of Results
• Two New SOCP Relaxations
• The SOCP-C Relaxation
• The SOCP-Q Relaxation

73
The SOCP-Q Relaxation
Include all cycle inequalities
True SOCP
a
b
Clique of size n
c
d
Define an SOCP Constraint using C 1
(S xi)2 n (S Xij)
SOCP-Q strictly dominates LP-S
SOCP-Q strictly dominates SOCP-C
74
4-Neighbourhood MRF
Test SOCP-C
50 binary MRFs of size 30x30
u N (0,1)
P N (0,s2)
75
4-Neighbourhood MRF
s 2.5
76
8-Neighbourhood MRF
Test SOCP-Q
50 binary MRFs of size 30x30
u N (0,1)
P N (0,s2)
77
8-Neighbourhood MRF
s 1.125
78
Conclusions

• Large class of SOCP/QP dominated by LP-S
• New SOCP relaxations dominate LP-S
• More experimental results in poster

79
Future Work

• Comparison with cycle inequalities
• Determine best SOC constraints
• Develop efficient algorithms for new relaxations

80
Questions ??