Convex Relaxations

1
Convex Relaxations
Ben Recht

• May 2, 2004

2
Outline

• Lagrangian Duality
• Linear Programming and Combinatorics
• Non-convex quadratic programming
• Positivstellensatz and Polynomial Programming

3
Lagrangian Duality

• General Problem

4
Lagrangian Duality

• General Problem
• Lagrangian

5
Lagrangian Duality

• From Calculus search for
• Lagrangian

6
Lagrangian Duality

• Equivalent Optimization
• sup is infinite unless constraints satisfied
• Lagrangian

7
Lagrangian Duality

• Equivalent optimization
• Consider
• This is the dual problem

8
Visualization
f(x)
(g(x), h(x))
optimum
Search over half spaces containing the epigraph
of the function
(m,l,1)
9
Visualization
f(x)
(m,l,1)
10
Visualization
f(x)
(m,l,1)
11
Visualization
duality gap
12
Linear Programming Duality

• Lagrangian

13
Linear Programming Duality

• Minimize with respect to x

14
Linear Programming Duality

• Minimize with respect to x

15
Linear Programming Duality

• Minimize with respect to x

Either 0 or -1
16
Linear Programming Duality

• Minimize with respect to x

Either 0 or -1
Independent of x
17
Linear Programming Duality

• Form the Dual

18
Linear Programming Duality

• Primal
• Dual

19
Integer Programming

• Primal

20
Integer Programming

• Primal
• Dual

21
Integer Programming

• Primal
• Dual

the same dual dual dual is just the LP without
integer constraints
22
Integer Programming and Combinatorics

• Primal Dual Methods (shortest path, network
  flows)
• Total Unimodularity
• Guarantee integer solutions
• Total Dual Integrality and Min-max Relations
• Prove problem in NPÅcoNP
• Branch and Bound, Branch and Cut

23
Quadratic Programming

• Problem is convex only when the Ai are positive
  semidefinite.

24
Nonconvex Quadratically Constrained Quadratic
Programs

• Consider the general problem no assumption on
  definiteness.

25
NCQ2P

• Consider the general problem no assumption on
  definiteness.

26
NCQ2P

• Simplified presentation

27
NCQ2P

• Form the Lagrangian

28
NCQ2P

• Form the Lagrangian

29
NCQ2P

• Form the Lagrangian

Taking inf over x gives 0 or -1
30
NCQ2P

• Dual
• This is a semidefinite program

31
NCQ2P

• Dual
• Dual Dual

32
NCQ2P
SDP Relaxation
33
Dual Dual
34
Dual Dual
35
Dual Dual
36
Dual Dual
37
Dual Dual

• Recovered same relaxation
• This technique doesnt generalize (duality does!)

38
Bounding the gap

• For Aº0

39
Bounding the gap

• For Aº0

40
Bounding the gap

• For Aº0

41
Bounding the gap

• For Aº0

42
Bounding the gap

• For Aº0
• Take xsign(y), yN (0,Z). Then

43
The MAX-CUT Relaxation

• Invented by Goemans and Williamson
• Guarantees accuracy of 88 for the MAX-CUT
  problem. An algorithm with accuracy of 95 would
  prove PNP.
• Specific instance of the A0 matrix in the
  relaxation we discussed.
• Generalizes to MAX-2-SAT, MAX-SAT, graph
  coloring, MAX-DICUT, etc.

44
MAX-CUT

• Let G(V,E) be a graph and let wE! R be an
  arbitrary function. A cut in the graph is a
  partition of the vertices into two disjoint sets
  V1 and V2 such that V1 V2 V. Let F(V1)
  denote the set of edges which have exactly one
  node in V1.
• The weight of the cut is defined w(F) ?f2 F
  w(f)
• Problem find the partition which maximizes w.

45

• Graph G(V,E)
• Maximum-Cut

46

• Graph G(V,E)
• Maximum-Cut

47

• Graph G(V,E)
• Maximum-Cut

48

• Graph G(V,E)
• Maximum-Cut
• Easy for bipartite graphs. In general, NP-Hard

49
Petersen Graph

• Classic Counterexample
• Maximum-Cut 12

50
Petersen Graph

• Classic Counterexample
• Maximum-Cut 12

51
Algorithm 1 Erdos

• Expected Error is 50

52
Algorithm 1 Erdos

• Expected Error is 50
• Flip a coin for each node

53
Algorithm 1 Erdos

• Expected Error is 50
• Flip a coin for each node
• Probability edge is cut1/2

54
Algorithm 1 Erdos

• Expected Error is 50
• Flip a coin for each node
• Probability edge is cut1/2
• State of the art until 1993

25 error
55
Graph Laplacians

• The Laplacian is the V V matrix defined by
• where Adj(v) is the set of vertices adjacent to
  v.

56
MAX-CUT as an IQP

• We can write the MAX-CUT problem as
• Or, using the Laplacian, we can write this as
• and use the SDP techniques to find an approximate
  answer

57
Analysis

• FACT 1 For -1 x 1,
• 2/? arccos(x) ?(1-x)
• with ? 0.87856
• FACT 2 If y is drawn randomly from a Gaussian
  with zero mean and covariance Z
• Prsign(yi) ? sign(yj) 1/? arccos(Zij)

58
Analysis (continued)

• For any edge e2 E, let ?e denote the indicator
  function for e in the cut. Then the expected
  value of a cut is
• So we have Ecut ? (1/4)Tr(LZ) ? MAX CUT(G)

59
LPs and emptiness
A2x gt b2

• To prove there is no intersection, must find
  positive l1 and l2 such that for all x

A1x lt b1
l1(b1 - A1x) l2(A2x b2) lt 0
60
LPs and emptiness
A2x gt b2

• To prove there is no intersection, must find
  positive l1 and l2 such that for all x

A1x lt b1
l1(b1 - A1x) l2(A2x b2) lt 0
Main Idea Positive combinations of positive
terms cannot be negative!
61
Functions and Emptiness
g2(x) gt 0

• To prove there is no intersection, find positive
  functions l1(x) and l2(x) such that for all x

g1(x) lt 0
l1(x)(-g1(x)) l2(x)g2(x) lt 0
Main Idea Positive combinations of positive
terms cannot be negative!
62
Generalizing Lagrangian Duality

• Recall
• where

63
Positivstellensatz

• Let f1,,fn, g1,,gm be functions in a class F
• Then W if and only if there exist mI(x)0 and
  lj(x) in F such that

64
Positivstellensatz

• True for smooth functions
• True for polynomials
• Here, search for multipliers can be posed as a
  semidefinite program
• Hierarchies of approximations

65
Polynomials and Emptiness
g2(x) gt 0

• To prove there is no intersection, find positive
  functions l1(x) and l2(x) such that for all x
• When g1 and g2 are polynomials, we can assume l1
  and l2 are polynomials

g1(x) lt 0
l1(x)(-g1(x)) l2(x)g2(x) lt 0
g(x) a4 x4 a3 x3 a2 x2 a1 x a0
66
Cones and Ideals

• The cone generated by polynomials f1,,fn is the
  set of polynomials
• where the ab(x) are all positive
• N.B. If all of the fi are nonnegative everywhere,
  then any polynomial in the cone is nonnegative
  everywhere

67
Cones and Ideals

• The ideal generated by polynomials g1,,gn is the
  set of polynomials
• where the q(x) are arbitrary
• N.B. If all of the gi are zero everywhere, then
  any polynomial in the ideal is zero everywhere

68
Cones and Ideals

• The cone generated by f1,,fn is (ab(x)gt0)
• The ideal generated by g1,,gn is

69
Positivstellensatz

• Let f1,,fn, g1,,gm be polynomials and
• Then W if and only if there exists a F(x) in
  the cone of the fis and a G(x) in the ideal of
  the gis such that

70
Same Interpretation

• If ai gt 0 and bj 0 and
• with all li positive, then either one of the ai
  is negative or one of the bj is not zero.

Main Idea Positive combinations of positive
terms cannot be negative!
71
SOS/SDP Theorem

• One can search for certificates to prove that W
  using semidefinite programming (Parillo 2000).
• Let x be a vector of all monomials in L
  variables of degree less than or equal to N/2.
  Write
• Then if Qº0, p is a positive polynomial and
  hence W.

72
Two kinds of complexity

• Degree of polynomial multipliers
• Conditioning of numerical search
• The interplay between the two can be quite
  intricate

73
Looks hard is easy
f(x) - y lt 0

• Not convex, not separable by a line

y - f(x) c gt 0
74
Looks hard is easy
f(x) - y lt 0

• Not convex, not separable by a line
• Proof
• (-f(x) y) (y - f(x) - c) -c
• Fragility is the size of c

y - f(x) - c gt 0
75
More involved
(x-1)2 (y-1)2 lt 1
x3-8x-2y 0
76
More involved

• Explicitly find a and b such that

(x-1)2 (y-1)2 lt 1
(x-1)2 (y-1)2 1 (ax b)(x3 8x 2y) lt 0
x3-8x-2y 0
77
More involved

• Explicitly find a and b such that
• Using SDP we find a certificate
• b 0.31625 and a -0.1517

(x-1)2 (y-1)2 lt 1
(x-1)2 (y-1)2 1 (ax b)(x3 8x 2y) lt 0
x3-8x-2y 0
78
Pretty Darned Hard

• 2nd order multipliers
• Ill conditioning

xgt0, ygt0
x2yxy2-3xy1lt0
79
Looks easy is hard

• Does the square leave the ellipse?

Square x2 lt 1, y2 lt 1
Ellipse x2y2 - xy x y gt 4.5
80
Looks easy is hard

• Does the square leave the ellipse?
• Need quadratic multipliers for proof
• 3 corners equidistant from the ellipse boundary

Square x2 lt 1, y2 lt 1
Ellipse x2y2 - xy x y gt 4.5
81
Is there a crash?

• Given a line in the plane
• a0 a1x a2y 0
• Question does it hit a corner of the square?
• x21, y21

82
Is there a crash?

• Given a line in the plane
• a0 a1x a2y 0
• Question does it hit a corner of the
  square? x21, y21
• If it hits a corner, then the problem is fragile

83
Is there a crash?

• Given a line in the plane
• a0 a1x a2y 0
• Question does it hit a corner of the
  square? x21, y21
• Inside the square, quadratic multipliers are
  needed.
• This is the first nontrivial duality gap for
  NCQ2P

84
Software

• SOS tools
• Yalmip
• SeDuMi
• Write your own (see Boyd and Vandenberghe)

85
Problem 1 Spin Glasses

• Try to solve problem 13.1 in NMM using a
  semidefinite relaxation. Estimate the duality
  gap using your primal and dual solutions.
  Compare your answer to the one returned by
  simulated annealing.
• Now assume that the spins are coupled together on
  an 10x10 square lattice. Estimate the lowest
  energy state using simulated annealing and the
  semidefinite relaxation and compare the results.

86
Problem 2 MAX 2 SAT

• A clause is the OR of two Boolean variables or
  their negation. (e.g. xÇy)
• Given a Boolean expression which is the AND of a
  bunch of clauses with two variables, the MAX 2
  SAT problem is to determine how the maximum
  number of clauses that can be simultaneously
  satisfied.

87
Problem 2 Continued

• Let xi be true if xi zi where zi 1
• Let v(x) (1zx)/2 and v(x)(1-zx)/2
• Show that v(x)1 if x is TRUE and 0 if x is FALSE
• Let
• Given a clause, show that C(x,y) is TRUE if and
  only if v(x,y)1. Show that C(x,y) is false if
  and only if C(x,y)0.
• Combine these facts to write MAX 2 SAT as an
  integer quadratic program.

88
Problem 2 Continued

• Use the analysis from above to show that the
  semidefinite relaxation of the quadratic program
  returns an ?-approximation of the MAX 2 SAT
  problem.