Survey Propagation: an Algorithm for Satisfiability

1
Survey Propagation an Algorithm for
Satisfiability
A. Braunstein, M. Mezard, R. Zecchina
Hannaneh Hajishirzi hajishir_at_uiuc.edu
2
Content

• SAT Problem
• Warning Passing
• Survey Propagation
• Belief Propagation

3
SAT Problem

• CNF
• Clause a (zi1 ? zi2 ? ? zik)
• And Jair 1 if zir xir
• Jair -1 if zir xir

4
Factor Graph

• Bipartite Graph
• Variables ? Variable nodes
• Clauses ? Function nodes
• Edge(a - i) variable i appears in clause a

5
Variable Node
Function Node
6
Some Notions

• V(i) neighbors of variable i
• V(i) clauses that i appears un-negated
• V- (i) clauses that i appears negated
• V(i)\b set V(i) without a node b

7
Warning Passing
j
hj?a
ua?i
a (i ? j ? k ? l)
i
k
a
hk?a
hl?a
l
ua?i can be a warning
8
Formulae

• hj?a Cavity Field
• ua?i Cavity Bias

1
9
WP algorithm

• T 0 Randomly initialize cavity bias(ua?i )
• For t 1 to t tmax
• 2.1 Update all ua?i(t) sequentially. (update
  method)
• 2.2 If ua?i(t) ua?i(t-1) for all edges,
• ua?i ua?i(t). Goto 3.
• If t tmax Return UN-CONVERGED.
• else output cavity biases ua?i.

10
Update Method

• Update(ua?i )
• For every j ? V(a) \ i, compute hj?a
• Using hj?a, compute ua?i

11
Convergence

• Local Field Hi
• Contradiction Number ci

12
b
a
1
1
0
0
1
2
0
0
c
0
3
0
d
13
b
a
1
1
1
2
1
0
0
c
0
0
3
0
d
14
b
a
1
1
1
2
-1
0
0
c
0
0
3
0
d
15
b
a
1
1
1
2
-1
1
0
0
c
0
1
1
3
0
d
16
WP on Tree

• If Factor graph Tree
• WP converges
• ? i ci gt 0 ? UNSAT
• Otherwise SAT

17
1. Remove edge a-i2. Level of a 0 ?u 13.
Level of a 1 ?u 04. Level of
a r can be determined from us at level
r-2
18
Solving SAT with WP
1. Run WP. 2. Check if there are
conflicts. 3. Fix all constrained variables
Clean the graph. 4. Choose randomly an un-biased
variable,Fix it to one value (0, 1), Clean the
graph. 5. Run again WP on the new instance.
19
Example
x1 1, x2 0 x3 1, x4 1 x7 ? 0, 1
20
Example
x1 1, x2 0 x3 1, x4 1 x7 ? 0, 1
x5 1
x6 1
1
1
3
21
Clustering

• Random SAT with M ? N
• ? lt ?d 3.921
• Set of all satisfying assignments is connected (1
  cluster)

Local Search
22
Clustering

• ?d lt ? lt ?c 4.267 (hard-SAT region)
• - Set of all satisfying assignments becomes
  divided into subsets (clusters)
• - Proliferation of meta-stable clusters

Difficult for Local Search Methods
23
Clustering

• ? ?c
• Number of Clusters 0
• ? gt ?c
• Almost unsatisfiable
• ? Survey Propagation tries to solve SAT problem
  in hard-SAT region.

24
Definition of Surveys

• Survey ua?i

ua?i in cluster l
Ncl Number of clusters
?(x, y) 1 if x y,otherwise 0
25
Surveys

• u ? 0, 1
• Binomial Distribution

For any set of cavity biases Q(u, v, w,)
Joint Probability
26
1
27
Jaj 1
2
Jaj -1 V(i) V-(i) exchange
28
(No Transcript)
29
Factorized Form
Normalization Factor
Constraint
All values for all ub?j
30
Compute C

• Class u ? ub?j 1, b ? Vua(j)
• 2. Class s ? ub?j 1, b ? Vsa(j)

31
Compute C
3. Class 0 ?ub?j 0, b ? V (j)
32
Closed Formula
33
SP Algorithm

• 1. T 0 Randomly initialize cavity bias(?a?i )
• 2. For t 1 to t tmax
• 2.1 Update all ?a?i(t) sequentially. (update
  method)
• 2.2 If ?a?i(t) - ?a?i(t-1) lt ? for all edges,
• ?a?i ?a?i(t). Goto 3.
• 3. If t tmax Return UN-CONVERGED.
• else output cavity biases ?a?i.

34
Discussion on SP

• If factor-graph tree, the same as WP
• No proof of convergence
• Experimental converges where WP doesnt
• Useful for large N

35
Find SAT Assignment

• Simple Algorithm (at most 2N SP calls)
• Fix one variable
• Run SP for size N-1
• If sub-problem is SAT keep the assignment
• Otherwise change the value
• Drawbacks
• Imprecision of determining if problem is
  Satisfiable or not
• Not using information from surveys

36
Properties of Variables
Fraction of clusters that xj is positive
37
Properties of Variables
38
Categories of Variables

• Under-constrained
• Fix it Affects internal structure of clusters
• Biased
• Fix it Few clusters are eliminated
• Balanced
• Fix it Enormous effect

or
and
39
Search Algorithm

• 1. Run SP
• 2. Evaluate all Wi, Wi-, Wi0.
• 3. Simplify Search
• 3.1 If (exists ? ltgt 0) , fix with largest Wi
  - Wi-
• 3.2 If (all ? 0) , Run WalkSAT
• 3.3 SP doesnt converge, Probably UNSAT
• 4. If Done, output SAT
• If no contradiction ? goto 1

40
BP in SAT Problem

• ?i?a(xi)
• variable takes xi, in the absence of clause a
• ?a?i(xi)
• Clause a becomes satisfied, given value for xi

For all values of variables xj
1 if Xxi satisfies aOtherwise 0
41
BP in SAT Problem

• ?i?a(xi) Probability that variable takes xi and
  violates clause a
• ?a?i Probability that all variables in a
  except i, violate a

i appears un-negated
42
BP in SAT Problem
Probability that xj 1
43
Example
? 1 1, ? 2 0 ? 3 1, ? 4 1 ? 7 3/4
?i?a
?a?i
? 5 1/2
? 6 3/4
3
44
Null Message

• Null message onto a variable means
• Receives no warning
• Under-constrained
• 3 states for a variable
• 0, 1, (joker state)

45
New BP

• No warning from Vua(i) Vsa(i)
• No warning from Vsa(i), but at least 1 from
  Vua(i)

46
New BP

• No warning from Vua(i), but at least 1 from
  Vsa(i)
• At least 1 warning from Vsa(i), and 1 from Vua(i)

New Formula
47
Loopy Factor Graph
2 possible solutions for WP,2 generalized
assignments (1, 1, ), (0, 0, 1)
In SP?a?1 ?b?2 x ?a?2 ?b?1 y?c?1
?c?2 0?c?3 (1-x)2y2 / (1-xy)2
48
When is SP useful?

• Maybe in small cases WP is better
• Useful in difficult cases for WP
• When messages are sent according to different
  clusters
• When graph is not well connected

SP performs better
49
Conclusion

• Message Passing Methods for SAT
• Advantage of SP
• More theoretical empirical work needed