Design and Analysis of Computer Algorithm Lecture 10

1
Design and Analysis of Computer AlgorithmLecture
10

• Pradondet Nilagupta
• Department of Computer Engineering

2
Acknowledgement

• This lecture note has been summarized from
  lecture note on Data Structure and Algorithm,
  Design and Analysis of Computer Algorithm all
  over the world. I cant remember where those
  slide come from. However, Id like to thank all
  professors who create such a good work on those
  lecture notes. Without those lectures, this slide
  cant be finished.

3
The theory of NP-completeness

• Tractable and intractable problems
• NP-complete problems

4
Classifying problems

• Here Classify problems as tractable or
  intractable.
• Problem is tractable if there exists a polynomial
  bound algorithm that solves it.
• An algorithm is polynomial bound if its worst
  case growth rate can be bound by a polynomial
  p(n) in the size n of the problem

5
What constitutes reasonable time?

• Standard working definition polynomial time
• On an input of size n the worst-case running time
  is O(nk) for some constant k
• Polynomial time O(n2), O(n3), O(1), O(n lg n)
• Not in polynomial time O(2n), O(nn), O(n!)

6
Polynomial-Time Algorithms

• Are some problems solvable in polynomial time?
• Of course every algorithm weve studied provides
  polynomial-time solution to some problem
• We define P to be the class of problems solvable
  in polynomial time
• Are all problems solvable in polynomial time?
• No Turings Halting Problem is not solvable by
  any computer, no matter how much time is given
• Such problems are clearly intractable, not in P

7
Intractable problems

• Problem is intractable if it is not tractable.
• Any algorithm that solves it is not polynomial
  bound.
• It has a worst case growth rate f(n) which cannot
  be bound by a polynomial p(n) in the size n of
  the problem.
• For intractable problems the bounds are

8
Why is this classification useful?

• If problem is intractable, no point in trying to
  find an efficient algorithm
• Any algorithm too slow for large inputs.
• To solve use approximations, heuristics, etc.
• Sometimes we need to solve only a restricted
  version of the problem.
• If restricted problem tractable design an
  algorithm for restricted problem

9
Intractable problems

• Turing showed some problems so hard that no
  algorithm can solve them (undecidable)
• Other researchers showed some decidable problems
  from automata, mathematical logic, etc. are
  intractable
• These problems are so hard that they cannot be
  solved in polynomial time by a nondeterministic
  computer

10
Hard practical problems

• Many practical problems for which no one has yet
  found a polynomial bound algorithm.
• Examples traveling salesperson, knapsack, graph
  coloring, etc.
• Most design automation problems such as testing
  and routing.
• Many networks, database and graph problems.

11
How are they solved?

• A variety of algorithms based on backtracking,
  branch and bound, etc.
• None can be shown to be polynomial bound
• Problems can be solved by a polynomial bound
  verification algorithm

12
The theory of NP completeness

• The theory of NP-completeness enables showing
  that these problems are at least as hard as
  NP-complete problems
• Practical implication of knowing problem is
  NP-complete is that it is probably intractable (
  whether it is or not has not been proved yet)
• So any algorithm that solves it will probably be
  very slow for large inputs

13
We need to define

• Decision problems
• The class P
• Nondeterministic algorithms
• The class NP
• The concept of polynomial transformations
• The class of NP-complete problems

14
The theory of NP-Completeness

• Decision problems
• Converting optimization problems into decision
  problems
• The relationship between an optimization problem
  and its decision version
• The class P
• Verification algorithms
• The class NP

15
Decision Problems

• A decision problem answers yes or no for a given
  input
• Examples
• Is there a path from s to t of length at most k?
• Does graph G contain a Hamiltonian cycle?

16
A decision problem HAMILTONIAN CYCLE

• A hamiltonian cycle of a graph G is a cycle that
  includes each vertex of the graph exactly once.
• Problem Given a graph G, does G have a
  hamiltonian cycle?

17
Converting to decision problems

• Optimization problems can be converted to
  decision problems (typically) by adding a bound
  B on the value to optimize, and asking the
  question
• Is there a solution whose value is at most B?
  (for a minimization problem)
• Is there a solution whose value is at least B?
  (for a maximization problem)

18
An optimization problem traveling salesman (TS)

• Given
• A finite set Cc1,...,cm of cities,
• A distance function d(ci, cj) of nonnegative
  numbers.
• Find the length of the minimum distance tour
  which includes every city exactly once

19
A decision problem traveling salesman

• Given a finite set Cc1,...,cm of cities, a
  distance function d(ci, cj) of nonnegative
  numbers and a bound B
• Is there a tour of all the cities (in which each
  city is visited exactly once) with total length
  at most B?

20
The relation between

• If we have a solution to the optimization problem
  we can compare the solution to the bound and
  answer yes or no.
• Therefore if the optimization problem is
  tractable so is the decision problem
• If the decision problem is hard the
  optimization problems are also hard

21
Class of Problems P and NP

• Definition The class P
• P is the class of decision problems that are
  polynomially bounded.
• there exist a deterministic algorithm
• Definition The class NP
• NP is the class of decision problems for which
  there is a polynomially bounded non-deterministic
  algorithm.
• The name NP comes from Non-deterministic
  Polynomially bounded.
• there exist a non-deterministic algorithm
• Theorem P ? NP

22
The goal of verification algorithms

• The goal of a verification algorithm is to verify
  a yes answer to a decision problems input.
• The inputs to the verification algorithm are the
  original input and a certificate (possible
  solution)

23
Example

• A verification algorithm for TS, verifies that a
  given TS tour has length at most B

24
A verification algorithm for PATH

• Given the problem PATH (does there exist a path
  of length k or less in a graph G between vertices
  u and v?), and a certificate p.
• It is simple to verify that the length of p is at
  most k (we have to also check that p is indeed a
  path from u to v).

25
Verification Algorithms

• Other problems like HAMILTONIAN CYCLE are not
  known to have polynomial bound algorithms but
  given a hamiltonian cycle, it is easy to verify
  that the cycle is indeed hamiltonian in
  polynomial time.
• A verification algorithm, takes a problem
  instance x and verifies it, if there exists a
  certificate y such that the answer for x with
  certificate y is yes

26
Polynomial bound verification algorithms

• Given a decision problem d.
• A verification algorithm for d is polynomial
  bound if given an input x to d,
• there exists a certificate y, such that
  yO(xc) where c is a constant,
• and a polynomial bound algorithm A(x, y) that
  verifies an answer yes for d with input x

27
The class NP

• NP is the class of decision problems for which
  there is a polynomial bounded verification
  algorithm
• It can be shown that
• all decision problems in P, and
• decision problems such as traveling salesman and
  knapsack are also in NP

28
A non-deterministic algorithm

• The non-deterministic guessing phase.
• Some completely arbitrary string s, proposed
  solution
• each time the algorithm is run the string may
  differ
• The deterministic verifying phase.
• a deterministic algorithm takes the input of the
  problem and the proposed solution s, and
• return value true or false
• The output step.
• If the verifying phase returned true, the
  algorithm outputs yes. Otherwise, there is no
  output.

29
P and NP

• Summary so far
• P problems that can be solved in polynomial
  time
• NP problems for which a solution can be
  verified in polynomial time
• Unknown whether P NP (most suspect not)
• Hamiltonian-cycle problem is in NP
• Cannot solve in polynomial time
• Easy to verify solution in polynomial time (How?)

30
A Problem Which is in NP

• Can solve variant of TSP which is in form of a
  decision problem
• TSP Given a complete directed graph G with cost
  for each edge, and an integer k. Return YES, if
  there is a tour with total distance ? k NO
  otherwise
• Can be solved in polynomial time with
  nondeterministic computer
• How?
• Cannot be converted to polynomial time algorithm
  for regular computer
• Why?

31
NP-Complete Problems

• We will see that NP-Complete problems are the
  hardest problems in NP
• If any one NP-Complete problem can be solved in
  polynomial time
• then every NP-Complete problem can be solved in
  polynomial time
• and in fact every problem in NP can be solved in
  polynomial time (which would show P NP)
• Thus solve hamiltonian-cycle in O(n100) time,
  youve proved that P NP. Retire rich famous.

32
The Class NP-Complete (1/2)

• A problem Q is NP-complete
• if it is in NP and
• it is NP-hard.
• A problem Q is NP-hard
• if every problem in NP
• is reducible to Q.

Theorem Let A be in NP-Complete, and B is in
NP. If A ?P B, then B is also NP-complete.
33
The Class NP-Complete (2/2)

• A problem P is reducible to a problem Q if
• there exists a polynomial reduction function T
  such that
• For every string x,
• if x is a yes input for P, then T(x) is a yes
  input for Q
• if x is a no input for P, then T(x) is a no input
  for Q.
• T can be computed in polynomially bounded time.

34
Class P and Class Relationships

• Problems that are solvable in polynomial time on
  a regular computer are said to be in class P
• All problems in P are solvable in p-time on
  nondeterministic computer
• Some problems in NP are NP-complete
• e.g., Clique problem for undirected graphs
• All problems solvable in exponential time is an
  even bigger class
• Note that all problems solvable in p-time are
  certainly solvable in exponential time

35
Theoreticians View of World
Exponential time problems
NP problems
TOH
NP-Complete problems
TSP
P problems
SORTING
36
Polynomial Reductions

• Problem P is reducible to Q
• P ?p Q
• Transforming inputs of P to inputs of Q
• Reducibility relation is transitive.

37
Reduction

• The crux of NP-Completeness is reducibility
• Informally, a problem P can be reduced to another
  problem Q if any instance of P can be easily
  rephrased as an instance of Q, the solution to
  which provides a solution to the instance of P
• What do you suppose easily means?
• This rephrasing is called transformation
• Intuitively If P reduces to Q, P is no harder
  to solve than Q

38
Reducibility

• An example
• P Given a set of Booleans, is at least one TRUE?
• Q Given a set of integers, is their sum
  positive?
• Transformation (x1, x2, , xn) (y1, y2, , yn)
  where yi 1 if xi TRUE, yi 0 if xi FALSE
• Another example
• Solving linear equations is reducible to solving
  quadratic equations
• How can we easily use a quadratic-equation solver
  to solve linear equations?

39
Using Reductions

• If P is polynomial-time reducible to Q, we denote
  this P ?p Q
• Definition of NP-Hard and NP-Complete
• If all problems R ? NP are reducible to P, then P
  is NP-Hard
• We say P is NP-Complete if P is NP-Hard and P ?
  NP
• If P ?p Q and P is NP-Complete, Q is alsoNP -
  Complete
• This is the key idea you should take away today

40
Why Prove NP-Completeness?

• Though nobody has proven that P ! NP, if you
  prove a problem NP-Complete, most people accept
  that it is probably intractable
• Therefore it can be important to prove that a
  problem is NP-Complete
• Dont need to come up with an efficient algorithm
• Can instead work on approximation algorithms

41
Proving NP-Completeness

• What steps do we have to take to prove a problem
  P is NP-Complete?
• Pick a known NP-Complete problem Q
• Reduce Q to P
• Describe a transformation that maps instances of
  Q to instances of P, s.t. yes for P yes for
  Q
• Prove the transformation works
• Prove it runs in polynomial time
• Oh yeah, prove P ? NP (What if you cant?)

42
If you tell me that this graph is 3-colourable,
it is very difficult for me to check whether you
are right.
43
But if you tell me that this graph is 3-colorable
and give me a solution, it is very easy for me to
verify whether you are right.
Loosely speaking, problems that are difficult to
compute, but easy to verify are known as
Non-deterministic Polynomial.
44
Cooks Theorem

• Any NP problem can be converted to SAT in
  polynomial time.

45
The SAT Problem

• One of the first problems to be proved
  NP-Complete was satisfiability (SAT)
• Given a Boolean expression on n variables, can we
  assign values such that the expression is TRUE?
• Ex ((x1 ?x2) ? ?((?x1 ? x3) ? x4)) ??x2
• Cooks Theorem The satisfiability problem is
  NP-Complete
• Note Argue from first principles, not reduction
• Proof not here

46
Conjunctive Normal Form

• Even if the form of the Boolean expression is
  simplified, the problem may be NP-Complete
• Literal an occurrence of a Boolean or its
  negation
• A Boolean formula is in conjunctive normal form,
  or CNF, if it is an AND of clauses, each of which
  is an OR of literals
• Ex (x1 ? ?x2) ? (?x1 ? x3 ? x4) ? (?x5)
• 3-CNF each clause has exactly 3 distinct
  literals
• Ex (x1 ? ?x2 ? ?x3) ? (?x1 ? x3 ? x4) ? (?x5 ?
  x3 ? x4)
• Notice true if at least one literal in each
  clause is true

47
The 3-CNF Problem

• Satisfiability of Boolean formulas in 3-CNF form
  (the 3-CNF Problem) is NP-Complete
• Proof Nope
• The reason we care about the 3-CNF problem is
  that it is relatively easy to reduce to others
• Thus by proving 3-CNF NP-Complete we can prove
  many seemingly unrelated problems NP-Complete

48
3-CNF ? Clique

• What is a clique of a graph G?
• A a subset of vertices fully connected to each
  other, i.e. a complete subgraph of G
• The clique problem how large is the maximum-size
  clique in a graph?
• Can we turn this into a decision problem?
• A Yes, we call this the k-clique problem
• Is the k-clique problem within NP?

this graph contains a 4-clique
49
3-CNF ? Clique

• What should the reduction do?
• A Transform a 3-CNF formula to a graph, for
  which a k-clique will exist (for some k) iff the
  3-CNF formula is satisfiable

50
3-CNF ? Clique

• The reduction
• Let B C1 ? C2 ? ? Ck be a 3-CNF formula with
  k clauses, each of which has 3 distinct literals
• For each clause put a triple of vertices in the
  graph, one for each literal
• Put an edge between two vertices if they are in
  different triples and their literals are
  consistent, meaning not each others negation
• Run an example B (x ? ?y ? ?z) ? (?x ? y ? z
  ) ? (x ? y ? z )

51
3-CNF ? Clique

• Prove the reduction works
• If B has a satisfying assignment, then each
  clause has at least one literal (vertex) that
  evaluates to 1
• Picking one such true literal from each clause
  gives a set V of k vertices. V is a clique
  (Why?)
• If G has a clique V of size k, it must contain
  one vertex in each triple (clause) (Why?)
• We can assign 1 to each literal corresponding
  with a vertex in V, without fear of contradiction

52
Clique ? Vertex Cover

• A vertex cover for a graph G is a set of vertices
  incident to every edge in G
• The vertex cover problem what is the minimum
  size vertex cover in G?
• Restated as a decision problem does a vertex
  cover of size k exist in G?
• Thm vertex cover is NP-Complete

53
Clique ? Vertex Cover

• First, show vertex cover in NP (How?)
• Next, reduce k-clique to vertex cover
• The complement GC of a graph G contains exactly
  those edges not in G
• Compute GC in polynomial time
• G has a clique of size k iff GC has a vertex
  cover of size V - k

54
Clique ? Vertex Cover

• Claim If G has a clique of size k, GC has a
  vertex cover of size V - k
• Let V be the k-clique
• Then V - V is a vertex cover in GC
• Let (u,v) be any edge in GC
• Then u and v cannot both be in V (Why?)
• Thus at least one of u or v is in V-V (why?), so
  edge (u, v) is covered by V-V
• Since true for any edge in GC, V-V is a vertex
  cover

55
Clique ? Vertex Cover

• Claim If GC has a vertex cover V ? V, with V
  V - k, then G has a clique of size k
• For all u,v ? V, if (u,v) ? GC then u ? V or v
  ? V or both (Why?)
• Contrapositive if u ? V and v ? V, then (u,v)
  ? E
• In other words, all vertices in V-V are
  connected by an edge, thus V-V is a clique
• Since V - V k, the size of the clique is k

56
General Comments

• Literally hundreds of problems have been shown to
  be NP-Complete
• Some reductions are profound, some are
  comparatively easy, many are easy once the key
  insight is given
• You can expect a simple NP-Completeness proof on
  the final

57
ExampleNP-Complete Problems(1/2)

• Vertex Cover (VC)
• Instance Graph G(V,E) and integer k
• Question Does there exist a vertex cover of size
  at most k?
• (V? V is a vertex cover if for each
  (u,v)?E, either u?Vor v?V).
• Independent Set (IND)
• Instance Graph G(V,E) and integer k
• Question Does G have an independent set of size
  at least k?
• (V? V is an independent set if for any
  u,v?V, (u,v)?E.)
• Clique
• Instance Graph G(V,E) and integer k
• Question Does G have a clique of size at least
  k?
• (V? V is a clique if for any u,v?V,
  (u,v)?E.)

58
ExampleNP-Complete Problems(2/2)

• Hamiltonian Path (HP)
• Instance Graph G(V,E) and integer k
• Question Does there exist a Hamiltonian Path of
  G?
• That is, does ? a simple path of length
  V - 1?
• 3 Color Problem (3COL)
• Instance Graph G(V,E)
• Question Can we color the nodes of G with three
  three colors such that no two adjacent nodes
  of G have the same color?
• Subset-sum
• Instance Given a set of integers
• Question does there exist a subset that adds up
  to some target T?
• Partition
• Bin Packing
• Integer Linear Programming (ILP)