Notes

1
Notes

• This set of slides (Handout 7) is missing the
  material on Reductions and Greed that was
  presented in class. Those slides are still under
  construction, and will be posted as Handout 6.
• The quiz on Thursday of 8th week will cover Greed
  and Dynamic Programming, and through HW 4

2
Topological Sorting

• Prelude to shortest paths
• Generic scheduling problem
• Input
• Set of tasks T1, T2, T3, , Tn
• Example getting dressed in the morning put on
  shoes, socks, shirt, pants, belt,
• Set of dependencies T1?T2, T3 ? T4, T5 ? T1,
• Example must put on socks before shoes, pants
  before belt,
• Want
• ordering of tasks which is consistent with
  dependencies
• Problem representation Directed Acyclic Graph
• Vertices tasks Directed Edges dependencies
• Acyclic if cycle of dependencies, no
  solution possible

3
Topological Sorting

• TOP_SORT PROBLEM Given a DAG G(V,E) with
  Vn, assign labels 1,...,n to vi Î V s.t. if
  v has label k, all vertices reachable from v have
  labels gt k
• Induction idea
• Know how to label DAGs with lt n vertices
• Claim A DAG G always has some vertex with
  indegree 0
• Take an arbitrary vertex v. If v doesnt have
  indegree 0, traverse any incoming edge to reach
  a predecessor of v. If this vertex doesnt have
  indegree 0, traverse any incoming edge to reach
  a predecessor, etc.
• Eventually, this process will either identify a
  vertex with indegree 0, or else reach a vertex
  that has been reached previously (a
  contradiction, given that G is acyclic).
• Inductive approach Find v with indegree(v)
  0, give it lowest available label, then delete v
  (and incident edges), update degrees of remaining
  vertices, and repeat

4
Dynamic Programming Key Phrases

• As we discuss shortest paths, and then the
  dynamic programming approach, keep in mind the
  following key phrases
• Principle of Optimality Any subsolution of an
  optimal solution is itself an optimal solution
• Overlapping Subproblems Can exploit a
  polynomially bounded number of possible
  subproblems
• Bottom-Up D/Q / Table (cache) of Subproblem
  Solutions avoid recomputation by tabulating
  subproblem solutions
• Relaxation / Successive Approximation Often,
  the DP approach makes multiple passes, each time
  solving a less restricted version of the original
  problem instance
• Eventually, solve completely unrestricted
  original problem instance

5
Single-Source Shortest Paths

• Given G(V,E) a directed graph, L E ? Â a
  length function, and a distinguished source s Î
  V. Find the shortest path in G from s to each vi
  Î V, vi ¹ s.
• Let L(i,j) length of SP from vi to vj
• Lemma Suppose S Ì V contains s vi and L(1,w)
  is known "w Î S. For vi Ï S
• Let D(vi) minw Î S L(1,w) L(w,i) ()
• L(1,w) path length, with path restricted to
  vertices of S
• L(w,i) edge length
• Let vv minimize D(vi) over all nodes vi Ï S
  ()
• Then L(1,v) D(vv).
• Notation D(v) length of the 1-to-v SP that
  uses only vertices of S (except for v). (D(v)
  is not necessarily the same as L(1,v), because
  the path is restricted!)

6
Single-Source Shortest Paths

• Proof of Lemma To prove equality, prove and ³
  
• L(1,v) D(v) is obvious because D(v) is the
  minimum length of a restricted path, while
  L(1,v) is unrestricted
• Let L(1,v) be the shortest path sv1, v2, , vr
  v from the source v1 to v. Let vj be the first
  vertex in this SP that is not in S
• Then L(1,v) L(1, vj-1) L(vj-1, vj)
  L(vj,v)
• // else, not a shortest path
• ³ D(vj)
  L(vj,v)
• // D(vj) minimizes over all w
  in S, including vj-1
• ³ D(v)
  L(vj,v)
• // because both vj and vv Ï S,
  but vv chosen first
• ³ D(v)
• // since L(vj,v) ³ 0. Lemma
  ? Dijkstras Algorithm

7
A Fact About Shortest Paths

• Triangle Inequality
• ?(u,v) ? ?(u,x) ?(x,v)
• (shortest path distances induce a metric)

u
v
x
8
Shortest Path Formulations

• Given a graph G(V,E) and w E ? ?
• (1 to 2) s-t Find a shortest path
  from s to t
• (1 to all) single-source Find a
  shortest path from a source s to every other
  vertex v ?V
• (All to all) all-pairs Find a shortest
  path from every vertex to every other vertex
• Weight of path ltv1,...,vkgt ?
  w(vi,vi1)
• Sometimes no negative edges
• Examples of negative edges travel
  inducements, exothermic reactions in chemistry,
  unprofitable transactions in arbitrage,
• Always no negative cycles
• Makes the shortest-path problem well-defined

lt0
9
Shortest Paths

• First case All edges have positive length
• Length of (vi,vj ) edge dij
• Condition 1 dij gt 0
• Condition 2 di djk dik for some i,j,k
• else shortest-path problem would be trivial
• Observation 1 Length of a path gt length of any
  of its subpaths
• Observation 2 Any subpath of a shortest path is
  itself a shortest path Principle of
  Optimality
• Observation 3 Any shortest path contains n-1
  edges pigeonhole principle assumes no
  negative cycles n nodes total

10
Shortest Paths

• Scenario All shortest paths from v0 source to
  other nodes are ordered by increasing length
• P1 P2 Pn-1
• Index nodes accordingly
• Algorithm Find P1, then find P2, etc.
• Q How many edges are there in P1 ?
• Exactly 1 edge, else can find a subpath that is
  shorter
• Q How many edges are there in Pk ?
• At most k edges, else can find k (shorter)
  subpaths, which would contradict the definition
  of Pk
• Observation 4 Pk contains k edges
• To find P1 only look at one-edge paths (min
  P1)
• To find P2 only look at one- and two-edge paths
• But, need only consider two-edge paths of form
  d01 d1i
• Else would have ³ 1 paths shorter than P2, a
  contradiction

11
Another Presentation of Dijkstras Algorithm

• Terminology
• Permanent label true SP distance from v0 to vi
• Temporary label restricted SP distance from v0
  to vi (going through only existing
  permanently-labeled nodes)
• Permanently labeled nodes set S in previous
  development
• Dijkstras Algorithm
• 0. All vertices vi, i 1,, n-1, receive
  temporary labels li with value d0i
• LOOP
• 1. Among all temporary labels, pick lk minI li
  and change lk to lk (i.e., make lk s label
  permanent) // stop if no temporary labels left
• 2. Replace all temporary labels of vk s
  neighbors, using
• li min (li , lk dki)

12
Prims Algorithm vs. Dijkstras Algorithm

• Prim Iteratively add edge eij
  to T,
• such that vi Î T,
  vj Ï T,
• and dij is
  minimum
• Dijkstra Iteratively add edge eij to
  T,
• such that vi Î T,
  vj Ï T,
• and li dij is
  minimum
• Both are building trees, in very similar ways!
• Prim Minimum Spanning Tree
• Dijkstra Shortest Path Tree
• What kind of tree does the following algorithm
  build?
• Prim-Dijkstra Iteratively add edge eij to T,
• such that vi Î T,
  vj Ï T,
• and c li dij
  is minimum
• // 0 c
  1

13
Bellman-Ford Algorithm

• Idea Successive Approximation / Relaxation
• Find SP using 1 edges
• Find SP using 2 edges
• Find SP using n-1 edges ? have true shortest
  paths
• Let lj(k) denote shortest v0 vj pathlength
  using k edges
• Then, li(1) d0j " j 1, , n-1 // dij
  if no i-j edge
• In general, lj(k1) min lj(k) , mini (li(k)
  dij)
• lj(k) dont need k1 arcs
• mini (li(k) dij) view as length-k SP plus a
  single edge

14
Bellman-Ford vs. Dijkstra
B
4
A
8
1
3
S
C
2
2
D
Pass 1 2
3
4 Label A 8
min(8, 34, ? 1, 2 ?) 7 min(7, 34, 41,
2?) 5 min(5, 34, 41, 2? ) 5
B 3 min(3, 84, ??, 2?) 3 min(3,
74, 4?, 2?) 3 min(3, 54, 4?, 2?) 3
C ? min(?, 81, 3?, 22) 4
min(4, 71, 3?, 22) 4 min(4, 51, 3?,
22) 4 D 2 min(2, 8?, 3?,
?2) 2 min(2, 7?, 3?, 42) 2 min(2,
5?, 3?, 42) 2
15
Bellman-Ford vs. Dijkstra
B
4
A
8
1
3
S
C
2
2
D
Pass 1 2
3
4 Label A 8 min(8, 2 ?) 8
min(8, 34) 7 min(7, 41) 5
B 3 min(3, 2? ) 3 C ?
min(?, 22) 4 min(4, 3?) 4
D 2
16
Special Case DAGs (24.2)

• Longest-Path Problem well-defined only when
  there are no cycles
• DAG topologically sort the vertices
• ? labels v1, , vn s.t. all edges directed from
  vi to vj, i lt j
• Let li denote longest v0 vj pathlength
• l0 0
• l1 d01 // dij - if no i-j edge
• l2 max(d01 d12 , d02)
• In general, lk maxjltk (lj djk)
• Shortest pathlength in DAG replace max by min,
  use dij if no i-j edge

17
DAG Shortest Paths Complexity (24.2)

• Bellman-Ford O(VE)
• Topological sort O(VE) (DFS)
• Will never relax edges out of vertex v until have
  done all edges in to v
• Runtime O(VE)
• Application PERT (program evaluation and review
  technique) critical path is the longest path
  through the DAG

18
All-Pairs Shortest Paths (25.1-25.2)

• Directed graph G (V,E), weight E ? ?
• Goal Create n ? n matrix of SP distances ?(u,v)
• Running Bellman-Ford once from each vertex
• O( ) O( ) on dense
  graphs
• Adjacency-matrix representation of graph
• n ? n matrix W (wij) of edge weights
• assume wii 0 ?i,
• SP to self has no edges, as long as there are no
  negative cycles

19
Simple APSP Dynamic Programming (25.1)

• dij(m) weight of s-p from i to j with ? m edges
• dij(0) 0 if i j and dij(0) ? if i ?
  j
• dij(m) minkdik(m-1) wkj
• Runtime O( n4)
• n-1 passes, each computing n2 ds in O(n) time

? m-1
j
i
? m-1
20
Matrix Multiplication (25.1)

• Similar C A ? B, two n ? n matrices
• cij ?k aik ? bkj O(n3) operations
• replacing ? min
• ? ?
  
• gives cij mink aik bkj
• D(m) D(m-1) ? W
• identity matrix is D(0)
• Cannot use Strassens because no subtraction
• Time is still O(n ? n3 ) O(n4 )
• Repeated squaring W2n Wn ? Wn (addition
  chains)
• Compute W, W2 , W4 ,..., W2k , k log n ?
  O(n3 log n)

21
Floyd-Warshall Algorithm (26.2/25.2)

• Also DP, but even faster (by another log n actor
  ? O(n3))
• cij(m) weight of SP from i to j with
  intermediate vertices in the set 1, 2, ..., m
  ? ?(i, j) cij(n)
• DP compute cij(n) in terms of smaller cij(n-1)
• cij(0) wij
• cij(m) min cij(m) , cim(m-1) cmj(m-1)
• intermediate nodes in 1,
  2, ..., m

cmj(m-1)
m
cim(m-1)
j
i
cij(m-1)
22
Floyd-Warshall Algorithm (26.2/25.2)

• Difference from previous we do not check all
  possible intermediate vertices.
• for m1..n do for i1..n do for j 1..n do
• cij(m) min cij(m-1) , cim(m-1)
  cmj(m-1)
• Runtime O(n3 )
• Transitive Closure G of graph G
• (i,j) ? G iff ? path from i to j in G
• Adjacency matrix, elements on 0,1
• Floyd-Warshall with min ? OR ,
  ? AND
• Runtime O(n3 )
• Useful in many problems

23

• BEGIN DIGRESSION
• (alternative presentation, can be skipped)

24
CLRS Notation Bellman-Ford (24.1) SSSP in
General Graphs

• Essentially a BFS based algorithm
• Shortest paths (tree) easy to reconstruct
• for each v ?V do dv ? ? ds ? 0
• //
  initialization
• for i 1,...,V-1 do
• for each edge (u,v) ? E do
• dv ? mindv, duw(u,v)
• //
  relaxation
• for each v ?V do if dvgt du w(u,v) then no
  solution
• //
  negative cycle checking
• for each v ?V, dv ?(s,v) // have true
  shortest paths

25
Bellman-Ford Analysis (CLRS 24.1)

• Runtime O(VE)
• Correctness
• Lemma dv ? ?(s,v)
• Initially true
• Let dv du w(u,v)
• by triangle inequality, for first violation
• dv lt ?(s,v) ? ?(s,u)w(u,v) ?
  d(u)w(u,v)
• After V-1 passes all d values are ?s if there
  are no negative cycles
• s ? v1 ? v2 ? ... ? v (some shortest path)
• After i-th iteration ds,vi is correct and
  final

26
Dijkstra Yet Again (CLRS 24.3)

• Faster than Bellman-Ford because can exploit
  having only non-negative weights
• Like BFS, but uses priority queue
• for each v ?V do dv ? ? ds ? 0
• S ? ? Q ? V
• While Q ? ? do
• u ? Extract-Min(Q)
• S ? S u
• for v adjacent to u do
• dv ? mindv, duw(u,v)
• (relaxation Decrease-Key)

27
Dijkstra Runtime (24.3)

• Extract-Min executed V times
• Decrease-Key executed E times
• Time V?T(Extract-Min)
• // finddelete
  O(log V))
• E ?T(Decrease-Key)
• // deleteadd O(log V))
• Binary Heap E ? log V (30 years
  ago)
• Fibonacci Heap E V ? log V (10 years ago)
• Optimal time algorithm found 1 year ago runs in
  time O(E) (Mikel Thorup)

28
Dijkstra Correctness (24.3)

• Same Lemma as for Bellman-Ford dv ? ?(s,v)
• Thm Whenever u is added to S, du ?(s,u)
• Proof
• Assume that u is the first vertex s.t. du gt
  ?(s,u)
• Let y be first vertex ? V-S that is on actual
  shortest s-u path ? dy ?(s,y)
• For ys predecessor x, dx ?(s,x)
• At the moment that we put x in S, dy gets value
  ?(s,y)
• du gt ?(s,u) ?(s,y) ?(y,u) dy ?(y,u) ?
  dy

S
u
s
y
Q
x
29

• END DIGRESSION
• (alternative presentation, can be skipped)

30
Dynamic Programming (CLRS 15)

• Third Major Paradigm so far
• DQ, Greed are previous paradigms
• Dynamic Programming metatechnique (not a
  particular algorithm)
• Programming refers to use of a tableau in the
  method (cf. mathematical programming), not to
  writing of code

31
Longest Common Subsequence (CLRS 15.4)

• Problem Given x1..m and y1..n, find LCS
• x A B C B D A B
• 
  ? B C B A
• y B D C A B A
• Brute-force algorithm
• for every subsequence of x, check if it is in y
• O(n2m ) time
• 2m subsequences of x (each element is either in
  or out of the subsequence)
• O(n) for scanning y with x-subsequence (m ? n)

32
Recurrent Formula for LCS (15.4)

• Let ci,j length of LCS of Xix1..i,Yj
  y1..j
• Then cm,n length of LCS of x and y
• Theorem
• 
  if xi yj
• 
  otherwise
• Proof xi yj ?
• LCS(Xi,Yj) LCS(Xi-1,Yj-1) xi

33
DP Properties(15.3)

• Any part of the optimal answer is also optimal
• A subsequence of LCS(X,Y) is the LCS for some
  subsequences of X and Y.
• Subproblems overlap
• LCS(Xm,Yn-1) and LCS(Xm-1,Yn) have common
  subproblem LCS(Xm-1,Yn-1)
• There are polynomially few subproblems in total
  mn for LCS
• Unlike divide and conquer

34
DP (CLRS 15.3)

• After computing solution of a subproblem, store
  in table
• Time O(mn)
• When computing ci,j we need O(1) time if we
  have
• xi, yj
• ci,j-1
• ci-1,j
• ci-1,j-1

35
DP Table for LCS
y B D C A B A
x A B C B D A B
36
DP Table for LCS
y B D C A B A
x A B C B D A B
0 0 0 0 0 0 0 1
1 0 1 0 1 0 1 0 1 0 1
37
DP Table for LCS
y B D C A B A
x A B C B D A B
0 0 0 0 0 0 0 1 1 0
1 1 0 1 1 0 1 2 0
1 0 1
38
DP Table for LCS
y B D C A B A
x A B C B D A B
0 0 0 0 0 0 0 0 0
0 0 1 1 1 0 1 1 1
1 2 2 0 1 1 2 2 2
2 0 1 1 2 2 3 3 0
1 2 2 2 3 3 0 1 2
2 3 3 4 0 1 2 2 3
4 4
39
Optimal Polygon Triangulation

• Polygon has sides and vertices
• Polygon is simple not self-intersecting.
• Polygon P is convex if any line segment with ends
  in P lies entirely in P
• Triangulation of P is partition of P with chords
  into triangles.
• Problem Given a convex polygon and weight
  function defined on triangles (e.g. the
  perimeter). Find triangulation of minimum weight
  (of minimum total length).
• of triangles Always have n-2 triangles with
  n-3 chords

40
Optimal Polygon Triangulation

• Optimal sub-triangulation of optimal
  triangulation
• Recurrent formula
• ti,j the weight of the optimal triangulation
  of the polygon
• ltvi-1,vi,...,vjgt If ij, then ti,j0
  else
• Runtime O(n3) and space is O(n2)

vi
vi
vi-1
vi-1
vj
vj
vk
vk