CS 3343: Analysis of Algorithms - PowerPoint PPT Presentation

PPT – CS 3343: Analysis of Algorithms PowerPoint presentation | free to download - id: 6d54ca-NDZkM

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

CS 3343: Analysis of Algorithms

Description:

CS 3343: Analysis of Algorithms Lecture 17: Intro to Dynamic Programming In the next few lectures Two important algorithm design techniques Dynamic programming Greedy ... – PowerPoint PPT presentation

Number of Views:98
Avg rating:3.0/5.0
Slides: 67
Provided by: Jian131
Category:
Tags:
Transcript and Presenter's Notes

Title: CS 3343: Analysis of Algorithms

1
CS 3343 Analysis of Algorithms
• Lecture 17 Intro to Dynamic Programming

2
In the next few lectures
• Two important algorithm design techniques
• Dynamic programming
• Greedy algorithm
• Meta algorithms, not actual algorithms (like
divide-and-conquer)
• Very useful in practice
• Once you understand them, it is often easier to
reinvent certain algorithms than trying to look
them up!

3
Optimization Problems
• An important and practical class of computational
problems.
• For each problem instance, there are many
feasible solutions (often exponential)
• Each solution has a value (or cost)
• The goal is to find a solution with the optimal
value
• For most of these, the best known algorithm runs
in exponential time.
• Industry would pay dearly to have faster
algorithms.
• For some of them, there are quick dynamic
programming or greedy algorithms.
• For the rest, you may have to consider acceptable
solutions rather than optimal solutions

4
Dynamic programming
• The word programming is historical and predates
computer programming
• A very powerful, general tool for solving certain
optimization problems
• Once understood it is relatively easy to apply.
• It looks like magic until you have seen enough
examples

5
A motivating example Fibonacci numbers
• 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89,
• F(0) 1
• F(1) 1
• F(n) F(n-1) f(n-2)
• How to compute F(n)?

6
A recursive algorithm
• function fib(n)
• if (n 0 or n 1) return 1
• else return fib(n-1) fib(n-2)
• Whats the time complexity?

7
• Time complexity between 2n/2 and 2n

T(n) F(n) O(1.6n)
8
An iterative algorithm
• function fib(n)
• F0 1 F1 1
• for i 2 to n
• Fi Fi-1 Fi-2
• Return Fn
• Time complexity?

9
• Problem with the recursive Fib algorithm
• Each subproblem was solved for many times!
• Solution avoid solving the same subproblem more
than once
• (1) pre-compute all subproblems that may be
needed later
• (2) Compute on demand, but memorize the solution
to avoid recomputing
• Can you always speedup a recursive algorithm by
making it an iterative algorithm?
• E.g., merge sort
• No. since there is no overlap between the two
sub-problems

10
Another motivating example Shortest path problem
start
goal
• Find the shortest path from start to goal
• An optimization problem

11
Possible approaches
• Brute-force algorithm
• Enumerate all possible paths and compare
• How many?
• Greedy algorithm
• Always use the currently shortest edge
• Does not necessarily lead to the optimal solution
• Can you think about a recursive strategy?

12
Optimal subpaths
• Claim if a path start?goal is optimal, any
sub-path x?y, where x, y is on the optimal path,
is also optimal.
• If the subpath b between x and y is not the
shortest
• we can replace it with a shorter one, b
• which will reduce the total length of the new
path
• the optimal path from start to goal is not the
• Hence, the subpath b must be the shortest among
all paths from x to y

a b c is shortest b lt b a b c lt a b
c
13
Shortest path problem
• SP(start, goal) min

dist(start, A) SP(A, goal) dist(start, B)
SP(B, goal) dist(start, C) SP(C, goal)
14
A special shortest path problem
S

m
G
n
Each edge has a length (cost). We need to get to
G from S. Can only move to the right or bottom.
Aim find a path with the minimum total length
15
Recursive solution
n

• Suppose weve found the shortest path
• It must use one of the two edges
• (m, n-1) to (m, n) Case 1
• (m-1, n) to (m, n) Case 2
• If case 1
• find shortest path from (0, 0) to (m, n-1)
• SP(0, 0, m, n-1) dist(m, n-1, m, n) is the
overall shortest path
• If case 2
• find shortest path from (0, 0) to (m-1, n)
• SP(0, 0, m, n-1) dist(m, n-1, m, n) is the
overall shortest path
• We dont know which case is true
• But if weve found the two paths, we can compare
• Actual shortest path is the one with shorter
overall length

m
16
Recursive solution
Let F(i, j) SP(0, 0, i, j). gt F(m, n) is
length of SP from (0, 0) to (m, n)
n

m
17
• To find shortest path from (0, 0) to (m, n), we
need to find shortest path to both (m-1, n) and
(m, n-1)
• If we use recursive call, some subpaths will be
recomputed for many times
• Strategy solve subproblems in certain order such
that redundant computation can be avoided

m
n
18
Dynamic Programming
Let F(i, j) SP(0, 0, i, j). gt F(m, n) is
length of SP from (0, 0) to (m, n)
n

F(i-1, j) dist(i-1, j, i, j) F(i, j)
min F(i, j-1) dist(i, j-1, i, j)
m
19
Dynamic programming illustration
S
9
1
2
3
3
12
13
15
0

5
3
3
3
3
2
5
2
3
6
8
13
15
5
2
3
3
9
3
4
2
3
2
9
7
11
13
16
6
2
3
7
4
6
3
3
3
11
14
17
20
13
4
6
3
1
3
2
3
2
1
17
17
18
20
17
G
F(i-1, j) dist(i-1, j, i, j) F(i, j) min
F(i, j-1) dist(i, j-1, i, j)
20
Trace back
S
9
1
2
3
3
12
13
15
0

5
3
3
3
3
2
5
2
3
6
8
13
15
5
2
3
3
9
3
4
2
3
2
9
7
11
13
16
6
2
3
7
4
6
3
3
3
11
14
17
20
13
4
6
3
1
3
2
3
2
1
17
17
18
20
17
G
Shortest path has length 20 What is the actual
shortest path?
21
Elements of dynamic programming
• Optimal sub-structures
• Optimal solutions to the original problem
contains optimal solutions to sub-problems
• Overlapping sub-problems
• Some sub-problems appear in many solutions
• Memorization and reuse
• Carefully choose the order that sub-problems are
solved, such that each sub-problem will be solved
for at most once and the solution can be reused

22
Two steps to dynamic programming
• Formulate the solution as a recurrence relation
of solutions to subproblems.
• Specify an order to solve the subproblems so you
always have what you need.
• Bottom-up
• Tabulate the solutions to all subproblems before
they are used
• What if you cannot determine the order easily, of
if not all subproblems are needed?
• Top-down
• Compute when needed.
• Remember the ones youve computed

23
Question
• If instead of shortest path, we want (simple)
longest path, can we still use dynamic
programming?
• Simple means no cycle allowed
• Not in general, since no optimal substructure
• Even if smt is the optimal path from s to t,
sm may not be optimal from s to m
• Yes for acyclic graphs

24
Example problems that can be solved by dynamic
programming
• Sequence alignment problem (several different
versions)
• Shortest path in general graphs
• Knapsack (several different versions)
• Scheduling
• etc.
• Well discuss alignment, knapsack, and scheduling
problems
• Shortest path in graph algorithms

25
Sequence alignment
• Compare two strings to see if they are similar
• We need to define similarity
• Very useful in many applications
• Comparing DNA sequences, articles, source code,
etc.
• Example Longest Common Subsequence problem (LCS)

26
Common subsequence
• A subsequence of a string is the string with zero
or more chars left out
• A common subsequence of two strings
• A subsequence of both strings
• Ex x A B C B D A B , y B D C A B A
• B C and A A are both common subsequences of x
and y

27
Longest Common Subsequence
• Given two sequences x1 . . m and y1 . . n,
find a longest subsequence common to them both.

28
Brute-force LCS algorithm
Check every subsequence of x1 . . m to see if
it is also a subsequence of y1 . . n.
• Analysis
• 2m subsequences of x (each bit-vector of length m
determines a distinct subsequence of x).
• Hence, the runtime would be exponential !
• Towards a better algorithm a DP strategy
• Key optimal substructure and overlapping
sub-problems
• First well find the length of LCS. Later well
modify the algorithm to find LCS itself.

29
Optimal substructure
• Notice that the LCS problem has optimal
substructure parts of the final solution are
solutions of subproblems.
• If z LCS(x, y), then any prefix of z is an LCS
of a prefix of x and a prefix of y.
• Subproblems find LCS of pairs of prefixes of x
and y

m
i
x
z
n
y
j
30
Recursive thinking
m
x
n
y
• Case 1 xmyn. There is an optimal LCS that
matches xm with yn.
• Case 2 xm? yn. At most one of them is in LCS
• Case 2.1 xm not in LCS
• Case 2.2 yn not in LCS

31
Recursive thinking
m
x
n
y
• Case 1 xmyn
• LCS(x, y) LCS(x1..m-1, y1..n-1) xm
• Case 2 xm ? yn
• LCS(x, y) LCS(x1..m-1, y1..n) or
• LCS(x1..m, y1..n-1), whichever is
longer

Reduce both sequences by 1 char
concatenate
Reduce either sequence by 1 char
32
Finding length of LCS
m
x
n
y
• Let ci, j be the length of LCS(x1..i,
y1..j)
• gt cm, n is the length of LCS(x, y)
• If xm yn
• cm, n cm-1, n-1 1
• If xm ! yn
• cm, n max cm-1, n, cm, n-1

33
Generalize recursive formulation
34
Recursive algorithm for LCS
LCS(x, y, i, j) if xi y j then ci, j ?
LCS(x, y, i1, j1) 1 else ci, j ?
max LCS(x, y, i1, j), LCS(x, y, i, j1)
Worst-case xi ¹ y j, in which case the
algorithm evaluates two subproblems, each with
only one parameter decremented.
35
Recursion tree
m 3, n 4
3,4
2,4
3,3
1,4
3,2
2,3
2,3
1,3
2,2
1,3
2,2
Height m n ? work potentially exponential.
36
DP Algorithm
• Key find out the correct order to solve the
sub-problems
• Total number of sub-problems m n

n
0
j

C(i, j)

0
i
m
37
DP Algorithm
• LCS-Length(X, Y)
• 1. m length(X) // get the of symbols in X
• 2. n length(Y) // get the of symbols in Y
• 3. for i 1 to m ci,0 0 // special case
Y0
• 4. for j 1 to n c0,j 0 // special case
X0
• 5. for i 1 to m // for all Xi
• 6. for j 1 to n // for all Yj
• 7. if ( Xi Yj)
• 8. ci,j ci-1,j-1 1
• 9. else ci,j max( ci-1,j, ci,j-1 )
• 10. return c

38
LCS Example
• Well see how LCS algorithm works on the
following example
• X ABCB
• Y BDCAB

What is the LCS of X and Y?
LCS(X, Y) BCB X A B C B Y B D C
A B
39
LCS Example (0)
ABCB BDCAB
j 0 1 2 3 4
5
i
Yj
B
B
A
C
D
Xi
0
A
1
B
2
3
C
4
B
X ABCB m X 4 Y BDCAB n Y
5 Allocate array c5,6
40
LCS Example (1)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
B
2
0
3
C
0
4
B
0
for i 1 to m ci,0 0 for j 1 to n
c0,j 0
41
LCS Example (2)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
42
LCS Example (3)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
43
LCS Example (4)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
1
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
44
LCS Example (5)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
1
1
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
45
LCS Example (6)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
1
0
1
B
2
0
1
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
46
LCS Example (7)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
1
1
1
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
47
LCS Example (8)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
1
1
1
2
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
48
LCS Example (9)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
2
1
1
1
1
3
C
0
1
1
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
49
LCS Example (10)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
50
LCS Example (11)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
51
LCS Example (12)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
1
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
52
LCS Example (13)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
1
1
2
2
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
53
LCS Example (14)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
54
LCS Algorithm Running Time
• LCS algorithm calculates the values of each entry
of the array cm,n
• So what is the running time?

O(mn) since each ci,j is calculated in
constant time, and there are mn elements in the
array
55
How to find actual LCS
• The algorithm just found the length of LCS, but
not LCS itself.
• How to find the actual LCS?
• For each ci,j we know how it was acquired
• A match happens only when the first equation is
taken
• So we can start from cm,n and go backwards,
remember xi whenever ci,j ci-1, j-11.

2
2
For example, here ci,j ci-1,j-1 1 213
2
3
56
Finding LCS
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
Time for trace back O(mn).
57
Finding LCS (2)
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
LCS (reversed order)
B
C
B
LCS (straight order)
B C B (this string turned out to be a
palindrome)
58
LCS as a longest path problem
B
B
A
C
D
A
1
1
1
B
1
C
B
1
1
59
LCS as a longest path problem
B
B
A
C
D
0
0
0
0
0
0
A
1
0
0
0
0
1
1
1
1
B
0
1
1
1
1
2
1
C
0
1
1
2
2
2
B
1
1
0
1
1
1
2
3
60
A more general problem
• Aligning two strings, such that
• Match 1
• Mismatch 0 (or other scores)
• Insertion/deletion -1
• Aligning ABBC with CABC
• LCS 3 ABC
• Best alignment

61
• Let F(i, j) be the best alignment score between
X1..i and Y1..j.
• F(m, n) is the best alignment score between X and
Y
• Recurrence
• F(i, j) max

F(i-1, j-1) ?(i, j) F(i-1,j) 1 F(i, j-1) 1
Match/Mismatch
Insertion on Y
Insertion on X
?(i, j) 1 if XiYj and 0 otherwise.
62
Alignment Example
ABBC CABC
j 0 1 2 3 4
i
C
C
B
A
Yj
Xi
0
A
1
B
2
3
B
4
C
X ABBC m X 4 Y CABC n Y
4 Allocate array F5,5
63
Alignment Example
ABBC CABC
j 0 1 2 3 4
Yj
i
C
C
B
A
Xi
0
0
-1
-2
-3
-4
A
-1
0
0
-1
-2
1
B
2
-2
-1
0
1
0
3
B
-3
-2
-1
1
1
4
C
-4
-2
-2
0
2
F(i-1, j-1) ?(i, j) F(i-1,j) 1 F(i, j-1) 1
Match/Mismatch
F(i, j) max
Insertion on Y
Insertion on X
?(i, j) 1 if XiYj and 0 otherwise.
64
Alignment Example
ABBC CABC
j 0 1 2 3 4
Yj
i
C
C
B
A
Xi
0
0
-1
-2
-3
-4
A
-1
0
0
-1
-2
1
B
2
-2
-1
0
1
0
3
B
-3
-2
-1
1
1
4
C
-4
-2
-2
0
2
F(i-1, j-1) ?(i, j) F(i-1,j) 1 F(i, j-1) 1
Match/Mismatch
F(i, j) max
Insertion on Y
Insertion on X
?(i, j) 1 if XiYj and 0 otherwise.
65
Alignment as a longest path problem
C
C
B
A

A
B
B
C
-1
1
0
66
Alignment as a longest path problem
C
C
B
A

A
B
B
C
-1
1
0