CS 3343: Analysis of Algorithms - PowerPoint PPT Presentation

Loading...

PPT – CS 3343: Analysis of Algorithms PowerPoint presentation | free to download - id: 6d54ca-NDZkM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

CS 3343: Analysis of Algorithms

Description:

CS 3343: Analysis of Algorithms Lecture 17: Intro to Dynamic Programming In the next few lectures Two important algorithm design techniques Dynamic programming Greedy ... – PowerPoint PPT presentation

Number of Views:98
Avg rating:3.0/5.0
Slides: 67
Provided by: Jian131
Learn more at: http://www.cs.utsa.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: CS 3343: Analysis of Algorithms


1
CS 3343 Analysis of Algorithms
  • Lecture 17 Intro to Dynamic Programming

2
In the next few lectures
  • Two important algorithm design techniques
  • Dynamic programming
  • Greedy algorithm
  • Meta algorithms, not actual algorithms (like
    divide-and-conquer)
  • Very useful in practice
  • Once you understand them, it is often easier to
    reinvent certain algorithms than trying to look
    them up!

3
Optimization Problems
  • An important and practical class of computational
    problems.
  • For each problem instance, there are many
    feasible solutions (often exponential)
  • Each solution has a value (or cost)
  • The goal is to find a solution with the optimal
    value
  • For most of these, the best known algorithm runs
    in exponential time.
  • Industry would pay dearly to have faster
    algorithms.
  • For some of them, there are quick dynamic
    programming or greedy algorithms.
  • For the rest, you may have to consider acceptable
    solutions rather than optimal solutions

4
Dynamic programming
  • The word programming is historical and predates
    computer programming
  • A very powerful, general tool for solving certain
    optimization problems
  • Once understood it is relatively easy to apply.
  • It looks like magic until you have seen enough
    examples

5
A motivating example Fibonacci numbers
  • 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89,
  • F(0) 1
  • F(1) 1
  • F(n) F(n-1) f(n-2)
  • How to compute F(n)?

6
A recursive algorithm
  • function fib(n)
  • if (n 0 or n 1) return 1
  • else return fib(n-1) fib(n-2)
  • Whats the time complexity?

7
  • Time complexity between 2n/2 and 2n

T(n) F(n) O(1.6n)
8
An iterative algorithm
  • function fib(n)
  • F0 1 F1 1
  • for i 2 to n
  • Fi Fi-1 Fi-2
  • Return Fn
  • Time complexity?

9
  • Problem with the recursive Fib algorithm
  • Each subproblem was solved for many times!
  • Solution avoid solving the same subproblem more
    than once
  • (1) pre-compute all subproblems that may be
    needed later
  • (2) Compute on demand, but memorize the solution
    to avoid recomputing
  • Can you always speedup a recursive algorithm by
    making it an iterative algorithm?
  • E.g., merge sort
  • No. since there is no overlap between the two
    sub-problems

10
Another motivating example Shortest path problem
start
goal
  • Find the shortest path from start to goal
  • An optimization problem

11
Possible approaches
  • Brute-force algorithm
  • Enumerate all possible paths and compare
  • How many?
  • Greedy algorithm
  • Always use the currently shortest edge
  • Does not necessarily lead to the optimal solution
  • Can you think about a recursive strategy?

12
Optimal subpaths
  • Claim if a path start?goal is optimal, any
    sub-path x?y, where x, y is on the optimal path,
    is also optimal.
  • Proof by contradiction
  • If the subpath b between x and y is not the
    shortest
  • we can replace it with a shorter one, b
  • which will reduce the total length of the new
    path
  • the optimal path from start to goal is not the
    shortest gt contradiction!
  • Hence, the subpath b must be the shortest among
    all paths from x to y

a b c is shortest b lt b a b c lt a b
c
13
Shortest path problem
  • SP(start, goal) min

dist(start, A) SP(A, goal) dist(start, B)
SP(B, goal) dist(start, C) SP(C, goal)
14
A special shortest path problem
S










m
G
n
Each edge has a length (cost). We need to get to
G from S. Can only move to the right or bottom.
Aim find a path with the minimum total length
15
Recursive solution
n





  • Suppose weve found the shortest path
  • It must use one of the two edges
  • (m, n-1) to (m, n) Case 1
  • (m-1, n) to (m, n) Case 2
  • If case 1
  • find shortest path from (0, 0) to (m, n-1)
  • SP(0, 0, m, n-1) dist(m, n-1, m, n) is the
    overall shortest path
  • If case 2
  • find shortest path from (0, 0) to (m-1, n)
  • SP(0, 0, m, n-1) dist(m, n-1, m, n) is the
    overall shortest path
  • We dont know which case is true
  • But if weve found the two paths, we can compare
  • Actual shortest path is the one with shorter
    overall length

m
16
Recursive solution
Let F(i, j) SP(0, 0, i, j). gt F(m, n) is
length of SP from (0, 0) to (m, n)
n






m
17
  • To find shortest path from (0, 0) to (m, n), we
    need to find shortest path to both (m-1, n) and
    (m, n-1)
  • If we use recursive call, some subpaths will be
    recomputed for many times
  • Strategy solve subproblems in certain order such
    that redundant computation can be avoided











m
n
18
Dynamic Programming
Let F(i, j) SP(0, 0, i, j). gt F(m, n) is
length of SP from (0, 0) to (m, n)
n






F(i-1, j) dist(i-1, j, i, j) F(i, j)
min F(i, j-1) dist(i, j-1, i, j)
m
19
Dynamic programming illustration
S
9
1
2
3
3
12
13
15
0




5
3
3
3
3
2
5
2
3
6
8
13
15
5
2
3
3
9
3
4
2
3
2
9
7
11
13
16
6
2
3
7
4
6
3
3
3
11
14
17
20
13
4
6
3
1
3
2
3
2
1
17
17
18
20
17
G
F(i-1, j) dist(i-1, j, i, j) F(i, j) min
F(i, j-1) dist(i, j-1, i, j)
20
Trace back
S
9
1
2
3
3
12
13
15
0




5
3
3
3
3
2
5
2
3
6
8
13
15
5
2
3
3
9
3
4
2
3
2
9
7
11
13
16
6
2
3
7
4
6
3
3
3
11
14
17
20
13
4
6
3
1
3
2
3
2
1
17
17
18
20
17
G
Shortest path has length 20 What is the actual
shortest path?
21
Elements of dynamic programming
  • Optimal sub-structures
  • Optimal solutions to the original problem
    contains optimal solutions to sub-problems
  • Overlapping sub-problems
  • Some sub-problems appear in many solutions
  • Memorization and reuse
  • Carefully choose the order that sub-problems are
    solved, such that each sub-problem will be solved
    for at most once and the solution can be reused

22
Two steps to dynamic programming
  • Formulate the solution as a recurrence relation
    of solutions to subproblems.
  • Specify an order to solve the subproblems so you
    always have what you need.
  • Bottom-up
  • Tabulate the solutions to all subproblems before
    they are used
  • What if you cannot determine the order easily, of
    if not all subproblems are needed?
  • Top-down
  • Compute when needed.
  • Remember the ones youve computed

23
Question
  • If instead of shortest path, we want (simple)
    longest path, can we still use dynamic
    programming?
  • Simple means no cycle allowed
  • Not in general, since no optimal substructure
  • Even if smt is the optimal path from s to t,
    sm may not be optimal from s to m
  • Yes for acyclic graphs

24
Example problems that can be solved by dynamic
programming
  • Sequence alignment problem (several different
    versions)
  • Shortest path in general graphs
  • Knapsack (several different versions)
  • Scheduling
  • etc.
  • Well discuss alignment, knapsack, and scheduling
    problems
  • Shortest path in graph algorithms

25
Sequence alignment
  • Compare two strings to see if they are similar
  • We need to define similarity
  • Very useful in many applications
  • Comparing DNA sequences, articles, source code,
    etc.
  • Example Longest Common Subsequence problem (LCS)

26
Common subsequence
  • A subsequence of a string is the string with zero
    or more chars left out
  • A common subsequence of two strings
  • A subsequence of both strings
  • Ex x A B C B D A B , y B D C A B A
  • B C and A A are both common subsequences of x
    and y

27
Longest Common Subsequence
  • Given two sequences x1 . . m and y1 . . n,
    find a longest subsequence common to them both.

28
Brute-force LCS algorithm
Check every subsequence of x1 . . m to see if
it is also a subsequence of y1 . . n.
  • Analysis
  • 2m subsequences of x (each bit-vector of length m
    determines a distinct subsequence of x).
  • Hence, the runtime would be exponential !
  • Towards a better algorithm a DP strategy
  • Key optimal substructure and overlapping
    sub-problems
  • First well find the length of LCS. Later well
    modify the algorithm to find LCS itself.

29
Optimal substructure
  • Notice that the LCS problem has optimal
    substructure parts of the final solution are
    solutions of subproblems.
  • If z LCS(x, y), then any prefix of z is an LCS
    of a prefix of x and a prefix of y.
  • Subproblems find LCS of pairs of prefixes of x
    and y

m
i
x
z
n
y
j
30
Recursive thinking
m
x
n
y
  • Case 1 xmyn. There is an optimal LCS that
    matches xm with yn.
  • Case 2 xm? yn. At most one of them is in LCS
  • Case 2.1 xm not in LCS
  • Case 2.2 yn not in LCS

31
Recursive thinking
m
x
n
y
  • Case 1 xmyn
  • LCS(x, y) LCS(x1..m-1, y1..n-1) xm
  • Case 2 xm ? yn
  • LCS(x, y) LCS(x1..m-1, y1..n) or
  • LCS(x1..m, y1..n-1), whichever is
    longer

Reduce both sequences by 1 char
concatenate
Reduce either sequence by 1 char
32
Finding length of LCS
m
x
n
y
  • Let ci, j be the length of LCS(x1..i,
    y1..j)
  • gt cm, n is the length of LCS(x, y)
  • If xm yn
  • cm, n cm-1, n-1 1
  • If xm ! yn
  • cm, n max cm-1, n, cm, n-1

33
Generalize recursive formulation
34
Recursive algorithm for LCS
LCS(x, y, i, j) if xi y j then ci, j ?
LCS(x, y, i1, j1) 1 else ci, j ?
max LCS(x, y, i1, j), LCS(x, y, i, j1)
Worst-case xi ¹ y j, in which case the
algorithm evaluates two subproblems, each with
only one parameter decremented.
35
Recursion tree
m 3, n 4
3,4
2,4
3,3
1,4
3,2
2,3
2,3
1,3
2,2
1,3
2,2
Height m n ? work potentially exponential.
36
DP Algorithm
  • Key find out the correct order to solve the
    sub-problems
  • Total number of sub-problems m n

n
0
j


C(i, j)


0
i
m
37
DP Algorithm
  • LCS-Length(X, Y)
  • 1. m length(X) // get the of symbols in X
  • 2. n length(Y) // get the of symbols in Y
  • 3. for i 1 to m ci,0 0 // special case
    Y0
  • 4. for j 1 to n c0,j 0 // special case
    X0
  • 5. for i 1 to m // for all Xi
  • 6. for j 1 to n // for all Yj
  • 7. if ( Xi Yj)
  • 8. ci,j ci-1,j-1 1
  • 9. else ci,j max( ci-1,j, ci,j-1 )
  • 10. return c

38
LCS Example
  • Well see how LCS algorithm works on the
    following example
  • X ABCB
  • Y BDCAB

What is the LCS of X and Y?
LCS(X, Y) BCB X A B C B Y B D C
A B
39
LCS Example (0)
ABCB BDCAB
j 0 1 2 3 4
5
i
Yj
B
B
A
C
D
Xi
0
A
1
B
2
3
C
4
B
X ABCB m X 4 Y BDCAB n Y
5 Allocate array c5,6
40
LCS Example (1)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
B
2
0
3
C
0
4
B
0
for i 1 to m ci,0 0 for j 1 to n
c0,j 0
41
LCS Example (2)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
42
LCS Example (3)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
43
LCS Example (4)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
1
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
44
LCS Example (5)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
1
1
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
45
LCS Example (6)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
1
0
1
B
2
0
1
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
46
LCS Example (7)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
1
1
1
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
47
LCS Example (8)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
1
1
1
2
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
48
LCS Example (9)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
2
1
1
1
1
3
C
0
1
1
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
49
LCS Example (10)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
50
LCS Example (11)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
51
LCS Example (12)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
1
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
52
LCS Example (13)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
1
1
2
2
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
53
LCS Example (14)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
54
LCS Algorithm Running Time
  • LCS algorithm calculates the values of each entry
    of the array cm,n
  • So what is the running time?

O(mn) since each ci,j is calculated in
constant time, and there are mn elements in the
array
55
How to find actual LCS
  • The algorithm just found the length of LCS, but
    not LCS itself.
  • How to find the actual LCS?
  • For each ci,j we know how it was acquired
  • A match happens only when the first equation is
    taken
  • So we can start from cm,n and go backwards,
    remember xi whenever ci,j ci-1, j-11.

2
2
For example, here ci,j ci-1,j-1 1 213
2
3
56
Finding LCS
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
Time for trace back O(mn).
57
Finding LCS (2)
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
LCS (reversed order)
B
C
B
LCS (straight order)
B C B (this string turned out to be a
palindrome)
58
LCS as a longest path problem
B
B
A
C
D
A
1
1
1
B
1
C
B
1
1
59
LCS as a longest path problem
B
B
A
C
D
0
0
0
0
0
0
A
1
0
0
0
0
1
1
1
1
B
0
1
1
1
1
2
1
C
0
1
1
2
2
2
B
1
1
0
1
1
1
2
3
60
A more general problem
  • Aligning two strings, such that
  • Match 1
  • Mismatch 0 (or other scores)
  • Insertion/deletion -1
  • Aligning ABBC with CABC
  • LCS 3 ABC
  • Best alignment

61
  • Let F(i, j) be the best alignment score between
    X1..i and Y1..j.
  • F(m, n) is the best alignment score between X and
    Y
  • Recurrence
  • F(i, j) max

F(i-1, j-1) ?(i, j) F(i-1,j) 1 F(i, j-1) 1
Match/Mismatch
Insertion on Y
Insertion on X
?(i, j) 1 if XiYj and 0 otherwise.
62
Alignment Example
ABBC CABC
j 0 1 2 3 4
i
C
C
B
A
Yj
Xi
0
A
1
B
2
3
B
4
C
X ABBC m X 4 Y CABC n Y
4 Allocate array F5,5
63
Alignment Example
ABBC CABC
j 0 1 2 3 4
Yj
i
C
C
B
A
Xi
0
0
-1
-2
-3
-4
A
-1
0
0
-1
-2
1
B
2
-2
-1
0
1
0
3
B
-3
-2
-1
1
1
4
C
-4
-2
-2
0
2
F(i-1, j-1) ?(i, j) F(i-1,j) 1 F(i, j-1) 1
Match/Mismatch
F(i, j) max
Insertion on Y
Insertion on X
?(i, j) 1 if XiYj and 0 otherwise.
64
Alignment Example
ABBC CABC
j 0 1 2 3 4
Yj
i
C
C
B
A
Xi
0
0
-1
-2
-3
-4
A
-1
0
0
-1
-2
1
B
2
-2
-1
0
1
0
3
B
-3
-2
-1
1
1
4
C
-4
-2
-2
0
2
F(i-1, j-1) ?(i, j) F(i-1,j) 1 F(i, j-1) 1
Match/Mismatch
F(i, j) max
Insertion on Y
Insertion on X
?(i, j) 1 if XiYj and 0 otherwise.
65
Alignment as a longest path problem
C
C
B
A




A
B
B
C
-1
1
0
66
Alignment as a longest path problem
C
C
B
A




A
B
B
C
-1
1
0
About PowerShow.com