# ALGORITHM TYPES - PowerPoint PPT Presentation

PPT – ALGORITHM TYPES PowerPoint presentation | free to download - id: 68522b-M2I0Z

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## ALGORITHM TYPES

Description:

### ALGORITHM TYPES Greedy, Divide and Conquer, Dynamic Programming, and Backtracking Note the general strategy from the examples The classification is neither exhaustive ... – PowerPoint PPT presentation

Number of Views:41
Avg rating:3.0/5.0
Slides: 45
Provided by: Debasi3
Category:
Tags:
Transcript and Presenter's Notes

Title: ALGORITHM TYPES

1
ALGORITHM TYPES
• Greedy, Divide and Conquer, Dynamic Programming,
and Backtracking
• Note the general strategy from the examples
• The classification is neither exhaustive (there
may be other types), nor mutually exclusive
(one may combine)

2
PROBLEM 1 DYNAMIC PROGRAMMING STRATEGY
• In case the divide and conquer strategy can
divide the problem at a very small level, and
there are repetition of some calculations over
some components
• One can apply a bottom up approach calculate the
smaller components first and then keep combining
them until the highest level of the problem is
solved.
• Draw the recursion tree of Fibonacci-series
calculation, you will see example of such
repetitive calculations
• f(n) f(n-1) f(n-2), for ngt1 and f(n)1
otherwise
• fib(n) calculation
• n 1, 2, 3, 4, 5
• fib 1, 2, 3, 5, 8

3
DYNAMIC PROGRAMMING STRATEGY (Continued)
• Recursive fib(n)
• if (nlt1) return 1
• else return (fib(n-1) fib(n-2)).
• Time complexity exponential, O(kn) for some
kgt1.0
• Iterative fibonacci(n)
• fib(0) fib(1) 1
• for i2 through n do
• fib(i) fib(i-1) fib(i-2)
• end for
• return fib(n).
• Time complexity O(n), Space complexity O(n)

4
DYNAMIC PROGRAMMING STRATEGY (Continued)
• SpaceSaving-fibonacci(n)
• if (nlt1) return 1
• int last1, last2last1, result1
• for i2 through n do
• result last last2last
• last2last last
• lastresult
• end for
• return result.
• Time complexity O(n), Space complexity O(1)

5
DYNAMIC PROGRAMMING STRATEGY (Continued)
• The recursive call recalculates fib(1) 5 times,
fib(2) 3 times, fib(3) 2 times - in fib(5)
calculation. The complexity is exponential.
• In iterative calculation we avoid repetition by
storing the needed values in variables -
complexity of order n.
• Dynamic Programming approach consumes more memory
to store the result of calculations of lower
levels for the purpose of calculating the next
higher level. They are typically stored in a
table.

6
Problem 2 0-1 Knapsack Problem (not in the book)
• Given a set of objects with (Weight, Profit)
pair, and a Knapsack of limited weight capacity
(M), find a subset of objects for the knapsack to
maximize total profit P
• Sample Problem
• Objects (wt, p) (2, 1), (3, 2), (9, 20), (5,
3). M8
• Exhaustive Algorithm Try all subset of objects.
• How many?

total(0 lbs, 0) (2,1) total (2lbs,
1) (9,20) Illegal, total wtgt8
(2,1), (3,2) (5lbs, 3)
Total possibilities 2n For each object
present or absent 0 or 1 A bit string is a
good representation for a set, e.g.
000101100. n-bits
7
0-1 Knapsack Problem
• DP uses a representation for optimal profit P(j,
k)
• for the first j objects considered
• (in any arbitrary ordering of objects)
• with a variable knapsack limit k
• Develop the table for P(j, k)
• with rows j 1 n (objects), and
• for each row, go from k 0 M (the KS limit)
• Finally P(n, M) gets the result

w-gt 0 1 2 3 4 5 6 7 8
0 0 0 0 0 0 0 0 0
O1 0 0 1 1 1 gt 1
O1 O2 0 0 1 2 2 3 3 gt 3
O1 O2 O3 0 0 1 2 2 3 3 4 5
8
0-1 Knapsack Recurrence for computing P
• The recurrence, for all k0M, j0N
• P(j, k) (case 1) P(j-1, k),
• if wjgtk, wj is weight of the j-th object
• else
• P(j, k) (case2) max P(j-1, k-wj)pj,
• P(j-1, k)
• Explanation for the formula is quite intuitive
• Recurrence terminates
• P(0, k)0, P(j, 0)0, for all ks and js

Objects (wt, p) (2, 1), (3, 2), (5, 3).
M9
w-gt 0 1 2 3 4 5 6 7 8
0 0 0 0 0 0 0 0 0
O1 0 0 1 1 1 1 1 1 1
O1 O2 0 0 1 2 2 3 3 3 3
O1 O2 O3 0 0 1 2 2 P(3,5)3
9
Recurrence -gt recursive algorithm
• Algorithm P(j, k)
• Input Set of n objects with (wi, pi) pairs, and
knapsack limit M
• Output Maximum profit P for subset of objects
with total wt ltM
• If j lt 0 or k lt 0 then return 0 //recursion
termination
• else
• if wjgtk then
• return P(j-1, k)
• else
• return maxP(j-1, k-Wj) pj, P(j-1, k)
• End algorithm.
• Driver call P(n, M), for given n objects KS
limit M.
• This is not table building Ps are recursive
functions

Complexity ?
Exponential O(2n)
Why?
Same reason as in Fibonacci number
calculation Repeats computing Ps
10
0-1 Knapsack Recursive call tree
• Objects (wt, p) (2, 1), (3, 2), (5, 3).
M8

P(3,8)
P(2,8-5)
P(2,8)
P(1,8-3)
P(1,3-3)
P(1,8)
P(1,3)
w1 2gt0, not called
P(0,8)
P(0,3-2)
P(0,3)
P(0,5-2)
P(0,5)
P(0,8-2)
P(0,0-2)
P(0,0)
Each leaf has w0, and P(0,) returns a value, 0
to the caller above
w-gt 0 1 2 3 4 5 6 7 8
0 0 0 0 0 0 0 0 0
O1 0 0 1 1 1 gt 1
O1 O2 0 0 1 2 2 3 3 gt 3
O1 O2 O3 0 0 1 2 2 3 3 4 5
11
0-1 Knapsack DP-algorithm
Do not repeat the same computation, store them in
a table and reuse
• Algorithm P(j, k)
• Input Set of n objects with (wi, pi) pairs, and
knapsack limit M
• Output Maximum profit P for a subset of objects
with total wt ltM
• For all k, P(0, k) 0 For all j, P(j, 0) 0
//initialize
• For j1 to n do
• For k 1 to M do
• if wjgtk then
• P(j, k) P(j-1, k)
• else
• P(j, k) maxP(j-1, k-Wj) pj, P(j-1, k)
• End loops and algorithm

Complexity O(nM), pseudo-polynomial because M is
an input value. If M30.5 the table would be of
size O(10nM), or if M30.54 it would be O(100NM).
12
0-1 Knapsack Problem (Example)
• Objects (wt, p) (2, 1), (3, 2), (5, 3).
M8

w-gt 0 1 2 3 4 5 6 7 8
0 0 0 0 0 0 0 0 0
O1 0 0 1 1 1 gt 1
O1 O2 0 0 1 2 2 3 3 gt 3
O1 O2 O3 0 0 1 2 2 3 3 4 5
2
0
-5 lbs
(23)
13
What if M9
• Objects (wt, p) (2, 1), (3, 2), (5, 3). M9

w-gt 0 1 2 3 4 5 6 7 8 9
0 0 0 0 0 0 0 0 0 0
O1 0 0 1 1 1 1 1 1 1 1
O1 O2 0 0 1 2 2 3 3 3 3 3
O1 O2 O3 0 0 1 2 2 3 3 4 5 5
-5 lbs
(23)
HOW TO FIND KNAPSACK CONTENT FROM TABLE? SPACE
COMPLEXITY?
14
Memoisation algorithm 0-1 knapsack
• Algorithm P(j, k)
• If j lt 0 or k lt 0 then return 0 //recursion
termination
• else
• if Wjgtk then
• yM(j-1, k) if ylt0 y P(j-1, k) M(j-1,
k)y // P( ) is a recursive call
• return y
• else
• xM(j-1, k-Wj) if xlt0 x P(j-1, k-Wj)
M(j-1, k-Wj)x
• yM(j-1, k) if ylt0 y P(j-1, k) M(j-1,
k)y
• M(j, k) maxx pj, y
• return maxx pj, y
• End algorithm.
• Driver Initialize a global matrix M(0-gtn, 0-gtM)
with -1 call P(n, M)
• Complexity ?

15
Problem 3 Ordering of Matrix-chain
Multiplications
• A chain of matrices to be multiplied ABCD,
• dimensions A (5x1), B(1x4), C(4x3), and D(3x6).
• Resulting matrix will be of size (5x6).
• scalar (or integer) multiplications for (BC) is
1.4.3,
• and the resulting matrixs dimension is (1x3), 1
column and 3 rows.

16
Ordering of Matrix-chain Multiplications
• A (5x1), B(1x4), C(4x3), and D(3x6)
• There are multiple ways to order the
multiplication (A(BC))D, A(B(CD)), (AB)(CD),
((AB)C)D, and A((BC)D)
• note Resulting matrix would be the same
• but the efficiency of calculation may vary
drastically
• Efficiency depends on scalar multiplications
• In the case of (A(BC))D, it is 1.4.3 5.1.3
5.3.6 117
• In the case (A(B(CD)), it is 4.3.6 1.4.6
5.1.6 126
• Our problem here is to find the best such
ordering
• An exhaustive search over all orderings is too
expensive - Catalan number involving n!

17
Recurrence for Ordering of Matrix-chain
Multiplications
• For a sequence A1...(Aleft....Aright)...An, we
want to find the optimal break point for the
parenthesized sequence.
• Calculate for ( r-l1) number of cases and find
the minimum min(Al...Ai) (Ai1... Ar), with l?
i lt r,
• Recurrence for optimum scalar-multiplication
• M(l, r)
• minM(l, i) M(i1, r) rowl .coli .colr,
• with l? i lt r.

r 1 2 3 4 lft 1 0 15 35(2) 69(2) 2 0 12
42 3 0 24 4 0
Sample M(i,j)
18
Matrix-chain Recursive algorithm
• Recursive Algorithm M(l, r)
• if rlt l then return 0 // Recurrence
termination no scalar-mult
• else
• return minM(l, i) M(i1, r) rowl.coli
.colr ,
• for l?iltr
• end algorithm.
• Driver call M(1, n) for the final answer.

19
Recurrence for Ordering of Matrix-chain
Multiplications to Bottom-up Computation
• Recurrence for optimum scalar-multiplication
• M(l, r)
• minM(l, i) M(i1, r) rowl .coli .colr,
• with l? i lt r.
• To compute M(l,r), you need M(l,i) and M(i1,r)
available for ALL is
• E.g., For M(3,9), you need M(3,3), M(4,9), and
M(3,4), M(5,9),
• Need to compute smaller size Ms first
• Gradually increase size from 1, 2, 3, , n

20
Strategy for Ordering of Bottom-up Computation
• Recurrence for optimum number of
scalar-multiplications
• M(l, r)
• minM(l, i) M(i1, r) rowl .coli .colr
,
• with l? i lt r.
• Compute by increasing size r-l11, r-l12,
r-l13, r-l1n
• Start the calculation at the lowest level with
two matrices, AB, BC, CD etc.
• Really?
• Where does the recurrence terminate?
• Then calculate for the triplets, ABC, BCD etc.
• And so on

0, for lr
21
Ordering of Matrix-chain Multiplications (Example)
• r 1 2 3 4
• l
• 1 0 15 35(2) 69(2)
• 2 0 12 42
• 3 0 24
• 4 0

Triplets
Pairs
Singlet's
22
Matrix-chain DP-algorithm
• Input list of pairwise dimensions of matrices
• Output optimum number of scalar multiplications
• for all 1? i ? n do M(i,i) 0 // diagonal
elements 0
• for size 2 to n do // size of subsequence
• for l 1 to n-size1 do
• r lsize-1 //move along diagonal
• M(l,r) infinity //minimizer
• for i l to r-1 do
• x M(l, i) M(i1, r)
• rowl .coli .colr
• if x lt M(l, r) then M(l, r) x
• End.
• // Complexities?

23
Ordering of Matrix-chain Multiplications (Example)
A1 (5x3), A2 (3x1), A3 (1x4), A4 (4x6).
• r 1 2 3 4
• l
• 1 0
• 2 0
• 3 0
• 4 0

15
35(2)
69(2)
12
42(?)
Triplets
24
Pairs
How do you find out the actual matrix ordering?
Calculation goes diagonally. COMPLEXITY?
24
DP Ordering of Matrix-chain Multiplications
(Example)
• A1 (5x3), A2 (3x1), A3 (1x4), A4 (4x6).
• M(1,1) M(2, 2) M(3, 3) M(4, 4) 0
• M(1, 2) M(1,1) M(2,2) 5.3.1 0 0 15.
• M(1,3) mini1 M(1,1)M(2,3)5.3.4, i2
M(1,2)M(3,3)5.1.4
• min72, 35 35(2)
• M(1,4) min i1 M(1,1)M(2,4)5.3.6, i2
M(1,2)M(3,4)5.1.6,
• i3 M(1,3)M(4,4)5.4.6
min132, 69, 155 69(2)
• 69 comes from the break-point i2
(A1.A2)(A3.A4)
• You may need to recursively break the sub-parts
too, by looking at which value of i gave the min
value at that stage, e.g., for (1,3) it was i2
(A1.A2)A3

25
Ordering of Matrix-chain Multiplications (Example)
A1 (5x3), A2 (3x1), A3 (1x4), A4 (4x6).
• r 1 2 3 4
• l
• 1 0 15 35(2) 69(2)
• 2 0 12 42
• 3 0 24
• 4 0

Triplets
Pairs
For a chain of n matrices, Table sizeO(n2),
computing for each entryO(n) COMPLEXITY O(n2
n) O(n3)
A separate matrix B(i,j) keeping track of optimum
i, for actual matrix ordering
26
Computing the actual break points B(i,j)
r 1 2 3 4 5 6 l 1 - - (2) (2) (3) (3) 2
- - (3) (3) (4) 3 - - (4) (4) 4 - (3) (4
) 5 - - - 6 - - -
Backtrack on this table (A1..A3)(A4A6) ABCDEF
-gt (ABC)(DEF)
Then (A1..A3) -gt A1(A2 A3), (A4A6)
-gt A4(A5 A6) ABCDEF -gt (ABC)(DEF) -gt ((AB)C)
(D(EF))
27
Inductive Proof of Matrix-chain Recurrence
• Induction base
• if rlt l then absurd case, there is no matrix to
multiply return 0
• If r l, then only one matrix, no
multiplication return 0
• Inductive hypothesis
• For all sizeltk, and k1, assume M(l,r) returns
correct optimum
• Note size r-l1
• Inductive step, for sizek
• Consider all possible ways to break up (lr)
chain, for l?iltr
• Make sure to compute the resulting pair of
matrices multiplication rowl.coli .colr
• Since M(l,i) and M(i1,r) are correct smaller
size optimums, as per hypothesis, minM(l, i)
M(i1, r) rowl.coli .colr , for l?iltr is the
correct return value for M(l,r)

28
Be careful with the correctness of the Recurrence
behind Dynamic Programming
• Inductive step
• Consider all possible ways to break up (lr)
chain, for l?iltr
• Make sure to compute the resulting pair of
matrices multiplication rowl.coli .colr
• Since M(l,i) and M(i1,r) are correct smaller
size optimums, per hypothesis, minM(l, i)
M(i1, r) rowl.coli .colr , for l?iltr is the
correct return value for M(l,r)
• It is NOT always possible to combine smaller
steps to the larger one.
• 43129 (((43) (12)) 9) , but
• average(4,3,1,2,9) av( av(av(4,3), av(1,2)),
9), may NOT be true
• DP needs the correct formulation of a Recurrence
first,
• Then, bottom up combination such that smaller
problems contributes to the larger ones

29
Problem 4 Optimal Binary Search Tree
• Binary Search Problem
• Input sorted objects, and key
• Output the keys index in the list, or not
found
• Binary Search on a Tree Sorted list is organized
as a binary tree
• Recursively each root t is l t r, where l is
left child and r is right child

Example 1, 3, 4, 7, 9, 10, 13, 15
1
7
3
4
10
3
7
9
13
4
2
9
10
13
15
15
30
Problem 4 Optimal Binary Search Tree
• Problem Optimal Binary-search Tree Organization
• Input Sorted list, with elements access
frequency (how many times to be accessed/searched
as a key over a period of time)
• Output Optimal binary tree organization so that
total cost is minimal
• Cost for accessing each element (frequency
access steps of the element)
• Access steps for each node is its distance from
the root, plus one.

Example list sorted by object order (index of A)
A1(7), A2(10), A3(5), A4(8), and A5(4)
A1(7)
A3(5)
A2(10)
A3(5)
A4(8)
A1(7)
A4(8)
A5(4)
A2(10)
A5(4)
Cost 51
72 103
82 43 77
Cost 71 102 53 84 45
94
31
Problem 4 Optimal Binary Search Tree
• Problem Optimal Binary-search Tree Organization
• Input Sorted list, with elements access
frequency (how many times to be accessed/searched
as a key over a period of time)
• Output Optimal binary tree organization so that
the total cost is minimal
• Every optimization problem optimizes an objective
function total access cost
• Different tree organizations will have different
aggregate access steps over all nodes,
• because the depths of the nodes are different on
different tree.
• Problem We want to find the optimal aggregate
access steps, and a bin-search tree organization
that produces it.
• What do you think the strategy should be?
• Which node should be at root, and which nodes at
leaves level?

32
Optimal Binary Search Tree (Continued)
• Say, the optimal cost for a sub-tree is C(left,
right).
• Call this part as tree t
• Note, when this sub-tree t is a branch of a root
at higher level
• under a parent,
• i.e. within another higher level sub-tree h
• then EACH nodes access step in t will increase
by 1 in the higher level subtree h

A1 A2 Aleft Ai Aright An
Ap
Ai
t
Al
Ar
33
A1 A2 Aleft Ai Aright An
fi
C(i1, right)
C(left, i-1)
Now C(l, i-1) and C(i1, r) is no longer useful
to compute C(l, r), unless
34
Recurrence for Optimal Binary Search Tree
• If the i-th node is chosen as the root for this
sub-tree, then
• C(l, r) minl? i ? r
• f(i) C(l,
i-1) C(i1, r)
• åjli-1 f(j) additional cost for left
sub-tree
• åji1r f(j) additional cost for right
sub-tree
• minl ? i ? r åjlr f(j) C(l, i-1)
C(i1, r)
• We will use the above formula to compute C(1, n).
• We will start from one element sub-tree (size1),
• and finish with n element full tree

35
Optimal Binary Search Tree (Continued)
• Like matrix-chain multiplication-ordering we will
develop a triangular part of the cost matrix (rgt
l), and
• we will develop it diagonally (r l size)
• Note that our boundary condition is different
now,
• c(l, r)0 if l gt r (meaningless cost)
• We start from, l r single node trees
• (not with lr-1 pairs of matrices, as in
matrix-chain-mult case).
• Also, i now goes from left through right, and
• i is excluded from both the subtrees Cs.

36
Write the DP algorithm for Optimal Binary-search
Tree Organization problem
r -gt 1 2 3 4 5
l1
l2 0
l3 0 0
l4 0 0 0
l5 0 0 0 0
37
Optimal Binary Search Tree (Example)
• Keys A1(7), A2(10), A3(5), A4(8), and A5(4).
• Diagonals C(1, 1)7, C(2, 2)10, .
• C(1,2)mink1 C(1,0)C(2,2)f1f201017,
• k2 C(1,1) C(3,2) f1f2 7 0 17
min27, 24 24 (i2)
• j 1 2 3 4 5
• i
• 1 7 24(2) 34 55 67
• 2 0 10 20 41 51
• 3 0 5 18 26
• 4 0 8 16
• 5 0 4
• This algorithm O(n3). However, O(n2) is
feasible.

38
Problem5 All Pairs Shortest Path
• A variation of Djikstras algorithm. Called
Floyd-Warshals algorithm. Good for dense graph.
• Algorithm Floyd
• Copy the distance matrix in d1..n1..n
• for k1 through n do //consider each vertex as
updating candidate
• for i1 through n do
• for j1 through n do
• if (dik dkj lt dij) then
• dij dik dkj
• pathij k // last updated via k
• End algorithm.
• O(n3), for 3 loops.

39
Problem 6 Gene Alignment
Score 9 matches x 1 1 mismatch x -1 1 gap x
-2 6
Find a best-score alignment
40
Problem 6 Gene Alignment
s1 s2 s3 sn t1 t2 t3 ..tm
41
Problem 6 Gene Alignment
Note, initialization, Or, Recurrence
termination Alignment start from
(n,m)-corner, Stops at (0,0) corner
42
Problem 6 Gene Alignment
43
Problem 6 Gene Alignment
? Global alignment
We may want Local alignment ?
44
Problem 6 Gene Alignment
We may want Local alignment ?
? No negative score
Initialize with 0s. Alignment, starts from
highest value on the table, And stops with a
zero