Loading...

PPT – ALGORITHM TYPES PowerPoint presentation | free to download - id: 68522b-M2I0Z

The Adobe Flash plugin is needed to view this content

ALGORITHM TYPES

- Greedy, Divide and Conquer, Dynamic Programming,

and Backtracking - Note the general strategy from the examples
- The classification is neither exhaustive (there

may be other types), nor mutually exclusive

(one may combine)

PROBLEM 1 DYNAMIC PROGRAMMING STRATEGY

- In case the divide and conquer strategy can

divide the problem at a very small level, and

there are repetition of some calculations over

some components - One can apply a bottom up approach calculate the

smaller components first and then keep combining

them until the highest level of the problem is

solved. - Draw the recursion tree of Fibonacci-series

calculation, you will see example of such

repetitive calculations - f(n) f(n-1) f(n-2), for ngt1 and f(n)1

otherwise - fib(n) calculation
- n 1, 2, 3, 4, 5
- fib 1, 2, 3, 5, 8

DYNAMIC PROGRAMMING STRATEGY (Continued)

- Recursive fib(n)
- if (nlt1) return 1
- else return (fib(n-1) fib(n-2)).
- Time complexity exponential, O(kn) for some

kgt1.0 - Iterative fibonacci(n)
- fib(0) fib(1) 1
- for i2 through n do
- fib(i) fib(i-1) fib(i-2)
- end for
- return fib(n).
- Time complexity O(n), Space complexity O(n)

DYNAMIC PROGRAMMING STRATEGY (Continued)

- SpaceSaving-fibonacci(n)
- if (nlt1) return 1
- int last1, last2last1, result1
- for i2 through n do
- result last last2last
- last2last last
- lastresult
- end for
- return result.
- Time complexity O(n), Space complexity O(1)

DYNAMIC PROGRAMMING STRATEGY (Continued)

- The recursive call recalculates fib(1) 5 times,

fib(2) 3 times, fib(3) 2 times - in fib(5)

calculation. The complexity is exponential. - In iterative calculation we avoid repetition by

storing the needed values in variables -

complexity of order n. - Dynamic Programming approach consumes more memory

to store the result of calculations of lower

levels for the purpose of calculating the next

higher level. They are typically stored in a

table.

Problem 2 0-1 Knapsack Problem (not in the book)

- Given a set of objects with (Weight, Profit)

pair, and a Knapsack of limited weight capacity

(M), find a subset of objects for the knapsack to

maximize total profit P - Sample Problem
- Objects (wt, p) (2, 1), (3, 2), (9, 20), (5,

3). M8 - Exhaustive Algorithm Try all subset of objects.
- How many?

total(0 lbs, 0) (2,1) total (2lbs,

1) (9,20) Illegal, total wtgt8

(2,1), (3,2) (5lbs, 3)

Total possibilities 2n For each object

present or absent 0 or 1 A bit string is a

good representation for a set, e.g.

000101100. n-bits

0-1 Knapsack Problem

- DP uses a representation for optimal profit P(j,

k) - for the first j objects considered
- (in any arbitrary ordering of objects)
- with a variable knapsack limit k
- Develop the table for P(j, k)
- with rows j 1 n (objects), and
- for each row, go from k 0 M (the KS limit)
- Finally P(n, M) gets the result

w-gt 0 1 2 3 4 5 6 7 8

0 0 0 0 0 0 0 0 0

O1 0 0 1 1 1 gt 1

O1 O2 0 0 1 2 2 3 3 gt 3

O1 O2 O3 0 0 1 2 2 3 3 4 5

0-1 Knapsack Recurrence for computing P

- The recurrence, for all k0M, j0N
- P(j, k) (case 1) P(j-1, k),
- if wjgtk, wj is weight of the j-th object
- else
- P(j, k) (case2) max P(j-1, k-wj)pj,
- P(j-1, k)
- Explanation for the formula is quite intuitive
- Recurrence terminates
- P(0, k)0, P(j, 0)0, for all ks and js

Objects (wt, p) (2, 1), (3, 2), (5, 3).

M9

w-gt 0 1 2 3 4 5 6 7 8

0 0 0 0 0 0 0 0 0

O1 0 0 1 1 1 1 1 1 1

O1 O2 0 0 1 2 2 3 3 3 3

O1 O2 O3 0 0 1 2 2 P(3,5)3

Recurrence -gt recursive algorithm

- Algorithm P(j, k)
- Input Set of n objects with (wi, pi) pairs, and

knapsack limit M - Output Maximum profit P for subset of objects

with total wt ltM - If j lt 0 or k lt 0 then return 0 //recursion

termination - else
- if wjgtk then
- return P(j-1, k)
- else
- return maxP(j-1, k-Wj) pj, P(j-1, k)
- End algorithm.
- Driver call P(n, M), for given n objects KS

limit M. - This is not table building Ps are recursive

functions

Complexity ?

Exponential O(2n)

Why?

Same reason as in Fibonacci number

calculation Repeats computing Ps

0-1 Knapsack Recursive call tree

- Objects (wt, p) (2, 1), (3, 2), (5, 3).

M8

P(3,8)

P(2,8-5)

P(2,8)

P(1,8-3)

P(1,3-3)

P(1,8)

P(1,3)

w1 2gt0, not called

P(0,8)

P(0,3-2)

P(0,3)

P(0,5-2)

P(0,5)

P(0,8-2)

P(0,0-2)

P(0,0)

Each leaf has w0, and P(0,) returns a value, 0

to the caller above

w-gt 0 1 2 3 4 5 6 7 8

0 0 0 0 0 0 0 0 0

O1 0 0 1 1 1 gt 1

O1 O2 0 0 1 2 2 3 3 gt 3

O1 O2 O3 0 0 1 2 2 3 3 4 5

0-1 Knapsack DP-algorithm

Do not repeat the same computation, store them in

a table and reuse

- Algorithm P(j, k)
- Input Set of n objects with (wi, pi) pairs, and

knapsack limit M - Output Maximum profit P for a subset of objects

with total wt ltM - For all k, P(0, k) 0 For all j, P(j, 0) 0

//initialize - For j1 to n do
- For k 1 to M do
- if wjgtk then
- P(j, k) P(j-1, k)
- else
- P(j, k) maxP(j-1, k-Wj) pj, P(j-1, k)
- End loops and algorithm

Complexity O(nM), pseudo-polynomial because M is

an input value. If M30.5 the table would be of

size O(10nM), or if M30.54 it would be O(100NM).

0-1 Knapsack Problem (Example)

- Objects (wt, p) (2, 1), (3, 2), (5, 3).

M8

w-gt 0 1 2 3 4 5 6 7 8

0 0 0 0 0 0 0 0 0

O1 0 0 1 1 1 gt 1

O1 O2 0 0 1 2 2 3 3 gt 3

O1 O2 O3 0 0 1 2 2 3 3 4 5

2

0

-5 lbs

(23)

What if M9

- Objects (wt, p) (2, 1), (3, 2), (5, 3). M9

w-gt 0 1 2 3 4 5 6 7 8 9

0 0 0 0 0 0 0 0 0 0

O1 0 0 1 1 1 1 1 1 1 1

O1 O2 0 0 1 2 2 3 3 3 3 3

O1 O2 O3 0 0 1 2 2 3 3 4 5 5

-5 lbs

(23)

HOW TO FIND KNAPSACK CONTENT FROM TABLE? SPACE

COMPLEXITY?

Memoisation algorithm 0-1 knapsack

- Algorithm P(j, k)
- If j lt 0 or k lt 0 then return 0 //recursion

termination - else
- if Wjgtk then
- yM(j-1, k) if ylt0 y P(j-1, k) M(j-1,

k)y // P( ) is a recursive call - return y
- else
- xM(j-1, k-Wj) if xlt0 x P(j-1, k-Wj)

M(j-1, k-Wj)x - yM(j-1, k) if ylt0 y P(j-1, k) M(j-1,

k)y - M(j, k) maxx pj, y
- return maxx pj, y
- End algorithm.
- Driver Initialize a global matrix M(0-gtn, 0-gtM)

with -1 call P(n, M) - Complexity ?

Problem 3 Ordering of Matrix-chain

Multiplications

- A chain of matrices to be multiplied ABCD,
- dimensions A (5x1), B(1x4), C(4x3), and D(3x6).

- Resulting matrix will be of size (5x6).
- scalar (or integer) multiplications for (BC) is

1.4.3, - and the resulting matrixs dimension is (1x3), 1

column and 3 rows.

Ordering of Matrix-chain Multiplications

- A (5x1), B(1x4), C(4x3), and D(3x6)
- There are multiple ways to order the

multiplication (A(BC))D, A(B(CD)), (AB)(CD),

((AB)C)D, and A((BC)D) - note Resulting matrix would be the same
- but the efficiency of calculation may vary

drastically - Efficiency depends on scalar multiplications
- In the case of (A(BC))D, it is 1.4.3 5.1.3

5.3.6 117 - In the case (A(B(CD)), it is 4.3.6 1.4.6

5.1.6 126 - Our problem here is to find the best such

ordering - An exhaustive search over all orderings is too

expensive - Catalan number involving n!

Recurrence for Ordering of Matrix-chain

Multiplications

- For a sequence A1...(Aleft....Aright)...An, we

want to find the optimal break point for the

parenthesized sequence. - Calculate for ( r-l1) number of cases and find

the minimum min(Al...Ai) (Ai1... Ar), with l?

i lt r, - Recurrence for optimum scalar-multiplication
- M(l, r)
- minM(l, i) M(i1, r) rowl .coli .colr,
- with l? i lt r.

r 1 2 3 4 lft 1 0 15 35(2) 69(2) 2 0 12

42 3 0 24 4 0

Sample M(i,j)

Matrix-chain Recursive algorithm

- Recursive Algorithm M(l, r)
- if rlt l then return 0 // Recurrence

termination no scalar-mult - else
- return minM(l, i) M(i1, r) rowl.coli

.colr , - for l?iltr
- end algorithm.
- Driver call M(1, n) for the final answer.

Recurrence for Ordering of Matrix-chain

Multiplications to Bottom-up Computation

- Recurrence for optimum scalar-multiplication
- M(l, r)
- minM(l, i) M(i1, r) rowl .coli .colr,
- with l? i lt r.
- To compute M(l,r), you need M(l,i) and M(i1,r)

available for ALL is - E.g., For M(3,9), you need M(3,3), M(4,9), and

M(3,4), M(5,9), - Need to compute smaller size Ms first
- Gradually increase size from 1, 2, 3, , n

Strategy for Ordering of Bottom-up Computation

- Recurrence for optimum number of

scalar-multiplications - M(l, r)
- minM(l, i) M(i1, r) rowl .coli .colr

, - with l? i lt r.
- Compute by increasing size r-l11, r-l12,

r-l13, r-l1n - Start the calculation at the lowest level with

two matrices, AB, BC, CD etc. - Really?
- Where does the recurrence terminate?
- Then calculate for the triplets, ABC, BCD etc.
- And so on

0, for lr

Ordering of Matrix-chain Multiplications (Example)

- r 1 2 3 4
- l
- 1 0 15 35(2) 69(2)
- 2 0 12 42
- 3 0 24
- 4 0

Triplets

Pairs

Singlet's

Matrix-chain DP-algorithm

- Input list of pairwise dimensions of matrices
- Output optimum number of scalar multiplications
- for all 1? i ? n do M(i,i) 0 // diagonal

elements 0 - for size 2 to n do // size of subsequence
- for l 1 to n-size1 do
- r lsize-1 //move along diagonal
- M(l,r) infinity //minimizer
- for i l to r-1 do
- x M(l, i) M(i1, r)
- rowl .coli .colr
- if x lt M(l, r) then M(l, r) x
- End.
- // Complexities?

Ordering of Matrix-chain Multiplications (Example)

A1 (5x3), A2 (3x1), A3 (1x4), A4 (4x6).

- r 1 2 3 4
- l
- 1 0
- 2 0
- 3 0
- 4 0

15

35(2)

69(2)

12

42(?)

Triplets

24

Pairs

How do you find out the actual matrix ordering?

Calculation goes diagonally. COMPLEXITY?

DP Ordering of Matrix-chain Multiplications

(Example)

- A1 (5x3), A2 (3x1), A3 (1x4), A4 (4x6).
- M(1,1) M(2, 2) M(3, 3) M(4, 4) 0
- M(1, 2) M(1,1) M(2,2) 5.3.1 0 0 15.
- M(1,3) mini1 M(1,1)M(2,3)5.3.4, i2

M(1,2)M(3,3)5.1.4 - min72, 35 35(2)
- M(1,4) min i1 M(1,1)M(2,4)5.3.6, i2

M(1,2)M(3,4)5.1.6, - i3 M(1,3)M(4,4)5.4.6

min132, 69, 155 69(2) - 69 comes from the break-point i2

(A1.A2)(A3.A4) - You may need to recursively break the sub-parts

too, by looking at which value of i gave the min

value at that stage, e.g., for (1,3) it was i2

(A1.A2)A3

Ordering of Matrix-chain Multiplications (Example)

A1 (5x3), A2 (3x1), A3 (1x4), A4 (4x6).

- r 1 2 3 4
- l
- 1 0 15 35(2) 69(2)
- 2 0 12 42
- 3 0 24
- 4 0

Triplets

Pairs

For a chain of n matrices, Table sizeO(n2),

computing for each entryO(n) COMPLEXITY O(n2

n) O(n3)

A separate matrix B(i,j) keeping track of optimum

i, for actual matrix ordering

Computing the actual break points B(i,j)

r 1 2 3 4 5 6 l 1 - - (2) (2) (3) (3) 2

- - (3) (3) (4) 3 - - (4) (4) 4 - (3) (4

) 5 - - - 6 - - -

Backtrack on this table (A1..A3)(A4A6) ABCDEF

-gt (ABC)(DEF)

Then (A1..A3) -gt A1(A2 A3), (A4A6)

-gt A4(A5 A6) ABCDEF -gt (ABC)(DEF) -gt ((AB)C)

(D(EF))

Inductive Proof of Matrix-chain Recurrence

- Induction base
- if rlt l then absurd case, there is no matrix to

multiply return 0 - If r l, then only one matrix, no

multiplication return 0 - Inductive hypothesis
- For all sizeltk, and k1, assume M(l,r) returns

correct optimum - Note size r-l1
- Inductive step, for sizek
- Consider all possible ways to break up (lr)

chain, for l?iltr - Make sure to compute the resulting pair of

matrices multiplication rowl.coli .colr - Since M(l,i) and M(i1,r) are correct smaller

size optimums, as per hypothesis, minM(l, i)

M(i1, r) rowl.coli .colr , for l?iltr is the

correct return value for M(l,r)

Be careful with the correctness of the Recurrence

behind Dynamic Programming

- Inductive step
- Consider all possible ways to break up (lr)

chain, for l?iltr - Make sure to compute the resulting pair of

matrices multiplication rowl.coli .colr - Since M(l,i) and M(i1,r) are correct smaller

size optimums, per hypothesis, minM(l, i)

M(i1, r) rowl.coli .colr , for l?iltr is the

correct return value for M(l,r) - It is NOT always possible to combine smaller

steps to the larger one. - Addition, Multiplication are associative
- 43129 (((43) (12)) 9) , but
- average(4,3,1,2,9) av( av(av(4,3), av(1,2)),

9), may NOT be true - DP needs the correct formulation of a Recurrence

first, - Then, bottom up combination such that smaller

problems contributes to the larger ones

Problem 4 Optimal Binary Search Tree

- Binary Search Problem
- Input sorted objects, and key
- Output the keys index in the list, or not

found - Binary Search on a Tree Sorted list is organized

as a binary tree - Recursively each root t is l t r, where l is

left child and r is right child

Example 1, 3, 4, 7, 9, 10, 13, 15

1

7

3

4

10

3

7

9

13

4

2

9

10

13

15

15

Problem 4 Optimal Binary Search Tree

- Problem Optimal Binary-search Tree Organization
- Input Sorted list, with elements access

frequency (how many times to be accessed/searched

as a key over a period of time) - Output Optimal binary tree organization so that

total cost is minimal - Cost for accessing each element (frequency

access steps of the element) - Access steps for each node is its distance from

the root, plus one.

Example list sorted by object order (index of A)

A1(7), A2(10), A3(5), A4(8), and A5(4)

A1(7)

A3(5)

A2(10)

A3(5)

A4(8)

A1(7)

A4(8)

A5(4)

A2(10)

A5(4)

Cost 51

72 103

82 43 77

Cost 71 102 53 84 45

94

Problem 4 Optimal Binary Search Tree

- Problem Optimal Binary-search Tree Organization
- Input Sorted list, with elements access

frequency (how many times to be accessed/searched

as a key over a period of time) - Output Optimal binary tree organization so that

the total cost is minimal - Every optimization problem optimizes an objective

function total access cost - Different tree organizations will have different

aggregate access steps over all nodes, - because the depths of the nodes are different on

different tree. - Problem We want to find the optimal aggregate

access steps, and a bin-search tree organization

that produces it. - What do you think the strategy should be?
- Which node should be at root, and which nodes at

leaves level?

Optimal Binary Search Tree (Continued)

- Say, the optimal cost for a sub-tree is C(left,

right). - Call this part as tree t
- Note, when this sub-tree t is a branch of a root

at higher level - under a parent,
- i.e. within another higher level sub-tree h
- then EACH nodes access step in t will increase

by 1 in the higher level subtree h

A1 A2 Aleft Ai Aright An

Ap

Ai

t

Al

Ar

A1 A2 Aleft Ai Aright An

fi

C(i1, right)

C(left, i-1)

Now C(l, i-1) and C(i1, r) is no longer useful

to compute C(l, r), unless

Recurrence for Optimal Binary Search Tree

- If the i-th node is chosen as the root for this

sub-tree, then - C(l, r) minl? i ? r
- f(i) C(l,

i-1) C(i1, r) - åjli-1 f(j) additional cost for left

sub-tree - åji1r f(j) additional cost for right

sub-tree - minl ? i ? r åjlr f(j) C(l, i-1)

C(i1, r) - We will use the above formula to compute C(1, n).

- We will start from one element sub-tree (size1),
- gradually increase size,
- and finish with n element full tree

Optimal Binary Search Tree (Continued)

- Like matrix-chain multiplication-ordering we will

develop a triangular part of the cost matrix (rgt

l), and - we will develop it diagonally (r l size)
- Note that our boundary condition is different

now, - c(l, r)0 if l gt r (meaningless cost)
- We start from, l r single node trees
- (not with lr-1 pairs of matrices, as in

matrix-chain-mult case). - Also, i now goes from left through right, and

- i is excluded from both the subtrees Cs.

Write the DP algorithm for Optimal Binary-search

Tree Organization problem

r -gt 1 2 3 4 5

l1

l2 0

l3 0 0

l4 0 0 0

l5 0 0 0 0

Optimal Binary Search Tree (Example)

- Keys A1(7), A2(10), A3(5), A4(8), and A5(4).
- Diagonals C(1, 1)7, C(2, 2)10, .
- C(1,2)mink1 C(1,0)C(2,2)f1f201017,
- k2 C(1,1) C(3,2) f1f2 7 0 17

min27, 24 24 (i2) - j 1 2 3 4 5
- i
- 1 7 24(2) 34 55 67
- 2 0 10 20 41 51
- 3 0 5 18 26
- 4 0 8 16
- 5 0 4
- This algorithm O(n3). However, O(n2) is

feasible.

Problem5 All Pairs Shortest Path

- A variation of Djikstras algorithm. Called

Floyd-Warshals algorithm. Good for dense graph. - Algorithm Floyd
- Copy the distance matrix in d1..n1..n
- for k1 through n do //consider each vertex as

updating candidate - for i1 through n do
- for j1 through n do
- if (dik dkj lt dij) then
- dij dik dkj
- pathij k // last updated via k
- End algorithm.
- O(n3), for 3 loops.

Problem 6 Gene Alignment

Score 9 matches x 1 1 mismatch x -1 1 gap x

-2 6

Find a best-score alignment

Problem 6 Gene Alignment

s1 s2 s3 sn t1 t2 t3 ..tm

Problem 6 Gene Alignment

Note, initialization, Or, Recurrence

termination Alignment start from

(n,m)-corner, Stops at (0,0) corner

Problem 6 Gene Alignment

Problem 6 Gene Alignment

? Global alignment

We may want Local alignment ?

Problem 6 Gene Alignment

We may want Local alignment ?

? No negative score

Initialize with 0s. Alignment, starts from

highest value on the table, And stops with a

zero