Loading...

PPT – CS 3343: Analysis of Algorithms PowerPoint presentation | free to download - id: 796bb1-NjFkM

The Adobe Flash plugin is needed to view this content

CS 3343 Analysis of Algorithms

- Lecture 2 Asymptotic Notations

Outline

- Review of last lecture
- Order of growth
- Asymptotic notations
- Big O, big O, T

How to express algorithms?

- Nature language (e.g. English)
- Pseudocode
- Real programming languages

Increasing precision

Ease of expression

Describe the ideas of an algorithm in nature

language. Use pseudocode to clarify sufficiently

tricky details of the algorithm.

Insertion Sort

- InsertionSort(A, n) for j 2 to n

? Pre condition A1..j-1 is sorted

1. Find position i in A1..j-1 such that Ai

Aj lt Ai1 2. Insert Aj between Ai and

Ai1

? Post condition A1..j is sorted

j

1

sorted

Insertion Sort

- InsertionSort(A, n) for j 2 to n key

Aj i j - 1 - while (i gt 0) and (Ai gt key) Ai1

Ai i i 1 - Ai1 key

Use loop invariants to prove the correctness of

loops

- A loop invariant (LI) is a formal statement about

the variables in your program which holds true

throughout the loop - Proof by induction
- Initialization the LI is true prior to the 1st

iteration - Maintenance if the LI is true before the jth

iteration, it remains true before the (j1)th

iteration - Termination when the loop terminates, the

invariant shows the correctness of the algorithm

Loop invariants and correctness of insertion sort

- Claim at the start of each iteration of the for

loop, the subarray consists of the elements

originally in A1..j-1 but in sorted order. - Proof by induction

Prove correctness using loop invariants

- InsertionSort(A, n) for j 2 to n key

Aj i j - 1 ?Insert Aj into the sorted

sequence A1..j-1 - while (i gt 0) and (Ai gt key) Ai1

Ai i i 1 - Ai1 key

Loop invariant at the start of each iteration of

the for loop, the subarray A1..j-1 consists of

the elements originally in A1..j-1 but in

sorted order.

Initialization

- InsertionSort(A, n) for j 2 to n key

Aj i j - 1 ?Insert Aj into the sorted

sequence A1..j-1 - while (i gt 0) and (Ai gt key) Ai1

Ai i i 1 - Ai1 key

Subarray A1 is sorted. So loop invariant is

true before the loop starts.

Maintenance

- InsertionSort(A, n) for j 2 to n key

Aj i j - 1 ?Insert Aj into the sorted

sequence A1..j-1 - while (i gt 0) and (Ai gt key) Ai1

Ai i i 1 - Ai1 key

Assume loop variant is true prior to iteration j

Loop variant will be true before iteration j1

Termination

- InsertionSort(A, n) for j 2 to n key

Aj i j - 1 ?Insert Aj into the sorted

sequence A1..j-1 - while (i gt 0) and (Ai gt key) Ai1

Ai i i 1 - Ai1 key

The algorithm is correct!

Upon termination, A1..n contains all the

original elements of A in sorted order.

jn1

n

1

Sorted

Efficiency

- Correctness alone is not sufficient
- Brute-force algorithms exist for most problems
- E.g. use permutation to sort
- Problem too slow!
- How to measure efficiency?
- Accurate running time is not a good measure
- It depends on input, computer, and

implementation, etc.

Machine-independent

- A generic uniprocessor random-access machine

(RAM) model - No concurrent operations
- Each simple operation (e.g. , -, , , if, for)

takes 1 step. - Loops and subroutine calls are not simple

operations. - All memory equally expensive to access
- Constant word size
- Unless we are explicitly manipulating bits

Analysis of insertion Sort

- InsertionSort(A, n) for j 2 to n key

Aj i j - 1 while (i gt 0) and (Ai gt key)

Ai1 Ai i i - 1 Ai1

key

How many times will this line execute?

Analysis of insertion Sort

- InsertionSort(A, n) for j 2 to n key

Aj i j - 1 while (i gt 0) and (Ai gt key)

Ai1 Ai i i - 1 Ai1

key

How many times will this line execute?

Analysis of insertion Sort

- Statement cost time__
- InsertionSort(A, n)
- for j 2 to n c1 n
- key Aj c2 (n-1)
- i j - 1 c3 (n-1)
- while (i gt 0) and (Ai gt key) c4 S
- Ai1 Ai c5 (S-(n-1))
- i i - 1 c6 (S-(n-1))
- 0
- Ai1 key c7 (n-1)
- 0
- S t2 t3 tn where tj is number of while

expression evaluations for the jth for loop

iteration

Analyzing Insertion Sort

- T(n) c1n c2(n-1) c3(n-1) c4S c5(S -

(n-1)) c6(S - (n-1)) c7(n-1) c8S

c9n c10 - What can S be?
- Best case -- inner loop body never executed
- tj 1 ? S n - 1
- T(n) an b is a linear function
- Worst case -- inner loop body executed for all

previous elements - tj j ? S 2 3 n n(n1)/2 - 1
- T(n) an2 bn c is a quadratic function
- Average case
- Can assume that on average, we have to insert

Aj into the middle of A1..j-1, so tj j/2 - S n(n1)/4
- T(n) is still a quadratic function

T (n)

T (n2)

T (n2)

Analysis of insertion Sort

- Statement cost time__
- InsertionSort(A, n)
- for j 2 to n c1 n
- key Aj c2 (n-1)
- i j - 1 c3 (n-1)
- while (i gt 0) and (Ai gt key) c4 S
- Ai1 Ai c5 (S-(n-1))
- i i - 1 c6 (S-(n-1))
- 0
- Ai1 key c7 (n-1)
- 0

What are the basic operations (most executed

lines)?

Analysis of insertion Sort

- Statement cost time__
- InsertionSort(A, n)
- for j 2 to n c1 n
- key Aj c2 (n-1)
- i j - 1 c3 (n-1)
- while (i gt 0) and (Ai gt key) c4 S
- Ai1 Ai c5 (S-(n-1))
- i i - 1 c6 (S-(n-1))
- 0
- Ai1 key c7 (n-1)
- 0

Analysis of insertion Sort

- Statement cost time__
- InsertionSort(A, n)
- for j 2 to n c1 n
- key Aj c2 (n-1)
- i j - 1 c3 (n-1)
- while (i gt 0) and (Ai gt key) c4 S
- Ai1 Ai c5 (S-(n-1))
- i i - 1 c6 (S-(n-1))
- 0
- Ai1 key c7 (n-1)
- 0

What can S be?

Inner loop stops when Ai lt key, or i 0

i

j

1

Key

sorted

- S ? j2..n tj
- Best case
- Worst case
- Average case

Best case

Inner loop stops when Ai lt key, or i 0

i

j

1

Key

sorted

- Array already sorted
- S ? j2..n tj
- tj 1 for all j
- S n-1 T(n) T (n)

Worst case

Inner loop stops when Ai lt key

i

j

1

Key

sorted

- Array originally in reverse order sorted
- S ? j2..n tj
- tj j
- S ? j2..n j 2 3 n (n-1) (n2) / 2

T (n2)

Average case

Inner loop stops when Ai lt key

i

j

1

Key

sorted

- Array in random order
- S ? j2..n tj
- tj j / 2 on average
- S ? j2..n j/2 ½ ? j2..n j (n-1) (n2) / 4

T (n2)

What if we use binary search?

Answer still T(n2)

Asymptotic Analysis

- Running time depends on the size of the input
- Larger array takes more time to sort
- T(n) the time taken on input with size n
- Look at growth of T(n) as n?8.
- Asymptotic Analysis
- Size of input is generally defined as the number

of input elements - In some cases may be tricky

Asymptotic Analysis

- Ignore actual and abstract statement costs
- Order of growth is the interesting measure
- Highest-order term is what counts
- As the input size grows larger it is the high

order term that dominates

Comparison of functions

log2n n nlog2n n2 n3 2n n!

10 3.3 10 33 102 103 103 106

102 6.6 102 660 104 106 1030 10158

103 10 103 104 106 109

104 13 104 105 108 1012

105 17 105 106 1010 1015

106 20 106 107 1012 1018

For a super computer that does 1 trillion

operations per second, it will be longer than 1

billion years

Order of growth

- 1 ltlt log2n ltlt n ltlt nlog2n ltlt n2 ltlt n3 ltlt 2n ltlt n!
- (We are slightly abusing of the ltlt sign. It

means a smaller order of growth).

Exact analysis is hard!

- Worst-case and average-case are difficult to deal

with precisely, because the details are very

complicated

It may be easier to talk about upper and lower

bounds of the function.

Asymptotic notations

- O Big-Oh
- O Big-Omega
- T Theta
- o Small-oh
- ? Small-omega

Big O

- Informally, O (g(n)) is the set of all functions

with a smaller or same order of growth as g(n),

within a constant multiple - If we say f(n) is in O(g(n)), it means that g(n)

is an asymptotic upper bound of f(n) - Intuitively, it is like f(n) g(n)
- What is O(n2)?
- The set of all functions that grow slower than or

in the same order as n2

- So
- n ? O(n2)
- n2 ? O(n2)
- 1000n ? O(n2)
- n2 n ? O(n2)
- 100n2 n ? O(n2)
- But
- 1/1000 n3 ? O(n2)

Intuitively, O is like

small o

- Informally, o (g(n)) is the set of all functions

with a strictly smaller growth as g(n), within a

constant multiple - What is o(n2)?
- The set of all functions that grow slower than n2
- So
- 1000n ? o(n2)
- But
- n2 ? o(n2)

Intuitively, o is like lt

Big O

- Informally, O (g(n)) is the set of all functions

with a larger or same order of growth as g(n),

within a constant multiple - f(n) ? O(g(n)) means g(n) is an asymptotic lower

bound of f(n) - Intuitively, it is like g(n) f(n)
- So
- n2 ? O(n)
- 1/1000 n2 ? O(n)
- But
- 1000 n ? O(n2)

Intuitively, O is like

small ?

- Informally, ? (g(n)) is the set of all functions

with a larger order of growth as g(n), within a

constant multiple - So
- n2 ? ?(n)
- 1/1000 n2 ? ?(n)
- n2 ? ?(n2)

Intuitively, ? is like gt

Theta (T)

- Informally, T (g(n)) is the set of all functions

with the same order of growth as g(n), within a

constant multiple - f(n) ? T(g(n)) means g(n) is an asymptotically

tight bound of f(n) - Intuitively, it is like f(n) g(n)
- What is T(n2)?
- The set of all functions that grow in the same

order as n2

- So
- n2 ? T(n2)
- n2 n ? T(n2)
- 100n2 n ? T(n2)
- 100n2 log2n ? T(n2)
- But
- nlog2n ? T(n2)
- 1000n ? T(n2)
- 1/1000 n3 ? T(n2)

Intuitively, T is like

Tricky cases

- How about sqrt(n) and log2 n?
- How about log2 n and log10 n
- How about 2n and 3n
- How about 3n and n!?

Big-Oh

- Definition
- O(g(n)) f(n) ?? positive constants c and

n0 such that 0 f(n) cg(n) ? nn0 - lim n?8 g(n)/f(n) gt 0 (if the limit exists.)
- Abuse of notation (for convenience)
- f(n) O(g(n)) actually means f(n) ? O(g(n))

Big-Oh

- Claim f(n) 3n2 10n 5 ? O(n2)
- Proof by definition
- (Hint to prove this claim by definition, we

need to find some positive constants c and n0

such that f(n) lt cn2 for all n n0.) - (Note you just need to find one concrete

example of c and n0 satisfying the condition, but

it needs to be correct for all n n0. So do not

try to plug in a concrete value of n and show the

inequality holds.) - Proof
- 3n2 10n 5 ? 3n2 10n2 5, ? n 1
- ? 3n2 10n2 5n2,? n 1
- ? 18 n2, ? n 1
- If we let c 18 and n0 1, we have f(n) ? c

n2, ? n n0. - Therefore by definition, f(n) O(n2).

Big-Omega

- Definition
- O(g(n)) f(n) ?? positive constants c and n0

such that 0 cg(n) f(n) ? nn0 - lim n?8 f(n)/g(n) gt 0 (if the limit exists.)
- Abuse of notation (for convenience)
- f(n) O(g(n)) actually means f(n) ? O(g(n))

Big-Omega

- Claim f(n) n2 / 10 O(n)
- Proof by definition
- f(n) n2 / 10, g(n) n
- Need to find a c and a no to satisfy the

definition of f(n) ? O(g(n)), i.e., f(n) cg(n)

for n n0 - Proof
- n n2 / 10 when n 10
- If we let c 1 and n0 10, we have f(n) cn, ?

n n0. Therefore, by definition, n2 / 10 O(n).

Theta

- Definition
- T(g(n)) f(n) ?? positive constants c1, c2,

and n0 such that 0 ? c1 g(n) ? f(n) ? c2 g(n),

? n ? n0 - f(n) O(g(n)) and f(n) O(g(n))
- Abuse of notation (for convenience)
- f(n) T(g(n)) actually means f(n) ? T(g(n))
- T(1) means constant time.

Theta

- Claim f(n) 2n2 n T (n2)
- Proof by definition
- Need to find the three constants c1, c2, and n0

such that - c1n2 2n2n c2n2 for all n n0
- A simple solution is c1 2, c2 3, and n0 1

More Examples

- Prove n2 3n lg n is in O(n2)
- Need to find c and n0 such that
- n2 3n lg n lt cn2 for n n0
- Proof
- n2 3n lg n lt n2 3n2 n for n 1
- lt n2 3n2 n2 for n 1
- lt 5n2 for n 1
- Therefore by definition n2 3n lg n ? O(n2).
- (Alternatively n2 3n lg n lt n2 n2 n2

for n 10 - lt 3n2 for n 10)

More Examples

- Prove n2 3n lg n is in O(n2)
- Want to find c and n0 such that
- n2 3n lg n gt cn2 for n n0
- n2 3n lg n gt n2 for n 1
- n2 3n lg n O(n2) and n2 3n lg n O

(n2) - gt n2 3n lg n T(n2)

O, O, and T

The definitions imply a constant n0 beyond which

they are satisfied. We do not care about small

values of n.

Using limits to compare orders of growth

- 0
- lim f(n) / g(n) c gt 0
- 8

f(n) ? o(g(n))

f(n) ? O(g(n))

f(n) ? T (g(n))

n?8

f(n) ? O(g(n))

f(n) ? ? (g(n))

logarithms

- compare log2n and log10n
- logab logcb / logca
- log2n log10n / log102 3.3 log10n
- Therefore lim(log2n / log10 n) 3.3
- log2n T (log10n)

- Compare 2n and 3n
- lim 2n / 3n lim(2/3)n 0
- Therefore, 2n ? o(3n), and 3n ? ?(2n)
- How about 2n and 2n1?
- 2n / 2n1 ½, therefore 2n T (2n1)

n?8

n?8

L Hopitals rule

- You can apply this transformation as many times

as you want, as long as the condition holds

Condition If both lim f(n) and lim g(n) are ??

or 0

lim f(n) / g(n) lim f(n) / g(n)

n?8

n?8

- Compare n0.5 and log n
- lim n0.5 / log n ?
- (n0.5) 0.5 n-0.5
- (log n) 1 / n
- lim (n-0.5 / 1/n) lim(n0.5) 8
- Therefore, log n ? o(n0.5)
- In fact, log n ? o(ne), for any e gt 0

n?8

Stirlings formula

(constant)

- Compare 2n and n!
- Therefore, 2n o(n!)
- Compare nn and n!
- Therefore, nn ?(n!)
- How about log (n!)?

(No Transcript)

More advanced dominance ranking

Asymptotic notations

- O Big-Oh
- O Big-Omega
- T Theta
- o Small-oh
- ? Small-omega
- Intuitively

- O is like ?
- o is like lt

- ? is like ?
- ? is like gt

- ? is like