# CS 3343: Analysis of Algorithms - PowerPoint PPT Presentation

PPT – CS 3343: Analysis of Algorithms PowerPoint presentation | free to download - id: 796bb1-NjFkM

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## CS 3343: Analysis of Algorithms

Description:

### CS 3343: Analysis of Algorithms Lecture 2: Asymptotic Notations Outline Review of last lecture Order of growth Asymptotic notations Big O, big , How to express ... – PowerPoint PPT presentation

Number of Views:13
Avg rating:3.0/5.0
Slides: 58
Provided by: DavidL418
Category:
Tags:
Transcript and Presenter's Notes

Title: CS 3343: Analysis of Algorithms

1
CS 3343 Analysis of Algorithms
• Lecture 2 Asymptotic Notations

2
Outline
• Review of last lecture
• Order of growth
• Asymptotic notations
• Big O, big O, T

3
How to express algorithms?
• Nature language (e.g. English)
• Pseudocode
• Real programming languages

Increasing precision
Ease of expression
Describe the ideas of an algorithm in nature
language. Use pseudocode to clarify sufficiently
tricky details of the algorithm.
4
Insertion Sort
• InsertionSort(A, n) for j 2 to n

? Pre condition A1..j-1 is sorted
1. Find position i in A1..j-1 such that Ai
Aj lt Ai1 2. Insert Aj between Ai and
Ai1
? Post condition A1..j is sorted
j
1
sorted
5
Insertion Sort
• InsertionSort(A, n) for j 2 to n key
Aj i j - 1
• while (i gt 0) and (Ai gt key) Ai1
Ai i i 1
• Ai1 key

6
Use loop invariants to prove the correctness of
loops
• A loop invariant (LI) is a formal statement about
the variables in your program which holds true
throughout the loop
• Proof by induction
• Initialization the LI is true prior to the 1st
iteration
• Maintenance if the LI is true before the jth
iteration, it remains true before the (j1)th
iteration
• Termination when the loop terminates, the
invariant shows the correctness of the algorithm

7
Loop invariants and correctness of insertion sort
• Claim at the start of each iteration of the for
loop, the subarray consists of the elements
originally in A1..j-1 but in sorted order.
• Proof by induction

8
Prove correctness using loop invariants
• InsertionSort(A, n) for j 2 to n key
Aj i j - 1 ?Insert Aj into the sorted
sequence A1..j-1
• while (i gt 0) and (Ai gt key) Ai1
Ai i i 1
• Ai1 key

Loop invariant at the start of each iteration of
the for loop, the subarray A1..j-1 consists of
the elements originally in A1..j-1 but in
sorted order.
9
Initialization
• InsertionSort(A, n) for j 2 to n key
Aj i j - 1 ?Insert Aj into the sorted
sequence A1..j-1
• while (i gt 0) and (Ai gt key) Ai1
Ai i i 1
• Ai1 key

Subarray A1 is sorted. So loop invariant is
true before the loop starts.
10
Maintenance
• InsertionSort(A, n) for j 2 to n key
Aj i j - 1 ?Insert Aj into the sorted
sequence A1..j-1
• while (i gt 0) and (Ai gt key) Ai1
Ai i i 1
• Ai1 key

Assume loop variant is true prior to iteration j
Loop variant will be true before iteration j1
11
Termination
• InsertionSort(A, n) for j 2 to n key
Aj i j - 1 ?Insert Aj into the sorted
sequence A1..j-1
• while (i gt 0) and (Ai gt key) Ai1
Ai i i 1
• Ai1 key

The algorithm is correct!
Upon termination, A1..n contains all the
original elements of A in sorted order.
jn1
n
1
Sorted
12
Efficiency
• Correctness alone is not sufficient
• Brute-force algorithms exist for most problems
• E.g. use permutation to sort
• Problem too slow!
• How to measure efficiency?
• Accurate running time is not a good measure
• It depends on input, computer, and
implementation, etc.

13
Machine-independent
• A generic uniprocessor random-access machine
(RAM) model
• No concurrent operations
• Each simple operation (e.g. , -, , , if, for)
takes 1 step.
• Loops and subroutine calls are not simple
operations.
• All memory equally expensive to access
• Constant word size
• Unless we are explicitly manipulating bits

14
Analysis of insertion Sort
• InsertionSort(A, n) for j 2 to n key
Aj i j - 1 while (i gt 0) and (Ai gt key)
Ai1 Ai i i - 1 Ai1
key

How many times will this line execute?
15
Analysis of insertion Sort
• InsertionSort(A, n) for j 2 to n key
Aj i j - 1 while (i gt 0) and (Ai gt key)
Ai1 Ai i i - 1 Ai1
key

How many times will this line execute?
16
Analysis of insertion Sort
• Statement cost time__
• InsertionSort(A, n)
• for j 2 to n c1 n
• key Aj c2 (n-1)
• i j - 1 c3 (n-1)
• while (i gt 0) and (Ai gt key) c4 S
• Ai1 Ai c5 (S-(n-1))
• i i - 1 c6 (S-(n-1))
• 0
• Ai1 key c7 (n-1)
• 0
• S t2 t3 tn where tj is number of while
expression evaluations for the jth for loop
iteration

17
Analyzing Insertion Sort
• T(n) c1n c2(n-1) c3(n-1) c4S c5(S -
(n-1)) c6(S - (n-1)) c7(n-1) c8S
c9n c10
• What can S be?
• Best case -- inner loop body never executed
• tj 1 ? S n - 1
• T(n) an b is a linear function
• Worst case -- inner loop body executed for all
previous elements
• tj j ? S 2 3 n n(n1)/2 - 1
• T(n) an2 bn c is a quadratic function
• Average case
• Can assume that on average, we have to insert
Aj into the middle of A1..j-1, so tj j/2
• S n(n1)/4
• T(n) is still a quadratic function

T (n)
T (n2)
T (n2)
18
Analysis of insertion Sort
• Statement cost time__
• InsertionSort(A, n)
• for j 2 to n c1 n
• key Aj c2 (n-1)
• i j - 1 c3 (n-1)
• while (i gt 0) and (Ai gt key) c4 S
• Ai1 Ai c5 (S-(n-1))
• i i - 1 c6 (S-(n-1))
• 0
• Ai1 key c7 (n-1)
• 0

What are the basic operations (most executed
lines)?
19
Analysis of insertion Sort
• Statement cost time__
• InsertionSort(A, n)
• for j 2 to n c1 n
• key Aj c2 (n-1)
• i j - 1 c3 (n-1)
• while (i gt 0) and (Ai gt key) c4 S
• Ai1 Ai c5 (S-(n-1))
• i i - 1 c6 (S-(n-1))
• 0
• Ai1 key c7 (n-1)
• 0

20
Analysis of insertion Sort
• Statement cost time__
• InsertionSort(A, n)
• for j 2 to n c1 n
• key Aj c2 (n-1)
• i j - 1 c3 (n-1)
• while (i gt 0) and (Ai gt key) c4 S
• Ai1 Ai c5 (S-(n-1))
• i i - 1 c6 (S-(n-1))
• 0
• Ai1 key c7 (n-1)
• 0

21
What can S be?
Inner loop stops when Ai lt key, or i 0
i
j
1
Key
sorted
• S ? j2..n tj
• Best case
• Worst case
• Average case

22
Best case
Inner loop stops when Ai lt key, or i 0
i
j
1
Key
sorted
• Array already sorted
• S ? j2..n tj
• tj 1 for all j
• S n-1 T(n) T (n)

23
Worst case
Inner loop stops when Ai lt key
i
j
1
Key
sorted
• Array originally in reverse order sorted
• S ? j2..n tj
• tj j
• S ? j2..n j 2 3 n (n-1) (n2) / 2
T (n2)

24
Average case
Inner loop stops when Ai lt key
i
j
1
Key
sorted
• Array in random order
• S ? j2..n tj
• tj j / 2 on average
• S ? j2..n j/2 ½ ? j2..n j (n-1) (n2) / 4
T (n2)

What if we use binary search?
25
Asymptotic Analysis
• Running time depends on the size of the input
• Larger array takes more time to sort
• T(n) the time taken on input with size n
• Look at growth of T(n) as n?8.
• Asymptotic Analysis
• Size of input is generally defined as the number
of input elements
• In some cases may be tricky

26
Asymptotic Analysis
• Ignore actual and abstract statement costs
• Order of growth is the interesting measure
• Highest-order term is what counts
• As the input size grows larger it is the high
order term that dominates

27
Comparison of functions
log2n n nlog2n n2 n3 2n n!
10 3.3 10 33 102 103 103 106
102 6.6 102 660 104 106 1030 10158
103 10 103 104 106 109
104 13 104 105 108 1012
105 17 105 106 1010 1015
106 20 106 107 1012 1018
For a super computer that does 1 trillion
operations per second, it will be longer than 1
billion years
28
Order of growth
• 1 ltlt log2n ltlt n ltlt nlog2n ltlt n2 ltlt n3 ltlt 2n ltlt n!
• (We are slightly abusing of the ltlt sign. It
means a smaller order of growth).

29
Exact analysis is hard!
• Worst-case and average-case are difficult to deal
with precisely, because the details are very
complicated

It may be easier to talk about upper and lower
bounds of the function.
30
Asymptotic notations
• O Big-Oh
• O Big-Omega
• T Theta
• o Small-oh
• ? Small-omega

31
Big O
• Informally, O (g(n)) is the set of all functions
with a smaller or same order of growth as g(n),
within a constant multiple
• If we say f(n) is in O(g(n)), it means that g(n)
is an asymptotic upper bound of f(n)
• Intuitively, it is like f(n) g(n)
• What is O(n2)?
• The set of all functions that grow slower than or
in the same order as n2

32
• So
• n ? O(n2)
• n2 ? O(n2)
• 1000n ? O(n2)
• n2 n ? O(n2)
• 100n2 n ? O(n2)
• But
• 1/1000 n3 ? O(n2)

Intuitively, O is like
33
small o
• Informally, o (g(n)) is the set of all functions
with a strictly smaller growth as g(n), within a
constant multiple
• What is o(n2)?
• The set of all functions that grow slower than n2
• So
• 1000n ? o(n2)
• But
• n2 ? o(n2)

Intuitively, o is like lt
34
Big O
• Informally, O (g(n)) is the set of all functions
with a larger or same order of growth as g(n),
within a constant multiple
• f(n) ? O(g(n)) means g(n) is an asymptotic lower
bound of f(n)
• Intuitively, it is like g(n) f(n)
• So
• n2 ? O(n)
• 1/1000 n2 ? O(n)
• But
• 1000 n ? O(n2)

Intuitively, O is like
35
small ?
• Informally, ? (g(n)) is the set of all functions
with a larger order of growth as g(n), within a
constant multiple
• So
• n2 ? ?(n)
• 1/1000 n2 ? ?(n)
• n2 ? ?(n2)

Intuitively, ? is like gt
36
Theta (T)
• Informally, T (g(n)) is the set of all functions
with the same order of growth as g(n), within a
constant multiple
• f(n) ? T(g(n)) means g(n) is an asymptotically
tight bound of f(n)
• Intuitively, it is like f(n) g(n)
• What is T(n2)?
• The set of all functions that grow in the same
order as n2

37
• So
• n2 ? T(n2)
• n2 n ? T(n2)
• 100n2 n ? T(n2)
• 100n2 log2n ? T(n2)
• But
• nlog2n ? T(n2)
• 1000n ? T(n2)
• 1/1000 n3 ? T(n2)

Intuitively, T is like
38
Tricky cases
• How about sqrt(n) and log2 n?
• How about log2 n and log10 n
• How about 2n and 3n
• How about 3n and n!?

39
Big-Oh
• Definition
• O(g(n)) f(n) ?? positive constants c and
n0 such that 0 f(n) cg(n) ? nn0
• lim n?8 g(n)/f(n) gt 0 (if the limit exists.)
• Abuse of notation (for convenience)
• f(n) O(g(n)) actually means f(n) ? O(g(n))

40
Big-Oh
• Claim f(n) 3n2 10n 5 ? O(n2)
• Proof by definition
• (Hint to prove this claim by definition, we
need to find some positive constants c and n0
such that f(n) lt cn2 for all n n0.)
• (Note you just need to find one concrete
example of c and n0 satisfying the condition, but
it needs to be correct for all n n0. So do not
try to plug in a concrete value of n and show the
inequality holds.)
• Proof
• 3n2 10n 5 ? 3n2 10n2 5, ? n 1
• ? 3n2 10n2 5n2,? n 1
• ? 18 n2, ? n 1
• If we let c 18 and n0 1, we have f(n) ? c
n2, ? n n0.
• Therefore by definition, f(n) O(n2).

41
Big-Omega
• Definition
• O(g(n)) f(n) ?? positive constants c and n0
such that 0 cg(n) f(n) ? nn0
• lim n?8 f(n)/g(n) gt 0 (if the limit exists.)
• Abuse of notation (for convenience)
• f(n) O(g(n)) actually means f(n) ? O(g(n))

42
Big-Omega
• Claim f(n) n2 / 10 O(n)
• Proof by definition
• f(n) n2 / 10, g(n) n
• Need to find a c and a no to satisfy the
definition of f(n) ? O(g(n)), i.e., f(n) cg(n)
for n n0
• Proof
• n n2 / 10 when n 10
• If we let c 1 and n0 10, we have f(n) cn, ?
n n0. Therefore, by definition, n2 / 10 O(n).

43
Theta
• Definition
• T(g(n)) f(n) ?? positive constants c1, c2,
and n0 such that 0 ? c1 g(n) ? f(n) ? c2 g(n),
? n ? n0
• f(n) O(g(n)) and f(n) O(g(n))
• Abuse of notation (for convenience)
• f(n) T(g(n)) actually means f(n) ? T(g(n))
• T(1) means constant time.

44
Theta
• Claim f(n) 2n2 n T (n2)
• Proof by definition
• Need to find the three constants c1, c2, and n0
such that
• c1n2 2n2n c2n2 for all n n0
• A simple solution is c1 2, c2 3, and n0 1

45
More Examples
• Prove n2 3n lg n is in O(n2)
• Need to find c and n0 such that
• n2 3n lg n lt cn2 for n n0
• Proof
• n2 3n lg n lt n2 3n2 n for n 1
• lt n2 3n2 n2 for n 1
• lt 5n2 for n 1
• Therefore by definition n2 3n lg n ? O(n2).
• (Alternatively n2 3n lg n lt n2 n2 n2
for n 10
• lt 3n2 for n 10)

46
More Examples
• Prove n2 3n lg n is in O(n2)
• Want to find c and n0 such that
• n2 3n lg n gt cn2 for n n0
• n2 3n lg n gt n2 for n 1
• n2 3n lg n O(n2) and n2 3n lg n O
(n2)
• gt n2 3n lg n T(n2)

47
O, O, and T
The definitions imply a constant n0 beyond which
they are satisfied. We do not care about small
values of n.
48
Using limits to compare orders of growth
• 0
• lim f(n) / g(n) c gt 0
• 8

f(n) ? o(g(n))
f(n) ? O(g(n))
f(n) ? T (g(n))
n?8
f(n) ? O(g(n))
f(n) ? ? (g(n))
49
logarithms
• compare log2n and log10n
• logab logcb / logca
• log2n log10n / log102 3.3 log10n
• Therefore lim(log2n / log10 n) 3.3
• log2n T (log10n)

50
• Compare 2n and 3n
• lim 2n / 3n lim(2/3)n 0
• Therefore, 2n ? o(3n), and 3n ? ?(2n)
• How about 2n and 2n1?
• 2n / 2n1 ½, therefore 2n T (2n1)

n?8
n?8
51
L Hopitals rule
• You can apply this transformation as many times
as you want, as long as the condition holds

Condition If both lim f(n) and lim g(n) are ??
or 0
lim f(n) / g(n) lim f(n) / g(n)
n?8
n?8
52
• Compare n0.5 and log n
• lim n0.5 / log n ?
• (n0.5) 0.5 n-0.5
• (log n) 1 / n
• lim (n-0.5 / 1/n) lim(n0.5) 8
• Therefore, log n ? o(n0.5)
• In fact, log n ? o(ne), for any e gt 0

n?8
53
Stirlings formula
(constant)
54
• Compare 2n and n!
• Therefore, 2n o(n!)
• Compare nn and n!
• Therefore, nn ?(n!)
• How about log (n!)?

55
(No Transcript)
56
More advanced dominance ranking
57
Asymptotic notations
• O Big-Oh
• O Big-Omega
• T Theta
• o Small-oh
• ? Small-omega
• Intuitively
• O is like ?
• o is like lt
• ? is like ?
• ? is like gt
• ? is like