Analysis of Algorithm Efficiency

1
Analysis of Algorithm Efficiency

• Dr. Yingwu Zhu
• p5-11, p16-29, p43-53, p93-96

2
What is Algorithm?

• Any well-defined computational procedure that
• takes input some value or a set of values
• and
• produces output some value or a set of values
• Example sorting problems
• Input a sequence of numbers a1, a2, , an
• Output a permutation b1, b2, , bn of the
  input s.t. b1ltb2ltltbn

3
Algorithms Motivation

• At the heart of programs lie algorithms
• Algorithms as a key technology
• Think about
• mapping/navigation
• Google search
• word processing (spelling correction, layout)
• content delivery and streaming video
• games (graphics, rendering)
• big data (querying, learning)

4
Algorithms Motivation

• In a perfect world
• for each problem we would have an algorithm
• the algorithm would be the fastest possible
• What would CS look like in this world?

5
Algorithms Motivation

• Our world (fortunately) is not so perfect
• for many problems we know embarrassingly little
  about what the fastest algorithm is
• multiplying two integers or two matrices
• factoring an integer into primes
• determining shortest tour of given n cities
• for many problems we suspect fast algorithms are
  impossible (NP-complete problems)
• for some problems we have unexpected and clever
  algorithms (we will see many of these)

6
Analyzing Algorithms

• Given computing resources and input data
• how fast does an algorithm run?
• Time efficiency amount of time required to
  accomplish the task
• Our focus
• How much memory is required?
• Space efficiency amount of memory required
• Deals with the extra space the algorithm requires

7
Time Efficiency

• Time efficiency depends on
• size of input
• speed of machine
• quality of source code
• quality of compiler

So, we cannot express time efficiency
meaningfully in real time units such as seconds!
8
Empirical analysis of time efficiency

• Select a specific (typical) sample of inputs
• Use physical unit of time (e.g., milliseconds)
• or
• Count actual number of basic operations
  executions
• Analyze the empirical data
• Limitation results dependent on the particular
  computer and the sample of inputs

9
Theoretical analysis of time efficiency

• Time efficiency is analyzed by determining the
  number of repetitions of the basic operation as a
  function of input size
• Basic operation the operation that contributes
  most towards the running time of the algorithm
• T(n) copC(n)

10
Input size and basic operation examples
Problem Input size measure Basic operation
Searching for key in a list of n items Number of lists items, i.e. n Key comparison
Multiplication of two matrices Matrix dimensions or total number of elements Multiplication of two numbers
Checking primality of a given integer n nsize number of digits (in binary representation) Division
Typical graph problem vertices and/or edges Visiting a vertex or traversing an edge
11
Time Efficiency

• T(n) (approximated by) number of times the
  basic operation is executed.
• Not only depends on the input size n, but also
  depends on the arrangement of the input items
• Best case not informative
• Average case difficult to determine
• Worst case is used to measure an algorithms
  performance

12
Example Sequential Search

• Best case T(n)
• Worst case T(n)
• Average case T(n)
• Assume success search probability of p

13
Order of Growth

• Established framework for analyzing T(n)
• Order of growth as n?8
• Highest-order term is what counts
• Remember, we are doing asymptotic analysis
• As the input size grows larger it is the high
  order term that dominates
• disregard lower-order terms in running time
• disregard coefficient on highest order term
• Asymptotic notations ?, O, ?

14
Asymptotic Order of Growth

• A way of comparing functions that ignores
  constant factors and small input sizes
• O(g(n)) class of functions f(n) that grow no
  faster than g(n)
• T(g(n)) class of functions f(n) that grow at
  same rate as g(n)
• O(g(n)) class of functions f(n) that grow at
  least as fast as g(n)

15
Big-oh
16
Big-omega
17
Big-theta
18
Upper Bound Notation

• In general a function
• f(n) is O(g(n)) if there exist positive constants
  c and n0 such that f(n) ? c ? g(n) for all n ? n0
• Order of growth of f(n) ? order of growth of g(n)
  (within constant multiple)
• Formally
• O(g(n)) f(n) ? positive constants c and n0
  such that f(n) ? c ? g(n) ? n ? n0

19
Big-Oh

• Examples
• 10n is O(n2)
• 5n20 is O(n)

20
An Example Insertion Sort

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

21
An Example Insertion Sort
i ? j ? key ?Aj ? Aj1 ?
30
10
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

22
An Example Insertion Sort
i 2 j 1 key 10Aj 30 Aj1 10
30
10
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

23
An Example Insertion Sort
i 2 j 1 key 10Aj 30 Aj1 30
30
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

24
An Example Insertion Sort
i 2 j 1 key 10Aj 30 Aj1 30
30
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

25
An Example Insertion Sort
i 2 j 0 key 10Aj ? Aj1 30
30
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

26
An Example Insertion Sort
i 2 j 0 key 10Aj ? Aj1 30
30
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

27
An Example Insertion Sort
i 2 j 0 key 10Aj ? Aj1 10
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

28
An Example Insertion Sort
i 3 j 0 key 10Aj ? Aj1 10
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

29
An Example Insertion Sort
i 3 j 0 key 40Aj ? Aj1 10
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

30
An Example Insertion Sort
i 3 j 0 key 40Aj ? Aj1 10
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

31
An Example Insertion Sort
i 3 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

32
An Example Insertion Sort
i 3 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

33
An Example Insertion Sort
i 3 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

34
An Example Insertion Sort
i 4 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

35
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

36
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

37
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 20
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

38
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 20
10
30
40
20
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

39
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 40
10
30
40
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

40
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 40
10
30
40
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

41
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 40
10
30
40
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

42
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

43
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

44
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 30
10
30
30
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

45
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 30
10
30
30
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

46
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 30
10
30
30
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

47
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 30
10
30
30
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

48
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 20
10
20
30
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

49
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 20
10
20
30
40
1
2
3
4

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

Done!
50
Insertion Sort

• InsertionSort(A, n) for i 2 to n key
  Ai j i - 1 while (j gt 0) and (Aj gt key)
  Aj1 Aj j j - 1 Aj1
  key

How many times will this loop execute?
51
T(n) for Insertion Sort

• Worst case?
• Best case?

52
Insertion Sort Is O(n2)

• Proof
• Suppose runtime is an2 bn c
• If any of a, b, and c are less than 0 replace
  the constant with its absolute value
• an2 bn c ? (a b c)n2 (a b c)n (a
  b c)
• ? 3(a b c)n2 for n ? 1
• Let c 3(a b c) and let n0 1
• Question
• Is InsertionSort O(n3)?
• Is InsertionSort O(n)?

53
Big O Fact

• A polynomial of degree k is O(nk)
• Proof
• Suppose f(n) bknk bk-1nk-1 b1n b0
• Let ai bi
• f(n) ? aknk ak-1nk-1 a1n a0

54
Lower Bound Notation

• In general a function
• f(n) is ?(g(n)) if ? positive constants c and n0
  such that 0 ? c?g(n) ? f(n) ? n ? n0
• Proof
• Suppose run time is an b
• Assume a and b are positive (what if b is
  negative?)
• an ? an b
• Example n3 ?(n2)

55
Asymptotic Tight Bound

• A function f(n) is ?(g(n)) if ? positive
  constants c1, c2, and n0 such that c1 g(n) ?
  f(n) ? c2 g(n) ? n ? n0
• Theorem
• f(n) is ?(g(n)) iff f(n) is both O(g(n)) and
  ?(g(n))
• Proof
• Engineering
• Drop low-order items ignore leading constants
• Example 3n3 90n2 -5n 9999 ?(n3)

56
Using limits for comparing orders of growth
T(n) has a smaller order of growth than g(n)
T(n) has the same order of growth than g(n)
T(n) has a larger order of growth than g(n)
Example compare the orders of growth of
and n2
57
Practical Complexity
58
Practical Complexity
59
Practical Complexity
60
Practical Complexity
61
Properties of Big-O

• f(n) ? O(f(n))
• f(n) ? O(g(n)) iff g(n) ??(f(n))
• If f (n) ? O(g (n)) and g(n) ? O(h(n)) , then
  f(n) ? O(h(n)) Note similarity with a b
• If f1(n) ? O(g1(n)) and f2(n) ? O(g2(n)) , then
• f1(n) f2(n) ? O(maxg1(n),
  g2(n))

62
Basic asymptotic efficiency classes
1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial
63
Summary Analysis for Algorithms

• Focus on worst case
• well-suited to rigorous analysis, simple
• Measure time/space complexity using asymptotic
  notation (big-oh notation)
• disregard lower-order terms in running time
• disregard coefficient on highest order term
• Non-recursive algorithms
• Recursive algorithms

64
T(n) for nonrecursive algorithms

• General Plan for Analysis
• Decide on parameter n indicating input size
• Identify algorithms basic operation
• Determine worst, average, and best cases for
  input of size n
• Set up a sum for the number of times the basic
  operation is executed
• Simplify the sum using standard formulas and
  rules
• Disregard low-order terms coefficient on the
  highest-order term

65
Useful summation formulas and rules

• ?l?i?u1 111 u - l 1
• In particular, ?1?i?n1 n - 1 1 n ?
  ?(n)
• ?1?i?n i 12n n(n1)/2 ? n2/2 ? ?(n2)
• ?1?i?n i2 1222n2 n(n1)(2n1)/6 ? n3/3 ?
  ?(n3)
• ?0?i?n ai 1 a an (an1 - 1)/(a - 1)
  for any a ? 1
• In particular, ?0?i?n 2i 20 21
  2n 2n1 - 1 ? ?(2n )
• ?(ai bi ) ?ai ?bi ?cai c?ai
  ?l?i?uai ?l?i?mai ?m1?i?uai

66
Example 1 Maximum element
67
Example 2 Element uniqueness problem
68
Example 3 Matrix multiplication
69
Plan for Analysis of Recursive Algorithms

• Decide on a parameter indicating an inputs
  size.
• Identify the algorithms basic operation.
• Check whether the number of times the basic op.
  is executed may vary on different inputs of the
  same size. (If it may, the worst, average, and
  best cases must be investigated separately.)
• Set up a recurrence relation with an appropriate
  initial condition expressing the number of times
  the basic op. is executed.
• Solve the recurrence (or, at the very least,
  establish its solutions order of growth) by
  backward substitutions or another method.

70
Example 1 Recursive evaluation of n!

• Definition n ! 1 ? 2 ? ?(n-1) ? n for n 1
  and 0! 1
• Recursive definition of n! F(n) F(n-1) ? n
  for n 1 and
• 
  F(0) 1
• Size
• Basic operation
• Recurrence relation

71
Example 2 The Tower of Hanoi Puzzle

• 
  

Recurrence for number of moves
n disks of different sizes and 3 pegs the goal
is to move all the disks to the third peg using
the 2nd one as an auxiliary if necessary. Move
on disk at a time and donot place a larger disk
on top of a smaller one!
72
The Tower of Hanoi Puzzle

• T(n) 2T(n-1) 1
• T(1) 1

73
Example 3 Binary Search
74
Master Theorem

• T(n) aT(n/b) f (n), where f(n) ? ?(nd),
  d ? 0
• Master Theorem
  If a lt bd, T(n) ? ?(nd)
• If a bd,
  T(n) ? ?(nd log n)
• If a gt bd,
  T(n) ? ?(nlog ba )