ITCS6114: Algorithms and Data Structures

1
ITCS6114 Algorithms and Data Structures

• Introduction
• Proof by Induction
• Asymptotic Notation

2
The Course

• Purpose a rigorous introduction to the design
  and analysis of algorithms
• Not a lab or programming course
• Not a math course, either
• Suggested Textbooks
• Introduction to The Design Analysis of
  Algorithms, Anany Levitin
• Introduction to Algorithms, Cormen, Leiserson,
  Rivest, Stein

3
The Course

• Instructor Zbigniew W. Ras
• ras_at_uncc.edu
• Office Woodward Hall 430C
• Office hours Tuesday 1200-130pm
• TA Ayman Hajja     
• ahajja_at_uncc.edu Office Woodward Hall 404
  (Future Computing Lab)
• Office hours Tuesday, Thursday 1130-130pm

4
The Course

• Grading policy
• Midterm 30 points
• Project 30 points
• Final 30 points
• Participation 10 points
• Grade A from 86 to 100 points, Grade B from 71 to
  85 points, Grade C from 56 to 70.

5
Review Induction

• Suppose
• S(k) is true for fixed constant k
• Often k 0
• S(n) ? S(n1) for all n gt k
• Then S(n) is true for all n gt k

6
Proof By Induction

• Claim S(n) is true for all n gt k
• Basis
• Show formula is true when n k
• Inductive hypothesis
• Assume formula is true for an arbitrary n
• Step
• Show that formula is then true for n1

7
Induction Example Gaussian Closed Form

• Prove 1 2 3 n n(n1) / 2
• Basis
• If n 0, then 0 0(01) / 2
• Inductive hypothesis
• Assume 1 2 3 n n(n1) / 2
• Step (show true for n1)
• 1 2 n n1 (1 2 n) (n1)
• n(n1)/2 n1 n(n1) 2(n1)/2
• (n1)(n2)/2 (n1)(n1 1) / 2

8
Induction ExampleGeometric Closed Form

• Prove a0 a1 an (an1 - 1)/(a - 1) for
  all a ? 1
• Basis show that a0 (a01 - 1)/(a - 1)
• a0 1 (a1 - 1)/(a - 1)
• Inductive hypothesis
• Assume a0 a1 an (an1 - 1)/(a - 1)
• Step (show true for n1)
• a0 a1 an1 a0 a1 an an1
• (an1 - 1)/(a - 1) an1 (an11 - 1)/(a - 1)

9
Induction

• Weve been using weak induction
• Strong induction also holds
• Basis show S(0)
• Hypothesis assume S(k) holds for arbitrary k lt
  n
• Step Show S(n1) follows
• Another variation
• Basis show S(0), S(1)
• Hypothesis assume S(n) and S(n1) are true
• Step show S(n2) follows

10
Asymptotic Performance

• In this course, we care most about asymptotic
  performance
• How does the algorithm behave as the problem size
  gets very large?
• Running time
• Memory/storage requirements

11
Asymptotic Notation

• By now you should have an intuitive feel for
  asymptotic (big-O) notation
• What does O(n) running time mean? O(n2)?O(n lg
  n)?
• How does asymptotic running time relate to
  asymptotic memory usage?
• Our first task is to define this notation more
  formally and completely

12
Input Size

• Time and space complexity
• This is generally a function of the input size
• E.g., sorting, multiplication
• How we characterize input size depends
• Sorting number of input items
• Multiplication total number of bits
• Graph algorithms number of nodes edges
• Etc

13
Analysis

• Worst case
• Provides an upper bound on running time
• An absolute guarantee
• Average case
• Provides the expected running time
• Very useful, but treat with care what is
  average?
• Random (equally likely) inputs
• Real-life inputs

14
An Example Insertion Sort
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 14
8/14/2015
15
An Example Insertion Sort
i ? j ? key ?Aj ? Aj1 ?
30
10
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 15
8/14/2015
16
An Example Insertion Sort
i 2 j 1 key 10Aj 30 Aj1 10
30
10
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 16
8/14/2015
17
An Example Insertion Sort
i 2 j 1 key 10Aj 30 Aj1 30
30
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 17
8/14/2015
18
An Example Insertion Sort
i 2 j 1 key 10Aj 30 Aj1 30
30
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 18
8/14/2015
19
An Example Insertion Sort
i 2 j 0 key 10Aj ? Aj1 30
30
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 19
8/14/2015
20
An Example Insertion Sort
i 2 j 0 key 10Aj ? Aj1 30
30
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 20
8/14/2015
21
An Example Insertion Sort
i 2 j 0 key 10Aj ? Aj1 10
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 21
8/14/2015
22
An Example Insertion Sort
i 3 j 0 key 10Aj ? Aj1 10
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 22
8/14/2015
23
An Example Insertion Sort
i 3 j 0 key 40Aj ? Aj1 10
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 23
8/14/2015
24
An Example Insertion Sort
i 3 j 0 key 40Aj ? Aj1 10
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 24
8/14/2015
25
An Example Insertion Sort
i 3 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 25
8/14/2015
26
An Example Insertion Sort
i 3 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 26
8/14/2015
27
An Example Insertion Sort
i 3 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 27
8/14/2015
28
An Example Insertion Sort
i 4 j 2 key 40Aj 30 Aj1 40
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 28
8/14/2015
29
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 29
8/14/2015
30
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 30
8/14/2015
31
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 20
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 31
8/14/2015
32
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 20
10
30
40
20
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 32
8/14/2015
33
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 40
10
30
40
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 33
8/14/2015
34
An Example Insertion Sort
i 4 j 3 key 20Aj 40 Aj1 40
10
30
40
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 34
8/14/2015
35
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 35
8/14/2015
36
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 40
10
30
40
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 36
8/14/2015
37
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 30
10
30
30
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 37
8/14/2015
38
An Example Insertion Sort
i 4 j 2 key 20Aj 30 Aj1 30
10
30
30
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 38
8/14/2015
39
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 30
10
30
30
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 39
8/14/2015
40
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 30
10
30
30
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 40
8/14/2015
41
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 20
10
20
30
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
David Luebke 41
8/14/2015
42
An Example Insertion Sort
i 4 j 1 key 20Aj 10 Aj1 20
10
20
30
40
1
2
3
4
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
Done!
David Luebke 42
8/14/2015
43
Animating Insertion Sort

• Check out the Animator, a java applet at
  http//www.sorting-algorithms.com/insertion-sort
• Try it out with random, ascending, and descending
  inputs

David Luebke 43
8/14/2015
44
Insertion Sort
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt key)
Aj1 Aj j j - 1 Aj1
key
How many times will this loop execute?
David Luebke 44
8/14/2015
45
Analysis

• Simplifications
• Ignore actual and abstract statement costs
• Order of growth is the interesting measure
• Highest-order term is what counts
• Remember, we are doing asymptotic analysis
• As the input size grows larger it is the high
  order term that dominates

David Luebke 45
8/14/2015
46
Upper Bound Notation

• We say InsertionSorts run time is O(n2)
• Properly we should say run time is in O(n2)
• Read O as Big-O (youll also hear it as
  order)
• In general a function
• f(n) is O(g(n)) if there exist positive constants
  c and n0 such that f(n) ? c ? g(n) for all n ? n0
• Formally
• O(g(n)) f(n) ? positive constants c and n0
  such that f(n) ? c ? g(n) ? n ? n0

David Luebke 46
8/14/2015
47
Insertion Sort Is O(n2)

• Proof
• Suppose runtime is an2 bn c
• If any of a, b, and c are less than 0 replace
  the constant with its absolute value
• an2 bn c ? (a b c)n2 (a b c)n (a
  b c)
• ? 3(a b c)n2 for n ? 1
• Let c 3(a b c) and let n0 1
• Question
• Is InsertionSort O(n3)?
• Is InsertionSort O(n)?

David Luebke 47
8/14/2015
48
Lower Bound Notation

• We say InsertionSorts run time is ?(n)
• In general a function
• f(n) is ?(g(n)) if ? positive constants c and n0
  such that 0 ? c?g(n) ? f(n) ? n ? n0
• Proof
• Suppose run time is an b
• Assume a and b are positive (what if b is
  negative?)
• an ? an b

David Luebke 48
8/14/2015
49
Asymptotic Tight Bound

• A function f(n) is ?(g(n)) if ? positive
  constants c1, c2, and n0 such that c1 g(n) ?
  f(n) ? c2 g(n) ? n ? n0
• Theorem
• f(n) is ?(g(n)) iff f(n) is both O(g(n)) and
  ?(g(n))

David Luebke 49
8/14/2015
50
Practical Complexity
David Luebke 50
8/14/2015
51
Practical Complexity
David Luebke 51
8/14/2015
52
Practical Complexity
David Luebke 52
8/14/2015
53
Practical Complexity
David Luebke 53
8/14/2015
54
Practical Complexity
David Luebke 54
8/14/2015
55
Other Asymptotic Notations

• A function f(n) is o(g(n)) if ? positive
  constants c and n0 such that f(n) lt c g(n) ? n
  ? n0
• A function f(n) is ?(g(n)) if ? positive
  constants c and n0 such that c g(n) lt f(n) ? n
  ? n0
• Intuitively,

• o() is like lt
• O() is like ?

• ?() is like gt
• ?() is like ?

• ?() is like

David Luebke 55
8/14/2015