Analysis of Algorithms

Aaron Tan

http//www.comp.nus.edu.sg/tantc/cs1101.html

Introduction to Analysis of Algorithms

- After you have read and studied this chapter, you

should be able to - Know what is analysis of algorithms (complexity

analysis) - Know the definition and uses of big-O notation
- How to analyse the running time of an algorithm

Introduction (1/2)

- Two aspects on writing efficient codes
- Programming techniques
- Implementation of algorithms
- Practitioners viewpoint
- Asymptotic analysis (big-O notation, etc.)
- Analysis and design of algorithms
- Theoreticians viewpoint

Asymptotic analysis keeps the students head in

the clouds, while attention to implementation

details keeps his feet on the grounds.

Introduction (2/2)

- Programming techniques versus Algorithm design

int f1 (int n) int a, sum0 for (a1

altn a) sum a return sum

Sum of Two Elements (1/5)

- Given this problem
- A sorted list of integers list and a value is

given. Write a program to find the indices of

(any) two distinct elements in the list whose sum

is equal to the given value. - Example
- list 2, 3, 8, 12, 15, 19, 22, 24
- sum 23
- answer elements 8 (at subscript 2) and 15 (at

subscript 4)

Sum of Two Elements (2/5)

- Algorithm A
- list 2, 3, 8, 12, 15, 19, 22, 24
- sum 23
- answer elements 8 (at subscript 2) and 15 (at

subscript 4)

n size of list for x from 0 to n 2 for y

from x 1 to n 1 if ((listx

listy) sum) then found! (answers

are x and y)

Sum of Two Elements (3/5)

- Code for algorithm A

import java.util. class SumOfTwoA

public static void main(String args)

Scanner scanner new Scanner(System.in)

int list 2, 3, 8, 12, 15, 19, 22, 24

int n list.length

System.out.print("Enter sum ") int sum

scanner.nextInt() boolean found

false for (int x 0 x lt n-1 !found

x) for (int y x1 y lt n

!found y) if (listx

listy sum)

System.out.println("Indices at " x " and "

y) found true

Sum of Two Elements (4/5)

- Algorithm A used nested loop and scans some

elements many times. - Algorithm B can you use a single loop to examine

each element at most once? - If this can be done, it will be more efficient

than Algorithm A.

Sum of Two Elements (5/5)

- Code for algorithm B

The Race

100 metres ahead

Who will reach the finishing line first?

Complexity Analysis (1/2)

- Complexity analysis to measure and predict the

behavior (running time, storage space) of an

algorithm. - We will focus on running time here.
- Inexact, but provides a good basis for

comparisons. - We want to have a good judgment on how an

algorithm will perform if the problem size gets

very big.

Complexity Analysis (2/2)

- Problem size is defined based on the problem on

hand. - Examples of problem size
- Number of elements in an array (for sorting

problems). - Length of the strings in an anagram problem.
- Number of discs in the Tower of Hanoi problem.
- Number of cities in the Traveling Salesman

Problem (TSP).

Definition (1/4)

- Assume problem size is n and T(n) is the running

time. - Upper bound Big-O notation
- Definition
- T(n) O(f(n)) if there are constants c and n0
- such that T(n) ? cf(n) when n ? n0
- We read the equal sign as is a member of (?),

because O(f(n)) is a set of functions. - We may also say that T(n) is bounded above by

f(n).

Definition (2/4)

Definition (3/4)

- The functions relative rates of growth are

compared. - For instance, compare f(n) n2 with g(n)

1000n. - Although at some points f(n) is smaller than

g(n), f(n) actually grows at a faster rate than

g(n). (Hence, an algorithm with running time

complexity of f(n) is slower than another with

running time complexity of g(n) in this example.) - Hence, g(n) O(f(n)).
- The definition says that eventually there is some

point n0 past which cf(n) is always larger or

equal to g(n). - Here, we can make c 1 and n0 1000.

Definition (4/4)

- Besides the big-O (upper bound) analysis, there

are the Omega ? (lower bound) analysis, the Theta

? (tight bounds) analysis, and others.

Exercises (1/4)

- f(n) 1 2 3 n
- Show that f(n) O(n2)
- Proof
- 1 2 3 n n(n1)/2
- n2/2 n/2
- ? n2/2 n2/2
- n2
- The above is the running time of basic sorting

algorithms such as bubblesort, insertion sort,

selection sort.

Exercises (2/4)

- f(n) 17 n n/3
- Show that f(n) O(n)
- Proof 17 n n/3 ? 3n O(n)
- f(n) n4 n2 20n 100
- Show that f(n) O(n4)
- Proof n4 n2 20n 100 ? 4n4 O(n4)
- From the two examples above, it can be seen that

an expression is dominated by the term of the

highest degree.

Exercises (3/4)

- Tower of Hanoi
- Algorithm
- Tower(n, source, temp, dest)
- if (n gt 0)
- tower(n-1, source, dest, temp)
- move disc from source to dest
- tower(n-1, temp, source, dest)
- Let T(n) number of move to solve a tower of

n discs.

Exercises (4/4)

- Tower of Hanoi (cont.)
- Let T(n) number of move to solve a tower of n

discs. - Prove that T(n) 2n 1.
- T(0) 0
- T(n) T(n 1) 1 T(n 1)
- 2 ? T(n 1) 1
- 2 ? (2n1 1) 1
- 2 ? 2n1 2 1
- 2n 1
- Hence T(n) O(2n)

Some Common Series

The Conversation

- Boss Your program is too slow! Rewrite it!
- You But why? All we need to do is to buy faster

computer!

Is this really the solution?

Complexity Classes (1/4)

- There are some common complexity classes.
- In analysis of algorithm, log refers to log2, or

sometimes written simply as lg.

Notation Name

O(1) Constant.

O(log n) Logarithmic.

O(n) Linear.

O(n log n) Linearithmic, loglinear, quasilinear or supralinear.

O(n2) Quadratic.

O(nc), c gt 1 Polynomial, sometimes called algebraic. Examples O(n2), O(n3), O(n4).

O(cn) Exponential, sometimes called geometric. Examples O(2n), O(3n).

O(n!) Factorial, sometimes called combinatorial.

n is problem size.

Complexity Classes (2/4)

- Algorithms of polynomial running times are

desirable.

Complexity Classes (3/4)

Complexity Classes (4/4)

Analysing Simple Codes (1/4)

- Some rules.
- Basic operations are those that can be computed

in O(1) or constant time. - Examples are assignment statements, comparison

statements, and simple arithmetic operations.

Analysing Simple Codes (2/4)

- Code fragment 1
- temp x
- x y
- y temp
- Running time 3 statements O(1)
- Code fragment 2
- p list.size()
- for (int i 0 i lt p i)
- listi 3
- Running time 1 p statements O(p)

Analysing Simple Codes (3/4)

- Code fragment 3
- if (x lt y)
- a 1 b 2 c 3
- else
- a 2 b 4 c 8 d 13 e 51
- Running time max3, 5 statements 5 O(1)

In code fragments 2 and 3, we consider only

assignment statements as our basic operations.

Even if we include the loop test operation (i lt

p) and update operation (i) in fragment 2, and

the if test operation (x lt y) in fragment 3, it

will not affect the final result in big-O

notation, since they are each of constant time.

Analysing Simple Codes (4/4)

- Code fragment 4
- sum 0.0
- for (int k 0 k lt n k)
- sum arrayk
- avg sum/n
- Running time 1 n 1 statements n 2 O(n)

- Code fragment 5
- for (int i 0 i lt n i)
- for (int j 0 j lt i j)
- sum matrixij
- Running time 0 1 2 (n-1) n(n-1)/2

O(n2)

General Rules (1/3)

- Rule 1 Loops
- The running time of a loop is at most the running

time of the statements inside the loop times the

number of iterations.

General Rules (2/3)

- Rule 2 Nested loops
- Analyse these inside out. The total running time

of a statement inside a group of nested loops is

the running time of the statement multiplied by

the product of the sizes of all the loops.

General Rules (3/3)

- Rule 3 Selection statements
- For the fragment
- if (condition) S1
- else S2
- the running time of an if-else statement is

never more than the running time of the condition

test plus the larger of the running times of S1

and S2.

Worst-case Analysis

- We may analyze an algorithm/code based on the

best-case, average-case and worst-case scenarios. - Average-case and worst-case analysis are usually

better indicators of performance than best-case

analysis. - Worst-case is usually easier to determine than

average-case.

Running Time of Some Known Algorithms

- The following are worst-case running time of some

known algorithms on arrays. The problem size, n,

is the number of elements in the array. - Sequential search (linear search) in an array

O(n). - Binary search in a sorted array O(lg n).
- Simple sorts (bubblesort, selection sort,

insertion sort) O(n2). - Mergesort O(n lg n).

Sequential Search vs Binary Search (1/2)

- Sequential/linear search Start from first

element, visit each element to see if it matches

the search item.

public static int linearSearch(int list, int

searchValue) for (int i 0 i lt

list.length i) if (listi

searchValue) return i

return -1

Sequential Search vs Binary Search (2/2)

- Binary search
- Works for sorted array.
- Examine middle element, and eliminate half of the

array.

public static int binarySearch(int list, int

searchValue) int left 0 int

right list.length - 1 int mid

while (left lt right) mid (left

right)/2 if (listmid

searchValue) return mid

else if (listmid lt searchValue)

left mid 1 else

right mid - 1 return -1

Analysis of Sequential Search

- Assume
- An array with n elements.
- Basic operation is the comparison operation.
- Best-case
- When the key is found at the first element.

Running time O(1). - Worst-case
- When the key is found at the last element, or

when the key is not found. Running time O(n). - Average case
- Assuming that the chance of every element that

matches the key is equal, then on average the key

is found after n/2 compare operations. Running

time O(n).

Analysis of Binary Search

- Assume
- An array with n elements.
- Basic operation is the comparison operation.
- Best-case
- When the key is found at the middle element.

Running time O(1). - Worst-case
- Running time O(lg n). Why?
- If you start with the value n, how many times can

you half it until it becomes 1? - Examples Starting with 8, it takes 3 halving to

get it to 1 starting with 32, it takes 5

halving starting with 1024, it takes 10 halving.

Analysis of Sort Algorithms

- For comparison-based sorting algorithms, the

basic operations used in analysis is - The number of comparisons, or
- The number of swaps (exchanges).
- Worst-case analysis
- All the three basic sorts selection sort,

bubble sort, and insertion sort have worst-case

running time of O(n2), where n is the array size. - What is the worst-case scenario for bubble sort?

For selection sort? For insertion sort?

Maximum Subsequence Sum (1/6)

- Given this problem
- Given (possibly negative) integers a0, a1, a2, ,

an-1, find the maximum value of - (For convenience, the maximum subsequence sum is

0 if all the integers are negative.) - Example
- list -2, 11, -4, 13, -5, -2
- answer 20 (a1 through a3).
- Many algorithms to solve this problem.

Maximum Subsequence Sum (2/6)

- Algorithm 1

public static int maxSubseqSum(int list)

int thisSum, maxSum maxSum 0

for (int i 0 i lt list.length i)

for (int j 0 j lt list.length j)

thisSum 0 for (int k

i k lt j k) thisSum

listk // count this line if

(thisSum gt maxSum) maxSum

thisSum return maxSum

Maximum Subsequence Sum (3/6)

- Algorithm 1 Analysis

How many times is line thisSum listk

executed?

Maximum Subsequence Sum (4/6)

- Algorithm 2

Maximum Subsequence Sum (5/6)

- Algorithm 2 Analysis

Algorithm 2 avoids the cubic running time O(n3)

by removing the inner-most for-k loop in

algorithm 1. New running-time complexity is O(n2).

Maximum Subsequence Sum (6/6)

- Algorithm 3

End of file