CSCE 210 Data Structures and Algorithms

1
CSCE 210Data Structures and Algorithms

• Prof. Amr Goneid
• AUC
• Part 8. Introduction to the Analysis of Algorithms

2
Introduction to the Analysis of Algorithms

• Algorithms
• Analysis of Algorithms
• Time Complexity
• Bounds and the Big-O
• Types of Complexities
• Rules for Big-O
• Examples of Algorithm Analysis

3
1. Algorithms

• An Algorithm is a procedure to do a certain task
• An Algorithm is supposed to solve a general,
  well-specified problem, e.g. Sorting Problem
• Input A sequence of keys a1 , a2 , , an
• output A permutation (re-ordering) of the
  input,
• a1 , a2 , , an such that a1 a2
  an
• An instance of the problem might be sorting an
  array of names or sorting an array of integers.
• An algorithm is supposed to solve all instances
  of the problem

4
Example Selection Sort Algorithm

• void selectsort (itemType a , int n)
• int i , j , m
• for (i 0 i lt n-1 i)
• m i
• for ( j i1 j lt n j)
• if (aj ? am) m j
• swap (ai , am)

5
Algorithms

• An algorithm should have a clear and Transparent
  purpose
• An Algorithm should be Correct, i.e. solves the
  problem correctly.
• An Algorithm should be Complete, i.e. solve all
  instances of the problem
• An algorithm should be Writeable, i.e. we should
  be able to express its procedure with available
  implementation language.

6
Algorithms

• An algorithm should be Maintainable, i.e. easy to
  debug and modify.
• An algorithm is supposed to be Easy to use.
• An Algorithm is supposed to be Efficient.
• An efficient algorithm uses minimum of resources
  (space and time).
• In particular, it should solve the problem in the
  minimum amount of time.

7
2. Analysis of Algorithms

• The main goal is to determine the cost of running
  an algorithm and how to reduce that cost. Cost is
  expressed as Complexity
• Time Complexity
• Space Complexity

8
Analysis of Algorithms

• Time Complexity
• Depends on
• - Machine Speed
• - Size of Data and Number of Operations
  needed (n)
• Space Complexity
• Depends on
• - Size of Data
• - Size of Program

9
3. Time Complexity

• Expressed as T(n) number of operations
  required.
• (n) is the Problem Size
• n could be the number of specific operations, or
  the size of data (e.g. an array) or both.

10
Number of Operations T(n)
Example (1) Factorial Function int factorial
(int n) int i , f f 1 if ( n gt 0
) for (i 1 i lt n i) f i
return f Let T(n) Number of
multiplications. For a given n , then T(n) n
(always)
11
Complexity of the Factorial Algorithm

• Because T(n) n always, then T(n) ?(n)

T(n)
?(n)
n
12
Number of Operations T(n)

• Example (2) Linear Search
• int linSearch (const int a , int target, int n)
• for (int i 0 i lt n i)
• if (ai target) return i
• return -1
• T(n) number of array element comparisons.
• Best case T(n) 1
• Worst case T(n) n

13
Complexity of the Linear Search Algorithm

• T(n) 1 in the best case. T(n) n in the worst
  case
• We write that as T(n) ?(1) and T(n) O(n)

T(n)
O(n)
?(1)
n
14
4. Bounds and the Big-O

• If an algorithm always costs T(n) f(n) for the
  same (n) independent of the data, it is an Exact
  algorithm. In this case we say T(n) ?(f(n)), or
  Big ?.
• The factorial function is an example where T(n)
  ?(n)

15
Bounds

• If the cost T(n) of an algorithm for a given size
  (n) changes with the data, it is not an exact
  algorithm. In this case, we find the Best Case
  (Lower Bound) T(n) ?(f(n)) or Big ? and the
  Worst Case (Upper Bound) T(n) O(f(n)) or Big O
• The linear search function is an example where
  T(n) ?(1) and T(n) O(n)

16
Constants do not matter

• If T(n) constant function of (n), i.e.
• T(n) c f(n)
• we still say that T(n) is O(f(n)) or ?(f(n)) or
  ?(f(n)).
• Examples
• T(n) 4 (best case) then T(n) ?(1)
• T(n) 6 n2 (worst case) then T(n) O(n2)
• T(n) 3 n (always) then T(n) ?(n)

17
Constants do not matter

• T(n) 4 (best case) then T(n) ?(1)
• T(n) 6 n2 (worst case) then T(n) O(n2)
• T(n) 3 n (always) then T(n) ?(n)

is of
Number of operations
Complexity
18
5. Types of Complexities

• Constant Complexity
• T(n) constant independent of (n)
• Runs in constant amount of time ? O(1)
• Example cout ltlt a00
• Polynomial Complexity
• T(n)amnm a2n2 a1n1 a0
• If m1, then O(a1na0) ? O(n)
• If m gt 1, then ? O(nm) as nm dominates

19
Polynomials Complexities
O(n3)
O(n2)
Log T(n)
O(n)
n
20
Types of Complexities

• Logarithmic Complexity
• Log2nm is equivalent to n2m
• Reduces the problem to half ? O(log2n)
• Example Binary Search
• T(n) O(log2n)
• Much faster than Linear Search which has T(n)
  O(n)

21
Linear vs Logarithmic Complexities
O(n)
T(n)
O(log2n)
n
22
Types of Complexities

• Exponential
• Example List all the subsets of a set of n
  elements a,b,c
• a,b,c, a,b,a,c,b,c,a,b,c,
• Number of operations T(n) O(2n)
• Exponential expansion of the problem ? O(an)
  where a is a constant greater than 1

23
Exponential Vs Polynomial
O(2n)
Log T(n)
O(n3)
O(n)
n
24
Types of Complexities

• Factorial time Algorithms
• Example
• Traveling salesperson problem (TSP)
• Find the best route to take in visiting n cities
  away from home. What are the number of possible
  routes? For 3 cities (A,B,C)

25
Possible routes in a TSP

• Number of operations 3!6, Hence T(n) n!
• Expansion of the problem ? O(n!)

26
Exponential Vs Factorial
O(nn)
O(n!)
Log T(n)
O(2n)
n
27
Execution Time Example

• Example
• For the exponential algorithm of listing all
  subsets of a given set, assume the set size to be
  of 1024 elements
• Number of operations is 21024 about 1.810308
• If we can list a subset every nanosecond the
  process will take 5.7 10291 yr!!!

28
Polynomial Non-polynomial Times

• P (Polynomial) Times
• O(1), O(log n), O(log n)2, O(n) , O(n logn),
• O(n2), O(n3), .
• NP (Non-Polynomial) Times
• O(2n) , O(en) , O(n!) , O(nn) , ..

29
6. Rules for Big-O
30
Exercises

• Which function has smaller complexity ?
• f 100 n4 g n5
• f log(log n3) g log n
• f n2 g n log n
• f 50 n5 n2 n g n5
• f en g n!

31
Examples Find T(n)

• y x x ai y 2
• i 1 while ( x ! ai i lt n) i
• for (i 1 i lt n i)
• for ( j 1 j lt n j) k
• if (a1,1 0)
• for (i 1 i lt n i)
• for (j 1 j lt n j) ai,j 0
• else
• for (i 1 i lt n i) ai,i 1

32
7. Examples of Algorithm Analysis

• for (i 1 i lt n/2 i)
• O(1)
• for (j 1 j lt nn j)
• O(1)
• T(n) (n2 1) n/2 n3/2 n/2 Hence T(n)
  O(n3)
• _________________________________________________
  ________
• for (i 1 i lt n/2 i)
• O(1)
• for (j 1 j lt nn j)
• O(1)
• T(n) (n/2) n2 Hence T(n) O(n2)

33
Infinite Loop?

• int k , n , d
• cout ltlt Enter n cin gtgt n
• k n
• for ( )
• d k 2 cout ltlt d k / 2
• if (k 0) break
• Each iteration cuts the value of (k) by half and
  the final iteration reduces k to zero. Hence the
  number of iterations T(n) ? log2 n ? 1
• Hence T(n) O(log2 n)

34
A function to reverse an array(1)

• A function to reverse an array a .
• Version(1) using a temporary array b .
• int aN
• void reverse_array (int a , int n)
• int bN
• for (int i 0 i lt n i) bi an-i-1
• for (int i 0 i lt n i) ai bi
• Consider T(n) to be the number array accesses,
  then T(n) 2n 2n 4n
• Hence T(n) O(n), with extra space bN

35
A function to reverse an array(2)

• A function to reverse an array a .
• Version(2) using swapping.
• int aN
• void reverse_array (int a , int n)
• int temp
• for (int i 0 i lt n/2 i)
• temp ai ai an-i-1 an-i-1
  temp
• Consider T(n) to be the number array accesses,
  then T(n) 4(n/2) 2n
• Hence T(n) O(n), without extra space

36
Index of the minimum element

• A function to return the index of the minimum
  element
• int index_of_min ( int a , int s , int e )
• int imin s
• for (int i s1 i lt e i)
• if (ai lt aimin) imin i
• return imin
• Consider T(n) to be the number of times we
  compare array elements. When invoked as
  index_of_min (a, 0 , n-1 ), then T(n) e
  (s1) 1 e - s n-1
• Hence T(n) O(n)

37
Analysis of Selection Sort

• void SelSort(int a , int n)
• int m, temp
• for (int i 0 i lt n-1 i)
• m index_of_min(a, i, n-1)
• temp ai ai am am temp
• T(n) number of array comparisons. For a given
  iteration (i),
• Ti(n) n-1-i and T(n) T0(n) T1(n) .
  Tn-2(n)
• (n-1) (n-2) 1 n(n-1)/2 0.5 n2
  0.5 n O(n2)
• This cost is the same for any data of size (n)
  (exact algorithm)

Costs n-1-i
38
SelectSort vs QuickSort
Selectsort O(n2)
T(n)
Quicksort O(n log2n)
n
39
Analysis of Binary Search

• int BinSearch( int a , int n, int x)
• int L, M, H bool found false
• L 0 H n
• while ( (L lt H) ! found)
• M (L H)/2 // Approximate Middle
• if (x aM) found true // Match
• else if (x gt aM) L M1 // Discard left
  half
• else H M 1 // Discard right half
• if (found) return M else return -1
• In the best case, a match occurs in the first
  iteration, thus T(n) ?(1). In the
• worst case (not found) half of the array is
  discarded in each iteration.
• Hence T(n) log2 n 1 O(log2 n)

40
Linear vs Binary Search
O(n)
T(n)
Linear Search
O(log2n)
Binary Search
n
41
Learn on your own about

• Proving Correctness of algorithms
• Analyzing Recursive Functions
• Standard Algorithms in C