8. Analysis of Algorithms

1
8. Analysis of Algorithms

• We will focus on time complexity, or the length
  of time a particular algorithm takes to solve a
  particular problem
• We begin by considering different algorithms for
  simple computational tasks such as searching or
  sorting and comparing how long they take to
  execute

2
Sequential Search Algorithm

• Problem Is a key x in the array S of n keys?
•  
• Inputs positive integer n, array of keys S
  indexed from 1 to n and a key x.
•  
• Outputs location, the location of x in S (0 if x
  is not in S).

3
Sequential Search procedure

• procedure seqsearch (n integer
• S array1..n of keytype
• x keytype)
• var location index
•   begin
• location1
• while location ? n and Slocation ? x do
• location location 1
• end
• if location gt n then
• location 0
• end
• end

4
Binary Search Algorithm

• Problem Is the key x in the sorted array S of n
  keys?
• Inputs positive integer n, sorted array of keys
  S indexed from 1 to n and a key x.
• Outputs location, the location of x in S (0 if x
  is not in S).
• Assumes the array is initially sorted
• At each step, the number of possible positions is
  cut in half

5
Binary search procedure

• procedure binsearch (n integer
• S array1..n of keytype
• x keytype)
• var location index
• var
• low, high, mid index
• begin
• low 1 highn
• location0
• while low ? high and location 0 do
• mid (low high) div 2
• if x Smid then location mid
• else if x lt Smid then high mid - 1
• else low mid 1
• end
• end
• end

6
The number of comparisons done by Sequential
Search and Binary search when x is larger than
all the array items. 
7
Time Complexity analysis

• How does the time taken to solve a problem scale
  with the input size?
• Array sum algorithm
• Problem Add all the numbers in the array s of n
  numbers
• Inputs positive integer n, array of keys S
  indexed from 1 to n
• Outputs sum, the sum of the numbers in S
•  

8

• Array sum function
• function sum ( n integer S array1..n of
  number) number)
• var
• i index
• begin
• sum 0
• for i 1 to n do
• sum sum Si
• end
• end
•  
• Basic operation the addition of an item in the
  array to sum.
• Input size n, the number of items in the array
• Time complexity function T(n) n

9
Time complexity function T(n)

• In general, a complexity function can be any
  function from the nonnegative integers (input
  size) to the nonnegative reals (time taken in
  some units)
• There are different kinds of time complexity
• average case time complexity
• worst case time complexity

10
Worst-case time complexity W(n)

• Often, the time taken for a particular algorithm
  depends on the input it operates on
• The worst-case time complexity reflects the
  longest time it could possibly take to finish
• For example, the sequential search is measured in
  terms of how many comparisons are needed. If
  were lucky, the number we are searching for is
  the first element. But if the number isnt in
  the array, we have to make n comparisons, so
• W(n) n

11
 

• Sorting algorithms
• Exchange sort algorithm
• Problem Sort n keys in ascending order
• Inputs positive integer n, array of keys S
  indexed from 1 to n.
• Outputs the array S containing the keys in
  ascending order.
• Basic idea go through list of numbers, find
  smallest and put in first position repeat for
  second with remaining numbers, iterate.

12
Exchange sort procedure

• procedure exchangesort (n integer
• var S array1..n of keytype)
• var
• i,j index
• begin
• for i 1 to n-1 do
• for j i1 to n do
• if Sj lt Si then
• exchange Si and Sj
• end
• end
• end
• end

13
Time complexity of exchange sort

• Basic operation the comparison of Sj with Si
• Input size n, the number of items to be sorted
•  
• T(n) (n - 1) (n - 2) (n -3) . . . . . 1
  
• (n - 1) n/ 2

14
Another sorting algorithm

• Now let us look at another way of dealing with
  sorting keys known as mergesort.
• This works by taking the initial array and then
  dividing it into two subarrays with (approx.)
  half the items in each subarray.
• Then each subarray is sorted, by recursively
  calling the same algorithm.
• Then the solutions to the subarrays are merged
  into a single sorted array.

15
Mergesort algorithm

• Problem sort n keys in ascending sequence
•  
• Inputs positive integer n, array of keys S
  indexed from 1 to n.
•  
• Outputs the array S containing the keys in
  ascending order.
• The same problem as for exchange sort

16
Mergesort procedure

• procedure mergesort (ninteger var S
  array1..n of keytype)
• const
• h n div 2
• m n h
• var
• U array 1..h of keytype
• V array1..m of keytype
•  begin
• if ngt1 then
• copy S1 through Sh to U
• copy Sh1 through Sn to V
• mergesort(h,U)
• mergesort(m,V)
• merge(h,m,U,V,S)
• end
• end

Calls itself twice!
17
Merge two arrays

• Problem merge two sorted arrays into one sorted
  array
• Inputs positive integers h and m, array of
  sorted keys U indexed from 1 to h, array of
  sorted keys V indexed from 1 to m
• Outputs an array S indexed from 1 to h m
  containing the keys in U and V in a single sorted
  array.

18
Merge procedure

• procedure merge (h, minteger U array 1..h
  of keytype
• V array1..m of keytype)
• var S array1..n of keytype
•  var i,j,k index
•  begin
• i 1 j1 k 1
• while ilt h and jlt m do
• if Ui lt Vj then Sk Ui i i1 else
  Sk Vj j j1
• end
• kk 1
• end
• if igth then copy Vj through Vm to Sk
  through Sh m
• else copy Ui through Uh to Sk through Sh
  m
• end
• end

19
Whats the longest merge could take?

• Basic operation the comparison of Ui with
  Vj.
• Input size h and m the number of items in each
  of the two input arrays.
• The worst case occurs when the loop is exited,
  because one of the indices, for example i, has
  reached its exit point h whilst the other index j
  has reached m 1, 1 less than its exit point.

20
Worst case time complexity of merge

• This can occur when the first m 1 items in V are
  placed first in S , followed by all h items in U,
  at which time the loop is exited because i
  h.
• For example, (1 2 3 8) (4 5 6 7)
• Therefore,
•   W(h, m) h m - 1

21
How long could the full sort take?

• Basic operation the comparison that takes place
  in merge.
• Input size n, the number of items in the array
  S.
• Time is mostly spent in merge procedure

22
Worst case time complexity of mergesort

• The total number of comparisons is
• the sum of the number of comparisons in the
  recursive call to mergesort with U as input,
• the number of comparisons in the recursive call
  to mergesort with V as input and
• the number of comparisons in the top-level call
  to merge.
• Therefore
•   W(n) W(h) W(m) h m 1

Time to sort U Time to sort V Time to merge
23
Slight simplification

• For simplicity, assume that n is a power of 2.
•  
• Then h n div 2 n/2 and
• m n h n/2.
• So h m n.
•  
• Thus we get
•  
• W(n) W(n/2) W(n/2) n 1
• 2W(n/2) n - 1

24
A recursive solution

• Putting all this together and doing a little
  arithmetic we can say that if n 2k, then 
• W(2k) 2 W(2k-1) 2k 1
• 2 2W(2k-2) 2k-1-1 2k 1
• 22W(2k-2) 2.2k-1 2 2k 1
• 22W(2k-2) 2.2k 2 -1
• 222W(2k-3)2k-2 1 2.2k 2 1
• 23W(2k-3) 3. 2k 22 2 1
• We can carry on recursively substituting like
  this and you can see that the term after j
  substitutions we will get
• 2jW(2k-j) j.2k 2j-1 2j-2 2j-3 -..-2 1

25
Work back to one element arrays

• Finally when j k, the process stops and we
  have
• W(2k) 2jW(2k-j) j.2k 2j-1 2j-2 2j-3 -..-2
  1
• 2kW(1) k2k 2k-12k-2 -..-2 1
• When the input size is 1, the terminal condition
  is met and no merging is done. So W(1) 0.
• Thus,
• W(2k) 0 k2k 1 2 222k-1
• k2k (2k- 1)/(2 1)
• k2k 2k1

26
Final time complexity of mergesort

• Now since we took n 2k, k log2 n, and so we
  can write
•   W(n) n log n n 1
• How does this compare to exchange sort?
• T(n) (n - 1) n/ 2
• Considerably faster when k ltlt 2k !
• Is this as fast as one can get?

27
Decision Trees

• We can get an idea of the optimal time complexity
  by considering decision trees.
• Lets return to the binary search example.
• Each comparison has three possible results
• 1) We found the key were searching for (stop)
• 2) Were looking for a lower number, so repeat
  the lower branch of remaining numbers
• 3) Were looking for a lower number, so repeat
  the lower branch of remaining numbers
• We can represent this with a decision tree

28
Binary search of the list

• 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41,
  43, 47
• Given a key K, is it in the list?

29
How many comparisons do we need?

• In this case (n15) we need at most four
  comparisons to find whether K is in the list.
• So a worst case lower bound for the number of
  comparisons is 4, which is 1 plus the depth of
  the binary tree
• The minimum depth of a binary tree with n nodes
  is ?log2 n?. The total number of nodes must
  exceed n.
• So the lower bound for the worst case of a
  binary search algorithm on a sorted input list of
  n elements is 
• 1 ?log2 n?

30
Weighing things

• Suppose that we are given eight coins and told to
  find the heavy coin among the eight with the
  assumption that they all look alike, and the
  other seven all have the same weight, and we must
  use a set of scales.
• There are two ways to proceed
• If we ignore the possibility that the scales
  might balance, then we get a binary decision
  tree.
• If we include a balanced outcome, we can solve
  it with a ternary decision tree. 

31
Binary tree algorithm

• Let each internal node of the tree represent the
  scales balance, with an equal number of coins on
  each side.
• If the left side goes down, then the heavy coin
  is on the left side of the balance.
• Otherwise, the heavy coin is on the right side of
  the balance.
• Each leaf represents one coin that is the heavy
  coin.
• Suppose we label the coins with the numbers 1,2,
  ..., 8.

32
Binary decision tree

• Each branching represents a result of a weighing
• Three weighings are needed
• 8 possible states, 2 x 2 x 2 possible outcomes

33
Ternary (3 pronged) tree algorithm

• Here we allow for the third possibility that the
  two pans might be exactly the same weight
• So we don't have to use all eight coins on the
  first weighing.
• The decision tree below shows one solution to the
  problem.

34
Ternary decision tree

• Each weighing has three possible outcomes
• This solution requires just two weighings!
• 8 possible states, 3 x 3 9 possible results

35
A harder example

• Suppose we have a set of 13 coins in which at
  most one coin is bad and a bad coin may be
  heavier or lighter than the other coins.
• The problem is to use a set of scales to find the
  bad coin if it exists and say whether it is heavy
  or light.
• We'll find a lower bound on the heights of
  decision trees for scales algorithms to solve the
  problem.

36
How many possible states?

• Any solution must tell whether a bad coin is
  heavy or light.
• Thus there are 27 possible situations
• no bad coin
• and
• the 13 pairs of possibilities
•   ?ith coin light, ith coin heavy?.

37
How many weighings for a ternary tree?

• Any decision tree for the problem must have at
  least 27 leaves.
• So a ternary decision tree of depth k must
  satisfy 3k ? 27, or k ? 3.
• This gives us a lower bound of 3, which seems
  just enough
• Now the big question
• Is there an algorithm to solve the problem,
  where the decision tree of the algorithm has
  depth 3?

38
Unfortunately, this is not possible

• Look at the different ways you could do the
  initial weighing.
• In each case the remaining possible conditions
  cannot be distinguished with just two more
  weighings.
• Therefore any decision tree for this problem must
  have depth 4 or more.
• However it is possible with 12 coins!

39
Back to the sorting problem

• What is the fewest number of comparison
  operations performed by any sorting algorithm
  that sorts by comparing elements in the list to
  be sorted?
• Assume that we have a set of n distinct numbers.
• Since there are n! possible arrangements of these
  numbers, it follows that any algorithm to sort a
  list of n numbers has n! possible input
  arrangements.
•  

40
How many comparisons do we need?

• Any decision tree for a comparison sorting
  algorithm must contain at least n! leaves, one
  leaf for each possible outcome of sorting one
  arrangement.
• We know that a binary tree of depth d has at most
  2d leaves.
• So the depth d of the decision tree for any
  comparison sort of n items must satisfy the
  inequality
•  
• n! ? 2d.

41
Solve for d

• We can solve this inequality for the natural
  number d by taking logs and using the ceiling
  function as follows
•  
• log2n! ? d
•  
• log2 n! ? ? d
• The strange brackets indicate the first integer
  greater than or equal to the number inside

42
The worst case lower bound

• Any algorithm that sorts by comparing elements
  must use at least ? log2 n! ? comparisons in the
  worst case to sort n items.
• If we calculate ? log2 n! ? by first computing
  n! and then taking the log and the ceiling, then
  the calculation is quite difficult for large
  values of n.
• But ? log2 n! ? is "approximately" the same as
  n log2 n. (Sterlings formula)
• This was basically the result for mergesort!

43

• The travelling salesperson problem
• (a problem which scales very badly!)
• There are n cities labelled 1, ...,n for which
  the distance di,j between any two cities i and j
  is known.
• What is the shortest route, starting and ending
  at city 1, to include each city?  

44
How many possible routes?

• Clearly there are
• n-1 ways to choose the first city,
• n-2 ways to choose the next and so on until he
  only has one city left to visit.
• So the number of possible routes is
• n ? (n-1) ? (n-2) ?......? 1 (n-1)!  

45
A possible solution

• begin
• min_dist total distance of any one route
• for each of the (n-1)! different routes R
• do begin
• compute total distance D of R
• if D lt min_dist then min_dist D
• end
• end.
•  
• T(n) n! for this solution to the travelling
  salesman problem

46
How long does it take?

• To compute the total distance per route requires
  n additions, with (n-1)! routes this makes n!
  additions in total.
• If n 30 then we would have 1020 computations.
• Fastest computers can do 109 additions per
  second.
• So it would take 1011 seconds, a few thousand
  years, to run.
• Increase n to 100 and the time needed exceeds the
  lifespan of the universe.

47
Efficient algorithms

• Algorithms should be designed to avoid
  unnecessary repeated calculations
• Example
• We have a class of n students whose exam marks
  are stored in an array X.
• We want to compute these marks as percentages.
• Assume the marks are out of 70. Then there are a
  couple ways to solve it

48
Two possible algorithms

•  
• Algorithm A Algorithm B
•  
• total 70 factor 100/70
• for i 1 to n do for i 1 to n do
• X(i) X(i) (100/total) X(i) X(i)
  factor
•  
•  
•  Which is faster?

49

• Comparison of time complexity
• In A, there is one multiplication and one
  division each time through the loop,
• so TA(n) 2n
•  
• For B, there is one division outside the loop and
  then one multiplication inside,
• so TB(n) n1
•  
• A smart compiler will fix this for you, but its
  better to program it right the first time!