COMP108 Algorithmic Foundations Algorithm efficiency Searching/Sorting

1
COMP108Algorithmic FoundationsAlgorithm
efficiency Searching/Sorting
Prudence Wong http//www.csc.liv.ac.uk/pwong/tea
ching/comp108
2
Learning outcomes

• Able to carry out simple asymptotic analysis of
  algorithms
• See some examples of polynomial time and
  exponential time algorithms
• Able to apply searching/sorting algorithms and
  derive their time complexities

3
Time Complexity Analysis

• How fast is the algorithm?

Code the algorithm and run the program, then
measure the running time

1. Depend on the speed of the computer
2. Waste time coding and testing if the algorithm is
   slow

Identify some important operations/steps and
count how many times these operations/steps
needed to executed
4
Time Complexity Analysis

• How to measure efficiency?

Number of operations usually expressed in terms
of input size

• If we doubled the input size, how much longer
  would the algorithm take?If we trebled the input
  size, how much longer would it take?

5
Why efficiency matters?

• speed of computation by hardware has been
  improved
• efficiency still matters
• ambition for computer applications grow with
  computer power
• demand a great increase in speed of computation

6
Amount of data handled matches speed increase?

• When computation speed vastly increased, can we
  handle much more data?

• Suppose
• an algorithm takes n2 comparisons to sort n
  numbers
• we need 1 sec to sort 5 numbers (25 comparisons)
• computing speed increases by factor of 100
• Using 1 sec, we can now perform 100x25
  comparisons,i.e., to sort 50 numbers
• With 100 times speedup, only sort 10 times more
  numbers!

7
Time Complexity Analysis
input n sum 0 for i 1 to n do begin sum
sum i end output sum

• Important operation of summation addition
• How many additions this algorithm requires?

We need n additions (depend on the input size n)
8
Space Complexity Analysis
input n sum 0 for i 1 to n do begin sum
sum i end output sum
We need to store the number n, the variable sum,
and the index i gt needs 3 memory space
In this example, the memory space does not depend
on the input size n
In other case, it may depend on the input size n
9
Look for improvement
input n sum n(n1)/2 output sum
Mathematical formula gives us an alternative to
find the sum of first n numbers 1 2 ... n
n(n1)/2
We only need 3 operations 1 addition, 1
multiplication, and 1 division (no matter what
the input size n is)
10
Finding the minimum...
11
Finding min among 3 numbers
Input 3 numbers a, b, c Output the min value of
these 3 numbers Algorithm
Input a, b, c
a ? b ?
N
Y
a ? c ?
b ? c ?
Y
N
Y
N
Output a
Output b
Output c
Output c
12
Finding min among 3 numbers
Example 1
Input a, b, c
a3, b5, c2
Y
N
a ? b ?
a ? c ?
b ? c ?
Y
N
Y
N
Output a
Output b
Output c
Output c
2 is output
13
Finding min among 3 numbers
Example 2
Input a, b, c
a6, b5, c7
Y
N
a ? b ?
a ? c ?
b ? c ?
Y
N
Y
N
Output a
Output b
Output c
Output c
5 is output
14
Time Complexity Analysis
input a, b, c if (a ? b) then if (a ? c) then
output a else output c else if (b ? c)
then output b else output c

• Important operation comparison
• How many comparison this algorithm requires?

It takes 2 comparisons
15
Finding min among n numbers
Input a0, a1, ...
Any systematic approach?
a0 ? a1 ?
N
Y
a0 ? a2 ?
a1 ? a2 ?
Y
Y
N
N
a0 ? a3 ?
a1 ? a3 ?
a2 ? a3 ?
a2 ? a3 ?
16
Finding min among n numbers
Input a0, a1, ..., an-1
min 0 i 1
i i1
i lt n?
N
Y
Output amin
ai lt amin?
Y
min i
N
17
Finding min among n numbers
input a0, a1, ..., an-1 min 0 i 1 while
(i lt n) do begin if (ai lt amin) then
min i i i 1 end output amin
Example
a50,30,40,20,10
iteration min amin
before
1
2
3
4
50
0
30
1
1
30
20
3
10
4
18
Finding min among n numbers
n-1

• Time complexity ?? comparisons

input a0, a1, ..., an-1 min 0 i 1 while
(i lt n) do begin if (ai lt amin) then
min i i i 1 end output amin
19
Finding min using for-loop

• Rewrite the above while-loop into a for-loop

input a0, a1, ..., an-1 min 0 for i 1
to n-1 do begin if (ai lt amin) then min
i end output amin
20
Time complexity- Big O notation
21
Which algorithm is the fastest?

• Consider a problem that can be solved by 5
  algorithms A1, A2, A3, A4, A5 using different
  number of operations (time complexity).

f1(n) 50n 20
f2(n) 10 n log2 n 100
f3(n) n2 - 3n 6
f4(n) 2n2
f5(n) 2n/8 - n/4 2
f5(n)
f3(n)
f1(n)
Depends on the size of the problem!
22
f1(n) 50n 20 f2(n) 10 n log2n 100 f3(n)
n2 - 3n 6 f4(n) 2n2 f5(n) 2n/8 - n/4 2
23
What do we observe?

• There is huge difference between
• functions involving powers of n (e.g., n, n log
  n, n2, called polynomial functions) and
• functions involving powering by n (e.g., 2n,
  called exponential functions)
• Among polynomial functions, those with same order
  of power are more comparable
• e.g., f3(n) n2 - 3n 6 and f4(n) 2n2

24
Growth of functions
25
Relative growth rate
2n
n2
n
log n
c
n
26
Hierarchy of functions

• We can define a hierarchy of functions each
  having a greater order of magnitude than its
  predecessor

1
log n
n n2 n3 ... nk ...
2n
constant
logarithmic
polynomial
exponential

• We can further refine the hierarchy by inserting
  n log n between n and n2,n2 log n between n2 and
  n3, and so on.

27
Hierarchy of functions (2)
1
log n
n n2 n3 ... nk ...
2n
constant
logarithmic
polynomial
exponential

• Important as we move from left to right,
  successive functions have greater order of
  magnitude than the previous ones.
• As n increases, the values of the later functions
  increase more rapidly than the values of the
  earlier ones.
• Relative growth rates increase

28
Hierarchy of functions (3)
1
log n
n n2 n3 ... nk ...
2n
constant
logarithmic
polynomial
exponential

• Now, when we have a function, we can assign the
  function to some function in the hierarchy
• For example, f(n) 2n3 5n2 4n 7The term
  with the highest power is 2n3.The growth rate of
  f(n) is dominated by n3.
• This concept is captured by Big-O notation

29
Big-O notation

• f(n) O(g(n)) read as f(n) is of order g(n)
• Roughly speaking, this means f(n) is at most a
  constant times g(n) for all large n
• Examples
• 2n3 O(n3)
• 3n2 O(n2)
• 2n log n O(n log n)
• n3 n2 O(n3)
• function on L.H.S and function on R.H.S are said
  to have the same order of magnitude

30
Big-O notation - formal definition

• f(n) O(g(n))
• There exists a constant c and no such that f(n) ?
  c g(n) for all n gt no
• ??c ?no ?ngtno then f(n) ? c g(n)

31
f(n) O(g(n) Graphical illustration
?constant c no such that?ngtno, f(n) ? c g(n)
c g(n)
f(n)
n0
32
Example n60 O(n)
2n
n 60
n
c2, n060
n0
?constant c no such that ?ngtno, f(n) ? c g(n)
33
Example 2nlog2n100?n1000 O(n log n)
3 n log n
c3, n0600
f(n)
n log n
n0
?constant c no such that ?ngtno, f(n) ? c g(n)
34
f(n) O(g(n))?
c g(n)
f(n)
value of f(n) for nltn0does NOT matter
n0
?constant c no such that ?ngtno, f(n) ? c g(n)
35
Which one is the fastest?

• Usually we are only interested in the asymptotic
  time complexity, i.e., when n is large
• O(log n) lt O(?n) lt O(n) lt O(n log n) lt O(n2) lt
  O(2n)

36
Proof of order of magnitude

• Show that 2n2 4n is O(n2)
• Since n ? n2 for all ngt1,we have 2n2 4n ? 2n2
  4n2 ? 6n2 for all ngt1.
• Therefore, by definition, 2n2 4n is O(n2).

• Show that n3 n2 log n n is O(n3)
• Since n ? n3 and log n ? n for all ngt1,we have
  n2 log n ? n3, andn3 n2 log n n ? 3n3 for
  all ngt1.
• Therefore, by definition, n3 n2 log n n is
  O(n3).

37
Exercise

• Determine the order of magnitude of the following
  functions.
• n3 3n2 3
• 4n2 log n n3 5n2 n
• 2n2 n2 log n
• 6n2 2n

O(n3) O(n3) O(n2 log n) O(2n)
38
Exercise (2)

• Prove the order of magnitude
• n3 3n2 3
• n3 n3 ?n
• 3n2 ? n3 ?n?3
• 3 ? n3 ?n?2
• ? n3 3n2 3 ? 3n3 ?n?3
• 4n2 log n n3 5n2 n
• 4n2 log n ? 4n3 ?n?1
• n3 n3 ?n
• 5n2 ? n3 ?n?5
• n?n3 ?n?1
• ? 4n2 log n n3 5n2 n ? 7n3 ?n?5

• n3 n3 ?n
• 3n2 ? 3n3 ?n?1
• 3 ? 3n3 ?n?1
• ? n33n23 ? 7n3 ?n?1

• 4n2 log n ? 4n3 ?n?1
• n3 n3 ?n
• 5n2 ? 5n3 ?n?1
• n?n3 ?n?1
• ? 4n2log nn35n2n ? 11n3 ?n?1

39
Exercise (2) contd

• Prove the order of magnitude
• 2n2 n2 log n
• 2n2 ? n2 log n ?n?4
• n2 log n n2 log n ?n
• ? 2n2 n2 log n ? 2n2 log n ?n?4
• 6n2 2n
• 6n2 ? 62n ?n?1
• 2n 2n ?n
• ? 6n2 2n ? 72n ?n?1

• 2n2 ? 2n2log n ?n?2
• n2 log n n2 log n ?n
• ? 2n2n2log n ? 3n2log n ?n?2

40
More Exercise

• Are the followings correct?
• n2log n n3 3n2 3 O(n2)?
• n 1000 O(n)?
• 6n20 2n O(n20)?
• n3 5n2 log n n O(n2 log n) ?

O(n3)
YES
O(2n)
O(n3)
41
Some algorithms we learnt
input n sum 0 for i 1 to n do begin sum
sum i end output sum

• Computing sum of the first n numbers

O(n)
O(1)
input n sum n(n1)/2 output sum
O(?)
O(?)
min 0 for i 1 to n-1 do if (ai lt amin)
then min i output amin
Finding the min value among n numbers
O(n)
O(?)
42
More exercise

• What is the time complexity of the following
  pseudo code?

for i 1 to 2n do for j 1 to n do x x
1
O(n2)
O(?)
The outer loop iterates for 2n times. The inner
loop iterates for n times for each i. Total 2n
n 2n2.
43
Learning outcomes

• Able to carry out simple asymptotic analysis of
  algorithms
• See some examples of polynomial time and
  exponential time algorithms
• Able to apply searching/sorting algorithms and
  derive their time complexities

44
More polynomial time algorithms - searching
45
Searching

• Input a sequence of n numbers a0, a1, , an-1
  and a number X
• Output determine whether X is in the sequence or
  not
• Algorithm (Linear search)
• Starting from i0, compare X with ai one by one
  as long as i lt n.
• Stop and report "Found!" when X ai .
• Repeat and report "Not Found!" when i gt n.

46
Linear Search
To find 7

• 12 34 2 9 7 5 six numbers7 number X
• 12 34 2 9 7 5 7
• 12 34 2 9 7 5 7
• 12 34 2 9 7 5 7
• 12 34 2 9 7 5 7 found!

47
Linear Search (2)
To find 10

• 12 34 2 9 7 510
• 12 34 2 9 7 5 10
• 12 34 2 9 7 5 10
• 12 34 2 9 7 5 10
• 12 34 2 9 7 5 10
• 12 34 2 9 7 5 10 not found!

48
Linear Search (3)

• i 0
• while i lt n do
• begin
• if X ai then
• report "Found!" and stop
• else
• i i1
• end
• report "Not Found!"

49
Time Complexity
i 0 while i lt n do begin if X ai then
report "Found!" stop else i
i1 end report "Not Found!"

• Important operation of searching comparison
• How many comparisons this algorithm requires?

Best case X is the 1st no., 1 comparison,
O(1) Worst case X is the last no. OR X is not
found, n comparisons, O(n)
50
Improve Time Complexity?
If the numbers are pre-sorted, then we can
improve the time complexity of searching by
binary search.
51
Binary Search

• more efficient way of searching when the sequence
  of numbers is pre-sorted
• Input a sequence of n sorted numbers a0, a1, ,
  an-1 in ascending order and a number X
• Idea of algorithm
• compare X with number in the middle
• then focus on only the first half or the second
  half (depend on whether X is smaller or greater
  than the middle number)
• reduce the amount of numbers to be searched by
  half

52
Binary Search (2)
To find 24

• 3 7 11 12 15 19 24 33 41 55 10 nos 24 X
• 19 24 33 41 55 24
• 19 24 24
• 24 24 found!

53
Binary Search (3)
To find 30

• 3 7 11 12 15 19 24 33 41 55 10 nos 30 X
• 19 24 33 41 55 30
• 19 24 30
• 24 30 not found!

54
Binary Search (4)

• first0, lastn-1
• while (first lt last) dobegin
• mid ?(firstlast)/2?
• if (X amid) report "Found!" stop
• else if (X lt amid) last mid-1
• else first mid1end
• report "Not Found!"

? ? is the floor function, truncate the decimal
part
55
Time Complexity
Best caseX is the number in the middlegt 1
comparison, O(1)-time Worst caseat most
?log2n?1 comparisons, O(log n)-time Why? Every
comparison reduces the amount of numbers by at
least half E.g., 16 gt 8 gt 4 gt 2 gt 1
first0, lastn-1 while (first lt last) dobegin
mid ?(firstlast)/2? if (X amid)
report "Found!" stop else if (X lt
amid) last mid-1 else first
mid1end report "Not Found!"
56
Binary search vs Linear search

• Time complexity of linear search is O(n)
• Time complexity of binary search is O(log n)
• Therefore, binary search is more efficient than
  linear search

57
Search for a pattern

• Weve seen how to search a number over a sequence
  of numbers
• What about searching a pattern of characters over
  some text?

Example
N O B O D Y _ N O T I C E _ H I M
text
N O T
pattern
N O B O D Y _ N O T I C E _ H I M
substring
58
String Matching

• Given a string of n characters called the text
  and a string of m characters (m?n) called the
  pattern.
• We want to determine if the text contains a
  substring matching the pattern.

Example
N O B O D Y _ N O T I C E _ H I M
text
N O T
pattern
N O B O D Y _ N O T I C E _ H I M
substring
59
Example
T
N O B O D Y _ N O T I C E _ H I M
N O T
P
N O T
N O T
bolded match underlined not match un-bolded
not considered
N O T
N O T
N O T
N O T
N O T
60
The algorithm

• The algorithm scans over the text position by
  position.
• For each position i, it checks whether the
  pattern P0..m-1 appears in Ti..im-1
• If the pattern exists, then report found
• Else continue with the next position i1
• If repeating until the end without success,
  report not found

61
Match pattern with Ti..im-1

• j 0
• while (j lt m PjTij) do
• j j 1
• if (j m) then
• report "found!" stop

Ti Ti1 Ti2 Ti3 Tim-1 P0 P1 P2
P3 Pm-1
62
Match for all positions

• for i 0 to n-m do
• begin
• // check if P0..m-1 match with Ti..im-1
• end
• report "Not found!"

63
Match for all positions

• for i 0 to n-m do
• begin
• j 0
• while (j lt m PjTij) do
• j j 1
• if (j m) then
• report "found!" stop
• end
• report "Not found!"

64
Time Complexity
How many comparisons this algorithm requires?
Best casepattern appears in the beginning of
the text, O(m)-time Worst case pattern appears
at the end of the text OR pattern does not exist,
O(nm)-time
for i 0 to n-m do begin j 0 while j lt m
PjTij do j j 1 if j m
then report "found!" stop end report "Not
found!"
65
More polynomial time algorithms - sorting
66
Sorting

• Input a sequence of n numbers a0, a1, , an-1
• Output arrange the n numbers into ascending
  order, i.e., from smallest to largest
• Example If the input contains 5 numbers 132, 56,
  43, 200, 10, then the output should be 10, 43,
  56, 132, 200

There are many sorting algorithmsinsertion
sort, selection sort, bubble sort, merge sort,
quick sort
67
Selection Sort

• find minimum key from the input sequence
• delete it from input sequence
• append it to resulting sequence
• repeat until nothing left in input sequence

68
Selection Sort - Example

• sort (34, 10, 64, 51, 32, 21) in ascending order
• Sorted part Unsorted part Swapped
• 34 10 64 51 32 21 10, 34
• 10 34 64 51 32 21 21, 34
• 10 21 64 51 32 34 32, 64
• 10 21 32 51 64 34 51, 34
• 10 21 32 34 64 51 51, 64
• 10 21 32 34 51 64 --
• 10 21 32 34 51 64

69
Selection Sort Algorithm

• for i 0 to n-2 do
• begin
• // find the index of the minimum number
• // in the range ai to an-1
• swap ai and amin
• end

70
Selection Sort Algorithm
how to swap two entries of an array? using a temp
variable? not using a temp variable?

• for i 0 to n-2 do
• begin
• min i
• for j i1 to n-1 do
• if aj lt amin then
• min j
• swap ai and amin
• end

71
Algorithm Analysis
for i 0 to n-2 do begin min i for j i1
to n-1 do if aj lt amin then min
j swap ai and amin end

• The algorithm consists of a nested for-loop.
• For each iteration of the outer i-loop,there is
  an inner j-loop.

Total number of comparisons (n-1) (n-2)
1 n(n-1)/2
i of comparisons in inner loop
0 n-1
1 n-2
...
n-2 1
O(n2)-time
72
Bubble Sort

• starting from the last element, swap adjacent
  items if they are not in ascending order
• when first item is reached, the first item is the
  smallest
• repeat the above steps for the remaining items to
  find the second smallest item, and so on

73
Bubble Sort - Example

• (34 10 64 51 32 21)round
• 34 10 64 51 32 21
• 1 34 10 64 51 21 32
• 34 10 64 21 51 32
• 34 10 21 64 51 32 ?dont need to swap
• 34 10 21 64 51 32
• 10 34 21 64 51 32
• 2 10 34 21 64 32 51
• 10 34 21 32 64 51 ?dont need to swap
• 10 34 21 32 64 51
• 10 21 34 32 64 51

underlined being considered italic sorted
74
Bubble Sort - Example (2)

• round
• 10 21 34 32 64 51
• 3 10 21 34 32 51 64 ?dont need to swap
• 10 21 34 32 51 64
• 10 21 32 34 51 64 ?dont need to swap
• 4 10 21 32 34 51 64 ?dont need to swap
• 10 21 32 34 51 64 ?dont need to swap
• 5 10 21 32 34 51 64

underlined being considered italic sorted
75
Bubble Sort Algorithm
the smallest will be moved to ai

• for i 0 to n-2 do
• for j n-1 downto i1 do
• if (aj lt aj-1)
• swap aj aj-1

start from an-1, check up to ai1
34 10 64 51 32 21
i 0
j 5
j 4
j 3
j 2
j 1
i 1
j 5
j 4
j 3
j 2
76
Algorithm Analysis

• The algorithm consists of a nested for-loop.

for i 0 to n-2 do for j n-1 downto i1 do
if (aj lt aj-1) swap aj aj-1
Total number of comparisons (n-1) (n-2)
1 n(n-1)/2
i of comparisons in inner loop
0 n-1
1 n-2
...
n-2 1
O(n2)-time
77
Sorting

• Input a sequence of n numbers a0, a1, , an-1
• Output arrange the n numbers into ascending
  order, i.e., from smallest to largest

We have learnt these sorting algorithmsselection
sort, bubble sort Next insertion sort
(optional, self-study)
78
Insertion Sort (optional, self-study)

• look at elements one by one
• build up sorted list by inserting the element at
  the correct location

optional
79
Example

• sort (34, 8, 64, 51, 32, 21) in ascending order
• Sorted part Unsorted part int moved
• 34 8 64 51 32 21
• 34 8 64 51 32 21 -
• 8 34 64 51 32 21 34
• 8 34 64 51 32 21 -
• 8 34 51 64 32 21 64
• 8 32 34 51 64 21 34, 51, 64
• 8 21 32 34 51 64 32, 34, 51, 64

optional
80
Insertion Sort Algorithm

• for i 1 to n-1 do
• begin key ai
• pos 0 while (apos lt key) (pos lt i) do
  pos pos 1
• shift apos, , ai-1 to the right
• apos key
• end

using linear search to find the correct position
for key
i.e., move ai-1 to ai, ai-2 to ai-1, ,
apos to apos1
finally, place key (the original ai) in apos
optional
81
Algorithm Analysis
for i 1 to n-1 do begin key ai pos 0
while (apos lt key) (pos lt i) do pos
pos 1 shift apos, , ai-1 to the right
apos key end

• Worst case input
• input is sorted in descending order
• Then, for ai
• finding the position takes i comparisons

i of comparisons in the while loop
1 1
2 2
...
n-1 n-1
total number of comparisons 1 2 n-1
(n-1)n/2
O(n2)-time
optional
82
Selection, Bubble, Insertion Sort

• All three algorithms have time complexity O(n2)
  in the worst case.
• Are there any more efficient sorting algorithms?
  YES, we will learn them later.
• What is the time complexity of the fastest
  comparison-based sorting algorithm?O(n log n)

83
Some exponential time algorithms Traveling
Salesman Problem, Knapsack Problem
84
Traveling Salesman Problem

• Input There are n cities.
• Output Find the shortest tour from a particular
  city that visit each city exactly once before
  returning to the city where it started.

This is known as Hamiltonian circuit
85
Example
2
a
b
To find a Hamiltonian circuit from a to a
7
8
3
5
c
d
1
Tour Length a -gt b -gt c -gt d -gt a 2 8 1 7
18 a -gt b -gt d -gt c -gt a 2 3 1 5 11 a
-gt c -gt b -gt d -gt a 5 8 3 7 23 a -gt c -gt
d -gt b -gt a 5 1 3 2 11 a -gt d -gt b -gt c
-gt a 7 3 8 5 23 a -gt d -gt c -gt b -gt a 7
1 8 2 18
86
Idea and Analysis

• A Hamiltonian circuit can be represented by a
  sequence of n1 cities v1, v2, , vn, v1, where
  the first and the last are the same, and all the
  others are distinct.
• Exhaustive search approach Find all tours in
  this form, compute the tour length and find the
  shortest among them.

How many possible tours to consider?
(n-1)! (n-1)(n-2)1
N.B. (n-1)! is exponential in terms of n refer
to batch 1 notes, slides 64,65
87
Knapsack Problem

• Input Given n items with weights w1, w2, , wn
  and values v1, v2, , vn, and a knapsack with
  capacity W.
• Output Find the most valuable subset of items
  that can fit into the knapsack.
• Application A transport plane is to deliver the
  most valuable set of items to a remote location
  without exceeding its capacity.

88
Example
w 7 v 42
capacity 10
w 5 v 25
w 4 v 40
w 3 v 12
item 1
item 2
item 3
item 4
knapsack
89
Idea and Analysis

• Exhaustive search approach Try every subset of
  the set of n given items, compute total weight of
  each subset and compute total value of those
  subsets that do NOT exceed knapsack's capacity.

How many subsets to consider?
2n, why?
90
Exercises (1)

• Suppose you have forgotten a password with 5
  characters. You only remember
• the 5 characters are all distinct
• the 5 characters are B, D, M, P, Y
• If you want to try all possible combinations, how
  many of them in total?
• What if the 5 characters can be any of the 26
  upper case alphabet?

5! 120
2625242322 or 26!/21!
91
Exercises (2)

• Suppose the password also contains 2 digits,
  i.e., 7 characters in total
• all characters are all distinct
• the 5 alphabets are B, D, M, P, Y
• the digit is either 0 or 1
• How many combinations are there?

7! 5040
92
Exercises (3)

• What if the password is in the form adaaada?
• a means alphabet, d means digit
• all characters are all distinct
• the 5 alphabets are B, D, M, P, Y
• the digit is either 0 or 1
• How many combinations are there?

5! 2! 240
93
Learning outcomes

• Able to carry out simple asymptotic analysis of
  algorithms
• See some examples of polynomial time and
  exponential time algorithms
• Able to apply searching/sorting algorithms and
  derive their time complexities