Design and Analysis of Algorithm Lecture on Divide and Conquer

1
Design and Analysis of AlgorithmLecture
onDivide and Conquer

• Pradondet Nilagupta pom_at_ku.ac.th
• Department of Computer Engineering
• Kasetsart University

2
Acknowledgement

• This lecture note has been summarized from
  lecture note on Data Structure and Algorithm,
  Design and Analysis of Computer Algorithm all
  over the world. I cant remember where those
  slide come from. However, Id like to thank all
  professors who create such a good work on those
  lecture notes. Without those lectures, this slide
  cant be finished.

3
Review What is an algorithm?

• A step by step procedure for performing some task
  in a finite amount of time
• A precise set of instructions for achieving a
  particular goal

4
Review General Concepts

• Algorithm strategy
• Approach to solving a problem
• May combine several approaches
• Algorithm structure
• Iterative ? execute action in loop
• Recursive ? reapply action to subproblem(s)
• Problem type
• Satisfying ? find any satisfactory solution
• Optimization ? find best solutions (vs. cost
  metric)

5
Review What you need to know about algorithms?

• Running time
• Memory requirements
• Understand how the complexity of an algorithm is
  measured
• Develop techniques to measure these complexities
  experimentally
• In other words, measure the efficiency of an
  algorithm!

6
Review Measures of efficiency

• Three main measures
• Best
• Worst
• Average
• E.g. consider a list
• Best first item in the list
• Worst last in the list, or not in the list
• Average in the middle of the list

7
Review An example of an algorithm (pseudocode)

• BEGIN
• Get WAGES
• IF WAGES lt100
• Tax wages0.25
• ELSE
• Tax 1000.25 (wages-100)0.4
• END IF
• Display Tax
• END
• It calculates tax at 25 on the first 100 of
  earnings then at 40 on subsequent earnings.

8
Some Algorithm Strategies

• Recursive algorithms
• Backtracking algorithms
• Divide and conquer algorithms
• Dynamic programming algorithms
• Greedy algorithms
• Brute force algorithms
• Branch and bound algorithms
• Heuristic algorithms

9
Divide-and-Conquer

• A general methodology for using recursion to
  design efficient algorithms
• It solves a problem by
• Diving the data into parts
• Finding sub solutions for each of the parts
• Constructing the final answer from the sub
  solutions

10
Divide and Conquer

• Based on dividing problem into subproblems
• Approach
• Divide problem into smaller subproblems
• Subproblems must be of same type
• Subproblems do not need to overlap
• Solve each subproblem recursively
• Combine solutions to solve original problem
• Usually contains two or more recursive calls

11
Divide-and-conquer technique
a problem of size n
subproblem 2 of size n/2
subproblem 1 of size n/2
a solution to subproblem 1
a solution to subproblem 2
a solution to the original problem
12
Divide and Conquer Algorithms

• Based on dividing problem into subproblems
• Divide problem into sub-problems
• Subproblems must be of same type
• Subproblems do not need to overlap
• Conquer by solving sub-problems recursively. If
  the sub-problems are small enough, solve them in
  brute force fashion
• Combine the solutions of sub-problems into a
  solution of the original problem (tricky part)

13
D-A-C

• For Divide-and-Conquer algorithms the running
  time is mainly affected by 3 criteria
• The number of sub-instances into which a problem
  is split.
• The ratio of initial problem size to sub-problem
  size.
• The number of steps required to divide the
  initial instance and to combine sub-solutions.

14
An example of D-A-C

• Given an array of n values find, in order, the
  lowest two values efficiently.
• To make life easier we assume n is at least 2 and
  that it is always divisible by 2
• Have you understood the problem??
• The solution should be 2 followed by 4

2 10 9 6 4 5
What if??
2 8 2 7 9 3
15
Possible Solutions

• Sort the elements and take the two first
• We get more information than we require (not very
  efficient!!)
• Find the lowest, then find the second lowest
• Better than the first but still we have to go
  through the array two times
• Be careful not to count the lowest value twice
• Search through the array keeping track of the
  lowest two found so far
• Lets see how the divide and conquer technique
  works!

16
Divide and conquer approach

• There are many ways to divide the array
• It seems reasonable that the division should be
  related to the problem
• Division based on 2!!
• Divide the array into two parts?
• Divide the array into pairs?
• Go ahead with the second option!

17
The algorithm

• Lets assume the following array
• We divide the values into pairs
• We sort each pair
• Get the first pair (both lowest values!)

2 6 7 3 5 6 9 2 4 1
2 6 7 3 5 6 9 2 4 1
2 6 3 7 5 6 2 9 1 4
18
The algorithm (2)

• We compare these values (2 and 6) with the values
  of the next pair (3 and 7)
• Lowest 2,3
• The next one (5 and 6)
• Lowest 2,3
• The next one (2 and 9)
• Lowest 2,2
• The next one (1 and 4)
• Lowest 1,2

2 6 3 7 5 6 2 9 1 4
19
Example Divide and Conquer

• Binary Search
• Heap Construction
• Tower of Hanoi
• Exponentiation
• Fibonnacci Sequence
• Quick Sort
• Merge Sort
• Multiplying large Integers
• Matrix Multiplications
• Closest Pairs

20
Exponentiation

• Power can be easily computed in O(logn) time as
  follows
• long Power (int x, int n)
• if (n0) return 1
• else
• y Power (x, n/2)
• if (n 2 0) return(yy) // n even
• else return (yyx)

21
Time Complexity Example 1 Fibonnacci Sequence

• Fibonnacci sequence F(n) F(n-1) F(n-2)

Approach 1 int Fib(n) if (n lt 2) return
(n) else return (Fib (n-1) Fib
(n-2))
Approach 2 int Fib(n) if (n lt 2) return
(n) else int F1 1 F2 0
for (i 2 i lt n i)
F F1 F2 F2 F1
F1 F return
(F)
22
Fibonnacci Sequence

• Approach 3 the complexity is O(logn)
• Note that we can rewrite
• F(n) 1 1 F(n-1)
• F(n-1) 1 0 F(n-2)
• Thus,
• 
  2
• F(n) 1 1 F(n-1) 1
  1 F(n-2)
• 
  
• F(n-1) 1 0 F(n-2) 1 0
  F(n-3)
• (n-1)
• 1 1 F(1)
• 1 0 F(0)
• Therefore, computing F(n) is reduced to
  computing
• (n-1)

23
Heap Construction

• Construct heap of T gt
• Construct heap T1,
• Construct heap T2
• Adjust heap with root T

T
T1
T2
24
Divide and Conquer Examples

• Quicksort
• Partition array into two parts around pivot
• Recursively quicksort each part of array
• Concatenate solutions
• Mergesort
• Partition array into two parts
• Recursively mergesort each half
• Merge two sorted arrays into single sorted array

25
Quick Sort
26
Sorting Problem Revisited

• Given an unsorted array
• Goal sort it

5 2 4 7 1 3 2 6
1 2 2 3 4 5 6 7
27
Quick Sort

• Approach
• Select pivot value (near median of list)
• Partition elements (into 2 lists) using pivot
  value
• Recursively sort both resulting lists
• Concatenate resulting lists
• For efficiency pivot needs to partition list
  evenly
• Performance
• O( n log(n) ) average case
• O( n2 ) worst case

28
Quick Sort Algorithm

• If list below size K
• Sort w/ other algorithm
• Else pick pivot x and partition S into
• L elements lt x
• E elements x
• G elements gt x
• Quicksort L G
• Concatenate L, E G
• If not sorting in place

x
x
L
G
E
x
29
Quick Sort Code

• void quickSort(int a, int x, int y) int
  pivotIndex if ((y x) gt 0)
  pivotIndex partionList(a, x, y) quickSort(a,
  x, pivotIndex 1)
• quickSort(a, pivotIndex1, y)
• int partionList(int a, int x, int y)
• // partitions list and returns index of
  pivot

Lower end of array region to be sorted
Upper end of array region to be sorted
30
Quick Sort Example
7 2 8 5 4
2 4 5 7 8
2 5 4
7
8
2 4 5
7
8
5 4
2
4 5
2
4
5
4
5
Partition Sort
Result
31
Quick Sort Code

• int partitionList(int a, int x, int y)
• int pivot ax
• int left x
• int right y
• while (left lt right)
• while ((aleft lt pivot) (left lt
  right))
• left
• while (aright gt pivot)
• right--
• if (left lt right)
• swap(a, left, right)
• swap(a, x, right)
• return right

Use first element as pivot
Partition elements in array relative to value of
pivot
Place pivot in middle of partitioned array,
return index of pivot
32
Merge Sort
33
Merge Sort

• Approach
• Partition list of elements into 2 lists
• Recursively sort both lists
• Given 2 sorted lists, merge into 1 sorted list
• Examine head of both lists
• Move smaller to end of new list
• Performance
• O( n log(n) ) average / worst case

34
Merge Example
2 4 5
2 7
7
4 5 8
8
2 4 5 7
2
7
8
4 5 8
2 4 5 7 8
2 4
7
5 8
35
Merge Sort Example
2 4 5 7 8
7 2 8 5 4
4 5 8
8 5 4
2 7
7 2
8
4 5
8
5 4
2
7
2
7
4
4
5
5
Split
Merge
36
Merge Sort Code

• void mergeSort(int a, int x, int y) int
  mid (x y) / 2
• if (y x) return
• mergeSort(a, x, mid)
• mergeSort(a, mid1, y)
• merge(a, x, y, mid)
• void merge(int a, int x, int y, int mid)
• // merges 2 adjacent sorted lists in array

Upper end of array region to be sorted
Lower end of array region to be sorted
37
Merge Sort Code
Upper end of 1st array region

• void merge (int a, int x, int y, int mid)
• int size y x
• int left x
• int right mid1
• int tmp int j
• for (j 0 j lt size j)
• if (left gt mid) tmpj aright
• else if (right gt y) (aleft lt
  aright)
• tmpj aleft
• else tmpj aright
• for (j 0 j lt size j)
• axj tmpj

Upper end of 2nd array region
Lower end of 1st array region
Copy smaller of two elements at head of 2 array
regions to tmp buffer, then move on
Copy merged array back
38
Mergesort Divide Step
Step 1 Divide
5 2 4 7 1 3 2 6
5 2 4 7 1 3 2 6
5 2 4 7 1 3 2 6
5 2 4 7 1 3 2 6
log(n) divisions to split an array of size n into
single elements
39
Mergesort Conquer Step

• Step 2 Conquer

5 2 4 7 1 3 2 6
O(n)
2 5 4 7 1 3 2 6
O(n)
2 4 5 7 1 2 3 6
O(n)
1 2 2 3 4 5 6 7
O(n)
O(n logn)
logn iterations, each iteration takes O(n) time.
Total Time
40
Mergesort Combine Step

• Step 3 Combine
• 2 arrays of size 1 can be easily merged to form a
  sorted array of size 2
• 2 sorted arrays of size n and m can be merged in
  O(nm) time to form a sorted array of size nm

5 2 2 5
41
Mergesort Combine Step
Combining 2 arrays of size 4
2 4 5 7
2 3 6
2 4 5 7
1 2 3 6
1 2
1
4 5 7
3 6
4 5 7
2 3 6
1 2 2 3
1 2 2
Etcetera
4 5 7
6
1 2 2 3 4
1 2 2 3 4 5 6 7
42
Mergesort Example
20
4
7
6
1
3
9
5
Divide
20
4
7
6
1
3
9
5
20
4
7
6
1
3
9
5
1
3
9
5
20
4
7
6
4
20
6
7
1
3
5
9
Conquer
4
6
7
20
1
3
5
9
1
3
4
5
6
7
9
20
43
MergeSort Running Time

• The problem is simplified to baby steps
• for the ith merging iteration, the complexity of
  the problem is O(n)
• number of iterations is O(log n)
• running time O(n logn)

44
Integer Multiplication
45
Integer Arithmetic

• Add. Given two n-digit integers a and b, compute
  a b.
• O(n) bit operations.
• Multiply. Given two n-digit integers a and b,
  compute a b.
• Brute force solution ?(n2) bit operations.

Multiply
Add
46
Divide-and-Conquer Multiplication

• To multiply two n-digit integers
• Multiply four ½n-digit integers.
• Add two ½n-digit integers, and shift to obtain
  result.

assumes n is a power of 2
47
Karatsuba Multiplication

• To multiply two n-digit integers
• Add two ½n digit integers.
• Multiply three ½n-digit integers.
• Add, subtract, and shift ½n-digit integers to
  obtain result.

B
C
A
C
A
48
Karatsuba Multiplication (cont)

• Theorem. Karatsuba-Ofman, 1962 Can multiply
  two n-digit integers in O(n1.585) bit operations.

49
Karatsuba Recursion Tree
n
T(n)
3(n/2)
T(n/2)
T(n/2)
T(n/2)
9(n/4)
T(n/4)
T(n/4)
T(n/4)
T(n/4)
T(n/4)
T(n/4)
T(n/4)
T(n/4)
T(n/4)
. . .
. . .
3k (n / 2k)
T(n / 2k)
. . .
. . .
3 lg n (2)
T(2)
T(2)
T(2)
T(2)
T(2)
T(2)
T(2)
T(2)
50
Multiplying Large Integers

• High school multiplications

eg)
1 1 0 0 1 1
?
1 0 1 0 0 1
A an-1 an-2 a1 a0
1 1 0 0 1 1
B bn-1 bn-2 b1 b0
?
0 0 0 0 0 0
0 0 0 0 0 0
1 1 0 0 1 1
Time O(N2)
0 0 0 0 0 0
1 1 0 0 1 1
51
Multiplying Large Integers (cont)

• A1 an-1 an-2 an/2 A0 an/2-1 a1 a0
• B1 bn-1 bn-2 bn/2 B0 bn/2-1 b1 b0

Define X A1 A0 Y B1 B0
AB A1B12n (XY A1B1 A0B0) 2n/2 A0B0
Complexity? An integer multiplication of two n
bit integer is reduced to gt three integer
multiplications of n/2 bits, and 6 additions of n
bits
T(n) 3T(n/2) O(n) O(nlog23) O(n1.59)
52
Multiplying integers(Decimal numbers)
A
A1
2 3 6 7
2 3 6 7 8 1 8 3
8 1 8 3
A0
B1
B0
B
1 3 0 4
6 4 9 1
1 3 0 4 6 4 9 1

• AB ( A1104 A0 ) ( B1104 B0 )
• A1B1108 (A1B0A0B1)104 A0B0
• 4 multiplications
• 3 additions
• 2 shiftings

T(n) 4T(n/2) O(n) T(1) O(1) T(n) O(n2)
53
Multiplying integers(Decimal numbers)D-A-C
Approach

• Let X ( A1 A0 ) , Y ( B1 B0 )
• XY A1B1 A1B0 A0B1 A0B0
• XY - A1B1 - A0B0
• A1B0A0B1
• AB ( A1104 A0 ) ( B1104 B0 )
• A1B1108 (A1B0A0B1)104 A0B0
• A1B1108 (XY - A1B1 - A0B0)104 A0B0
• Recursively compute XY, A1B1 , A0B0,
• 3 multiplications
• 6 additions
• 2 shiftings

T(n) 3T(n/2) O(n) T(1) O(1) T(n) nlog23
O(n1.59)
54
Matrix Multiplication
55
Matrix Multiplication

• Matrix multiplication. Given two n-by-n matrices
  A and B, compute C AB.
• Brute force. ?(n3) arithmetic operations.
• Fundamental question. Can we improve upon brute
  force?

56
Matrix Multiplication Warmup

• Divide-and-conquer.
• Divide partition A and B into ½n-by-½n blocks.
• Conquer multiply 8 ½n-by-½n recursively.
• Combine add appropriate products using 4 matrix
  additions.

57
Matrix Multiplication Key Idea

• Key idea. multiply 2-by-2 block matrices with
  only 7 multiplications.
• 7 multiplications.
• 18 10 8 additions (or subtractions).

58
Fast Matrix Multiplication

• Fast matrix multiplication. (Strassen, 1969)
• Divide partition A and B into ½n-by-½n blocks.
• Compute 14 ½n-by-½n matrices via 10 matrix
  additions.
• Conquer multiply 7 ½n-by-½n matrices
  recursively.
• Combine 7 products into 4 terms using 8 matrix
  additions.
• Analysis.
• Assume n is a power of 2.
• T(n) arithmetic operations.

59
Fast Matrix Multiplication in Practice

• Implementation issues.
• Sparsity.
• Caching effects.
• Numerical stability.
• Odd matrix dimensions.
• Crossover to classical algorithm around n 128.
• Common misperception "Strassen is only a
  theoretical curiosity."
• Advanced Computation Group at Apple Computer
  reports 8x speedup on G4 Velocity Engine when n
  2,500.
• Range of instances where it's useful is a subject
  of controversy.

60
Fast Matrix Multiplication in Theory

• Q. Multiply two 2-by-2 matrices with only 7
  scalar multiplications?
• A. Yes! Strassen, 1969
• Q. Multiply two 2-by-2 matrices with only 6
  scalar multiplications?
• A. Impossible. Hopcroft and Kerr, 1971
• Q. Two 3-by-3 matrices with only 21 scalar
  multiplications?
• A. Also impossible.
• Q. Two 70-by-70 matrices with only 143,640
  scalar multiplications?
• A. Yes! Pan, 1980

61
Fast Matrix Multiplication in Theory

• Decimal wars.
• December, 1979 O(n2.521813).
• January, 1980 O(n2.521801).
• Best known. O(n2.376) Coppersmith-Winograd,
  1987.
• Conjecture. O(n2?) for any ? gt 0.
• Caveat. Theoretical improvements to Strassen are
  progressively less practical.

62
Observations

• Comparison n 70
• Direct multiplication 703 343,000
• Strassen 70lg 7 is approximately 150,000
• Crossover point typically around n 20
• Hopcroft and Kerr have shown 7 multiplications
  are necessary to multiply 2 by 2 matrices
• But we can do better with larger matrices
• Best lower bound is W(n2) (since there are n2
  entries)
• Matrix multiplication can be used in some graph
  algorithms as a fundamental step, so theoretical
  improvements in the efficiency of this operation
  can lead to theoretical improvements in those
  algorithms

63
Closest Pair of Points
64
Closest Pair of Points

• Closest pair. Given n points in the plane, find
  a pair with smallest Euclidean distance between
  them.
• Fundamental geometric primitive.
• Graphics, computer vision, geographic information
  systems, molecular modeling, air traffic control.
• Special case of nearest neighbor, Euclidean MST,
  Voronoi.
• Brute force. Check all pairs of points p and q
  with ?(n2) comparisons.
• 1-D version. O(n log n) easy if points are on a
  line.
• Assumption. No two points have same x coordinate.

fast closest pair inspired fast algorithms for
these problems
to make presentation cleaner
65
Closest Pair of Points First Attempt

• Divide. Sub-divide region into 4 quadrants.

L
66
Closest Pair of Points First Attempt

• Divide. Sub-divide region into 4 quadrants.
• Obstacle. Impossible to ensure n/4 points in
  each piece.

L
67
Closest Pair of Points

• Algorithm.
• Divide draw vertical line L so that roughly ½n
  points on each side.

L
68
Closest Pair of Points

• Algorithm.
• Divide draw vertical line L so that roughly ½n
  points on each side.
• Conquer find closest pair in each side
  recursively.

L
69
Closest Pair of Points

• Algorithm.
• Divide draw vertical line L so that roughly ½n
  points on each side.
• Conquer find closest pair in each side
  recursively.
• Combine find closest pair with one point in
  each side.
• Return best of 3 solutions.

seems like ?(n2)
L
70
Closest Pair of Points

• Find closest pair with one point in each side,
  assuming that distance lt ?.

L
? min(12, 21)
71
Closest Pair of Points

• Find closest pair with one point in each side,
  assuming that distance lt ?.
• Observation only need to consider points within
  ? of line L.

L
? min(12, 21)
?
72
Closest Pair of Points

• Find closest pair with one point in each side,
  assuming that distance lt ?.
• Observation only need to consider points within
  ? of line L.
• Sort points in 2?-strip by their y coordinate.

L
7
6
5
4
? min(12, 21)
3
2
1
?
73
Closest Pair of Points

• Find closest pair with one point in each side,
  assuming that distance lt ?.
• Observation only need to consider points within
  ? of line L.
• Sort points in 2?-strip by their y coordinate.
• Only check distances of those within 11 positions
  in sorted list!

L
7
6
5
4
? min(12, 21)
3
2
1
?
74
Closest Pair of Points

• Def. Let si be the point in the 2?-strip,
  withthe ith smallest y-coordinate.
• Claim. If i j ? 12, then the distance
  betweensi and sj is at least ?.
• Pf.
• No two points lie in same ½?-by-½? box.
• Two points at least 2 rows aparthave distance ?
  2(½?). ?
• Fact. Still true if we replace 12 with 7.

j
39
31
½?
2 rows
30
½?
29
½?
i
28
27
26
25
?
?
75
Closest Pair Algorithm
Closest-Pair(p1, , pn) Compute separation
line L such that half the points are on one
side and half on the other side. ?1
Closest-Pair(left half) ?2
Closest-Pair(right half) ? min(?1, ?2)
Delete all points further than ? from separation
line L Sort remaining points by
y-coordinate. Scan points in y-order and
compare distance between each point and next
11 neighbors. If any of these distances is
less than ?, update ?. return ?.
O(n log n)
2T(n / 2)
O(n)
O(n log n)
O(n)
76
Closest Pair of Points Analysis

• Running time.
• Q. Can we achieve O(n log n)?
• A. Yes. Don't sort points in strip from scratch
  each time.
• Each recursive returns two lists all points
  sorted by y coordinate, and all points sorted by
  x coordinate.
• Sort by merging two pre-sorted lists.

77
Other Applications

• Find median (Probabilistic)
• Convex Hull
• Reversing array
• Maximum consecutive subsequence
• Etc