Divide and conquer

1

• LECTURE 8
• Divide and conquer

2
In the previous lecture we saw

• how to analyze recursive algorithms
• write a recurrence relation for the running time
• solve the recurrence relation by forward or
  backward substitution
• how to solve problems by decrease and conquer
• decrease by a constant/variable
• decrease by a constant/variable factor
• sometimes decrease and conquer lead to more
  efficient algorithms than brute force techniques

3
Outline

• Basic idea of divide and conquer
• Examples
• Master theorem
• Mergesort
• Quicksort

4
Basic idea of divide and conquer

• The problem is divided in several smaller
  instances of the same problem
• The subproblems must be independent (each one
  will be solved at most once)
• They should be of about the same size
• These subproblems are solved (by applying the
  same strategy or directly if their size is
  small enough)
• If the subproblem size is less than a given value
  (critical size) it is solved directly, otherwise
  it is solved recursively
• If necessary, the solutions obtained for the
  subproblems are combined

5
Basic idea of divide and conquer

• Divideconquer (n)
• IF nltnc THEN ltsolve P(n) directly to obtain rgt
• ELSE
• ltdivide P(n) in P(n1), , P(nk)gt
• FOR i1,k DO
• ri Divideconquer(ni)
• ENDFOR
• ltcombine r1, rk to obtain rgt
• ENDIF
• RETURN r

6
Example 1

• Compute the maximum of an array x1..n

n8, k2
3 2 7 5 1 6 4 5
3 2 7 5
1 6 4 5
Divide
Conquer
3 2
7 5
1 6
4 5
3 7
6 5
Combine
7 6
7
7
Example 1

• Algorithm
• Maximum(xleft..right)
• IF n1 then RETURN xleft
• ELSE
• m(leftright) DIV 2
• max1maximum(xleft..m)
• max2maximum(xm1..right)
• if max1gtmax2
• THEN RETURN max1
• ELSE RETURN max2
• ENDIF
• ENDIF

Efficiency analysis Problem size n Dominant
operation comparison Recurrence relation
0,
n1 T(n) T(n/2)T(n-n/2)1, ngt1
8
Example 1

• Backward substitution
• T(2m) 2T(2m-1)1
• T(2m-1)2T(2m-2)1 2
• T(2)2T(1)1 2m-1
• T(1)0
• ----------------------------
• T(n)12m-12m-1n-1

0,
n1 T(n) T(n/2)T(n-n/2)1,
ngt1 Particular case n2m 0,
n1 T(n)
2T(n/2)1, ngt1
9
Example 1

• General case.
• Proof by complete mathematical induction
• Basis step. n1 gtT(n)0n-1
• Inductive step.
• Let us suppose that T(k)k-1 for all kltn.
• Then
• T(n)n/2-1n-n/2-11n-1
• Thus T(n) n-1 gt
• T(n) belongs to T(n).

0,
n1 T(n) T(n/2)T(n-n/2)1,
ngt1 Particular case n2m gt
T(n)n-1
10
Example 1

• General case.
• Smoothness rule
• If T(n) belongs to ?(f(n)) for nbm
• T(n) is eventually nondecreasing (for ngtn0
  it is nondecreasing)
• f(n) is smooth (f(cn) belongs to ?(f(n))
  for any positive constant c)
• then T(n) belongs to ?(f(n)) for all n

• Remarks.
• All functions that do not grow too fast are
  smooth (polynomial and logarithmic)
• For our example T(n) is eventually nondecreasing,
  f(n)n is smooth, thus T(n) is from ?(n)

11
Example 2 binary search

• Check if a given value, v, is an element of an
  increasingly sorted array, x1..n (xiltxi1)

x1 xm-1 xm xm1 xn
vltxm
xmv
vgtxm
x1 xm-1 xm xm1 xm-1
xm1 xm-1 xm xm1 xn
True
xmv
leftgtright (empty array)
True
xleft..xright
False
12
Example 2 binary search

• Recursive variant
• binsearch(xleft..right,v)
• IF leftgtright THEN RETURN False
• ELSE
• m(leftright) DIV 2
• IF vxm THEN RETURN True
• ELSE
• IF vltxm
• THEN RETURN binsearch(xleft..m-1,v)
• ELSE RETURN binsearch(xm1..right,v)
  
• ENDIF
• ENDIF
• ENDIF

Remarks nc0 k2 Only one of the two
subproblems is solved This is rather a
decrease conquer approach
13
Example 2 binary search

• First iterative variant
• binsearch(x1..n,v)
• left1
• rightn
• WHILE leftltright DO
• m(leftright) DIV 2
• IF vxm THEN RETURN True
• ELSE
• IF vltxm
• THEN rightm-1
• ELSE leftm1
• ENDIF / ENDIF/ ENDWHILE
• RETURN False

Second iterative variant binsearch(x1..n,v)
left1 rightn WHILE leftltright DO
m(leftright) DIV 2 IF vltxm
THEN rightm ELSE leftm1
ENDIF / ENDWHILE IF xleftv THEN RETURN
True ELSE RETURN False
ENDIF
14
Example 2 binary search

• Second iterative variant
• binsearch(x1..n,v)
• left1
• rightn
• WHILE leftltright DO
• m(leftright) DIV 2
• IF vltxm
• THEN rightm
• ELSE leftm1
• ENDIF / ENDWHILE
• IF xleftv THEN RETURN True
• ELSE RETURN False
• ENDIF

• Correctness
• Precondition ngt1
• Postcondition
• returns True if v is in x1..n and False
  otherwise
• Loop invariant if v is in x1..n then it is
  in xleft..right
• left1, rightn gt the loop invariant is true
• It remains true after the execution of the loop
  body
• when rightleft it implies the postcondition

15
Example 2 binary search

• Second iterative variant
• binsearch(x1..n,v)
• left1
• rightn
• WHILE leftltright DO
• m(leftright) DIV 2
• IF vltxm
• THEN rightm
• ELSE leftm1
• ENDIF / ENDWHILE
• IF xleftv THEN RETURN True
• ELSE RETURN False
• ENDIF

Efficiency Worst case analysis (n2m)
1 n1 T(n) T(n/2)1
ngt1 T(n)T(n/2)1 T(n/2)T(n/4)1 T(2)T(1)
1 T(1)1 T(n)lg n1
O(lg n)
16
Master theorem

• Let us consider the following recurrence
  relation
• T0
  nltnc
• T(n)
• kT(n/m)TDC(n) ngtnc
• If TDC(n) belongs to ?(nd) (dgt0) then
• ?(nd) if
  kltmd
• T(n) belongs to ?(nd lgn) if kmd
• 
  ?(nlgk/lgm) if kgtmd
• A similar result holds for O and O notations

17
Master theorem

• Usefulness
• It can be applied in the analysis of divide
  conquer algorithms
• It avoids solving the recurrence relation
• In most practical applications the running time
  of the divide and combine steps has a polynomial
  order of growth
• It gives the efficiency class but it does not
  give the constants involved in the running time
• Example maximum computation
• k2 (division in two subproblems)
• m2 (each subproblem size is almost n/2)
• d0 (the divide and combine steps are of
  constant cost)
• Since kgtmd by applying the third case of the
  master theorem we obtain that T(n) belongs to
  ?(nlg k/lg m) ?(n)

18
Efficient sorting

• Elementary sorting methods belong to O(n2)
• Idea to increase the efficiency of sorting
  process
• divide the sequence in contiguous subsequences
  (usually two)
• sort each subsequence
• combine the sorted subsequences to obtain the
  sorted sequence

Merge sort
Quicksort
By position
Divide
By value
Combine
Merging
Concatenation
19
Merge sort

• Basic idea
• Divide x1..n in two subarrays x1..n/2 and
  xn/21..n
• Sort each subarray
• Merge the elements of x1..n/2 and
  xn/21..n and construct the sorted temporary
  array t1..n . Transfer the content of the
  temporary array in x1..n
• Remarks
• Critical value 1 (an array containing one
  element is already sorted)

20
Merge sort
21
Mergesort

• Algorithm
• mergesort(xleft..right)
• IF leftltright THEN
• m(leftright) DIV 2
• xleft..mmergesort(xleft..m)
• xm1..rightmergesort(xm1..right)
• xleft..rightmerge(xleft..m,xm1..right
  )
• ENDIF
• RETURN xleft..right
• Remark
• the algorithm will be called as
  mergesort(x1..n)

22
Mergesort
// transfer the last elements of the first
array (if it is the case) WHILE iltm DO
kk1 tkxi ii1 ENDWHILE //
transfer the last elements of the second array
(if it is the case) WHILE jltright DO
kk1 tkxj jj1 ENDWHILE RETU
RN t1..k

• Merge step
• merge (xleft..m,xm1..right)
• ileft jm1 k0
• // scan simultaneously the array
• // and transfer the smallest element
• WHILE iltm AND jltright DO
• IF xiltxj
• THEN kk1
• tkxi
• ii1
• ELSE kk1
• tkxj
• jj1
• ENDIF
• ENDWHILE

23
Mergesort

• The merge step can be used independently of the
  sorting process
• Variant of merge based on sentinels
• Merge two sorted arrays a1..p and b1..q
• Add large values to the end of each array
  ap1?, bq1 ?

Merge(a1..p,b1..q) ap1 ? bq1 ?
i1 j1 FOR k1,pq DO IF
ailtbj THEN ckai
ii1 ELSE
ckbj jj1
ENDIF / ENDFOR RETURN c1..pq
Efficiency analysis of merge step Dominant
operation comparison T(p,q)pq In mergesort
(pn/2, qn-n/2) T(n)ltn/2n-n/2n Thus
T(n) belongs to O(n)
24
Mergesort

• Efficiency analysis
• 0
  n1
• T(n)
• T(n/2)T(n-n/2)TM(n) ngt1
• Since k2, m2, d1 (TM(n) is from O(n)) it
  follows (by the second case of the Master
  theorem) that T(n) belongs to O(nlgn). In fact
  T(n) is from ?(nlgn)
• Remark.
• The main disadvantage of merge sort is the fact
  that it uses an auxiliary memory space of the
  size of the array
• If in the merge step the comparison is lt then
  the mergesort is stable

25
Quicksort

• Idea
• Divide the array x1..n in two subarrays x1..q
  and xq1..n such that all elements of x1..q
  are smaller than the elements of xq1..n
• Sort each subarray
• Concatenate the sorted subarrays

26
Quicksort

• An element xq having the properties
• xqgtxi, for all iltq
• xqltxi, for all igtq
• is called pivot
• A pivot is placed on its final position
• A good pivot divides the array in two subarrays
  of almost the same size
• Sometimes
• The pivot divides the array in an unbalanced
  manner
• Does not exist a pivot gt we must create one by
  swapping some elements

Example 1
3 1 2 4 7 5 8
Divide
3 1 2
7 5 8
1 2 3
5 7 8
1 2 3 4 5 7 8
Combine
27
Quicksort

• A position q having the property
• xiltxi, for all 1ltiltq and all q1ltjltn
• is called partitioning position
• A good partitioning position divides the array in
  two subarrays of almost the same size
• Sometimes
• The partitioning position divides the array in
  an unbalanced manner
• Does not exist such a partitioning position gt we
  must create one by swapping some elements

Example 2
3 1 2 7 5 4 8
Divide
3 1 2
7 5 4 8
1 2 3
4 5 7 8
1 2 3 4 5 7 8
Combine
28
Quicksort
The variant which use a pivot quicksort1(xle..r
i) IF leltri THEN qpivot(xle..ri)
xle..q-1quicksort1(xle..q-1)
xq1..riquicksort1(xq1..ri) RETURN
xle..ri
The variant which use a partitioning
position quicksort2(xle..ri) IF leltri THEN
qpartition(xle..ri) xle..qquicksort2(xle
..q) xq1..riquicksort2(xq1..ri) RETURN
xle..ri
29
Quicksort

• Constructing a pivot
• Choose a value from the array (the first one, the
  last one or a random one)
• Rearrange the elements of the array such that all
  elements which are smaller than the pivot value
  are before the elements which are larger than
  the pivot value
• Place the pivot value on its final position (such
  that all elements on its left are smaller than it
  and all elements on its right are larger than it)
• Idea for rearranging the elements
• use two pointers one starting from the first
  element and the other starting from the last
  element
• Increase/decrease the pointers until an inversion
  is found
• Repair the inversion
• Continue the process until the pointers cross
  each other

30
Quicksort
How to construct a pivot

• Choose the pivot value 4 (the last one)
• Place a sentinel on the first position (only for
  the initial array)
• i0, j7
• i2, j6
• i3, j4
• i4, j3 (the pointers crossed)
• The pivot is placed on its final position

0 1 2 3 4 5 6 7
1 7 5 3 8 2 4
4 1 7 5 3 8 2 4
4 1 7 5 3 8 2 4
4 1 2 5 3 8 7 4
4 1 2 3 5 8 7 4
4 1 2 3 4 8 7 5
31
Quicksort

• Remarks
• xright plays the role of a sentinel at the
  right
• At the left margin we can place explicitly a
  sentinel on x0 (only for the initial array
  x1..n)
• The conditions xigtv, xjltv allow to stop the
  search when the sentinels are encountered. Also
  it allows obtaining a balanced splitting when the
  array contains equal elements.
• At the end of the while loop the pointers
  satisfy either ij or ij1

• pivot(xleft..right)
• vxright
• ileft-1
• jright
• WHILE iltj DO
• REPEAT ii1 UNTIL xigtv
• REPEAT jj-1 UNTIL xjltv
• IF iltj THEN xi?xj ENDIF
• ENDWHILE
• xi ? xright
• RETURN i

32
Quicksort
Correctness Loop invariant If iltj then
xkltv for kleft..i xkgtv
for kj..right If igtj then xkltv for
kleft..i xkgtv for
kj1..right

• pivot(xleft..right)
• vxright
• ileft-1
• jright
• WHILE iltj DO
• REPEAT ii1 UNTIL xigtv
• REPEAT jj-1 UNTIL xjltv
• IF iltj THEN xi ?xj ENDIF
• ENDWHILE
• xi ?xright
• RETURN i

33
Quicksort
Efficiency Input size nright-left1 Dominant
operation comparison T(n)nc,
c0 if ij and c1 if
ij1 Thus T(n) belongs to ?(n)

• pivot(xleft..right)
• vxright
• ileft-1
• jright
• WHILE iltj DO
• REPEAT ii1 UNTIL xigtv
• REPEAT jj-1 UNTIL xjltv
• IF iltj THEN xi ?xj ENDIF
• ENDWHILE
• xi ?xright
• RETURN i

34
Quicksort

• Remark the pivot position does not always
  divide the array in a balanced manner
• Balanced manner
• the array is divided in two sequences of size
  almost n/2
• If each partition is balanced then the algorithm
  executes few operations (this is the best case)
• Unbalanced manner
• The array is divided in a subsequence of (n-1)
  elements and an empty subsequence
• If each partition is unbalanced then the
  algorithm executes much more operations (this is
  the worst case)

35
Quicksort

• Backward substitution
• T(n)T(n-1)(n1)
• T(n-1)T(n-2)n
• T(2)T(1)3
• T(1)0
• ---------------------
• T(n)(n1)(n2)/2-3
• Thus in the worst case quicksort is ?(n2)

Worst case analysis 0
if n1 T(n) T(n-1)n1, if ngt1
36
Quicksort
Best case analysis 0,
if n1 T(n) 2T(n/2)n, if ngt1

• By applying the second case of master theorem
  (for k2,m2,d1) one obtains that in the best
  case quicksort is ?(nlgn)

Thus quicksort belong to O(nlgn) and O(n2) An
average case analysis should be useful
37
Quicksort

• Average case analysis.
• Hypotheses
• Each partitioning step needs at most (n1)
  comparisons
• There are n possible positions for the pivot.
  Each has the same probability to be selected
  (Prob(q)1/n)
• If the pivot is on the position q then the number
  of comparisons satisfies
• Tq(n)T(q-1)T(n-q)(n1)

38
Quicksort

• The average number of comparisons is
• Ta(n)(T1(n)Tn(n))/n
• ((Ta(0)Ta(n-1))(Ta(1)Ta(n-2))(Ta(n-
  1)Ta(0)))/n (n1)
• 2(Ta(0)Ta(1)Ta(n-1))/n(n1)
• Thus
• n Ta(n) 2(Ta(0)Ta(1)Ta(n-1))n(n1)
• (n-1)Ta(n-1) 2(Ta(0)Ta(1)Ta(n-2))(n-1)n
• --------------------------------------------------
  ---------------
• By computing the difference
• nTa(n)(n1)Ta(n-1)2n
• Ta(n)(n1)/n Ta(n-1)2

39
Quicksort

• Average case analysis.
• By backward substitution
• Ta(n) (n1)/n Ta(n-1)2
• Ta(n-1) n/(n-1) Ta(n-2)2 (n1)/n
• Ta(n-2) (n-1)/(n-2) Ta(n-3)2 (n1)/(n-1)
• Ta(2) 3/2 Ta(1)2 (n1)/3
• Ta(1) 0
  (n1)/2
• --------------------------------------------------
  ---
• Ta(n) 22(n1)(1/n1/(n-1)1/3) 2(n1)(ln
  n-ln 3)2
• In the average case the complexity order is nlgn

40
Quicksort -variants
3 7 5 2 1 4 8 v3 3 7
5 2 1 4 8 i2, j5 3 1 5 2 7 4
8 i3, j4 3 1 2 5 7 4 8
i4, j3 Partitioning position
3 Complexity order of partition O(n)

• Finding a partitioning position
• partition(xleft..right)
• vxleft
• ileft
• jright1
• WHILE iltj DO
• REPEAT ii1 UNTIL xigtv
• REPEAT jj-1 UNTIL xjltv
• IF iltj THEN xi ?xj
• ENDIF
• ENDWHILE
• RETURN j

41
Next lecture will be on

• greedy strategy
• and its applications