Title: Randomized Algorithms
1ADVANCED ALGORITHM ANALYSISLECTURE
2RANDOMIZED ALGORITHMS
 Prof. Vuda Sreenivasarao
 Bahir Dar UniversityETHIOPIA
2Randomized Algorithms
 Objectives
 1.Randomized Algorithms
 1.1.Basic concept of Randomized algorithms with
example.  1.2.Probabilistic inequalities in analysis with
examples.  1.3.Amortized analysis with examples.
 1.4.Competitive analysis with examples.
3Deterministic Algorithms
 Input
Output  Goal to prove that the algorithm solves the
problem correctly always and quickly typically
the number of steps should be polynomial in the
size of the input.
Algorithm
4A short list of categories
 Algorithm types we will consider include
 Simple recursive algorithms
 Backtracking algorithms
 Divide and conquer algorithms
 Dynamic programming algorithms
 Greedy algorithms
 Branch and bound algorithms
 Brute force algorithms
 Randomized algorithms
5Randomized algorithms
 Randomization. Allow fair coin flip in unit
time.  Why randomize? Can lead to simplest, fastest, or
only known algorithm for a particular problem.  Examples Symmetry breaking protocols, graph
algorithms, quicksort, hashing, load balancing,
Monte Carlo integration, cryptography.
6Randomized algorithms
 A randomized algorithm is just one that depends
on random numbers for its operation.  These are randomized algorithms
 Using random numbers to help find a solution to a
problem.  Using random numbers to improve a solution to a
problem.  These are related topics
 Getting or generating random numbers.
 Generating random data for testing (or other)
purposes.
71.Randomized Algorithms


 INPUT
OUTPUT  RANDOM NUMBERS
 In addition to input algorithm takes a source of
random numbers and makes random choices during
execution.  Random algorithms make decisions on rolls of the
dice.  Ex Quick sort, Quick Select and Hash tables.
ALGORITHM
8Why use randomness?
 Avoid worstcase behavior randomness can
(probabilistically) guarantee average case
behavior  Efficient approximate solutions to inflexible
problems.
9Making Decision
Flip a coin.
10Making Decision
Flip a coin!
An algorithm which flip coins is called a
randomized algorithm.
11Why Randomness?
Making decisions could be complicated.
A randomized algorithm is simpler.
Consider the minimum cut problem
Can be solved by max flow.
Randomized algorithm?
Pick a random edge and contract.
And repeat until two vertices left.
12Why Randomness?
Making good decisions could be expensive.
A randomized algorithm is faster.
Consider a sorting procedure.
5 9 13 8 11 6 7 10
5 6 7
8
9 10 11 13
Picking an element in the middle makes the
procedure very efficient, but it is expensive
(i.e. linear time) to find such an element.
Picking a random element will do.
13Why Randomness?
Making good decisions could be expensive.
A randomized algorithm is faster.
 Minimum spanning trees.
 A linear time randomized algorithm,
 known but no linear time deterministic
algorithm.  Primality testing
 A randomized polynomial time algorithm,
 but it takes thirty years to find a
deterministic one.  Volume estimation of a convex body
 A randomized polynomial time approximation
algorithm,  but no known deterministic polynomial time
approximation algorithm.
14Why Randomness?
In many practical problems, we need to deal with
HUGE input, and dont even have time to read it
once. But can we still do something useful?
Sub linear algorithm randomness is essential.
 Fingerprinting verifying equality of strings,
pattern matching.  The power of two choices load balancing,
hashing.  Random walk check connectivity in logspace.
15Advantages of randomized algorithms
 Simplicity.
 Performance.
 For many problems, a randomized algorithm is the
simplest, the fastest, or both.
16Scope of Randomized Algorithms
 Number theoretic algorithms primality testing
Monte Carlo.  Data structures Sorting, order statistics,
searching, computational geometry.  Algebraic identities Polynomial and matrix
identity verification. Interactive proof systems.  Mathematical programming Faster algorithms for
linear programming. Rounding linear program
solutions to integer program solutions.
17Scope of Randomized Algorithms
 Graph algorithms Minimum spanning trees shortest
paths, minimum cuts.  Counting and enumeration Matrix permanent.
Counting combinatorial structures.  Parallel and distributed computing Deadlock
avoidance, distributed consensus.  Probabilistic existence proofs Show that a
combinatorial object arises with nonzero
probability among objects drawn from a suitable
probability space.
18Randomized algorithms
 In a randomized algorithm (probabilistic
algorithm), we make some random choices.  2 types of randomized algorithms
 For an optimization problem, a randomized
algorithm gives an optimal solution. The average
case timecomplexity is more important than the
worst case timecomplexity.  For a decision problem, a randomized algorithm
may make mistakes. The probability of producing
wrong solutions is very small.
19Types of Random Algorithms
 Las Vegas
 Guaranteed to produce correct answer, but running
time is probabilistic.  Monte Carlo
 Running time bounded by input size, but answer
may be wrong.
20 Las Vegas or Monte Carlo?
21Las Vegas
 Always gives the true answer.
 Running time is random.
 Running time is bounded.
 Quick sort is a Las Vegas algorithm.
 A Las Vegas algorithm always produces the correct
answer its running time is a random variable
whose expectation is bounded (say by a
polynomial).
22Monte Carlo
 It may produce incorrect answer!
 We are able to bound its probability.
 By running it many times on independent random
variables, we can make the failure probability
arbitrarily small at the expense of running time.  A Monte Carlo algorithm runs for a fixed number
of steps and produces an answer that is correct
with probability ?1/2.
23RP Class ( randomized polynomial )
 Bounded polynomial time in the worst case.
 If the answer is Yes Pr return Yes gt ½.
 If the answer is No Pr return Yes 0.
 ½ is not actually important.
24PP Class ( probabilistic polynomial )
 Bounded polynomial time in worst case.
 If the answer is Yes Pr return Yes gt ½.
 If the answer is No Pr return Yes lt ½.
 Unfortunately the definition is weak because the
distance to ½ is important but is not considered.
25Routing Problem
 There are n computers.
 Each computer has a packet.
 Each packet has a destination D(i).
 Packets can not follow the same edge
simultaneously.  An oblivious algorithm is required.
 For any deterministic oblivious algorithm on a
network of N nodes each of out degree d, there is
an instance of permutation routing requiring
(N/d) ½.
26Routing Problem
 Pick random intermediate destination.
 Packet i first travels to the intermediate
destination and then to the final destination.  With probability at least 1(1/N), every packet
reaches its destination in 14n of fewer steps in
Qn.  The expected number of steps is 15n.
271.Randomized Algorithms
 EXAMPLE Expectation
 X() flips a coin.
 Heads One second to execute.
 Tails Three seconds.
 Let X be running time of one cell to X()
 with probability 0.5 X is 1.
 With probability 0.5X is 3.
 Here random variable is X.
 Expected value of XEX0.5x10.5x3 2 seconds
expected time.  Suppose we run X(),// take time X
 X(),// take
time Y  Total running time is TXY , here T is random
variable.  What is expected total time ET?
 Linearity of expectation EXYEXEY224
seconds expected time.
28Min_Cut Problem
 Definition
 Min_cut Problem is to find the minimum edge set
C such that removing C disconnects the graph.  Traditional Solution
 Maxflow The maximum amount of flow is equal to
the capacity of a minimum cut
29Example of Min_Cut
a
b
e.g. Min_Cut 2
30Intuition
 Let a graph G has n nodes and size of min_cut
k, that is C k then degree for each node gt
k  total number of edges in G gt nk/2.
 Randomized Min_Cut
 Input a graph G(V, E), V n
 Output min_cut C
 Repeat Pick any edge uniformly at random,
collapse it and remove selfloops  Until V down to 2.
 Running time is O(n2)
31Example of Randomized Min_Cut
min_cut 2
Or maybe
min_cut 4
32Las Vegas VS Monte Carlo
 Las Vegas Algorithm It always produces the
correct answer and the expected running time is
finite (e.s.p. randomized quick sort).  Monte Carlo Algorithm It may produce incorrect
answer but with bounded error probability (e.s.p.
randomized min_cut).
33Analysis
 Probability of the first edge C
 Prob (kn/2 k ) / (kn/2)
 (n2) / n
 Probability of the second edge C
 Prob (k(n1)/2 k ) / (k(n1)/2)
 (n3) / (n1)
min_cut
34Analysis
Iteration Probability of avoiding C
1 (n 2) / n
2 (n 3) / (n 1)
3 (n 4) / (n 2)
4 (n 5) / (n 3)
n  2 1 / 3
Prob. Of outputting C Pr gt
35Analysis
 Probability of getting a min_cut is at least
2/n(n1)  Might look like small, but gets bigger after
repeating the algorithm e.s.p. If algorithm is
running twice, probability of outputting C would
be  Pr 1 ( 1 ) 2
 Let r be the number of running times of
algorithm.  Total running time O(nr)
36Internet Minimum Cut
June 1999 Internet graph, Bill Cheswick http//res
earch.lumeta.com/ches/map/gallery/index.html
371.Randomized Algorithms
 EXAMPLE Hash Tables
 Random hash code maps each possible key to
randomly chosen bucket, but a keys random hash
code never changes.  Good model for how a good hash code will
perform  Assume hash table uses chaining , no duplicate
keys.  Perform find (k). K hashes to bucket b cost of
search is one birr , plus birr for every entry in
the bucket b whose key is not k.  Suppose there are n keys in table besides k.
 V1,V2,Vn Random variables for each key Ki,
Vi 1 if key Ki hashes to bucket b , Zero
otherwise.
381.Randomized Algorithms
 Cost of find(k) is T 1V1V2.Vn.
 Expectation cost is ET 1EV1EV2E
Vn  N buckets each key has 1/N probability of
hashing to bucket b 0  EVi1/N
 ET1n/N.
 If load factor C,ET O(1).
 Hash table operations take O(1) expected
amortized time.
39Contention Resolution in a Distributed System
 Contention resolution. Given n processes P1, ,
Pn, each competing for access to a shared
database. If two or more processes access the
database simultaneously, all processes are locked
out. Devise protocol to ensure all processes get
through on a regular basis.  Restriction. Processes can't communicate.
 Challenge Need symmetrybreaking
 paradigm.
P1
P2
...
Pn
40Contention Resolution Randomized Protocol
 Protocol. Each process requests access to the
database at time t with probability p 1/n.  Claim. Let Si, t event that process i
succeeds in accessing the database at time t.
Then 1/(e ? n) ? PrS(i, t) ? 1/(2n).  Pf. By independence, PrS(i, t) p
(1p)n1.  Setting p 1/n, we have PrS(i, t) 1/n (1 
1/n) n1. ?  Useful facts from calculus. As n increases from
2, the function  (1  1/n)n1 converges monotonically from 1/4 up
to 1/e  (1  1/n)n1 converges monotonically from 1/2
down to 1/e.
none of remaining n1 processes request access
process i requests access
between 1/e and 1/2
value that maximizes PrS(i, t)
41Contention Resolution Randomized Protocol
 Claim. The probability that process i fails to
access the database inen rounds is at most 1/e.
After e?n(c ln n) rounds, the probability is at
most nc.  Pf. Let Fi, t event that process i fails to
access database in rounds 1 through t. By
independence and previous claim, we havePrF(i,
t) ? (1  1/(en)) t.  Choose t ?e ? n?
 Choose t ?e ? n? ?c ln n?
421.Randomized Algorithms
 EXAMPLE Nuts and Bolts
 Suppose we are given n nuts and n bolts of
different sizes.  Each nut matches exactly one bolt and vice versa.
 The nuts and bolts are all almost exactly the
same size, so we cant tell if one bolt is bigger
than the other, or if one nut is bigger than the
other. If we try to match a nut witch a bolt,
however, the nut will be either too big, too
small, or just right for the bolt.  Our task is to match each nut to its
corresponding bolt.
431.Randomized Algorithms
 Suppose we want to find the nut that matches a
particular bolt.  The obvious algorithm test every nut until we
find a match requires exactly n1 tests in the
worst case.  We might have to check every bolt except one if
we get down the last bolt without finding a
match, we know that the last nut is the one were
looking for.  Intuitively, in the average case, this
algorithm will look at approximately n/2 nuts.
But what exactly does average case mean?
44Deterministic vs. Randomized Algorithms
 Normally, when we talk about the running time of
an algorithm, we mean the worstcase running
time. This is the maximum, over all problems of a
certain size, of the running time of that
algorithm on that input  On extremely rare occasions, we will also be
interested in the bestcase running time  The averagecase running time is best defined by
the expected value, over all inputs X of a
certain size, of the algorithms running time for
X
45Randomized Algorithms
 Two kinds of algorithms deterministic and
randomized.  A deterministic algorithm is one that always
behaves the same way given the same input the
input completely determines the sequence of
computations performed by the algorithm.  Randomized algorithms, on the other hand, base
their behavior not only on the input but also on
several random choices.  The same randomized algorithm, given the same
input multiple times, may perform different
computations in each invocation. This means,
among other things, that the running time of a
randomized algorithm on a given input is no
longer fixed, but is itself a random variable.
46EXAMPLE Nuts and Bolts
 Finding the nut that matches a given bolt.
 Uniformly is a technical term meaning that each
nut has exactly the same probability of being
chosen.  So if there are k nuts left to test, each one
will be chosen with probability 1/k.  Now whats the expected number of comparisons we
have to perform? Intuitively, it should be about
n2, but lets formalize our intuition.
47EXAMPLE Nuts and Bolts
 Let T(n) denote the number of comparisons our
algorithm uses to find a match for a single bolt
out of n nuts.  We still have some simple base cases T(1) 0
and T(2) 1, but when n gt 2, T(n) is a random
variable.  T(n) is always between 1 and n1 its actual
value depends on our algorithms random choices.
We are interested in the expected value or
expectation of T(n), which is defined as follows
48EXAMPLE Nuts and Bolts
 If the target nut is the kth nut tested, our
algorithm performs mink, n1 comparisons.  In particular, if the target nut is the last nut
chosen, we dont actually test it. Because we
choose the next nut to test uniformly at random,
the target nut is equally likelywith probability
exactly 1/nto be the first , second, third, or
kth bolt tested, for any k. Thus
49EXAMPLES
 Contention Resolution.
 Global Minimum Cut.
 Linearity of Expectation.
 MAX 3SATISFIABILITY.
 Universal Hashing.
 Chernoff Bounds.
 Load Balancing.
 Randomized DivideandConquer.
 Queuing problems.
50Randomized Algorithms Examples
 Verifying Matrix Multiplication
 Problem Given three nxn matrices ABC is AB
C?  Deterministic algorithm
 (A) Multiply A and B and check if equal to C.
 (B) Running time? O(n3) by straight forward
approach. O(n237) with fast matrix multi
plication (complicated and impractical).  Randomized algorithm
 (A) Pick a random n x 1 vector r.
 (B) Return the answer of the equality ABr Cr.
 (C) Running time? O(n2)!
51Quicksort Vs. Randomized Quicksort
 Quicksort
 (A) Pick a pivot element from array
 (B) Split array into 3 sub arrays those smaller
than pivot, those larger than pivot, and the
pivot itself.  (C) Recursively sort the sub arrays, and
concatenate them.  Randomized Quicksort
 (A) Pick a pivot element uniformly at random from
the array.  (B) Split array into 3 sub arrays those smaller
than pivot, those larger than pivot, and the
pivot itself.  (C) Recursively sort the sub arrays, and
concatenate them.
52Quicksort Vs. Randomized Quicksort
 Quicksort can take O(n2) time to sort array of
size n.  Randomized Quicksort sorts a given array of
length n in O(n log n) expected time.  Randomization can NOT eliminate the worstcase
but it can make it less likely!
53Examples
54 Examples
 Quicksort algorithm is very efficient in
practice, its worstcase running time is rather
slow. When sorting n elements, the number of
comparisons may be O(n2).  The worst case happens if the sizes of the sub
problems are not balanced.  we prove that if the pivot is selected uniformly
at random, the expected number of comparisons of
this randomized version of Quicksort is bounded
by O(n log n).
55AverageCase Analysis of Quicksort
 Let X total number of comparisons performed in
all calls to PARTITION  The total work done over the entire execution of
Quicksort is  O(ncX)O(nX)
 Need to estimate E(X)
56Review of Probabilities
57Review of Probabilities
(discrete case)
58Random Variables
 Def. (Discrete) random variable X a function
from a sample space S to the real numbers.  It associates a real number with each possible
outcome of an experiment. 
X(j)
59Random Variables
E.g. Toss a coin three times
define X numbers of heads
60Computing Probabilities Using Random Variables
61Expectation
 Expected value (expectation, mean) of a discrete
random variable X is  EX Sx x PrX x
 Average over all possible values of random
variable X
62Examples
Example X face of one fair dice EX 1?1/6
2?1/6 3?1/6 4?1/6 5?1/6 6?1/6 3.5
Example
63Indicator Random Variables
 Given a sample space S and an event A, we define
the indicator random variable IA associated
with A  IA 1 if A occurs
 0 if A does not occur
 The expected value of an indicator random
variable XAIA is  EXA Pr A
 Proof
 EXA EIA
1 ? PrA 0 ? PrA
PrA
64Examples
65Examples
66Quick Sort
67Quick Sort
Partition set into two using randomly chosen
pivot
68Quick Sort
69Quick Sort
70mincut Examples
 An efficient algorithm cannot check all
partitions.  A sample execution of Algorithm on a graph with
5 nodes. Throughout the execution, the edges of
one mincut of G are colored blue. At each step,
the red, dashed edge is contracted.
71Examples
 Algorithm Randomized MinCut(G (VE)).
 while the graph has more than two nodes do
 choose an edge e (u , v) uniformly at
random  contract e (the node that combines u and
v inherits all the node labels of u and v)  and remove selfloops note that parallel edges
are not removed  end while
 return the cut defined by the labels of one of
the remaining nodes.
72Probabilistic Analysis
73Probabilistic analysis
 Probabilistic analysis is the use of probability
in the analysis of problems.  Most commonly, we use probabilistic analysis to
analyze the running time of an algorithm.  Then we analyze our algorithm, computing an
expected running time.  The expectation is taken over the distribution of
the possible inputs. Thus we are, in effect,
averaging the running time over all possible
inputs.
74Probability Measures
 Random Variables Binomial and Geometric.
 Useful Probabilistic Bounds and Inequalities.
 A probability measure (Prob) is a mapping from a
set of events to the reals such that  For any event A
 0 ? Prob(A) ? 1
 Prob (all possible events) 1
 If A, B are mutually exclusive events, then
 Prob(A ? B) Prob (A) Prob (B)
75Conditional Probability
76Bayes Theorem
 If A1, , An are mutually exclusive and contain
all events then
77Random Variable A (Over Real Numbers)
78Random Variable A (contd)
 Prob Distribution Function
79Random Variable A (contd)
 If for Random Variables A,B
 Then A upper bounds B and B lower bounds A
80Expectation of Random Variable A
 A is also called average of A and mean of A
?A
81Variance of Random Variable A
82Variance of Random Variable A (contd)
83Discrete Random Variable A
84Discrete Random Variable A (contd)
85Discrete Random Variable A Over Nonnegative
Numbers
86PairWise Independent Random Variables
 A,B independent if
 Prob(A ? B) Prob(A) X Prob(B)
 Equivalent definition of independence
87B is Binomial Variable with Parameters n , p
88B is Binomial Variable with Parameters n , p
(contd)
89Probabilistic Inequalities
90Markov and Chebychev Probabilistic Inequalities
 Markov Inequality (uses only mean)
 Chebychev Inequality (uses mean and variance)
91Gaussian Density Function
92Normal Distribution
93Sums of Independently Distributed Variables
 Let Sn be the sum of n independently distributed
variables A1, , An  Each with mean and variance
 So Sn has mean ? and variance ?2
94Strong Law of Large Numbers Limiting to Normal
Distribution
 The probability density function of to normal
distribution ?(x)  Hence
 Prob
95Strong Law of Large Numbers (contd)
 So
 Prob

 (since 1 ?(x) ? ?(x)/x)
96Moments of Random Variable A
 nth Moments of Random Variable A
 Moment generating function
97Amortized analysis
98Amortized analysis
 After discussing algorithm design techniques
(Dynamic programming and Greedy algorithms) we
now return to data structures and discuss a new
analysis methodAmortized analysis.  Until now we have seen a number of data
structures and analyzed the worstcase running
time of each individual operation.  Sometimes the cost of an operation vary widely,
so that that worstcase running time is not
really a good cost measure.
99Amortized analysis
 Similarly, sometimes the cost of every single
operation is not so important.  the total cost of a series of operations are more
important.(e.g when using priority queue to sort)  We want to analyze running time of one single
operation averaged over a sequence of operations.  Again keep in mind Average" is over a sequence
of operations for any sequence  not average for some input distribution (as in
quicksort).  not average over random choices made by algorithm
(as in skiplists).
100Amortized Analysis
 Amortized analysis is a technique for analyzing
an algorithm's running time.  The average cost of a sequence of n operations on
a given Data Structure.
101Applications of amortized analysis
 Vectors/ tables
 Disjoint sets
 Priority queues
 Heaps, Binomial heaps, Fibonacci heaps
 splay trees
 unionfind.
 Red black trees
 Maximum flow
 Dynamic arrays / hash tables
102Difference between amortized and average cost
 To do averages we need to use probability
 For amortized analysis no such assumptions are
needed  We compute the average cost per operation for any
mix of n operations
103Amortized Analysis
 Not just consider one operation, but a sequence
of operations on a given data structure.  Average cost over a sequence of operations.
 Probabilistic analysis
 Average case running time average over all
possible inputs for one algorithm (operation).  If using probability, called expected running
time.  Amortized analysis
 No involvement of probability
 Average performance on a sequence of operations,
even some operation is expensive.  Guarantee average performance of each operation
among the sequence in worst case.
104Amortized Analysis
 In an amortized analysis the time required to
perform a sequence of data structure operations
is averaged over all the operations performed.  Amortized analysis differs from average case
analysis in that probability is not involved an
amortized analysis guarantees the average
performance of each operation in the worst case.
105Indirect Solution
10 yards / minute 100 yards / minute
(Source Mark Allen Weiss. Data Structures and
Algorithm Analysis in Java.
106Indirect Solution
10 yards / minute 100 yards / minute
Geometric Series?
The easy solution is indirect. It takes a kitten
5 minutes togo 50 yards, how far can the mother
go in 5 minutes?.... 500 yards!
(Source Mark Allen Weiss. Data Structures and
Algorithm Analysis in Java.
107Amortized Analysis
 The worstcase running time is not always the
same as the worst possible average running time.  Example
 Worst casetime is O(n)
 Amortized worstcase is O(1)
 This could be from a series of table inserts and
clears
(Source Arup Guha. CS2 Notes Summer 2007.
108Binomial Queue
 Binomial Trees
 B0 B1 B2
 B3
 B4
Each tree doubles the previous.
109Binomial Queue
 Binomial Queue
 A queue of binomial trees. A Forest.
 Each tree is essentially a heap constrained to
the format of a binary tree.  Example B0, B2, B3
 Insertion Create a B0 and merge
 Deletion remove minimum( the root) from tree Bk.
This leaves a queue of trees B0, B1, , Bk1.
Merge this queue with the rest of the original
queue.
(Source Mark Allen Weiss. Data Structures and
Algorithm Analysis in Java.
110Binomial Queue Example
 The Most important step is Merge
 Merge Rules (for two binomial queues)
 0 or 1 Bk trees ? just leave merged
 2 Bk trees ? meld into 1 Bk1 tree.
 3 Bk trees ? meld two into 1 Bk1, and leave the
third.  Insertion Runtime
 M1 steps, where M is smallest tree not present.
Worstcase is k2, when Bk1 is the smallest tree
not present. How does k relate to the total
number of nodes in the tree?  k lg n, thus (non amortized) worstcase time is
O(lg n).
111Binomial Queue Example
 Make Bin Q Problem Build a binomial queue of N
elements. (Like make Heap). How long should this
take?  Insertion worstcase runtime
 Worstcase O(lg n) for 1 insert? O(n lg n) n
inserts, but we want O(n) like make Heap  Try amortized analysis directly
 Considering each linking step of the merge. The
1st, 3rd, 5th, etc. odd steps require no linking
step because there will be no B0. So ½ of all
insertions require no linking, similarly ¼
require 1 linking steps.  We could continue down this path, but the itll
be come especially difficult for deletion (we
should learn an indirect analysis).
112Binomial Queue Example
 Indirect Analysis (time M 1)
 If no B0 ? cost is 1 (M is 0)
 Results in 1 B0 tree added to the forest
 If B0 but no B1 ? cost is 2 (M is 1)
 Results in same of trees (new B1 but B0 is
gone)  When cost is 3 (M is 2)
 Results in 1 less tree (new B2 but remove B0 and
B1)  .etc.
 When cost is c (M is c 1)
 Results in increase of 2 c trees
113Binomial Queue Example
 increase of 2 c trees
 How can we use this?
 Ti of trees after ith iteration
 T0 0 of trees initially
 Ci Cost of ith iteration
 Then, Ti Ti1 (2 Ci) ?
 Ci (Ti Ti1) 2
 This is only the ith iteration
114Binomial Queue Example
 Ci (Ti Ti1) 2
 To get all iterations
 C1 (T1 T0) 2
 C2 (T2 T1) 2

 Cn1 (Tn1 Tn2) 2
 Cn (Tn Tn1) 2
 n
 S Ci (Tn T0) 2n (T1 .. Tn1 cancel out)
 i1
115Binomial Queue Example
 n
 S Ci (Tn T0) 2n
 i1
 T0 0 and Tn is definitely not negative, so Tn
T0 is not negative.  n
 ?S Ci lt 2n
 i1
Thus, the total cost lt 2n ? make Bin Q
O(n) Since, make Bin Q consists of O(n) inserts,
then the amortized worstcase of each insert is
O(1).
116Amortized Analysis
 There exists three common techniques used in
amortized analysis  Aggeregate method
 Accounting method
 Potential method
1171.Aggeregate method
 1.Aggeregate method
 We show that a sequence of n operations take
worstcase time T(n) in total.  In the worst case, the average. cost, or
amortized cost, per operation is therefore  T(n) / n.
 In the aggregate method, all operations have the
same amortized cost.  The other two methods, the accounting tricky and
the potential function method, may assign
different amortized costs to diferent types of
operations.
118An example push and pop
 A sequence of operations OP1, OP2, OPm
 OPi several pops (from the stack) and
 one push (into the stack)
 ti time spent by OPi
 the average time per operation

119 Example a sequence of push and pop
 p pop , u push
tave (11311132)/8 13/8
1.625
120 Another example a sequence of push and pop
 p pop , u push
tave (12111161)/8 14/8
1.75
121Amortized Analysis
 Example Stack operations
 As we know, a normal stack is a data structure
with operations  Push Insert new element at top of stack
 Pop Delete top element from stack
 A stack can easily be implemented (using linked
list) such that Push and Pop takes O(1) time.  Consider the addition of another operation
 Multipop(k) Pop k elements off the stack.
 Analysis of a sequence of n operations
 One Multipop can take O(n) time ? O(n2) running
time.  Amortized running time of each operation is O(1)
? O(n) running time.  Each element can be popped at most once each time
it is pushed  Number of Pop operations (including the one done
by Multipop) is bounded by n  Total cost of n operations is O(n)
 Amortized cost of one operation is O(n)/n O(1).
 Notice that no probability involved.
122The Accounting Method
 Assign different charges to different operations.
 Some are charged more than actual cost, some
are charged less.  Amortized cost amount we charge.
 When amortized cost gt actual cost, store the
difference on a credit object.  Use credit later to pay for operations whose
actual cost gt amortized cost.  Need credit to never go negative.
123The Accounting Method
124Stack with MultiPop Operations
125Potential Method
 Like the accounting , but credit stored with the
entire structure.  Accounting method stores credit with specific
objects.  Potential method stores potential in the data
structure as a whole.  Can release potential to pay for future
operations.  Most flexible of the amortized analysis
methods.
126Potential Method
127Potential Method
128Stack with MultiPop Operations
129Skew heaps
Two skew heaps
Step 1 Merge the right paths. 5 right heavy nodes
130Step 2 Swap the children along the right path.
No right heavy node
131Amortized analysis of skew heaps
 meld merge swapping
 operations on a skew heap
 findmin(h) find the min of a skew heap h.
 insert(x, h) insert x into a skew heap h.
 deletemin(h) delete the min from a skew heap h.
 meld(h1, h2) meld two skew heaps h1 and h2.
 The first three operations can be implemented
by melding.
132Potential function of skew heaps
 wt(x) of children of node x, including x.
 heavy node x wt(x) ? wt(p(x))/2, where
 p(x) is the parent node of x.
 light node not a heavy node
 potential function ?i of right heavy nodes of
the skew heap.
133 Any path in an nnode tree contains at most
?log2n? light nodes.
light nodes ? ?log2n?
heavyk3? ?log2n? possible heavy nodes
of nodes n
 The number of right heavy nodes attached to the
left path is at most ?log2n ?.
134Amortized time
light ? ?log2n1? heavy k1
light ? ?log2n2? heavy k2
135(No Transcript)
136AVLtrees
height balance of node v hb(v)height of right
sub tree  height of left sub tree
137Before insertion, hb(B)hb(C)hb(E)0 hb(I)?0
the first nonzero from leaves.
138Amortized analysis of AVLtrees
 Consider a sequence of m insertions on an empty
AVLtree.  T0 an empty AVLtree.
 Ti the tree after the ith insertion.
 Li the length of the critical path involved in
the ith insertion.  X1 total of balance factor changing from 0 to
1 or 1 during these m insertions (rebalancing
cost)
139Case 1 Absorption
 The tree height is not increased, we need not
rebalance it.
Val(Ti)Val(Ti1)(Li?1)
140Case 2 Rebalancing the tree
141Case 2.1 single rotation
 After a right rotation on the sub tree rooted at
A
Val(Ti)Val(Ti1)(Li2)
142Case 2.2 double rotation
143Case 2.2 double rotation
 After a left rotation on the sub tree rooted at B
and a right rotation on the sub tree rooted at A
Val(Ti)Val(Ti1)(Li2)
144Case 3 Height increase
 Li is the height of the root.
Val(Ti)Val(Ti1)Li
145Amortized analysis of X1
146Example TwoStack System
 Suppose there are two stacks called A and B,
manipulated by the following operations  push(S, d) Push a datum d onto stack S.
 Real Cost 1.
 multipop(S, k) Removes min(k, S) elements
from stack S.  Real Cost min(k, S).
 transfer(k) Repeatedly pop elements from stack A
and push them onto stack B. until either k
elements have been moved, or A is empty.  Real Cost of elements moved min(k, A).
147Design
B
A
1481.Aggregate Method
 For a sequence of n operations, there are n
data pushed into either A or B. Therefore there
are n data popped out from either A or B and
n data transferred from A to B. Thus, the total
cost of the n operations is 3n. Thus.
Operation Real Cost Amortized Cost
Push(A, d) 1 3
Push(B, d) 1 3
Multi pop(A, k) min(k, A) 3
Multi pop(B, k) min(k, B) 3
Transfer(k) min(k, A) 3
149Illustration
B
A
1502.Accounting Method
 push(A, d) 3  This pays for the push and a
pop of the push a transfer and a pop.  push(B, d) 2  This pays for the push and a
pop.  multipop(S, k) 0
 transfer(k) 0
 After any n operations you will have 2A B
dollars in the bank.  Thus the bank account never goes negative.
Furthermore the amortized cost of the n
operations is O(n) (more precisely 3n). 
151Comparison
Operation Real Cost Aggregate Accounting Potential
Push(A, d) 1 3 3
Push(B, d) 1 3 2
Multipop(A, k) min(k, A) 3 0
Multipop(B, k) min(k, B) 3 0
Transfer(k) min(k, A) 3 0
1523.Potential Method
 Let F(A, B) 2A B, then
 c(push(A, d)) 1 DF 1 2 3
 c(push(B, d)) 1 DF 1 1 2
 c(multipop(A, k)) min(k, A) DF min(k,
A)  c(multipop(B, k)) min(k, B) DF 0
 c(transfer(k)) min(k, A) DF 0
 Sc Sc DF
 Sc Sc  DF Sc 3n O(n).
 If we can pick F so that F(Di) ? F(D0) for all i,
and that Sc is easy to compute, then Sc/n
upperbounds the average cost.
153Summary
Operation Real Cost Aggregate Accounting Potential
Push(A, d) 1 3 3 3
Push(B, d) 1 3 2 2
Multipop(A, k) min(k, A) 3 0 min(k, A)
Multipop(B, k) min(k, B) 3 0 0
Transfer(k) min(k, A) 3 0 0
154Competitive analysis
155On Bounds
 Worst Case.
 Average Case Running time over some distribution
of input. (Quicksort)  Amortized Analysis Worst case bound on sequence
of operations.  Competitive Analysis Compare the cost of an
online algorithm with an optimal perceptive
algorithm on any sequence of requests.
156Applications
 Resource Allocation
 Scheduling
 Memory Management
 Routing
 Robot Motion Planning
 Exploring an unknown terrain
 Finding a destination
 Computational Finance
157Selforganizing lists
 List L of n elements
 The operation ACCESS(x)costs rankL(x)distance of
x from the head of L.  L can be reordered by transposing adjacent
elements at a cost of 1.  Example
 Accessing the element with key 14 costs 4.
 Transposing 3 and 50 costs 1.
158Online and offline problems
 Definition. A sequence S of operations is
provided one at a time. For each operation, an
online algorithm A must execute the operation
immediately without any knowledge of future
operations (e.g., Tetris).  An offline algorithm may see the whole sequence
S in advance.  Goal Minimize the total cost CA(S).
159Worstcase analysis of selforganizing lists
 An adversary always accesses the tail (nth)
element of L. Then, for any online algorithm A,
we have  CA(S) O(Sn)
 in the worst case.
160Averagecase analysis of selforganizing lists
 Suppose that element x is accessed with
probability p(x). Then, we have  which is minimized when L is sorted in decreasing
order with respect to p.  Heuristic Keep a count of the number of times
each element is accessed, and maintain L in order
of decreasing count.
161The movetofront heuristic
 Practice Implementers discovered that the
movetofront (MTF)heuristic empirically produces
good results.  IDEA After accessing x, move x to the head of L
using transposes  cost 2 rankL(x) .
 The MTF heuristic responds well to locality in
the access sequence S.
162Competitive analysis
 Definition . An online algorithm A is
acompetitive if there exists a constant k such
that for any sequence S of operations,  CA(S) aCOPT(S) k,
 where OPT is the optimal offline algorithm
(Gods algorithm).
163MTF is O(1)competitive
 Theorem MTF is 4competitive for selforganizing
lists.
164Potential function
Define the potential function FLi ?R by F(Li)
2 (x, y) x ?Li y and y ?Li x 2
inversions
F(Li) 2
F(Li) 2 (E,C),
165Potential function
F(Li) 2 (E,C), (E,A),
F(Li) 2 (E,C), (E,A), (E,D),
F(Li) 2 (E,C), (E,A), (E,D), (E,B),
166Potential function
F(Li) 2 (E,C), (E,A), (E,D), (E,B), (D,B)
F(Li) 2 (E,C), (E,A), (E,D), (E,B), (D,B)
10.
Note that F(Li) 0 for i 0, 1, , F(L0) 0
if MTF and OPT start with the same list. How much
does F change from 1 transpose? A transpose
creates/destroys 1 inversion. ?F 2 .
167What happens on an access?
Suppose that operation i accesses element x, and
define
168What happens on an access?
169Amortized cost
 The amortized cost for the i th operation of MTF
with respect to F is  cici F(Li) F(Li1)
 2r 2(A B ti)
 2r 2(A (r1 A) ti)
 (since r A B 1)
 2r 4A 2r 2 2ti
 4A 2 2ti
 4(r ti)
 (since r A C 1
A 1)  4ci.
170The grand finale
171Appendix
 If we count transpositions that move x toward the
front as free(models splicing x in and out of
Lin constant time), then MTF is 2competitive.  What if L0?L0?
 Then, F(L0)might be T(n2)in the worst case.
 Thus, CMTF(S) 4 COPT(S) T(n2), which is
still 4competitive, since n2is constant as S
?8.
172Assignments
 1. Example on Traveling salesman problem.
 2.Show that RANDOMIZEDQUICKSORTs expected
running time is ?(n log n).  3.E XAMPLES on competitive Analysis.
173(No Transcript)
174(No Transcript)
175 Analysis
 Worst case computing times for two sorting
algorithms on random inputs.  Average case computing times for two sorting
algorithms on random inputs.  Comparison of Quick Sort and RQuick Sort on the
input aii, 1,ltiltn times are in
milliseconds. 

N 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Merge Sort 105.7 206.4 335.2 422.1 589.9 691.3 794.8 889.5 1067 1167
Quick Sort 41.6 97.1 158.6 244.9 397.8 383.8 497.3 568.9 616.2 738.1
N 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Merge Sort 72.8 167.2 275.1 378.5 500.6 607.6 723.4 811.5 949.2 1073.6
Quick Sort 36.6 85.1 138.9 205.7 269.0 339.4 411.0 487.7 556.3 645.2
N 1000 2000 3000 4000 5000
Quick Sort 195.5 759.2 1728 3165 4829
R Quick Sort 9.4 21.0 30.5 41.6 52.8
176Matrix multiplication
177(No Transcript)
178Pivot element
 our pivot element is near the center of the
sorted array (i.e. , the pivot is close to the
median element)
179 180 181Have a nice randomized life