# Unit%201:%20Algorithmic%20Fundamentals - PowerPoint PPT Presentation

View by Category
Title:

## Unit%201:%20Algorithmic%20Fundamentals

Description:

### Course contents: On algorithms Mathematical foundations Asymptotic notation Growth of functions Complexity Lower vs. upper bounds Recurrences Unit 1: Algorithmic ... – PowerPoint PPT presentation

Number of Views:112
Avg rating:3.0/5.0
Slides: 86
Provided by: edut1550
Category:
Tags:
Transcript and Presenter's Notes

Title: Unit%201:%20Algorithmic%20Fundamentals

1
Unit 1 Algorithmic Fundamentals
• Course contents
• On algorithms
• Mathematical foundations
• Asymptotic notation
• Growth of functions
• Complexity
• Lower vs. upper bounds
• Recurrences

2
On Algorithms
• Algorithm A well-defined procedure for
transforming some input to a desired output.
• Major concerns
• Correctness Does it halt? Is it correct? Is it
stable?
• Efficiency Time complexity? Space complexity?
• Worst case? Average case? (Best case?)
• Better algorithms?
• How Faster algorithms? Algorithms with less
space requirement?
• Optimality Prove that an algorithm is best
possible/optimal? Establish a lower bound?

3
Example Traveling Salesman Problem (TSP)
• Input A set of points (cities) P together with a
distance d(p, q) between any pair p, q ? P.
• Output What is the shortest circular route that
starts and ends at a given point and visits all
the points.
• Correct and efficient algorithms?

4
Nearest Neighbor Tour
1. pick and visit an initial point p0 2. P ? p0
3. i ? 0 4. while there are unvisited points
do 5. visit pi's closet unvisited point
pi1 6. i ? i 1 7. return to p0 from pi.
• Simple to implement and very efficient, but
incorrect!

5
A Correct, but Inefficient Algorithm
1. d ? ? 2. for each of the n! permutations ?i
of the n points 3. if (cost(?i) ? d) then
4. d ? cost(?i) 5. Tmin ?
?i 6. return Tmin.
• Correctness Tries all possible orderings of the
points ? Guarantees to end up with the shortest
possible tour.
• Efficiency Tries n! possible routes!
• 120 routes for 5 points, 3,628,800 routes for 10
points, 20 points?
• No known efficient, correct algorithm for TSP!
• TSP is NP-complete.

6
Example Sorting
• Input A sequence of n numbers lta1, a2, , angt.
• Output A permutation lta1', a2', , an'gt such
that a1' ? a2' ? ? an'.
• Input lt8, 6, 9, 7, 5, 2, 3gt
• Output lt2, 3, 5, 6, 7, 8, 9 gt
• Correct and efficient algorithms?

7
Insertion Sort
InsertionSort(A) 1. for j ? 2 to lengthA do 2.
key ? Aj 3. / Insert Aj into the
sorted sequence A1..j-1. / 4. i ? j - 1
5. while i gt 0 and Ai gt key do 6.
Ai1 ? Ai 7. i ? i - 1 8.
Ai1 ? key
8
Exact Analysis of Insertion Sort
• The for loop is executed (n-1) 1 times. (why?)
• tj of times the while loop test for value j
(i.e., 1 of elements that have to be slided
right to insert the j-th item).
• Step 5 is executed t2 t3 tn times.
• Step 6 is executed (t2 - 1) (t3 - 1) (tn
- 1) times.

9
Exact Analysis of Insertion Sort (contd)
• Best case If the input is already sorted, all
tj's are 1.
• Linear T(n) (c1 c2 c4 c5 c8)n -
(c2 c4 c5 c8)
• Worst case If the array is in reverse sorted
order, tj j, ? j.
• Quadratic T(n) (c5 /2 c6/ 2 c7/2 ) n2
(c1 c2 c4 c5 /2 c6 /2 c7/2 c8) n
(c2 c4 c5 c8)
• Exact analysis is often hard!

10
Asymptotic Analysis
• Asymptotic analysis looks at growth of T(n) as n
? ?.
• ? notation Drop low-order terms and ignore the
• E.g., 8n3 - 4n2 5n - 2 ?(n3).
• As n grows large, lower-order ? algorithms
outperform higher-order ones.
• Worst case input reverse sorted, while loop is
?(j)
• Average case all permutations equally likely,
while loop is ?(j / 2)

11
Merge Sort A Divide-and-Conquer Algorithm
MergeSort(A, p, r)
T(n) 1. If p lt r then
?(1) 2. q ? ? (pr)/2?
?(1) 3.
MergeSort (A, p, q)
T(n/2) 4. MergeSort (A, q 1, r)
T(n/2) 5. Merge(A, p, q, r)
?(n)
12
Recurrence
• Describes a function recursively in terms of
itself.
• Describes performance of recursive algorithms.
• Recurrence for merge sort

MergeSort(A, p, r)
T(n) 1. If p lt r then
?(1) 2. q ? ? (pr)/2?
?(1) 3. MergeSort (A, p,
q) T(n/2) 4. MergeSort
(A, q 1, r) T(n/2) 5.
Merge(A, p, q, r) ?(n)
13
Recursion Tree for Asympotatic Analysis
• ?(n lg n) grows more slowly than ?(n2).
• Thus merge sort asymptotically beats insertion
sort in the worst case. (insertion sort
stable/in-place merge sort stable/not in-place)

14
O Upper Bounding Function
• Def f(n) O(g(n)) if ? c gt0 and n0 gt 0 such that
0 ? f(n) ? cg(n) for all n ? n0.
• Intuition f(n) ? g(n) when we ignore constant
multiples and small values of n.
• How to show O (Big-Oh) relationships?
• f(n) O(g(n)) iff limn ? ? c for some
c ? 0.
• Remember L'Hopitals Rule?

15
? Lower Bounding Function
• Def f(n) ?(g(n)) if ? c gt0 and n0 gt 0 such that
0 ? cg(n) ? f(n) for all n ? n0.
• Intuition f(n) ? g(n) when we ignore
constant multiples and small values of n.
• How to show ? (Big-Omega) relationships?
• f(n) ?(g(n)) iff limn ? ? c for some
c ? 0.

16
? Tightly Bounding Function
• Def f(n) ?(g(n)) if ? c1, c2 gt0 and n0 gt 0 such
that 0 ? c1g(n) ? f(n) ? c2 g(n) for all n ? n0.
• Intuition f(n) g(n) when we ignore
constant multiples and small values of n.
• How to show ? relationships?
• Show both big Oh (O) and Big Omega (?)
relationships.
• f(n) ?(g(n)) iff limn ? ? c for some
c gt 0.

17
o, ? Untightly Upper, Lower Bounding Functions
• Little Oh o f(n) o(g(n)) if ? c gt 0, ? n0 gt 0
such that 0 ? f(n) lt cg(n) for all n ? n0.
• Intuition f(n) lt any constant multiple of
g(n) when we ignore small values of n.
• Little Omega ? f(n) ?(g(n)) if ? c gt 0, ? n0 gt
0 such that 0 ? cg(n) lt f(n) for all n ? n0.
• Intuition f(n) is gt any constant multiple of
g(n) when we ignore small values of n.
• How to show o (Little-Oh) and ?(Little-Omega)
relationships?
• f(n) o(g(n)) iff limn ? ? 0.
• f(n) ?(g(n)) iff limn ? ? ?.

18
Properties for Asymptotic Analysis
• An algorithm has worst-case run time O(f(n))
there is a constant c s.t. for every n big
enough, every execution on an input of size n
takes at most cf(n) time.
• An algorithm has worst-case run time ?(f(n))
there is a constant c s.t. for every n big
enough, at least one execution on an input of
size n takes at least cf(n) time.
• Transitivity If f(n) ?(g(n)) and g(n)
?(h(n)), then f(n) ?(h(n)), where ? O, o, ?,
?, or ?.
• Rule of sums ?(f(n) g(n)) ?(maxf(n),
g(n)), where ? O, o, ?, ?, or ?.
• Rule of sums f(n) g(n) ?(maxf(n), g(n)),
where ? O, ?, or ?.
• Rule of products If f1(n) ?(g1(n)) and f2(n)
?(g2(n)), then f1(n) f2(n) ?(g1(n) g2(n)),
where ? O, o, ?, ?, or ?.
• Transpose symmetry f(n) O(g(n)) iff g(n)
?(f(n)).
• Transpose symmetry f(n) o(g(n)) iff g(n)
?(f(n)).
• Reflexivity f(n) ?(f(n)), where ? O, ? , or
?.
• Symmetry f(n) ?(g(n)) iff g(n) ?(f(n)).

19
Asymptotic Functions
• Polynomial-time complexity O(p(n)), where n is
the input size and p(n) is a polynomial function
of n (p(n) nO(1)).

20
Runtime Comparison
• Run-time comparison Assume 1000 MIPS, 1
instruction/operation.

21
I cant find an efficient algorithm, I guess Im
just too dumb.
22
Mission Impossible
I cant find an efficient algorithm, because no
such algorithm is possible.
23
I cant find an efficient algorithm, but neither
can all these famous people.
24
Easy and Hard Problems
• We argue that the class of problems that can be
solved in polynomial time (denoted by P)
corresponds well with what we can feasibly
compute. But sometimes it is difficult to tell
when a particular problem is in P or not.
• Theoreticians spend a good deal of time trying to
determine whether particular problems are in P.
To demonstrate how difficult it can be.
• To make this determination, we will survey a
number of problems, some of which are known to be
in P, and some of which we think are (probably)
not in P. The difference between the two types
of problem can be surprisingly small. Throughout
the following, an ''easy'' problem is one that is
solvable in polynomial time, while a ''hard''
problem is one that we think cannot be solved in
polynomial time.

25
Eulerian Tour vs. Hamiltonian Tour
-- Easy
• Eulerian Tours
• INPUT A graph G (V, E).
• DECIDE Is there a path that crosses every edge
exactly once and returns to its starting point?
• Hamiltonian Tours
• INPUT A graph G (V, E).
• DECIDE Is there a path that visits every vertex
exactly once and returns to its starting point?

-- Hard
26
Some Facts
• Eulerian Tours
• A famous mathematical theorem comes to our
rescue. If the graph is connected and every
vertex has even degree, then the graph is
guaranteed to have such a tour. The algorithm to
find the tour is a little trickier, but still
doable in polynomial time.
• Hamiltonian Tours
• No one knows how to solve this problem in
polynomial time. The subtle distinction between
visiting edges and visiting vertices changes an
easy problem into a hard one.

27
Map Colorability
-- Easy
• Map 2-colorability
• INPUT A graph G(V, E).
• DECIDE Can this map be
• colored with 2 colors so that no
• two adjacent countries have the
• same color?
• Map 3-colorability
• INPUT A graph G(V, E).
• DECIDE Can this map be colored with 3 colors so
that no two adjacent countries have the same
color?
• Map 4-colorability

-- Hard
-- Easy
28
Some Facts
• Map 2-colorability
• To solve this problem, we simply color the first
country arbitrarily. This forces the colors of
neighboring countries to be the other color,
which in turn forces the color of the countries
neighboring those countries, and so on. If we
reach a country which borders two countries of
different color, we will know that the map cannot
be two-colored otherwise, we will produce a two
coloring. So this problem is easily solvable in
polynomial time.

29
Some Facts
• Map 3-colorability
• This problem seems very similar to the problem
above, however, it turns out to be much harder.
No one knows how this problem can be solved in
polynomial time. (In fact this problem is
NP-complete.)
• Map 4-colorability.
• Here we have an easy problem again. By a famous
theorem, any map can be four-colored. It turns
out that finding such a coloring is not that
difficult either.

30
Problem vs. Problem Instance
• When we say that a problem is hard, it means that
some instances of the problem are hard. It does
not mean that all problem instances are hard.
• For example, the following problem instance is
trivially 3-colorable

31
Longest Path vs. Shortest Path
-- Hard
• Longest Path
• INPUT A graph G (V, E), two vertices u, v of
V, and a weighting function on E.
• OUTPUT The longest path between u and v.
• Shortest Path
• INPUT A graph G (V, E), two vertices u, v of
V, and a weighting function on E.
• OUTPUT The shortest path between u and v.

No one is able to come up with a polynomial time
algorithm yet.
-- Easy
A greedy method will solve this problem easily.
32
Multiplication vs. Factoring
-- Easy
• Multiplication
• INPUT Integers x,y.
• OUTPUT The product x?y.
• Factoring (Un-multiplying)
• INPUT An integer n.
• OUTPUT If n is not prime, output two integers x,
y such that 1 lt x, y lt n and x ? y n.

-- Hard
Again, the problem of factoring is not known to
be in P. In this case, the hardness of a problem
turns out to be useful. Some cryptographic
algorithms depend on the assumption that
factoring is hard to ensure that a code cannot be
broken by a computer.
33
Boolean Formulas
• Formula evaluation
• INPUT A boolean formula (e.g. (x ?y) ? (z ??x))
and a value for all variables in the formula
(e.g. x 0, y 1, z 0).
• DECIDE The value of the formula. (e.g., 1, or
"true'' in this case).
• Satisfiability of boolean formula
• INPUT A boolean formula.
• DECIDE Do there exist values for all variables
that would make the formula true?
• Tautology
• INPUT A boolean formula.
• DECIDE Do all possible assignments of values to
variables make the formula true?

-- Easy
-- Hard
-- Harder
34
Facts
• Formula evaluation
• It's not too hard to think of what the algorithm
would be in this case. All we would have to do is
to substitute the values in for the various
variables, then simplify the formula to a single
value in multiple passes (e.g. in a pass simplify
1 ? 0 to 1). .
• Satisfiability of boolean formula
• Given that there are n different variables in the
formula, there are 2n possible assignments of 0/1
to the variables. This gives us an easy
exponential time algorithm simply try all
possible assignments. No one knows if there is a
way to be cleverer, and cut the running time down
to polynomial
• Tautology
• It turns out that this problem seems to be even
harder than the Satisfiability problem.

35
How Do You Judge an Algorithm?
• Issues Related to the analysis of Algorithms
• How to measure the goodness of an algorithm?
• How to measure the difficulty of a problem?
• How do we know that an algorithm is optimal?

36
The Complexity of an Algorithm
• The space complexity of a program is the amount
of memory that it needs to run to completion.
• Fixed space requirements does not depend on the
programs inputs and outputs -- usually ignored.
• Variable space requirement size depends on
execution of program (recursion, dynamic
allocated variables, etc.)
• The time complexity of a program is the amount of
computer time that it needs to run a computation.

37
Input (Problem) Size
• Input (problem) size and costs of operations The
size of an instance corresponds formally to the
number of bits needed to represent the instance
on a computer, using some precisely defined and
reasonably compact coding scheme.
• uniform cost function
• logarithmic cost function
• Example Compute x nn
• x 1 uniform logarithmic
• for i 1 to n do T(n) Q(n) T(n)
Q(n2?log n)
• x x n S(n) Q(1) S(n) Q(n ?
log n)

38
Complexity of an Algorithm
• Best case analysis too optimistic, not really
useful.
• Worst case analysis usually only yield a rough
upper bound.
• Average case analysis a probability distribution
of input is assumed, and the average of the cost
of all possible input patterns are calculated.
However, it is usually difficult than worst case
analysis and does not reflect the behavior of
some specific data patterns.
• Amortized analysis this is similar to average
case analysis except that no probability
distribution is assumed and it is applicable to
any input pattern (worst case result).
• Competitive analysis Used to measure the
performance of an on-line algorithm w.r.t. an
adversary or an optimal off-line algorithm.

39
Example Binary Search
• Given a sorted array A1..n and an item x in A.
What is the index of x in A?
• Usually, the best case analysis is the easiest,
the worst case the second easiest, and the
average analysis the hardest.

40
Another Example
• Given a stack S with 2 operations push(S, x),
and multipop(S, k), the cost of the two
operations are 1 and min(k, S) respectively.
What is the cost of a sequence of n operations on
an initially empty stack S?
• Best case n, 1 for each operation.
• Worst case O(n2), O(n) for each operation.
• Average case complicate and difficult to
analyze.
• Amortized analysis 2n, 2 for each operation.
(There are at most n push operations and hence at
most n items popped out of the stack.)

41
The Difficulty of a Problem
• Upper bound O(f(n)) means that for sufficiently
large inputs, running time T(n) is bounded by a
multiple of f(n)
• Existing algorithms (upper bounds).
• Lower bound W(f(n)) means that for sufficiently
large n, there is at least one input of size n
such that running time is at least a fraction of
f(n) for any algorithm that solves the problem.
• The inherent difficulty ? lower bound of
algorithms
• The lower bound of a method to solve a problem is
not necessary the lower bound of the problem.

42
(No Transcript)
43
Examples
• Sorting n elements into ascending order.
• O(n2), O(n?log n), etc. -- Upper bounds.
• O(n), O(n?log n), etc. -- Lower bounds.
• Lower bound matches upper bound.
• Multiplication of 2 matrices of size n by n.
• Straightforward algorithm O(n3).
• Strassen's algorithm O(n2.81).
• Best known sequential algorithm O(n2.376) ?
• Best known lower bound W(n2)
• The best algorithm for this problem is still open.

44
Complexity Classes
• DSPACE(S(n)) NSPACE(S(n)) The classes of
problems that can be solved by deterministic
nondeterministic Turing machines using S(n)
space.
• DTIME(T(n)) NTIME(T(n)) The classes of
problems that can be solved by deterministic
nondeterministic Turing machines using T(n)
time.
• Tractable problems Problems in P.
•  Intractable problems Problem not known to be in
P.
•  Efficient algorithms Algorithms in P.

45
Complexity Classes
• Assume P ? NP

NEXP
co-NEXP
EXP
PSPACE
NPC
P
co-NP
NP
46
NP-Complete Problems
• M. R. Garey, and D. S. Johnson
• Computers and Intractability A Guide to the
Theory of NP-Completeness
• W. H. Freeman and Company, 1979

47
Complexity of Algorithms and Problems
• Notations
• Symbol Meaning
• P a problem
• I a problem instance
• In the set of all problem instances of size
• A an algorithm for P
• AP the set of algorithms for problem P
• Pr(I) probability of instance I
• CA(I) cost of A with input I
• RA the set of all possible versions of a
randomized algorithm A

48
Formal Definitions

f(n)
49
Example Complexity of the Sorting Problem
• Assume comparison is used to determine the
order of keys.

50
Divide-and-Conquer Algorithms Revisited
• Divide the problem into a number of subproblems.
• Conquer the subproblems (solve them).
• Combine the subproblem solutions to get the
solution to the original problem.
• Merge sort T(n) 2T(n/2) ? (n) ? (n lg n).
• Divide the n-element sequence to be sorted into
two n/2-element sequence.
• Conquer sort the subproblems, recursively using
merge sort.
• Combine merge the resulting two sorted
n/2-element sequences.

51
Analyzing Divide-and-Conquer Algorithms
• Recurrence for a divide-and-conquer algorithms
• a of subproblems
• n/b size of the subproblems
• D(n) time to divide the problem of size n into
subproblems
• C(n) time to combine the subproblem solutions to
get the answer for the problem of size n
• Merge sort
• a 2 two subproblems
• n/b n/2 each subproblem has size ? n/2
• D(n) ?(1) compute midpoint of array
• C(n) ?(n) merging by scanning sorted subarrays

52
Divide-and-Conquer Binary Search
• Binary search on a sorted array
• Divide Check middle element.
• Conquer Search the subarray.
• Combine Trivial.
• Recurrence T(n) T(n/2) ?(1) ?(lg n).
• a 1 search one subarray
• n/b n/2 each subproblem has size ? n/2
• D(n) ?(1) compute midpoint of array
• C(n) ?(1) trivial

53
Solving Recurrences
• Three general methods for solving recurrences
• Iteration Convert the recurrence into a
summation by expanding some terms and then bound
the summation
• Substitution Guess a solution and verify it by
induction.
• Master Theorem if the recurrence has the form
• T(n) aT(n/b) f(n),
• then most likely there is a formula that
can be applied.
• Two simplifications that won't affect asymptotic
analysis
• Ignore floors and ceilings.
• Assume base cases are constant, i.e., T(n) ?(1)
for small n.

54
Solving Recurrences Iteration
• Example T(n) 4T(n/2) n.

55
Iteration by Using Recursion Trees
• Root computation (D(n) C(n)) at top level of
recursion.
• Node at level i Subproblem at level i in the
recursion.
• Height of tree level in the recursion.
• T(n) sum of all nodes in the tree.
• T(1)1 ? T(n) 4T(n/2) n n 2n 4n
2lgnn ?(n2).

56
Solving Recurrences Substitution (Guess Verify)
• Guess form of solution.
• Apply math. induction to find the constant and
verify solution.
• Use to find an upper or a lower bound.
• Example Guess T(n) 4T(n/2) n O(n3) (T(1)
1)
• Show T(n) ? cn3 for some c gt 0 (we must find c).
• 1. Basis T(2) 4T(1) 2 6 ? 23c (pick c
1)
• 2. Assume T(k) ? ck3 for k lt n, and prove T(n) ?
cn3
• T(n) 4 T(n/2) n
• ? 4 (c (n/2)3) n
• cn3/2 n
• cn3 - (cn3/2-n)
• ? cn3,
• where c ? 2 and n ? 1. (Pick c ?
2 for Steps 1 2!)
• Useful tricks subtract a lower order term,
change variables (e.g., T(n)

57
Pitfall in Substitution
• Example Guess T(n) 2T(n/2) n O(n) (wrong
guess!)
• Show T(n) ? cn for some c gt 0 (we must find c).
• Basis T(2) 2T(1) 2 4 ? 2 c (pick c 2)
• Assume T(k) ? ck for k lt n, and prove T(n) ? cn
• T(n) 2 T(n/2) n
• ? 2 (cn/2) n
• cn n
• O(n)
/ Wrong!! /
• What's wrong?
• How to fix? Subtracting a lower-order term may
help!

58
Fixing Wrong Substitution
• Guess T(n) 4T(n/2) n O(n2) (right guess!)
• Assume T(k) ? ck2 for k lt n, and prove T(n) ?
cn2.
• T(n) 4T (n/2) n
• ? 4c (n/2)2 n
• cn2 n
• O(n2)
/ Wrong!! /
• Fix by subtracting a lower-order term.
• Assume T(k) ? c1k2 - c2 k for k lt n, and prove
T(n) ? c1 n2 - c2 n.
• T(n) 4T(n/2) n
• ? 4 (c1(n/2)2 -
c2(n/2)) n
• c1n2 - 2c2n n
• ? c1n2 - c2n
(if c2 ? 1)
• Pick c1 big enough to handle initial conditions.

59
Solving Recurrence Relations
• In general, we would prefer to have an explicit
formula to compute the value of an rather than
conducting n iterations.
• For one class of recurrence relations, we can
obtain such formulas in a systematic way.
• Those are the recurrence relations that express
the terms of a sequence as linear combinations of
previous terms.

60
Solving Recurrence Relations
• Definition A linear homogeneous recurrence
relation of degree k with constant coefficients
is a recurrence relation of the form
• an c1an-1 c2an-2 ckan-k,
• Where c1, c2, , ck are real numbers, and ck ? 0.
• A sequence satisfying such a recurrence relation
is uniquely determined by the recurrence relation
and the k initial conditions
• a0 C0, a1 C1, a2 C2, , ak-1 Ck-1.

61
Solving Recurrence Relations
• Examples
• The recurrence relation Pn (1.05)Pn-1
• is a linear homogeneous recurrence relation of
degree one.
• The recurrence relation fn fn-1 fn-2
• is a linear homogeneous recurrence relation of
degree two.
• The recurrence relation an an-5
• is a linear homogeneous recurrence relation of
degree five.

62
Solving Recurrence Relations
• Basically, when solving such recurrence
relations, we try to find solutions of the form
an rn, where r is a constant.
• an rn is a solution of the recurrence
relationan c1an-1 c2an-2 ckan-k if and
only if
• rn c1rn-1 c2rn-2 ckrn-k.
• Divide this equation by rn-k and subtract the
right-hand side from the left
• rk - c1rk-1 - c2rk-2 - - ck-1r - ck 0
• This is called the characteristic equation of the
recurrence relation.

63
Solving Recurrence Relations
• The solutions of this equation are called the
characteristic roots of the recurrence relation.
• Let us consider linear homogeneous recurrence
relations of degree two.
• Theorem Let c1 and c2 be real numbers. Suppose
that r2 c1r c2 0 has two distinct roots r1
and r2.
• Then the sequence an is a solution of the
recurrence relation an c1an-1 c2an-2 if and
only if an ?1r1n ?2r2n for n 0, 1, 2, ,
where ?1 and ?2 are constants.

64
Solving Recurrence Relations
• Example What is the solution of the recurrence
relation an an-1 2an-2 with a0 2 and a1 7
?
• Solution The characteristic equation of the
recurrence relation is r2 r 2 0.
• Its roots are r 2 and r -1.
• Hence, the sequence an is a solution to the
recurrence relation if and only if
• an ?12n ?2(-1)n for some constants ?1 and
?2.

65
Solving Recurrence Relations
• Given the equation an ?12n ?2(-1)n and the
initial conditions a0 2 and a1 7, it follows
that
• a0 2 ?1 ?2
• a1 7 ?1?2 ?2 ?(-1)
• Solving these two equations gives us?1 3 and
?2 -1.
• Therefore, the solution to the recurrence
relation and initial conditions is the sequence
an with
• an 3?2n (-1)n.

66
Solving Recurrence Relations
• an rn is a solution of the linear homogeneous
recurrence relationan c1an-1 c2an-2
ckan-k
• if and only if
• rn c1rn-1 c2rn-2 ckrn-k.
• Divide this equation by rn-k and subtract the
right-hand side from the left
• rk - c1rk-1 - c2rk-2 - - ck-1r - ck 0
• This is called the characteristic equation of the
recurrence relation.

67
Solving Recurrence Relations
• The solutions of this equation are called the
characteristic roots of the recurrence relation.
• Let us consider linear homogeneous recurrence
relations of degree two.
• Theorem Let c1 and c2 be real numbers. Suppose
that r2 c1r c2 0 has two distinct roots r1
and r2.
• Then the sequence an is a solution of the
recurrence relation an c1an-1 c2an-2 if and
only if an ?1r1n ?2r2n for n 0, 1, 2, ,
where ?1 and ?2 are constants.

68
Solving Recurrence Relations
• Example Give an explicit formula for the
Fibonacci numbers.
• Solution The Fibonacci numbers satisfy the
recurrence relation fn fn-1 fn-2 with initial
conditions f0 0 and f1 1.
• The characteristic equation is r2 r 1 0.
• Its roots are

69
Solving Recurrence Relations
• Therefore, the Fibonacci numbers are given by
• for some constants ?1 and ?2.
• We can determine values for these constants so
that the sequence meets the conditions f0 0 and
f1 1

70
Solving Recurrence Relations
• The unique solution to this system of two
equations and two variables is
• So finally we obtained an explicit formula for
the Fibonacci numbers

71
Solving Recurrence Relations
• But what happens if the characteristic equation
has only one root?
• How can we then match our equation with the
initial conditions a0 and a1 ?
• Theorem Let c1 and c2 be real numbers with c2??
0. Suppose that r2 c1r c2 0 has only one
root r0. A sequence an is a solution of the
recurrence relation an c1an-1 c2an-2 if and
only if an ?1r0n ?2nr0n, for n 0, 1, 2, ,
where ?1 and ?2 are constants.

72
Solving Recurrence Relations
• Example What is the solution of the recurrence
relation an 6an-1 9an-2 with a0 1 and a1
6?
• Solution The only root of r2 6r 9 0 is r0
3.Hence, the solution to the recurrence
relation is
• an ?13n ?2n3n for some constants ?1 and ?2.
• To match the initial condition, we need
• a0 1 ?1a1 6 ?1?3 ?2?3
• Solving these equations yields ?1 1 and ?2 1.
• Consequently, the overall solution is given by
• an 3n n3n.

73
Multiple Roots
• If a is the root of the characteristic equation
with multiplicity m, there is a general solution

74
Getting a Particular Solution
• Mainly try and error. However, some very good
suggestions do exist
• Sometimes your initial guess "degenerates" and
gives contradictory conditions. Then try a
solution of higher degree.

75
Examples

76
A Total Example
• Once the general and particular solutions have
been found, they are added together to give the
total solution.
• The initial conditions are then used to determine
the constants of the general solution.

77
A Total Example

78
Pseudo Nonlinear Recurrence
• Range Transformations on the values of the
sequences
• Domain Transformations on the value of indices

79
Example

80
A Practical Example
•  Mergesort
• Split list in half
• Sort both halves
• Merge the two halves
• Analysis The recurrence relation is
• T(0) 0

81
Example Continued
• T(n) 2T(n/2) (n 1)

82
Example Continued

83
Example Continued
• (3) Impose the initial conditions on A2n n ? 2n
1
• (n0) A 0 1 U(0) 0 ? A -1
• So U(n) n ? 2n 2n 1 T(2n)
• Replace n by log n gives
•
• T(n) n?log n n 1 Q(n ? log n)

84
Master Theorem
• Let a ? 1 and b gt 1 be constants, f(n) be a
function, and T(n) be defined on nonnegative
integers as
• T(n) aT(n/b) f(n).
• Then, T(n) can be bounded asymptotically as
follows

85
Solving Recurrences by Using Master Method