2IL65 Algorithms

1
2IL65 Algorithms

• Fall 2009Lecture 1 Introduction

2
Algorithms

• Algorithma well-defined computational procedure
  that takes some value, or a set of values, as
  input and produces some value, or a set of
  values, as output.
• Algorithmsequence of computational steps that
  transform the input into the output.
• Algorithms researchdesign and analysis of
  algorithms and data structures for computational
  problems.

3
Data structures

• Data Structurea way to store and organize data
  to facilitate access and modifications.
• Abstract data typedescribes functionality (which
  operations are supported).
• Implementationa way to realize the desired
  functionality
• how is the data stored (array, linked list, )
• which algorithms implement the operations

4
The course

• Design and analysis of efficient algorithms for
  some basic computational problems.
• Basic algorithm design techniques and paradigms
• Algorithms analysis O-notation, recursions,
• Basic data structures
• Basic graph algorithms

5
Some administration first
before we really get started
6
Organization

• Lecturer Elena Mumford, HG 7.35, e.mumford_at_tue
  .nl
• Web page http//www.win.tue.nl/emumford/Algo/2IL
  65.html
• Book T.H. Cormen, C.E. Leiserson, R.L. Rivest
  and C. Stein. Introduction to Algorithms (2nd
  edition) mandatory

7
Prerequisites

• Being able to work with basic programming
  constructs such as linked lists, arrays, loops
• Being able to apply standard proving techniques
  such as proof by induction, proof by
  contradiction ...
• Being familiar with sums and logarithms such as
  discussed in Chapter 3 and Appendix A of the
  textbook.
• If you think you might lack any of this
  knowledge, please come and talk to me immediately
  so we can get you caught up.

8
Grading scheme 2IL65

• 7 homework assignments, the best 5 of which each
  count for 10 of the final grade.
• A written exam (closed book) which counts for the
  remaining 50 of the final grade.
• If you reach less than 50 of the possible points
  on the homework assignments, then you are not
  allowed to participate in the final exam nor in
  the second chance exam. You will fail the course
  and your next chance will be next year. Your
  grade will be the minimum of 5 and the grade you
  achieved. If you reach less than 50 of the
  points on the final exam, then you will fail the
  course, regardless of the points you collected
  with the homework assignments. However, you are
  allowed to participate in the second chance exam.
  The grade of the second chance exam replaces the
  grade for the first exam, that is, your homework
  assignments always count for 50 of your grade.

Do the homework assignments!
9
Homework Assignments

• Posted on web-page on Wednesdays after the
  lecture.
• Due Tuesdays at 2359 as .pdf in my electronic
  mailbox.
• Late assignments will not be accepted. Only 5
  out of 7 assignments count, hence there are no
  exceptions.
• Must be typeset.
• Graded within 36 hours.
• Any questions Stop by my office whenever you
  want (send email if you want to make sure that
  I have time)

10
Homework Assignments
11
Academic Dishonesty

• Academic Dishonesty All class work has to be
  done independently. You are of course allowed to
  discuss the material presented in class, homework
  assignments, or general solution strategies with
  me or your classmates, but you have to formulate
  and write up your solutions by yourself. You must
  not copy from the internet, your friends, or
  other textbooks. Problem solving is an important
  component of this course so it is really in your
  best interest to try and solve all problems by
  yourself. If you represent other people's work as
  your own then that constitutes fraud and will be
  dealt with accordingly.

12
Organization

• Components
• Lecture Wednesday 78 paviljoen-r-gebouw 0.14
• Big tutorial Monday 56 potentiaal
  13.03During this tutorial you have the
  opportunity to work on this week's home work
  assignment. I will be present to answer
  questions.
• Small tutorial Thursday 78
  paviljoen-r-gebouw 0.14 During these tutorials
  I will explain the solutions to the homework
  assignments of the previous week and answer any
  questions that arise.
• Check web-page for details

13
Schedule
14
Sorting
15
The sorting problem

• Input a sequence of n numbers a1, a2, , an
• Output a permutation of the input such that ai1
  ain
• The input is typically stored in arrays
• Numbers Keys
• Additional information (satellite data) may be
  stored with keys
• We will study several solutions algorithms for
  this problem

16
Describing algorithms

• A complete description of an algorithm consists
  of three parts
• the algorithm
• (expressed in whatever way is clearest and most
  concise, can be English and / or pseudocode)
• a proof of the algorithms correctness
• a derivation of the algorithms running time

17
InsertionSort

• Like sorting a hand of playing cards
• start with empty left hand, cards on table
• remove cards one by one, insert into correct
  position
• to find position, compare to cards in hand from
  right to left
• cards in hand are always sorted
• InsertionSort is
• a good algorithm to sort a small number of
  elements
• an incremental algorithm
• Incremental algorithmsprocess the input elements
  one-by-one and maintain the solution for the
  elements processed so far.

18
Incremental algorithms

• Incremental algorithmsprocess the input elements
  one-by-one and maintain the solution for the
  elements processed so far.
• In pseudocode
• IncAlg(A)
• ? incremental algorithm which computes the
  solution of a problem with input A
  x1,,xn
• initialize compute the solution for x1
• for i ? 2 to n
• do compute the solution for x1,,xj using
  the (already computed) solution for
  x1,,xj-1

Check book for more pseudocode conventions
comment
assignment (Pascal )
no begin - end, just indentation
19
InsertionSort

• InsertionSort(A)
• ? incremental algorithm that sorts array A1..n
  in non-decreasing order
• initialize sort A1
• for j ? 2 to lengthA
• do sort A1..j using the fact that A1..
  j-1 is already sorted

20
InsertionSort

• InsertionSort(A)
• ? incremental algorithm that sorts array A1..n
  in non-decreasing order
• initialize sort A1
• for j ? 2 to lengthA
• do key ? Aj
• i ? j -1
• while i gt 0 and Ai gt key
• do Ai1 ? Ai
• i ? i -1
• Ai 1 ? key

InsertionSort is an in place algorithm the
numbers are rearranged within the array with
only constant extra space.
21
Correctness proof

• Use a loop invariant to understand why an
  algorithm gives the correct answer.
• Loop invariant (for InsertionSort)At the start
  of each iteration of the outer for loop
  (indexed by j) the subarray A1..j-1 consists of
  the elements originally in A1..j-1 but in
  sorted order.

22
Correctness proof

• To proof correctness with a loop invariant we
  need to show three things
• InitializationInvariant is true prior to the
  first iteration of the loop.
• MaintenanceIf the invariant is true before an
  iteration of the loop, it remains true before the
  next iteration.
• TerminationWhen the loop terminates, the
  invariant (usually along with the reason that the
  loop terminated) gives us a useful property that
  helps show that the algorithm is correct.

23
Correctness proof

• InsertionSort(A)
• initialize sort A1
• for j ? 2 to lengthA
• do key ? Aj
• i ? j -1
• while i gt 0 and Ai gt key
• do Ai1 ? Ai
• i ? i -1
• Ai 1 ? key
• InitializationJust before the first iteration, j
  2 ? A1..j-1 A1, which is the element
  originally in A1, and it is trivially sorted.

Loop invariant At the start of each iteration of
the outer for loop (indexed by j) the subarray
A1..j-1 consists of the elements originally in
A1..j-1 but in sorted order.
24
Correctness proof

• InsertionSort(A)
• initialize sort A1
• for j ? 2 to lengthA
• do key ? Aj
• i ? j -1
• while i gt 0 and Ai gt key
• do Ai1 ? Ai
• i ? i -1
• Ai 1 ? key
• MaintenanceStrictly speaking need to prove loop
  invariant for inner while loop. Instead, note
  that body of while loop moves Aj-1, Aj-2,
  Aj-3, and so on, by one position to the right
  until proper position of key is found (which has
  value of Aj) ? invariant maintained.

Loop invariant At the start of each iteration of
the outer for loop (indexed by j) the subarray
A1..j-1 consists of the elements originally in
A1..j-1 but in sorted order.
25
Correctness proof

• InsertionSort(A)
• initialize sort A1
• for j ? 2 to lengthA
• do key ? Aj
• i ? j -1
• while i gt 0 and Ai gt key
• do Ai1 ? Ai
• i ? i -1
• Ai 1 ? key
• TerminationThe outer for loop ends when j gt n
  this is when j n1 ? j-1 n. Plug n for j-1
  in the loop invariant ? the subarray A1..n
  consists of the elements originally in A1..n in
  sorted order.

Loop invariant At the start of each iteration of
the outer for loop (indexed by j) the subarray
A1..j-1 consists of the elements originally in
A1..j-1 but in sorted order.
26
Another sorting algorithm
using a different paradigm
27
MergeSort

• A divide-and-conquer sorting algorithm.
• Divide-and-conquerbreak the problem into two or
  more subproblems, solve the subproblems
  recursively, and then combine these solutions to
  create a solution to the original problem.

28
Divide-and-conquer

• DCAlg(A)
• ? divide-and-conquer algorithm that computes the
  solution of a problem with input A x1,,xn
• if elements of A is small enough (for example
  1)
• then compute Sol (the solution for A)
  brute-force
• else
• split A in, for example, 2 non-empty
  subsets A1 and A2
• Sol1 ? DCAlg(A1)
• Sol2 ? DCAlg(A2)
• compute Sol (the solution for A) from
  Sol1 and Sol2
• return Sol

29
MergeSort

• MergeSort(A)
• ? divide-and-conquer algorithm that sorts array
  A1..n
• if lengthA 1
• then compute Sol (the solution for A)
  brute-force
• else
• split A in 2 non-empty subsets A1 and
  A2
• Sol1 ? MergeSort(A1)
• Sol2 ? MergeSort(A2)
• compute Sol (the solution for A) from
  Sol1 en Sol2

30
MergeSort

• MergeSort(A)
• ? divide-and-conquer algorithm that sorts array
  A1..n
• if lengthA 1
• then skip
• else
• n ? lengthA n1 ? n/2 n2 ? n/2
  
• copy A1.. n1 to auxiliary array
  A11.. n1
• copy An11..n to auxiliary
  array A21.. n2
• MergeSort(A1)
• MergeSort(A2)
• Merge(A, A1, A2)

31
MergeSort
32
MergeSort

• Merging

A
1
3
4
7
8
14
17
21
28
35
33
MergeSort correctness proof

• Induction on n ( of input elements)
• proof that the base case (n small) is solved
  correctly
• proof that if all subproblems are solved
  correctly, then the complete problem is solved
  correctly

34
MergeSort correctness proof

• MergeSort(A)
• if lengthA 1
• then skip
• else
• n ? lengthA n1 ? n/2 n2 ? n/2
  
• copy A1.. n1 to auxiliary
  array A11.. n1
• copy An11..n to auxiliary
  array A21.. n2
• MergeSort(A1)
• MergeSort(A2)
• Merge(A, A1, A2)

• Proof (by induction on n)
• Base case n 1, trivial ?
• Inductive step assume n gt 1. Note that n1 lt n
  and n2 lt n.
• Inductive hypothesis ? arrays A1 and A2 are
  sorted correctly
• Remains to show Merge(A, A1, A2) correctly
  constructs a sorted array A out of the sorted
  arrays A1 and A2 etc.

Lemma MergeSort sorts the array A1..n
correctly.
35
QuickSort
another divide-and-conquer sorting algorithm
36
QuickSort

• QuickSort(A)
• ? divide-and-conquer algorithm that sorts array
  A1..n
• if lengthA 1
• then skip
• else
• pivot ? A1
• move all Ai with Ai lt pivot into
  auxiliary array A1
• move all Ai with Ai gt pivot into
  auxiliary array A2
• move all Ai with Ai pivot into
  auxiliary array A3
• QuickSort(A1)
• QuickSort(A2)
• A ? A1 followed by A3 followed by A2

37
Analysis of algorithms
some informal thoughts for now
38
Analysis of algorithms

• Can we say something about the running time of an
  algorithm without implementing and testing it?

• InsertionSort(A)
• initialize sort A1
• for j ? 2 to lengthA
• do key ? Aj
• i ? j -1
• while i gt 0 and Ai gt key
• do Ai1 ? Ai
• i ? i -1
• Ai 1 ? key

39
Analysis of algorithms

• Analyze the running time as a function of n ( of
  input elements)
• best case
• average case
• worst case
• elementary operationsadd, subtract, multiply,
  divide, load, store, copy, conditional and
  unconditional branch, return

An algorithm has worst case running time T(n) if
for any input of size n the maximal number of
elementary operations executed is T(n).
40
Analysis of algorithms example
InsertionSort 15 n2 7n 2
MergeSort 300 n log n 50 n
n 1,000,000 InsertionSort 1.5 x 1013
MergeSort 6 x
109 2500 x faster !
41
Analysis of algorithms

• It is extremely important to have efficient
  algorithms for large inputs
• The rate of growth (or order of growth) of the
  running time is far more important than
  constants

InsertionSort T(n2) MergeSort T(n log n)
42
T-notation

• Intuition concentrate on the leading term,
  ignore constants
• 19 n3 17 n2 - 3n becomes T(n3)
• 2 n lg n 5 n1.1 - 5 becomes
• n - ¾ n vn becomes
• (precise definition next week )

T(n1.1)
---
43
Find the leading term

• lg35n vs. vn ?
• logarithmic functions grow slower than polynomial
  functions
• lga n grows slower than nb for all constants a
  gt 0 and b gt 0
• n100 vs. 2 n ?
• polynomial functions grow slower than exponential
  functions
• na grows slower than bn for all constants a gt 0
  and b gt 1

44
Some rules and notation

• log n denotes log2 n
• We have for a, b, c gt 0
• logc (ab) logc a logc b
• logc (ab) b logc a
• loga b logc b / logc a

45
Announcements

• This week
• big tutorial on Thursday 78 in
  Paviljoen-r-gebouw 0.14