CS 3343 Analysis of Algorithms

- Lecture 22 Minimum Spanning Tree

Outline

- Review of last lecture
- Prims algorithm and Kruskals algorithm for MST

in detail

Graph Review

- G (V, E)
- V a set of vertices
- E a set of edges
- Degree number of edges
- In-degree vs out-degree
- Types of graphs
- Directed / undirected
- Weighted / un-weighted
- Dense / sparse
- Connected / disconnected

Graph representations

- Adjacency matrix

A 1 2 3 4

1 0 1 1 0

2 0 0 1 0

3 0 0 0 0

4 0 0 1 0

1

2

4

3

How much storage does the adjacency matrix

require? A O(V2)

Graph representations

- Adjacency list

1

2

3

3

2

4

3

3

How much storage does the adjacency list

require? A O(VE)

Graph representations

- Undirected graph

1

2

4

3

2

3

1

3

1

2

4

3

Graph representations

- Weighted graph

1

5

6

2

4

9

4

3

2,5

3,6

1,5

3,9

1,6

2,9

4,4

3,4

Analysis of time complexity

- Convention
- Number of vertices n, V, V
- Number of edges m, E, E
- O(VE) is the same as O(nm) or O(VE)

Tradeoffs between the two representations

V n, E m

Adj Matrix Adj List

test (u, v) ? E T(1) O(n)

Degree(u) T(n) O(n)

Memory T(n2) T(nm)

Edge insertion T(1) T(1)

Edge deletion T(1) O(n)

Graph traversal T(n2) T(nm)

Both representations are very useful and have

different properties, although adjacency lists

are probably better for most problems

Minimum Spanning Tree

- Problem given a connected, undirected, weighted

graph

Minimum Spanning Tree

- Problem given a connected, undirected, weighted

graph, find a spanning tree using edges that

minimize the total weight

6

4

5

9

- A spanning tree is a tree that connects all

vertices - Number of edges ?
- A spanning tree has no designated root.

14

2

10

15

3

8

Minimum Spanning Tree

- MSTs satisfy the optimal substructure property

an optimal tree is composed of optimal subtrees - Let T be an MST of G with an edge (u,v) in the

middle - Removing (u,v) partitions T into two trees T1 and

T2 - w(T) w(u,v) w(T1) w(T2)
- Claim 1 T1 is an MST of G1 (V1, E1), and T2 is

an MST of G2 (V2, E2)

- Proof by contradiction
- if T1 is not optimal, we can replace T1 with a

better spanning tree, T1 - T1, T2 and (u, v) form a new spanning tree T
- W(T) lt W(T). Contradiction.

Minimum Spanning Tree

- MSTs satisfy the optimal substructure property

an optimal tree is composed of optimal subtrees - Let T be an MST of G with an edge (u,v) in the

middle - Removing (u,v) partitions T into two trees T1 and

T2 - w(T) w(u,v) w(T1) w(T2)
- Claim 2 (u, v) is the lightest edge connecting

G1 (V1, E1) and G2 (V2, E2)

- Proof by contradiction
- if (u, v) is not the lightest edge, we can remove

it, and reconnect T1 and T2 with a lighter edge

(x, y) - T1, T2 and (x, y) form a new spanning tree T
- W(T) lt W(T). Contradiction.

T2

T1

v

u

Algorithms

- Generic idea
- Compute MSTs for sub-graphs
- Connect two MSTs for sub-graphs with the lightest

edge - Two of the most well-known algorithms
- Prims algorithm
- Kruskals algorithm
- Lets first talk about the ideas behind the

algorithms without worrying about the

implementation and analysis

Prims algorithm

- Basic idea
- Start from an arbitrary single node
- A MST for a single node has no edge
- Gradually build up a single larger and larger MST

6

5

Not yet discovered

7

Fully explored nodes

Discovered but not fully explored nodes

Prims algorithm

- Basic idea
- Start from an arbitrary single node
- A MST for a single node has no edge
- Gradually build up a single larger and larger MST

2

6

5

9

Not yet discovered

4

7

Fully explored nodes

Discovered but not fully explored nodes

Prims algorithm

- Basic idea
- Start from an arbitrary single node
- A MST for a single node has no edge
- Gradually build up a single larger and larger MST

2

6

5

9

4

7

Prims algorithm in words

- Randomly pick a vertex as the initial tree T
- Gradually expand into a MST
- For each vertex that is not in T but directly

connected to some nodes in T - Compute its minimum distance to any vertex in T
- Select the vertex that is closest to T
- Add it to T

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Pseudocode for Prims algorithm

- Given G (V, E). Output a MST T.
- Randomly select a vertex v
- S v A V \ S T .
- While (A is not empty)
- find a vertex u ? A that connects to vertex v ?

S such that w(u, v) w(x, y), for any x ? A and

y ? S - S S U u A A \ u T T U (u, v).
- End
- Return T

Time complexity

- Given G (V, E). Output a MST T.
- Randomly select a vertex v
- S v A V \ S T .
- While (A is not empty)
- find a vertex u ? A that connects to vertex v ?

S such that w(u, v) w(x, y), for any x ? A and

y ? S - S S U u A A \ u T T U (u, v).
- End
- Return T

n vertices

- Time complexity n (time spent on purple line

per vertex) - Naïve test all edges gt T(n m)
- Improve keep the list of candidates in an array

gt T(n2) - Better with priority queue gt T(m log n)

Idea 1 naive

- find a vertex u ? A that connects to vertex v ?

S such that w(u, v) w(x, y), for any x ? A and

y ? S - min_weight infinity.
- For each edge (x, y) ? E
- if x ? A, y ? S, and w(x, y) lt min_weight
- u x v y min_weight w(x, y)

time spent per vertex T(m) Total time

complexity T(nm)

Idea 2 distance array

- // For each vertex v, dv is the min distance

from v to any node already in S - // pv is the parent node of v in the spanning

tree - For each v ? V
- dv infinity pv null
- Randomly select a v, dv 1 // dv1 just to

ensure proper start - S A V T .
- While (A is not empty)
- Search d to find the smallest du gt 0.
- S S U u A A \ u T T U (u, pu)
- du 0.
- For each v in adju
- if dv gt w(u, v)
- dv w(u, v)
- pv u
- End

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?8 8 1 8 8 8 8 8

p ?/ / / / / / / /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?8 8 0 8 8 8 8 8

p ?/ / / / / / / /

a

6

12

9

5

b

f

g

Discovered

7

14

15

8

c

e

h

10

3

Explored

d

Not discovered

a b c d e f g h

d ?8 14 0 3 8 8 8 8

p ?/ c / c / / / /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?8 14 0 0 8 8 8 8

p ?/ c / c / / / /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?8 8 0 0 10 8 8 8

p ?/ d / c d / / /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?8 0 0 0 10 8 8 8

p ?/ d / c d / / /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?6 0 0 0 10 5 8 8

p ?b d / c d b / /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?6 0 0 0 10 0 8 8

p ?b d / c d b / /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?6 0 0 0 7 0 9 8

p ?b d / c f b f /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?0 0 0 0 7 0 9 8

p ?b d / c f b f /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?0 0 0 0 0 0 9 8

p ?b d / c f b f /

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?0 0 0 0 0 0 9 15

p ?b d / c f b f e

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?0 0 0 0 0 0 0 15

p ?b d / c f b f e

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

d ?0 0 0 0 0 0 0 0

p ?b d / c f b f e

Time complexity?

- // For each vertex v, dv is the min distance

from v to any node already in S - // pv is the parent node of v in the spanning

tree - For each v ? V
- dv infinity pv null
- Randomly select a v, dv 1 // dv1 just to

ensure proper start - S . T . A V.
- While (A is not empty)
- Search d to find the smallest du gt 0.
- S S U u A A \ u T T U (u, pu)
- du 0.
- For each v in adju
- if dv gt w(u, v)
- dv w(u, v)
- pv u
- End

n vertices

T(n) per vertex

O(n) per vertex if using adj list ?(n) per vertex

if using adj matrix

Overall complexity T (n2)

Idea 3 priority queue (min-heap)

- Observation
- In idea 2, we need to search the distance array n

times, each time we only look for the minimum

element. - Distance array size n
- n x n n2
- Can we do better?
- Priority queue (heap) enables fast retrieval of

min (or max) elements - log (n) for most operations

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a b c d e f g h

?8 8 8 8 8 8 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ChangeKey

c b a d e f g h

?0 8 8 8 8 8 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ExctractMin

h b a d e f g

8 8 8 8 8 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ChangeKey

d b a h e f g

3 14 8 8 8 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ExctractMin

b g a h e f

?14 8 8 8 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Changekey

b e a h g f

?8 10 8 8 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ExtractMin

e f a h g

10 8 8 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Changekey

f e a h g

5 10 6 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ExtractMin

a e g h

6 10 8 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Changekey

a e g h

6 7 9 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ExtractMin

e h g

7 8 9

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ExtractMin

g h

9 8

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Changekey

g h

9 15

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

ExtractMin

h

15

Example

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Complete Prims Algorithm

- MST-Prim(G, w, r)
- Q VG
- for each u ? Q
- keyu ?
- pu null
- keyr 0
- while (Q not empty)
- u ExtractMin(Q)
- for each v ? Adju
- if (v ? Q and w(u,v) lt keyv)
- pv u
- ChangeKey(v, w(u,v))

Overall running time T(m log n)

n vertices

T(n) times

T(n2) times?

T(m) times (with adj list)

How often is ExtractMin() called? How often is

ChangeKey() called?

Kruskals algorithm in words

- Procedure
- Sort all edges into non-decreasing order
- Initially each node is in its own tree
- For each edge in the sorted list
- If the edge connects two separate trees, then
- join the two trees together with that edge

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Example

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

Time complexity

- Depend on implementation
- Pseudocode
- sort all edges according to weights
- T . tree(v) v for all v.
- for each edge (u, v)
- if tree(u) ! tree(v)
- T T U (u, v)
- union (tree(u), tree(v))

T(m log m) T(m log n)

m edges

Overall time complexity m log n m t n

u Naïve T(nm) Better implementation T(m log n)

Disjoint Set Union

- Use a linked list to represent tree elements,

with pointers back to root - treeu ! treev O(1)
- Union(treeu,treev) Copy elements of A into

set B by adjusting elements of A to point to B - Each union may take O(n) time
- Precisely n-1 unions
- O(n2)?

Disjoint Set Union

- Better strategy and analysis
- Always copy smaller list into larger list
- Size of combined list is at least twice the size

of smaller list - Each vertex copied at most logn times before a

single tree emerges - Total number of copy operations for n vertex is

therefore O(n log n) - Overall time complexity m log n m t n u
- t time for finding tree root is constant
- u time for n union operations is at most nlog

(n) - m log n m t n u T (m log n m n log n)

T (m log n) - Conclusion
- Kruskals algorithm runs in T(m log n) for both

dense and sparse graph

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a

b

c

d

e

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a

b

c

d

e

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a

b

c

d

e

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a

b

c

d

e

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

a

b

c

d

g

f

e

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

e

b

c

d

a

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

e

b

c

d

a

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

e

b

c

d

a

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

e

b

c

d

a

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

e

b

c

d

a

f

g

h

Example with disjoint set union

c-d 3 b-f 5 b-a 6 f-e 7 b-d 8 f-g

9 d-e 10 a-f 12 b-c 14 e-h 15

a

6

12

9

5

b

f

g

7

14

15

8

c

e

h

10

3

d

e

b

c

d

h

f

g

a

Disjoint Set Union

- Better strategy and analysis
- Always copy smaller list into larger list
- Size of combined list is at least twice the size

of smaller list - Each vertex copied at most logn times before a

single tree emerges - Total number of copy operations for n vertex is

therefore O(n log n) - Overall time complexity m log n m t n u
- t time for finding tree root is constant
- u time for n union operations is at most nlog

(n) - m log n m t n u T (m log n m n log n)

T (m log n) - Conclusion
- Kruskals algorithm runs in T(m log n) for both

dense and sparse graph - How about using counting sort?
- T(m n log n)

Summary

- Kruskals algorithm
- T(m log n)
- Possibly T(m n log n) with counting sort
- Prims algorithm
- With priority queue T(m log n)
- Assume graph represented by adj list
- With distance array T(n2)
- Adj list or adj matrix
- For sparse graphs priority queue wins
- For dense graphs distance array may be better