Agglomerative clustering (AC)

Clustering algorithms Part 2c

Pasi Fränti 25.3.2014 Speech Image Processing

Unit School of Computing University of Eastern

Finland Joensuu, FINLAND

Agglomerative clusteringCategorization by cost

function

- Single link
- Minimize distance of nearest vectors
- Complete link
- Minimize distance of two furthest vectors
- Wards method
- Minimize mean square error
- In Vector Quantization, known as Pairwise

Nearest Neighbor (PNN) method

We focus on this

Pseudo code

Pseudo code

PNN(X, M) ? C, P FOR i?1 TO N DO pi?i

ci?xi REPEAT a,b ? FindSmallestMergeCost() M

ergeClusters(a,b) m?m-1 UNTIL mM

O(N)

O(N2)

N times

T(N) O(N3)

Wards methodWard 1963 Journal of American

Statistical Association

Merge cost

Local optimization strategy

- Nearest neighbor search
- Find the cluster pair to be merged
- Update of NN pointers

Example of distance calculations

Example of the overall process

M5000

M50

M5000 M4999 M4998 . . . M50 . . M16 M15

M16

M15

Detailed example of the process

Example - 25 Clusters

MSE 1.01109

Example - 24 Clusters

MSE 1.03109

Example - 23 Clusters

MSE 1.06109

Example - 22 Clusters

MSE 1.09109

Example - 21 Clusters

MSE 1.12109

Example - 20 Clusters

MSE 1.16109

Example - 19 Clusters

MSE 1.19109

Example - 18 Clusters

MSE 1.23109

Example - 17 Clusters

MSE 1.26109

Example - 16 Clusters

MSE 1.30109

Example - 15 Clusters

MSE 1.34109

Storing distance matrix

- Maintain the distance matrix and update rows for

the changed cluster only! - Number of distance calculations reduces from

O(N2) to O(N) for each step. - Search of the minimum pair still requires O(N2)

time ? still O(N3) in total. - It also requires O(N2) memory.

Heap structure for fast searchKurita 1991

Pattern Recognition

- Search reduces O(N2) ? O(logN).
- In total O(N2 logN)

Store nearest neighbor (NN) pointersFränti et

al., 2000 IEEE Trans. Image Processing

Time complexity reduces to O(N 3) ? ? (?N 2)

Pseudo code

PNN(X, M) ? C, P FOR i?1 TO N DO pi?i

ci?xi FOR i?1 TO N DO NNi?

FindNearestCluster(i) REPEAT a ?

SmallestMergeCost(NN) b ? NNi MergeClusters(C,

P,NN,a,b,) UpdatePointers(C,NN) UNTIL mM

O(N)

O(N2)

O(N)

O(?N)

http//cs.uef.fi/pages/franti/research/pnn.txt

Example with NN pointersVirmajoki 2004

Pairwise Nearest Neighbor Method Revisited

ExampleStep 1

ExampleStep 2

ExampleStep 3

ExampleStep 4

ExampleFinal

Time complexities of the variants

Number of neighbors (t)

Processing time comparison

With NN pointers

AlgorithmLazy-PNN

T. Kaukoranta, P. Fränti and O. Nevalainen,

"Vector quantization by lazy pairwise nearest

neighbor method", Optical Engineering, 38 (11),

1862-1868, November 1999

Monotony property of merge cost Kaukoranta et

al., Optical Engineering, 1999

Merge costs values are monotonically increasing

d(Sa, Sb) ? d(Sa, Sc) ? d(Sb, Sc) ? d(Sa, Sc) ?

d(Sab, Sc)

Lazy variant of the PNN

- Store merge costs in heap.
- Update merge cost value only when it appears at

top of the heap. - Processing time reduces about 35.

Method Ref. Time complexity Additional data structure Space compl.

Trivial PNN 10 O(dN3) - O(N)

Distance matrix 6 O(dN2 N3) Distance matrix O(N2)

Kuritas method 5 O(dN2 N2logN) Dist. matrix heap O(N2)

?-PNN 1 O(d?N2) NN-table O(N)

Lazy-PNN 4 O(d?N2) NN-table O(N)

Combining PNN and K-means

K-means

AlgorithmIterative shrinking

P. Fränti and O. Virmajoki Iterative shrinking

method for clustering problemsPattern

Recognition, 39 (5), 761-765, May 2006.

Agglomerative clustering based on merging

Agglomeration based on cluster removalFränti

and Virmajoki, Pattern Recognition, 2006

Merge versus removal

Pseudo code of iterative shrinking (IS)

Cluster removal in practice

Find secondary cluster

Calculate removal cost for every vector

Partition updates

Complexity analysis

- Number of vectors per cluster

If we iterate until M1

Adding the processing time per vector

AlgorithmPNN with kNN-graph

P. Fränti, O. Virmajoki and V. Hautamäki, "Fast

agglomerative clustering using a k-nearest

neighbor graph". IEEE Trans. on Pattern Analysis

and Machine Intelligence, 28 (11), 1875-1881,

November 2006

Agglomerative clustering with kNN graph

Example of 2NN graph

Example of 4NN graph

Graph using double linked lists

Merging a and b

Effect on calculationsnumber of steps

STAGE Theoretical Theoretical Theoretical Observed Observed Observed

STAGE ?-PNN Single link Double link ?-PNN Single link Double link

Find pair N 1 1 8 357 3 3

Merge N k2 logN k2 ?k logN 8 367 200 305

Remove last N k logN LogN 8 349 102 45

Find neighbors N kN ?k 8 357 41 769 204

Update costs N (1?) ? ?/k?logN ? ?/k?logN 48 538 198 187

TOTAL O(?N2) O(kN2) O(?N?logN) 81 970 42 274 746

Processing time as function of k(number of

neighbors in graph)

Time distortion comparison

?-PNN (229 s) Trivial-PNN (gt9999 s)

Graph-PNN (1)

MSE 5.36

Graph-PNN (2)

- Graph created by MSP
- Graph created by D-n-C

Conclusions

- Simple to implement, good clustering quality
- Straightforward algorithm slow O(N3)
- Fast exact (yet simple) algorithm O(tN2)
- Beyond this possible
- O(tNlogN) complexity
- Complicated graph data structure
- Compromizes the exactness of the merge

Literature

- P. Fränti, T. Kaukoranta, D.-F. Shen and

K.-S. Chang, "Fast and memory efficient

implementation of the exact PNN", IEEE Trans. on

Image Processing, 9 (5), 773-777, May 2000. - P. Fränti, O. Virmajoki and V. Hautamäki, "Fast

agglomerative clustering using a k-nearest

neighbor graph". IEEE Trans. on Pattern Analysis

and Machine Intelligence, 28 (11), 1875-1881,

November 2006. - P. Fränti and O. Virmajoki, "Iterative shrinking

method for clustering problems", Pattern

Recognition, 39 (5), 761-765, May 2006. - T. Kaukoranta, P. Fränti and O. Nevalainen,

"Vector quantization by lazy pairwise nearest

neighbor method", Optical Engineering, 38 (11),

1862-1868, November 1999. - T. Kurita, "An efficient agglomerative clustering

algorithm using a heap", Pattern Recognition 24

(3) (1991) 205-209.

Literature

- J. Shanbehzadeh and P.O. Ogunbona, "On the

computational complexity of the LBG and PNN

algorithms". IEEE Transactions on Image

Processing 6 (4), 614-616, April 1997. - O. Virmajoki, P. Fränti and T. Kaukoranta,

"Practical methods for speeding-up the pairwise

nearest neighbor method ", Optical Engineering,

40 (11), 2495-2504, November 2001. - O. Virmajoki and P. Fränti, "Fast pairwise

nearest neighbor based algorithm for multilevel

thresholding", Journal of Electronic Imaging, 12

(4), 648-659, October 2003. - O. Virmajoki, Pairwise Nearest Neighbor Method

Revisited, PhD thesis, Computer Science,

University of Joensuu, 2004. - J.H. Ward, Hierarchical grouping to optimize an

objective function, J. Amer. Statist.Assoc. 58

(1963) 236-244.