Agglomerative clustering (AC) - PowerPoint PPT Presentation

About This Presentation
Title:

Agglomerative clustering (AC)

Description:

Speech and Image Processing Unit. Department of Computer Science ... Monotony property of merge cost function [Kaukoranta et al., Optical Engineering, 1997] ... – PowerPoint PPT presentation

Number of Views:154
Avg rating:3.0/5.0
Slides: 57
Provided by: csJoe
Category:

less

Transcript and Presenter's Notes

Title: Agglomerative clustering (AC)


1
Agglomerative clustering (AC)
Clustering algorithms Part 2c
Pasi Fränti 25.3.2014 Speech Image Processing
Unit School of Computing University of Eastern
Finland Joensuu, FINLAND
2
Agglomerative clusteringCategorization by cost
function
  • Single link
  • Minimize distance of nearest vectors
  • Complete link
  • Minimize distance of two furthest vectors
  • Wards method
  • Minimize mean square error
  • In Vector Quantization, known as Pairwise
    Nearest Neighbor (PNN) method

We focus on this
3
Pseudo code
4
Pseudo code
PNN(X, M) ? C, P FOR i?1 TO N DO pi?i
ci?xi REPEAT a,b ? FindSmallestMergeCost() M
ergeClusters(a,b) m?m-1 UNTIL mM
O(N)
O(N2)
N times
T(N) O(N3)
5
Wards methodWard 1963 Journal of American
Statistical Association
Merge cost
Local optimization strategy
  • Nearest neighbor search
  • Find the cluster pair to be merged
  • Update of NN pointers

6
Example of distance calculations
7
Example of the overall process
M5000
M50
M5000 M4999 M4998 . . . M50 . . M16 M15
M16
M15
8
Detailed example of the process
9
Example - 25 Clusters
MSE 1.01109
10
Example - 24 Clusters
MSE 1.03109
11
Example - 23 Clusters
MSE 1.06109
12
Example - 22 Clusters
MSE 1.09109
13
Example - 21 Clusters
MSE 1.12109
14
Example - 20 Clusters
MSE 1.16109
15
Example - 19 Clusters
MSE 1.19109
16
Example - 18 Clusters
MSE 1.23109
17
Example - 17 Clusters
MSE 1.26109
18
Example - 16 Clusters
MSE 1.30109
19
Example - 15 Clusters
MSE 1.34109
20
Storing distance matrix
  • Maintain the distance matrix and update rows for
    the changed cluster only!
  • Number of distance calculations reduces from
    O(N2) to O(N) for each step.
  • Search of the minimum pair still requires O(N2)
    time ? still O(N3) in total.
  • It also requires O(N2) memory.

21
Heap structure for fast searchKurita 1991
Pattern Recognition
  • Search reduces O(N2) ? O(logN).
  • In total O(N2 logN)

22
Store nearest neighbor (NN) pointersFränti et
al., 2000 IEEE Trans. Image Processing
Time complexity reduces to O(N 3) ? ? (?N 2)
23
Pseudo code
PNN(X, M) ? C, P FOR i?1 TO N DO pi?i
ci?xi FOR i?1 TO N DO NNi?
FindNearestCluster(i) REPEAT a ?
SmallestMergeCost(NN) b ? NNi MergeClusters(C,
P,NN,a,b,) UpdatePointers(C,NN) UNTIL mM
O(N)
O(N2)
O(N)
O(?N)
http//cs.uef.fi/pages/franti/research/pnn.txt
24
Example with NN pointersVirmajoki 2004
Pairwise Nearest Neighbor Method Revisited
25
ExampleStep 1
26
ExampleStep 2
27
ExampleStep 3
28
ExampleStep 4
29
ExampleFinal
30
Time complexities of the variants
31
Number of neighbors (t)
32
Processing time comparison
With NN pointers
33
AlgorithmLazy-PNN
T. Kaukoranta, P. Fränti and O. Nevalainen,
"Vector quantization by lazy pairwise nearest
neighbor method", Optical Engineering, 38 (11),
1862-1868, November 1999
34
Monotony property of merge cost Kaukoranta et
al., Optical Engineering, 1999
Merge costs values are monotonically increasing
d(Sa, Sb) ? d(Sa, Sc) ? d(Sb, Sc) ? d(Sa, Sc) ?
d(Sab, Sc)
35
Lazy variant of the PNN
  • Store merge costs in heap.
  • Update merge cost value only when it appears at
    top of the heap.
  • Processing time reduces about 35.

Method Ref. Time complexity Additional data structure Space compl.
Trivial PNN 10 O(dN3) - O(N)
Distance matrix 6 O(dN2 N3) Distance matrix O(N2)
Kuritas method 5 O(dN2 N2logN) Dist. matrix heap O(N2)
?-PNN 1 O(d?N2) NN-table O(N)
Lazy-PNN 4 O(d?N2) NN-table O(N)
36
Combining PNN and K-means
K-means
37
AlgorithmIterative shrinking
P. Fränti and O. Virmajoki Iterative shrinking
method for clustering problemsPattern
Recognition, 39 (5), 761-765, May 2006.
38
Agglomerative clustering based on merging
39
Agglomeration based on cluster removalFränti
and Virmajoki, Pattern Recognition, 2006
40
Merge versus removal
41
Pseudo code of iterative shrinking (IS)
42
Cluster removal in practice
Find secondary cluster
Calculate removal cost for every vector
43
Partition updates
44
Complexity analysis
  • Number of vectors per cluster

If we iterate until M1
Adding the processing time per vector
45
AlgorithmPNN with kNN-graph
P. Fränti, O. Virmajoki and V. Hautamäki, "Fast
agglomerative clustering using a k-nearest
neighbor graph". IEEE Trans. on Pattern Analysis
and Machine Intelligence, 28 (11), 1875-1881,
November 2006
46
Agglomerative clustering with kNN graph
47
Example of 2NN graph
48
Example of 4NN graph
49
Graph using double linked lists
50
Merging a and b
51
Effect on calculationsnumber of steps
STAGE Theoretical Theoretical Theoretical Observed Observed Observed
STAGE ?-PNN Single link Double link ?-PNN Single link Double link
Find pair N 1 1 8 357 3 3
Merge N k2 logN k2 ?k logN 8 367 200 305
Remove last N k logN LogN 8 349 102 45
Find neighbors N kN ?k 8 357 41 769 204
Update costs N (1?) ?  ?/k?logN ?  ?/k?logN 48 538 198 187
TOTAL O(?N2) O(kN2) O(?N?logN) 81 970 42 274 746
52
Processing time as function of k(number of
neighbors in graph)
53
Time distortion comparison
?-PNN (229 s) Trivial-PNN (gt9999 s)
Graph-PNN (1)
MSE 5.36
Graph-PNN (2)
  1. Graph created by MSP
  2. Graph created by D-n-C

54
Conclusions
  • Simple to implement, good clustering quality
  • Straightforward algorithm slow O(N3)
  • Fast exact (yet simple) algorithm O(tN2)
  • Beyond this possible
  • O(tNlogN) complexity
  • Complicated graph data structure
  • Compromizes the exactness of the merge

55
Literature
  1. P. Fränti, T. Kaukoranta, D.-F. Shen and
    K.-S. Chang, "Fast and memory efficient
    implementation of the exact PNN", IEEE Trans. on
    Image Processing, 9 (5), 773-777, May 2000.
  2. P. Fränti, O. Virmajoki and V. Hautamäki, "Fast
    agglomerative clustering using a k-nearest
    neighbor graph". IEEE Trans. on Pattern Analysis
    and Machine Intelligence, 28 (11), 1875-1881,
    November 2006.
  3. P. Fränti and O. Virmajoki, "Iterative shrinking
    method for clustering problems", Pattern
    Recognition, 39 (5), 761-765, May 2006.
  4. T. Kaukoranta, P. Fränti and O. Nevalainen,
    "Vector quantization by lazy pairwise nearest
    neighbor method", Optical Engineering, 38 (11),
    1862-1868, November 1999.
  5. T. Kurita, "An efficient agglomerative clustering
    algorithm using a heap", Pattern Recognition 24
    (3) (1991) 205-209.

56
Literature
  1. J. Shanbehzadeh and P.O. Ogunbona, "On the
    computational complexity of the LBG and PNN
    algorithms". IEEE Transactions on Image
    Processing 6 (4), 614-616, April 1997.
  2. O. Virmajoki, P. Fränti and T. Kaukoranta,
    "Practical methods for speeding-up the pairwise
    nearest neighbor method ", Optical Engineering,
    40 (11), 2495-2504, November 2001.
  3. O. Virmajoki and P. Fränti, "Fast pairwise
    nearest neighbor based algorithm for multilevel
    thresholding", Journal of Electronic Imaging, 12
    (4), 648-659, October 2003.
  4. O. Virmajoki, Pairwise Nearest Neighbor Method
    Revisited, PhD thesis, Computer Science,
    University of Joensuu, 2004.
  5. J.H. Ward, Hierarchical grouping to optimize an
    objective function, J. Amer. Statist.Assoc. 58
    (1963) 236-244.
Write a Comment
User Comments (0)
About PowerShow.com