Pattern Recognition: Statistical and Neural

1
Nanjing University of Science Technology
Pattern RecognitionStatistical and Neural
Lonnie C. Ludeman Lecture 28 Nov 9, 2005
2
Lecture 28 Topics

• Review Clustering Methods
• 2. Agglomerative Hierarchical Clustering Example
• 3. Introduction to Fuzzy Sets
• 4. Fuzzy Partitions
• 5. Define Hard and soft clusters

3
K-Means Clustering Algorithm
Basic Procedure
Randomly Select K cluster centers from Pattern
Space Distribute set of patterns to the cluster
center using minimum distance Compute new
Cluster centers for each cluster Continue this
process until the cluster centers do not change
or a maximum number of iterations is reached.
Review
4
Flow Diagram for K-Means Algorithm
Review
5
Iterative Self Organizing Data Analysis Technique
A
ISODATA Algorithm
Performs Clustering of unclassified quantitative
data with an unknown number of clusters Similar
to K-Means but with ability to merge and split
clusters thus giving flexibility in number of
clusters
Review
6
Hierarchical Clustering Dendrogram
Review
7
Example - Hierarchical Clustering
Given the following data
(a) Perform a Hierarchical Clustering of the
data (b) Give the results for 3 clusters.
8
Solution
Plot of data vectors
9
Calculate distances between each pair of original
data vectors
S5(10)
S7(10) S5(9)
U
Data sample x5 and x7 are the closest together
thus we combine them to give the following for 9
clusters
10
S5(9)
S5(10) U S7(10) S5(9)
Combine Closest Clusters
11
Compute distances between new clusters
Clusters S8(9) and S9(9) are the closest together
thus we combine them to give the following for 8
clusters
12
S8(8)
S5(8)
S8(9) U S9(9) S8(8)
Combine Closest Clusters
13
Compute distances between new clusters
Clusters S5(8) and S8(8) are the closest together
thus we combine them to give the following for 7
clusters
14
S8(8)
S5(8)
S5(7)
S5(8) U S8(8) S5(7)
Combine Closest Clusters
15
Compute distances between new clusters
Clusters S1(7) and S4(7) are the closest together
thus we combine them to give the following for 6
clusters
16
S5(7)
S1(6)
S1(7) U S4(7) S1(6)
Combine Closest Clusters
17
Continuing this process we see the following
combinations of clusters at the given levels
18
Level 5
19
Level 4
20
Level 3
21
Level 2
22
Level 1
23
Dendogram for Given Example
24
(b) Using the dendrogram determine the results
for just three clusters
From the dendrogram at level 3 we see the
following clusters
S5(3) 5, 7, 8, 9, 10 ) S2(3) 2,
6 S1(3) 1,4,3
Answer
Cl1 x5, x7, x8, x9, x10) Cl2 x2, x6
Cl3 x1, x4, x3
25
Introduction to Fuzzy Clustering
K-means, Hierarchical, and ISODATA clustering
algorithms are what we call Hard Clustering.
The assignment of clusters is a partitioning of
the data into mutually disjoint and exhaustive
non empty sets. Fuzzy clustering is a relaxation
of this property and provides another way of
solving the clustering problem. Before we present
the Fuzzy Clustering algorithm we first lay a
background by defining Fuzzy sets.
26
Given a set S composed of pattern vectors as
follows
S x1, x2, ... , xN
A proper subset of S is any nonempty collection
of pattern vectors. Examples follow B
x2, x4, xN C x4 D x1,
x3 , x5, xN-1
A x1, x2
27
Given the following Set
S x1, x2, ... , xk, ... , xN
We can also specify subsets by using the
characteristic function which is defined on the
set S as follows for the subset A
µA(xk) 1 if xk is in the subset A
0 if xk is not in the subset A
Characteristic Function for subset A
µA(.) µA(x1), µA(x2), ... , µA(xk), ... , (xN
)
28
Examples
A x1, x2 µA(xk) 1, 1, 0, 0, 0, ... ,
0
29
Examples
A x1, x2 µA(xk) 1, 1, 0, 0, 0, ... ,
0
B x2, x4, xN
µB(xk) 0, 1, 0, 1, 0, ... , 1
30
Examples
A x1, x2 µA(xk) 1, 1, 0, 0, 0, ... ,
0
B x2, x4, xN
µB(xk) 0, 1, 0, 1, 0, ... , 1
C x4
µC(xk) 0, 0, 0, 1, 0, ... , 0
31
Examples
A x1, x2 µA(xk) 1, 1, 0, 0, 0, ...
, 0
B x2, x4, xN
µB(xk) 0, 1, 0, 1, 0, ... , 1
C x4
µC(xk) 0, 0, 0, 1, 0, ... , 0
D x1, x3 , x5, xN-1
µD(xk) 1, 0, 1, 0, 1,...,1 , 0
32
Partitions of a set S
Given a set S of NS n-dimensional pattern
vectors
S xj j 1, 2, ... , NS
A partition of S is a set of M subsets of S, Sk ,
k1, 2, ... , M, that satisfy the following
conditions.
Note Thus Clusters can be specified as a
partition of the pattern space S.
33
Properties of subsets of a Partition
1. Sk ? F Not empty
2. Sk n Sj ? F Pairwise disjoint
S Exhaustive
3.
where F is the Null Set
34
Partition in terms of Characteristic Functions
µS
1
µS
2
. . .
. . .
. . .
µS
M
µS (xk) 0 or 1
Sum 1 for each column
j
for all k and j
35
Cl2
Cl3
Cl1
x1 x2 x3 x4 x5 x6 x7
Cl1 1 0 1 0 0 0 0 Cl2 0 1 0
0 0 0 0 Cl3 0 0 0 1 1 1 1
Hard Partition
36
A Fuzzy Set can be defined by extending the
concept of the characteristic function to allow
positive values between and including 0 and 1 as
follows
Given a set S x1, x2, ... , xN
A Fuzzy Subset F, of a set S, is defined by its
membership function
F µF(x1), µF(x2), ... , µF(xk), ... , µF(xN
)
where xk is from S and
37
S
Function defined on S
38
Example Define a Fuzzy set A by the following
membership function
Or equivalently
39
A Fuzzy Partition F, of a set S, is defined by
its membership functions for the fuzzy sets Fk
k 1, 2, ... , K
)
Fuzzy Partition
40
where
Each value bounded by 0 and 1 Sum of each columns
values 1 Sum of each row less than n
41
Hard or Crisp Clusters
P2
P1
x5
x1
x3
x4
x6
x2
Set Descriptions
P1 x1 , x2 , x3
P2 x4 , x5 , x6
42
Soft or Fuzzy Clusters
x5
x1
x3
Pattern Vectors
x4
x6
x2
Membership Functions
43
Hard or Crisp Partition
Soft or Fuzzy Partition
44
Membership Function for F1
Domain Pattern Vectors
Membership Function for F2
Domain Pattern Vectors
Membership Functions for Fuzzy Clusters
45
Note Sum of columns 1
46
Fuzzy Kitten
47
Lecture 28 Topics

• Reviewed Clustering Methods
• 2. Gave an Example using Agglomerative
  Hierarchical Clustering
• 3. Introduced Fuzzy Sets
• 4. Described Crisp and Fuzzy Partitions
• 5. Defined Hard and soft clusters

48
End of Lecture 28