Pattern Recognition: Statistical and Neural

1
Nanjing University of Science Technology
Pattern RecognitionStatistical and Neural
Lonnie C. Ludeman Lecture 24 Nov 2, 2005
2
Lecture 24 Topics

• Review and Motivation for Link Structure
• Present the Functional Link Artificial Neural
  Network.
• Simple Example- design using ANN and FLANN
• Performance for Neural Network Designs
• 5. Radial Basis Function Neural Networks
• 6. Problems, Advantages, Disadvantages, and
  promise of Artificial Neural Network Design

3
Generalized Linear Discriminant Functions
w1
x
w2
x

wj
x

g(x)
x

wM
x
Review 1
4
Patterns are linearly separated in the 3-dim
space
Separating plane
Review 2
5
Example Decision rule using one nonlinear
discriminant function g(x)
Given the following g(x) and decision rule
Illustrate the decision regions R1 and R2 where
we respectively classify as C1 and C2 for the
decision rule above
Review 3
6
Solution
R1 decide C1 R2 decide C2
every where else
every where else
Review 4
7
Find a generalized linear discriminant function
that separates the classes
Solution
d(x) w1f1(x) w2f2(x) w3f3(x)
w4f4(x) w5f5(x) w6f6(x)
wT f (x)
in the f space (linear)
Review 5
8
where
in the original pattern space (nonlinear)
Review 6
9
Decision Boundary in original pattern space
x2
from C1
2
from C2
1
3
4
x1
1
2
-1
Boundary
d(x) 0
-2
Review 7
10
Potential Function Approach Motivated by
electromagnetic theory
from C1
- from C2
Sample space
Review 8
11
Plot of Samples from the two classes
Review 9
12
Given Samples x from two classes C1 and C2
S1 S2
C1
C2
Define Total Potential Function
K(x) ? K(x, xk) - ? K(x, xk)
xk S1
xk S2
Potential Function
Decision Boundary
K(x) 0
Review 10
13
Algorithm converged in 1.75 passes through the
data to give final discriminant function as
Review 11
14
Functional Link Neural Network
15
Quadratic Functional Link
16
Fourier Series Functional Link
17
Principal Component Functional Link
fk(x), k1 to N are chosen as the eigen vectors
of the sample covariance matrix
18
Example Comparison of Neural Net and functional
link neural net
Given two pattern classes C1 and C2 with the
following four patterns and their desired outputs
19

1. Design an Artificial Neural Network to classify
   the two patterns given
2. Design a Functional Link Artificial Neural
   Network to classify the patterns given.
3. Compare the Neural Net and Functional Link
   Neural Net designs

20
(a) Solution Artificial Neural Net Design
Select the following structure
21
After training using the training set and the
backpropagation algorithm the design becomes
Values determined by neural net
22
(b) Solution Functional Link Artificial Neural
Net Design
23
A neural net was trained using the functional
link output patterns as new pattern samples
The resulting weights and structure are
24
(c) Comparison Artificial Neural Net (ANN) and
Functional Link Artificial Neural Net (FLANN
Designs
FLANN has simpler structure than the ANN with
only one neural element and Link. Fewer
iterations and computations in the training
algorithm for FLANN. FLANN design may be more
sensitive to errors in patterns.
25
Determining Performance of Neural Net Design
on Training Set
26
Determine Performance for Design using Training
Set
Classify each member of the training set using
the neural network design.
Classify each member of the testing set using
the neural network design.
27
Could use (a) Performance
Measure ETOT (b) The Confusion Matrix (c)
Probability of Error (d) Bayes Risk
28
(a) Local and global errors- Used in Neural Net
Design procedure
Local Measure
Global Measure
29
(b) Confusion Matrix- Example
Correct Classification
Incorrect Classification
30
(c) Probability of Error- Example Estimates of
Probabilities of being Correct
Estimate of Total Probability of Error
31
(d) Bayes Risk Estimate
32
Radial Basis Function (RBF) Artificial Neural
Network
Functional Link
33
Functional Form of RBF ANN
where
Examples of Nonlinearities
34
Design Using RBF ANN
Let F(x1, x2, , xn) represent the function
we wish to approximate. For pattern
classification F(x) represents the class
assignment or desired output (target value) for
each pattern vector x a member of the training set
Define the performance measure E by
E
We wish to Minimize E by selecting M, a ,b1,
b2, ... , bM and z1, z2, ... zM
35
Finding the Best Approximation using RBF ANN
Usually broken into two parts
(1st ) Find the number M of prototypes and the
prototypes zj j1, 2, ... , M by using a
clustering algorithm(Presented in Chapter 6) on
the training samples
(2nd ) With these fixed M and zj j1,2, ... ,
M find the a ,b1, b2, ... , bM that minimize E.
Notes You can use any minimization procedure
you wish. Training does not use the
Backpropagation Algorithm
36
Problems Using Neural Network Designs
Failure to converge
Selection of insufficient structure
Max iterations too small Lockup occurs
Limit cycles
Good performance on training set poor
performance on testing set
Training set not representative of variation Too
strict of a tolerance - grandmothering
37
Advantages of Neural Network Designs
Can obtain a design for very complicated
problems.
Parallel structure using identical elements
allows hardware or software implementation
Structure of Neural Network Design similar for
all problems.
38
Other problems that can be solved using Neural
Network Designs
System Identification Functional
Approximation Control Systems
Any problem that can be placed in the format of a
clearly defined desired output for different
given input vectors.
39
Famous Quotation
Neural network designs are the second best way
to solve all problems
40
Famous Quotation
Neural network designs are the second best way
to solve all problems
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
41
Famous Quotation
Neural network designs are the second best way
to solve all problems
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
The promise is that a Neural Network can be used
to solve all problems however, with the caveat
that there is always a better way to solve a
specific problem.
42
So what is the best way to solve a given problem
???
43
So what is the best way to solve a given problem
???
?
44
So what is the best way to solve a given problem
???
?
A design that uses and understands the structure
of the data !!!
45
Summary Lecture 24

• Reviewed and Motivated Link Structure
• Presented the Functional Link Artificial Neural
  Network.
• Presented Simple Example with designs using ANN
  and FLANN
• Described Performance Measures for Neural
  Network Designs
• 5. Presented Radial Basis Function Neural
  Networks

46
6. Discussed Problems, Advantages, Disadvantages,
and the Promise of Artificial Neural Network
Design
47
End of Lecture 24