Artificial Neural Networks 0909.560.01/0909.454.01 Fall 2004

1
Artificial Neural Networks0909.560.01/0909.454.01
Fall 2004
Lecture 7October 25, 2004

• Shreekanth Mandayam
• ECE Department
• Rowan University
• http//engineering.rowan.edu/shreek/fall04/ann/

2
Plan

• RBF Design Issues
• K-means clustering algorithm
• Adaptive techniques
• ANN Design Issues
• Input data processing
• Selection of training and test data -
  cross-validation
• Pre-processing Feature Extraction
• Approximation Theory
• Universal approximation
• Lab Project 3

3
RBF Network
Hidden Layer
Input Layer
Output Layer
x1
y1
Outputs
x2
Inputs
y2
x3
wij
1
j(t)
t
4
RBF - Center Selection
5
K-means Clustering Algorithm

• N data points, xi i 1, 2, , N
• At time-index, n, define K clusters with cluster
  centers cj(n) j 1, 2,
  , K
• Initialization At n0, let cj(n) xj j 1, 2,
  , K (i.e.
  choose the first K data points as cluster
  centers)
• Compute the Euclidean distance of each data point
  from the cluster center, d(xj , cj(n)) dij
• Assign xj to cluster cj(n) if dij mini,j
  dij
  i 1, 2, , N, j 1, 2, , K
• For each cluster j 1, 2, , K, update the
  cluster center cj(n1) mean xj
  ? cj(n)
• Repeat until cj(n1) - cj(n) lt e

6
Selection of Training and Test Data Method of
Cross-Validation
Trial 1 Trial 2 Trial 3 Trial 4

• Vary network parameters until total mean squared
  error is minimum for all trials
• Find network with the least mean squared output
  error

7
Feature Extraction

• Objective
• Increase information content
• Decrease vector length
• Parametric invariance
• Invariance by structure
• Invariance by training
• Invariance by transformation

8
Approximation TheoryDistance Measures

• Supremum Norm
• Infimum Norm
• Mean Squared Norm

9
Recall Metric Space

• Reflexivity
• Positivity
• Symmetry
• Triangle Inequality

10
Approximation Theory Terminology

• Compactness
• Closure

F
11
Approximation Theory Terminology

• Best Approximation
• Existence Set

E
ALL f
M
min
u0
12
Approximation Theory Terminology

• Denseness

13
Fundamental Problem
E
M
min
g
?

• Theorem 1 Every compact set is an existence set
  (Cheney)
• Theorem 2 Every existence set is a closed set
  (Braess)

14
Stone-Weierstrass Theorem

• Identity
• Separability
• Algebraic Closure

F
afbg
15
Lab Project 3 Radial Basis Function Neural
Networks

• http//engineering.rowan.edu/shreek/fall04/ann/la
  b3.html

16
Summary