Artificial Neural Networks

1
Artificial Neural Networks
EECP0720 Expert Systems Artificial Neural
Networks

• Sanun Srisuk 42973003
• sanun_at_mut.ac.th

2
Introduction
EECP0720 Expert Systems Artificial Neural
Networks

• Artificial neural networks (ANNs) provide a
  general, practical method for learning
  real-valued, discrete-valued, and vector-valued
  functions from examples. Algorithms such as
  BACKPROPAGATION use gradient descent to tune
  network parameters to best fit a training set of
  input-output pairs. ANN learning is robust to
  errors in the training data and has been
  successfully applied to problems such as face
  recognition/detection, speech recognition, and
  learning robot control strategies.

3
Autonomous Vehicle Steering
EECP0720 Expert Systems Artificial Neural
Networks
4
Characteristics of ANNs
EECP0720 Expert Systems Artificial Neural
Networks

• Instances are represented by many attribute-value
  pairs.
• The target function output may be
  discrete-valued, real-valued, or a vector of
  several real- or discrete-valued attributes.
• The training examples may contain errors.
• Long training times are acceptable.
• Fast evaluation of the learned target function
  may be required.
• The ability of humans to understand the learned
  target function is not important.

5
Perceptrons
EECP0720 Expert Systems Artificial Neural
Networks

• One type of ANN system is based on a unit called
  a perceptron.
• The perceptron function can sometimes be written
  as
• The space H of candidate hypotheses considered in
  perceptron learning is the set of all possible
  real-valued weight vectors.

6
Representational Power of Perceptrons
EECP0720 Expert Systems Artificial Neural
Networks
7
Decision surface
EECP0720 Expert Systems Artificial Neural
Networks
linear decision surface
nonlinear decision surface
Programming Example of Decision Surface
8
The Perceptron Training Rule
EECP0720 Expert Systems Artificial Neural
Networks

• One way to learn an acceptable weight vector is
  to begin with random weights, then iteratively
  apply the perceptron to each training example,
  modifying the perceptron weights whenever it
  misclassifies an example. This process is
  repeated, iterating through the training examples
  as many times as needed until the perceptron
  classifies all training examples correctly.
  Weights are modified at each step according to
  the perceptron training rule, which revises the
  weight associated with input according to
  the rule

9
Gradient Descent and Delta Rule
EECP0720 Expert Systems Artificial Neural
Networks

• The delta training rule is best understood by
  considering the task of training an unthresholded
  perceptron that is, a linear unit for which the
  output o is given by
• In order to derive a weight learning rule for
  linear units, let us begin by specifying a
  measure for the training error of a hypothesis
  (weight vector), relative to the training
  examples.

10
Visualizing the Hypothesis Space
EECP0720 Expert Systems Artificial Neural
Networks
initial weight vector by random
minimum error
11
Derivation of the Gradient Descent Rule
EECP0720 Expert Systems Artificial Neural
Networks

• The vector derivative is called the gradient of E
  with respect to , written
• The gradient specifies the direction that
  produces the steepest increase in E. The negative
  of this vector therefore gives the direction of
  steepest decrease. The training rule for gradient
  descent is

12
Derivation of the Gradient Descent Rule (cont.)
EECP0720 Expert Systems Artificial Neural
Networks

• The negative sign is presented because we want to
  move the weight vector in the direction that
  decreases E. This training rule can also written
  in its component form
• which makes it clear that steepest descent is
  achieved by altering each component of in
  proportion to .

13
Derivation of the Gradient Descent Rule (cont.)
EECP0720 Expert Systems Artificial Neural
Networks

• The vector of derivatives that form the
  gradient can be obtained by differentiating E

The weight update rule for standard gradient
descent can be summarized as
14
Stochastic Approximation to Gradient Descent
EECP0720 Expert Systems Artificial Neural
Networks
15
Summary of Perceptron
EECP0720 Expert Systems Artificial Neural
Networks

• Perceptron training rule guaranteed to succeed if
  
• training examples are linearly separable
• sufficiently small learning rate
• Linear unit training rule uses gradient descent
• guaranteed to converge to hypothesis with minimum
  squared error
• given sufficiently small learning rate
• even when training data contains noise

16
BACKPROPAGATION Algorithm
EECP0720 Expert Systems Artificial Neural
Networks
17
Error Function
EECP0720 Expert Systems Artificial Neural
Networks

• The Backpropagation algorithm learns the weights
  for a multilayer network, given a network with a
  fixed set of units and interconnections. It
  employs gradient descent to attempt to minimize
  the squared error between the network output
  values and the target values for those outputs.
  We begin by redefining E to sum the errors over
  all of the network output units
• where outputs is the set of output units in the
  network, and tkd and okd are the target and
  output values associated with the kth output unit
  and training example d.

18
Architecture of Backpropagation
EECP0720 Expert Systems Artificial Neural
Networks
19
Backpropagation Learning Algorithm
EECP0720 Expert Systems Artificial Neural
Networks
20
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems Artificial Neural
Networks
21
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems Artificial Neural
Networks
22
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems Artificial Neural
Networks
23
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems Artificial Neural
Networks
24
Face Detection using Neural Networks
EECP0720 Expert Systems Artificial Neural
Networks
Training Process
Face Database
Output1, for face database
Non-Face Database
Neural Network
Face or
Non-Face?
Output0, for non-face database
Testing Process
25
End of Presentation
EECP0720 Expert Systems Artificial Neural
Networks
26
Derivation of Backpropagation
EECP0720 Expert Systems Artificial Neural
Networks
27
Derivation of Backpropagation (cont.)
EECP0720 Expert Systems Artificial Neural
Networks