1 / 27

Artificial Neural Networks

EECP0720 Expert Systems Artificial Neural

Networks

- Sanun Srisuk 42973003
- sanun_at_mut.ac.th

Introduction

EECP0720 Expert Systems Artificial Neural

Networks

- Artificial neural networks (ANNs) provide a

general, practical method for learning

real-valued, discrete-valued, and vector-valued

functions from examples. Algorithms such as

BACKPROPAGATION use gradient descent to tune

network parameters to best fit a training set of

input-output pairs. ANN learning is robust to

errors in the training data and has been

successfully applied to problems such as face

recognition/detection, speech recognition, and

learning robot control strategies.

Autonomous Vehicle Steering

EECP0720 Expert Systems Artificial Neural

Networks

Characteristics of ANNs

EECP0720 Expert Systems Artificial Neural

Networks

- Instances are represented by many attribute-value

pairs. - The target function output may be

discrete-valued, real-valued, or a vector of

several real- or discrete-valued attributes. - The training examples may contain errors.
- Long training times are acceptable.
- Fast evaluation of the learned target function

may be required. - The ability of humans to understand the learned

target function is not important.

Perceptrons

EECP0720 Expert Systems Artificial Neural

Networks

- One type of ANN system is based on a unit called

a perceptron. - The perceptron function can sometimes be written

as - The space H of candidate hypotheses considered in

perceptron learning is the set of all possible

real-valued weight vectors.

Representational Power of Perceptrons

EECP0720 Expert Systems Artificial Neural

Networks

Decision surface

EECP0720 Expert Systems Artificial Neural

Networks

linear decision surface

nonlinear decision surface

Programming Example of Decision Surface

The Perceptron Training Rule

EECP0720 Expert Systems Artificial Neural

Networks

- One way to learn an acceptable weight vector is

to begin with random weights, then iteratively

apply the perceptron to each training example,

modifying the perceptron weights whenever it

misclassifies an example. This process is

repeated, iterating through the training examples

as many times as needed until the perceptron

classifies all training examples correctly.

Weights are modified at each step according to

the perceptron training rule, which revises the

weight associated with input according to

the rule

Gradient Descent and Delta Rule

EECP0720 Expert Systems Artificial Neural

Networks

- The delta training rule is best understood by

considering the task of training an unthresholded

perceptron that is, a linear unit for which the

output o is given by - In order to derive a weight learning rule for

linear units, let us begin by specifying a

measure for the training error of a hypothesis

(weight vector), relative to the training

examples.

Visualizing the Hypothesis Space

EECP0720 Expert Systems Artificial Neural

Networks

initial weight vector by random

minimum error

Derivation of the Gradient Descent Rule

EECP0720 Expert Systems Artificial Neural

Networks

- The vector derivative is called the gradient of E

with respect to , written - The gradient specifies the direction that

produces the steepest increase in E. The negative

of this vector therefore gives the direction of

steepest decrease. The training rule for gradient

descent is

Derivation of the Gradient Descent Rule (cont.)

EECP0720 Expert Systems Artificial Neural

Networks

- The negative sign is presented because we want to

move the weight vector in the direction that

decreases E. This training rule can also written

in its component form - which makes it clear that steepest descent is

achieved by altering each component of in

proportion to .

Derivation of the Gradient Descent Rule (cont.)

EECP0720 Expert Systems Artificial Neural

Networks

- The vector of derivatives that form the

gradient can be obtained by differentiating E

The weight update rule for standard gradient

descent can be summarized as

Stochastic Approximation to Gradient Descent

EECP0720 Expert Systems Artificial Neural

Networks

Summary of Perceptron

EECP0720 Expert Systems Artificial Neural

Networks

- Perceptron training rule guaranteed to succeed if

- training examples are linearly separable
- sufficiently small learning rate
- Linear unit training rule uses gradient descent
- guaranteed to converge to hypothesis with minimum

squared error - given sufficiently small learning rate
- even when training data contains noise

BACKPROPAGATION Algorithm

EECP0720 Expert Systems Artificial Neural

Networks

Error Function

EECP0720 Expert Systems Artificial Neural

Networks

- The Backpropagation algorithm learns the weights

for a multilayer network, given a network with a

fixed set of units and interconnections. It

employs gradient descent to attempt to minimize

the squared error between the network output

values and the target values for those outputs.

We begin by redefining E to sum the errors over

all of the network output units - where outputs is the set of output units in the

network, and tkd and okd are the target and

output values associated with the kth output unit

and training example d.

Architecture of Backpropagation

EECP0720 Expert Systems Artificial Neural

Networks

Backpropagation Learning Algorithm

EECP0720 Expert Systems Artificial Neural

Networks

Backpropagation Learning Algorithm (cont.)

EECP0720 Expert Systems Artificial Neural

Networks

Backpropagation Learning Algorithm (cont.)

EECP0720 Expert Systems Artificial Neural

Networks

Backpropagation Learning Algorithm (cont.)

EECP0720 Expert Systems Artificial Neural

Networks

Backpropagation Learning Algorithm (cont.)

EECP0720 Expert Systems Artificial Neural

Networks

Face Detection using Neural Networks

EECP0720 Expert Systems Artificial Neural

Networks

Training Process

Face Database

Output1, for face database

Non-Face Database

Neural Network

Face or

Non-Face?

Output0, for non-face database

Testing Process

End of Presentation

EECP0720 Expert Systems Artificial Neural

Networks

Derivation of Backpropagation

EECP0720 Expert Systems Artificial Neural

Networks

Derivation of Backpropagation (cont.)

EECP0720 Expert Systems Artificial Neural

Networks