Introduction to Artificial Neural Networks

1
Introduction to Artificial Neural Networks

• Andrew L. Nelson
• Visiting Research Faculty
• University of South Florida

2
Overview

• Outline to the left
• Current topic in red
• Introduction
• History and Origins
• Biologically Inspired
• Applications
• Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

3
References

• W. S. McCulloch, W. Pitts, "A logical calculus of
  the ideas immanent in nervous activity,"
  Bulletin of Mathematical Biophysics, vol. 5 pp.
  115-133, 1943.
• J. L. McClelland, D. E. Rumelhart, Parallel
  Distributed Processing Explorations in the
  Microstructure of Cognition, The MIT Press, 1986.
• C. M. Bishop, Neural Networks for Pattern
  Recognition, Oxford University Press, 1995.

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

4
Introduction

• Artificial Neural Networks (ANN)
• Connectionist computation
• Parallel distributed processing
• Computational models
• Biologically Inspired computational models
• Machine Learning
• Artificial intelligence

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

5
History

• McCulloch and Pitts introduced the Perceptron in
  1943.
• Simplified model of a biological neuron
• Fell out of favor in the late 1960's
• (Minsky and Papert)
• Perceptron limitations
• Resurgence in the mid 1980's
• Nonlinear Neuron Functions
• Back-propagation training

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

6
Summary of Applications

• Function approximation
• Pattern recognition
• Signal processing
• Modeling
• Control
• Machine learning

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

7
Biologically Inspired

• Electro-chemical signals
• Threshold output firing

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

8
The Perceptron

• Binary classifier functions
• Threshold activation function

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

9
The Perceptron Threshold Activation Function

• Binary classifier functions
• Threshold activation function

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

10
Linear Activation functions

• Output is scaled sum of inputs

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

11
Nonlinear Activation Functions

• Sigmoid Neuron unit function

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

12
Layered Networks

• .

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

13
SISO Single Hidden Layer Network

• Can represent and single input single output
  functions y f(x)

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

14
Training Data Set

• Adjust weights (w) to learn a given target
  function y f(x)
• Given a set of training data X?Y

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

15
Training Weights Error Back-Propagation (BP)

• Weight update formula

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

16
Error Back-Propagation (BP)

• Training error term e

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

17
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

18
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

19
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

20
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

21
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

22
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

23
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

24
BP Formulation

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

25
Example The XOR problem

• Single hidden layer 3 Sigmoid neurons
• 2 inputs, 1 output

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

x1 x2 y
Example 1 0 0 0
Example 2 0 1 1
Example 3 1 0 1
Example 4 1 1 0
Desired I/O table (XOR)
26
Example The XOR problem

• Training error over epoch

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

27
Example The XOR problem

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

initial_weights 0.0654 0.2017 0.0769
0.1782 0.0243 0.0806 0.0174 0.1270
0.0599 0.1184 0.1335 0.0737
0.1511 final_weights 4.6970 -4.6585
2.0932 5.5168 -5.7073 -3.2338 -0.1886
1.6164 -0.1929 -6.8066 6.8477 -1.6886
4.1531
28
Example The XOR problem

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples

Mapping produced by the trained neural net
x1 x2 y
Example 1 0 0 0.0824
Example 2 0 1 0.9095
Example 3 1 0 0.9470
Example 4 1 1 0.0464
29
Example Overtraining

• Single hidden layer 10 Sigmoid neurons
• 1 input, 1 output

• References
• Introduction
• History
• Biologically Inspired
• Applications
• The Perceptron
• Activation Functions
• Hidden Layer Networks
• Training with BP
• Examples