1 / 29

Artificial Neural Networks An Introduction

- S. Bapi Raju
- Dept. of Computer and
- Information Sciences,
- University of Hyderabad

OUTLINE

- Biological Neural Networks
- Applications of Artificial Neural Networks
- Taxonomy of Artificial Neural Networks
- Supervised and Unsupervised Artificial Neural

Networks - Basis function and Activation function
- Learning Rules
- Applications
- OCR, Load Forecasting, Condition Monitoring

Biological Neural Networks

- Study of Neural Networks originates in biological

systems - Human Brain contains over 100 billion neurons,

number of synapses is approximately 1000 times

that - in electronic circuit terms synaptic fan-in

fan-out is 1000, - switching time of a neuron is order of

milliseconds - But on a face recognition problem brain beats

fastest supercomputer in terms of number of

cycles of computation to arrive at answer - Neuronal Structure

- Cell body
- Dendrites for input
- Axon carries output to other dendrites
- Synapse-where they meet
- Activation signal (voltage) travels along axon

Need for ANN

- Standard Von Neumman Computing as existing

presently has some shortcomings. - Following are some desirable characteristics in

ANN - Learning Ability
- Generalization and Adaptation
- Distributed and Parallel representation
- Fault Tolerance
- Low Power requirements
- Performance comes not just from the computational

elements themselves but the manner of networked

interconnectedness of the decision process.

VonNeumann versus BiologicalComputer

ANN Applications

- Pattern Classification
- Speech Recognition, ECG/EEG classification, OCR

ANN Applications

- Clustering/Categorization
- Data mining, data compression

ANN Applications

- Function Approximation
- Noisy arbitrary function needs to be approximated

ANN Applications

- Prediction/Forecasting
- Given a function of time, predict the function

values for future time values, used in weather

prediction and stock market predictions

ANN Applications

- Optimization
- Several scientific and other problems can be

reduced to an optimization problem like the

Traveling Salesman Problem (TSP)

ANN Applications

- Content Based Retrieval
- Given the partial description of an object

retrieve the objects that match this

ANN Applications

- Control
- Model-reference adaptive control, set-point

control - Engine idle-speed control

Characteristics of ANN

- Biologically inspired computational units
- Also called as Connectionist Models or

Connectionist Architectures - Large number of simple processing elements
- Very large number of weighted connections between

elements. Information in the network is encoded

in the weights learned by the connections - Parallel and distributed control
- Connection weights are learned by automatic

training techniques

Artifical Neuron Working Model

- Objective is to create a model of functioning of

biological neuron to aid computation

- All signals at synapses are summed i.e. all the

excitatory and inhibitory influences and

represented by a net value h(.) - If the excitatory influences are dominant, then

the neuron fires, this is modeled by a simple

threshold function ?(.) - Certain inputs are fixed biases
- Output y leads to other neurons

McCulloch Pitts Model

More about the Model

- Activation Functions play a key role
- Simple thresholding (hard limiting)
- Squashing Function (sigmoid)
- Gaussian Function
- Linear Function
- Biases are also learnt

Different Kinds of Network Architectures

Learning Ability

- Mere Architecture is insufficient
- Learning Techniques also need to be formulated
- Learning is a process where connection weights

are adjusted - Learning is done by training from labeled

examples. This is the most powerful and useful

aspect of neural networks in their use as Black

Box classifiers. - Most commonly an input-output relationship is

learnt - Learning Paradigm needs to be specified
- Weight update in learning rules must be specified
- Learning Algorithm specifies step by step

procedure

Learning Theory

- Major Factors
- Learning Capacity This concerns the number of

patterns that can be learnt and the functions and

kinds of decision boundaries that can be formed - Sample Complexity This concerns the number of

the samples needed to learn with generalization.

Overfitting problem is to be avoided - Computational Complexity This concerns the

computation time needed to learn the concepts

embedded in the training samples. Generally the

computational complexity of learning is high.

Learning Issues

Major Learning Rules

- Error Correction Error signal (dy) used to

adjust the weights so that eventually desired

output d is produced

Perceptron Solving AND Problem

Major Learning Rules

- Error Correction in Mutlilayer Feedforward

Network

Geometric interpretation of the role of hidden

units in a 2D input space

Major Learning Rules

- Hebbianweights are adjusted by a factor

proportional to the activities of the neurons

associated

Orientation Selectivity of a Single Hebbian Neuron

Major Learning Rules

- Competitive Learning winner take all

(a) Before Learning (b) After

Learning

Summary of ANN Algorithms

(No Transcript)

Application to OCR System

- The main problem in the Handwritten Letter

recognition is that characters with variation in

thickness shape, rotation and different nature of

strokes need to be recognized as of being in the

different categories for each letter. - Sufficient number of sample training data is

required for each character to train the networks

A Sample set of characters in the NIST Data

OCR Process

OCR Example (continued)

- Two schemes shown at right
- First makes use of the feature extractors
- Second uses the image pixels directly

References

- A. K. Jain, J.Mao, K.Mohiuddin, ANN a Tutorial,

IEEE Computer, 1996 March, pp 31-44 (Figures and

Tables taken from this reference) - B. Yegnanarayana, Artificial Neural Networks,

Prentice Hall of India, 2001. - Y. M. Zurada, Inroduction to Artificial Neural

Systems, Jaico, 1999. - MATLAB neural networks toolbox and manual