Artificial%20Intelligence%20CSC%20361

1
Artificial Intelligence CSC 361
Dr. Yousef Al-Ohali Computer Science
Depart. CCIS King Saud University Saudi
Arabia yousef_at_ccis.edu.sa http//faculty.ksu.edu.
sa/YAlohali
2
Intelligent SystemsPart II Neural Nets
3
Developing Intelligent Program Systems

• Machine Learning Neural Nets
• Artificial Neural Networks Artificial Neural
  Networks are crude attempts to model the highly
  massive parallel and distributed processing we
  believe takes place in the brain.
• Two main areas of activity
• Biological Try to model biological neural
  systems.
• Computational develop powerful applications.

4
Developing Intelligent Program Systems

• Machine Learning Neural Nets
• Neural nets can be used to answer the following
• Pattern recognition Does that image contain a
  face?
• Classification problems Is this cell defective?
• Prediction Given these symptoms, the patient has
  disease X
• Forecasting predicting behavior of stock market
• Handwriting is character recognized?
• Optimization Find the shortest path for the TSP.

5
Developing Intelligent Program Systems

• Machine Learning Neural Nets
• Strength and Weaknesses of ANN
• Examples may be described by a large number of
  attributes (e.g., pixels in an image).
• Data may contain errors.
• The time for training may be extremely long.
• Evaluating the network for a new example is
  relatively fast.
• Interpretability of the final hypothesis is not
  relevant (the NN is treated as a black box).

6
Artificial Neural Networks

• Biological Neuron

7
The Neuron

• The neuron receives nerve impulses through its
  dendrites. It then sends the nerve impulses
  through its axon to the terminal buttons where
  neurotransmitters are released to simulate other
  neurons.

8
The neuron

• The unique components are
• Cell body or soma which contains the nucleus
• The dendrites
• The axon
• The synapses

9
The neuron - dendrites

• The dendrites are short fibers (surrounding the
  cell body) that receive messages
• The dendrites are very receptive to connections
  from other neurons.
• The dendrites carry signals from the synapses to
  the soma.

10
The neuron - axon

• The axon is a long extension from the soma that
  transmits messages
• Each neuron has only one axon.
• The axon carries action potentials from the soma
  to the synapses.

11
The neuron - synapses

• The synapses are the connections made by an axon
  to another neuron. They are tiny gaps between
  axons and dendrites (with chemical bridges) that
  transmit messages
• A synapse is called excitatory if it raises the
  local membrane potential of the post synaptic
  cell.
• Inhibitory if the potential is lowered.

12
Artificial Neural Networks

• History of ANNs

13
History of Artificial Neural Networks

• 1943 McCulloch and Pitts proposed a model of a
  neuron --gt Perceptron
• 1960s Widrow and Hoff explored Perceptron
  networks (which they called Adalines) and the
  delta rule.
• 1962 Rosenblatt proved the convergence of the
  perceptron training rule.
• 1969 Minsky and Papert showed that the
  Perceptron cannot deal with nonlinearly-separable
  data sets---even those that represent simple
  function such as X-OR.
• 1970-1985 Very little research on Neural Nets
• 1986 Invention of Backpropagation Rumelhart and
  McClelland, but also Parker and earlier on
  Werbos which can learn from nonlinearly-separable
  data sets.
• Since 1985 A lot of research in Neural Nets

14
Artificial Neural Networks

• artificial Neurons

15
Artificial Neuron

• Incoming signals to a unit are combined by
  summing their weighted values
• Output function Activation functions include
  Step function, Linear function, Sigmoid function,
  

16
Activation functions
Linear function
Sign function
Sigmoid (logistic) function
Step function
sign(x) 1, if x gt 0 -1, if x
lt 0
step(x) 1, if x gt threshold 0,
if x lt threshold (in picture above, threshold 0)
pl(x) x
sigmoid(x) 1/(1e-x)
Adding an extra input with activation a0 -1 and
weight W0,j t (called the bias weight) is
equivalent to having a threshold at t. This way
we can always assume a 0 threshold.
17
Real vs. Artificial Neurons
18
Neurons as Universal computing machine

• In 1943, McCulloch and Pitts showed that a
  synchronous assembly of such neurons is a
  universal computing machine. That is, any Boolean
  function can be implemented with threshold (step
  function) units.

19
Implementing AND
20
Implementing OR
21
Implementing NOT
22
Implementing more complex Boolean functions
23
Artificial Neural Networks

• When using ANN, we have to define
• Artificial Neuron Model
• ANN Architecture
• Learning mode

24
Artificial Neural Networks

• ANN Architecture

25
ANN Architecture

• Feedforward Links are unidirectional, and there
  are no cycles, i.e., the network is a directed
  acyclic graph (DAG). Units are arranged in
  layers, and each unit is linked only to units in
  the next layer. There is no internal state other
  than the weights.
• Recurrent Links can form arbitrary topologies,
  which can implement memory. Behavior can become
  unstable, oscillatory, or chaotic.

26
Artificial Neural NetworkFeedforward Network
27
Artificial Neural NetworkFeedForward Architecture

• Information flow unidirectional
• Multi-Layer Perceptron (MLP)
• Radial Basis Function (RBF)
• Kohonen Self-Organising Map (SOM)

28
Artificial Neural NetworkRecurrent Architecture

• Feedback connections
• Hopfield Neural Networks Associative memory
• Adaptive Resonance Theory (ART)

29
Artificial Neural NetworkLearning paradigms

• Supervised learning
• Teacher presents ANN input-output pairs,
• ANN weights adjusted according to error
• Classification
• Control
• Function approximation
• Associative memory
• Unsupervised learning
• no teacher
• Clustering

30
ANN capabilities

• Learning
• Approximate reasoning
• Generalisation capability
• Noise filtering
• Parallel processing
• Distributed knowledge base
• Fault tolerance

31
Main Problems with ANN

• Contrary to Expert sytems, with ANN the Knowledge
  base is not transparent (black box)
• Learning sometimes difficult/slow
• Limited storage capability

32
Some applications of ANNs

• Pronunciation NETtalk program (Sejnowski
  Rosenberg 1987) is a neural network that learns
  to pronounce written text maps characters
  strings into phonemes (basic sound elements) for
  learning speech from text
• Speech recognition
• Handwritten character recognitiona network
  designed to read zip codes on hand-addressed
  envelops
• ALVINN (Pomerleau) is a neural network used to
  control vehicles steering direction so as to
  follow road by staying in the middle of its lane
• Face recognition
• Backgammon learning program
• Forecasting e.g., predicting behavior of stock
  market

33
When to use ANNs?

• Input is high-dimensional discrete or real-valued
  (e.g. raw sensor input).
• Inputs can be highly correlated or independent.
• Output is discrete or real valued
• Output is a vector of values
• Possibly noisy data. Data may contain errors
• Form of target function is unknown
• Long training time are acceptable
• Fast evaluation of target function is required
• Human readability of learned target function is
  unimportant
• ? ANN is much like a black-box