CS621: Artificial Intelligence Lecture 27: Backpropagation applied to recognition problems; start of logic

1
CS621 Artificial IntelligenceLecture 27
Backpropagation applied to recognition problems
start of logic

• Pushpak Bhattacharyya
• Computer Science and Engineering Department
• IIT Bombay

2
Backpropagation algorithm
Output layer (m o/p neurons)
.
j
wji
.
i
Hidden layers
.
.
Input layer (n i/p neurons)

• Fully connected feed forward network
• Pure FF network (no jumping of connections over
  layers)

3
General Backpropagation Rule

• General weight updating rule
• Where

for outermost layer
for hidden layers
4
Local Minima

• Due to the Greedy nature of BP, it can get stuck
  in local minimum m and will never be able to
  reach the global minimum g as the error can only
  decrease by weight change.

5
Momentum factor

• Introduce momentum factor.
• Accelerates the movement out of the trough.
• Dampens oscillation inside the trough.
• Choosing ß If ß is large, we may jump over
  the minimum.

6
Symmetry breaking

• If mapping demands different weights, but we
  start with the same weights everywhere, then BP
  will never converge.

XOR n/w if we s started with identical weight
everywhere, BP will not converge
7
Backpropagation Applications
8
Feed Forward Network Architecture
Problem defined
O/P layer
Decided by trial error
Hidden layer
Problem defined
I/P layer
9
Digit Recognition Problem

• Digit recognition
• 7 segment display
• Segment being on/off defines a digit

2
1
3
7
6
4
5
10
9O 8O 7O . . . 2O 1O
Full connection
Hidden layer
Full connection
7O 6O 5O . . . 2O
1O Seg-7 Seg-6 Seg-5
Seg-2 Seg-1
11
Example - Character Recognition

• Output layer 26 neurons (all capital)
• First output neuron has the responsibility of
  detecting all forms of A
• Centralized representation of outputs
• In distributed representations, all output
  neurons participate in output

12
An application in Medical Domain
13
Expert System for Skin Diseases Diagnosis

• Bumpiness and scaliness of skin
• Mostly for symptom gathering and for developing
  diagnosis skills
• Not replacing doctors diagnosis

14
Architecture of the FF NN

• 96-20-10
• 96 input neurons, 20 hidden layer neurons, 10
  output neurons
• Inputs skin disease symptoms and their
  parameters
• Location, distribution, shape, arrangement,
  pattern, number of lesions, presence of an active
  norder, amount of scale, elevation of papuls,
  color, altered pigmentation, itching, pustules,
  lymphadenopathy, palmer thickening, results of
  microscopic examination, presence of herald
  pathc, result of dermatology test called KOH

15
Output

• 10 neurons indicative of the diseases
• psoriasis, pityriasis rubra pilaris, lichen
  planus, pityriasis rosea, tinea versicolor,
  dermatophytosis, cutaneous T-cell lymphoma,
  secondery syphilis, chronic contact dermatitis,
  soberrheic dermatitis

16
Training data

• Input specs of 10 model diseases from 250
  patients
• 0.5 is some specific symptom value is not knoiwn
• Trained using standard error backpropagation
  algorithm

17
Testing

• Previously unused symptom and disease data of 99
  patients
• Result
• Correct diagnosis achieved for 70 of
  papulosquamous group skin diseases
• Success rate above 80 for the remaining diseases
  except for psoriasis
• psoriasis diagnosed correctly only in 30 of the
  cases
• Psoriasis resembles other diseases within the
  papulosquamous group of diseases, and is somewhat
  difficult even for specialists to recognise.

18
Explanation capability

• Rule based systems reveal the explicit path of
  reasoning through the textual statements
• Connectionist expert systems reach conclusions
  through complex, non linear and simultaneous
  interaction of many units
• Analysing the effect of a single input or a
  single group of inputs would be difficult and
  would yield incor6rect results

19
Explanation contd.

• The hidden layer re-represents the data
• Outputs of hidden neurons are neither symtoms nor
  decisions

20
(No Transcript)
21
Discussion

• Symptoms and parameters contributing to the
  diagnosis found from the n/w
• Standard deviation, mean and other tests of
  significance used to arrive at the importance of
  contributing parameters
• The n/w acts as apprentice to the expert

22
Exercise

• Find the weakest condition for symmetry breaking.
  It is not the case that only when ALL weights are
  equal, the network faces the symmetry problem.

23
Logic
24
Logic and inferencing
Vision
NLP

• Search
• Reasoning
• Learning
• Knowledge

Expert Systems
Robotics
Planning
Obtaining implication of given facts and rules --
Hallmark of intelligence
25

• Inferencing through
• Deduction (General to specific)
• Induction (Specific to General)
• Abduction (Conclusion to hypothesis in absence of
  any other evidence to contrary)

Deduction Given All men are mortal
(rule) Shakespeare is a man (fact) To
prove Shakespeare is mortal (inference)
Induction Given Shakespeare is mortal
Newton is mortal (Observation) Dijkst
ra is mortal To prove All men are mortal
(Generalization)
26
If there is rain, then there will be no
picnic Fact1 There was rain Conclude There was
no picnic
Deduction
Fact2 There was no picnic Conclude There was no
rain (?)
Induction and abduction are fallible forms of
reasoning. Their conclusions are susceptible to
retraction
Two systems of logic 1) Propositional
calculus 2) Predicate calculus
27

• Propositions
• Stand for facts/assertions
• Declarative statements
• As opposed to interrogative statements
  (questions) or imperative statements (request,
  order)
• Operators
• gt and form a minimal set (can express other
  operations)
• - Prove it.
• Tautologies are formulae whose truth value is
  always T, whatever the assignment is

28

• Model
• In propositional calculus any formula with n
  propositions has 2n models (assignments)
• - Tautologies evaluate to T in all models.
• Examples
• 1)
• 2)
• e Morgan with AND

29
Semantic Tree/Tableau method of proving tautology
Start with the negation of the formula
- a - formula
a-formula
ß-formula
- ß - formula
a-formula
- a - formula
30
Example 2
X
(a - formula)
(a - formulae)
a-formula
(ß - formulae)
B
C
B
C
Contradictions in all paths
31
A puzzle(Zohar Manna, Mathematical Theory of
Computation, 1974)

• From Propositional Calculus

32
Tourist in a country of truth-sayers and liers

• Facts and Rules In a certain country, people
  either always speak the truth or always lie. A
  tourist T comes to a junction in the country and
  finds an inhabitant S of the country standing
  there. One of the roads at the junction leads to
  the capital of the country and the other does
  not. S can be asked only yes/no questions.
• Question What single yes/no question can T ask
  of S, so that the direction of the capital is
  revealed?

33
Diagrammatic representation
Capital
S (either always says the truth Or always lies)
T (tourist)
34
Deciding the Propositions a very difficult step-
needs human intelligence

• P Left road leads to capital
• Q S always speaks the truth

35
Meta Question What question should the tourist
ask

• The form of the question
• Very difficult needs human intelligence
• The tourist should ask
• Is R true?
• The answer is yes if and only if the left road
  leads to the capital
• The structure of R to be found as a function of P
  and Q

36
A more mechanical part use of truth table
P Q Ss Answer R
T T Yes T
T F Yes F
F T No F
F F No T
37
Get form of R quite mechanical

• From the truth table
• R is of the form (P x-nor Q) or (P Q)

38
Get R in English/Hindi/Hebrew

• Natural Language Generation non-trivial
• The question the tourist will ask is
• Is it true that the left road leads to the
  capital if and only if you speak the truth?
• Exercise A more well known form of this question
  asked by the tourist uses the X-OR operator
  instead of the X-Nor. What changes do you have to
  incorporate to the solution, to get that answer?