Ch12. Combining Inductive and Analytical Learning

1
Ch12. Combining Inductive and Analytical Learning

• ?????
• ????????
• ???
• 2000.11.15

2
Contents

• Motivation
• KBANN algorithm
• EBNN algorithm
• FOCL algorithm

3
Motivation (1/2)

• Inductive learning vs Analytical learning

Inductive learning
Analytical learning
Plentiful data No prior knowledge
Perfect prior knowledge Scarce data
4
Motivation (2/2)

• Goal
• Combine two mechanisms to obtain the benefits of
  both approaches
• Properties of combined method
• Given no domain theory, it should learn at least
  as effectively as purely inductive methods
• Given a perfect domain theory, it should learn at
  least as effectively as purely analytical methods
• Given an imperfect domain theory and imperfect
  training data, it should combine the two to
  outperform either purely inductive or purely
  analytical methods
• It should accommodate an unknown level of error
  in the training data
• It should accommodate an unknown level of error
  in the domain theory

5
KBANN (1/6)

• KBANN(Knowledge-Based Artificial Neural Network)
  algorithm
• Given
• A set of training examples
• A domain theory consisting of nonrecursive,
  propositional Horn clauses
• Determine
• An artificial neural network that fits the
  training examples, biased by the domain theory

6
KBANN (2/6)

• KBANN( Domain_Theory, Training_Examples )
• Analytical step Create an initial network
  equivalent to the domain theory
• For each instance attribute create a network
  input
• For each Horn clause in the Domain_Theory, create
  a network unit as follows
• Connect the inputs of this unit to the attributes
  tested by the clause antecedents
• For each non-negated antecedent of the clause,
  assign a weight of W to the corresponding sigmoid
  unit input
• For each negated antecedent of the clause, assign
  a weight of W to the corresponding sigmoid unit
  input
• Set the threshold weight w0 for this unit to
  (n-0.5)W, where n is the number of non-negated
  antecedents of the clause
• Add additional connections among the network
  units, connecting each network unit at depth i
  from the input layer to all network units at
  depth i 1. Assign random near-zero weights to
  these additional connections
• Inductive step Refine the initial network
• Apply the Backpropagation algorithm to adjust the
  initial network weights to fit the
  Training_Examples

7
KBANN (3/6)

• Example (the Cup learning task)
• Domain theory
• Cup ? Stable, Liftable, OpenVessel
• Stable ? BottomIsFlat
• Liftable ? Graspable, Light
• Graspable ? HasHandle
• OpenVessel ? HasConcavity, ConcavityPointsUp
• Training examples

8
KBANN (4/6)

• A neural network equivalent to the domain theory

9
KBANN (5/6)

• Result of inductively refining the initial network

10
KBANN (6/6)

• Hypothesis space search in KBANN

11
EBNN (1/6)

• TangentProp algorithm

12
EBNN (2/6)

• EBNN(Explanation-Based Neural Network) algorithm
• Given
• A set of training examples
• A domain theory represented by a set of
  previously trained neural networks
• Determine
• A new neural network that approximates the target
  function f
• This learned network is trained to fit both the
  training examples and training derivatives of f
  extracted from the domain theory

13
EBNN (3/6)

• EBNN algorithm
• Creates a new, fully connected feedforward
  network to represent the target function
• Initialized with small random weights
• For each training example ltxi,f(xi)gt
• Determines the corresponding training derivatives
  in a two-step process
• First, it uses the domain theory to predict the
  value of the target function for instance xi
  ( A(xi) )
• Second, the weights and activations of the domain
  theory networks are analyzed to extract the
  derivatives of A(xi) with respect to each of the
  components of xi
• Using a minor variant of the TangentProp
  algorithm to train the target network to fit the
  following error function

14
EBNN (4/6)
xi ith training instance A(x) the domain
theory prediction for input x xj jth component
of the vector x The coefficient c is a
normalizing constant whose value is chosen to
assure that for all i, 0 µj 1
15
EBNN (5/6)
16
EBNN (6/6)

• Hypothesis space search in EBNN

17
FOCL (1/2)

• FOCL algorithm

18
FOCL (2/2)

• Recognizing legal chess endgame positions
• 30 positive, 30 negative examples
• FOIL 86
• FOCL ( (using domain theory with 76
  accuracy)
• NYNEX telephone network diagnosis
• 500 training examples
• FOIL 90
• FOCL 98 (using domain theory with 95 accuracy)