Validating RuleBased Systems A Complete Methodology

1
The Rule Retranslation Problem and the
Validation Interface
Rainer Knauf Technical University of
Ilmenau School of Computer Science
Automation Ilmenau, Germany
Hans-Werner Kelbassa Paderborn University School
of Computer Science Paderborn, Germany
The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


2
Content

• Introduction
• Single Case Analysis Systems
• Multiple Case Analysis Systems
• Approaches to Solve the Retranslation Problem
• GINSBERGs Approach
• KNAUFs Approach
• Validation of Rule Retranslations
• The Proposed Two Stage Validation Process
• Summary Conclusion

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


3
1 Introduction

• Whats the objective of validation ?
• At a first view, a reliable statement on the
  usefulness of an intelligent system.
• But finally, we are interested in developing a
  better system with higher performance

• Whats should validation of rule-based systems
  aim at ?
• At a first view, a valid input/output behavior.
• But honestly, at a valid knowledge base at all.
• Even with a correct entire behavior the rule
  based approach (as any AI approach) looses its
  intension, if the rules dont reflect the
  causalities of the domain.
• This, of course, includes the correctness of
  rules that infer intermediate conclusions.

• Validation results usually reveal invalid
  input/output behavior.
• Thus, refining or a rule base by utilizing
  validation results is a hard problem.
  Its complexity increases with the number
  intermediate inference levels.
• We refer to the problem of translating validation
  results back to the rule language as the
  retranslation problem.
• Unfortunately, there is no satisfactory standard
  that solves that problem.

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


4

• First systems that perform rule refinement (and
  more or less retranslation)
• Historically, rule bases have been fixed with
  respect to a particular test case that indicates
  an invalidity single case analysis systems
• The insight, that the impact of any refinement on
  other cases has to be considered as well, led to
  multiple case analysis systems.

1.1 Single Case Analysis Systems

• TEIRESIAS (1982)
• guides the expert through the rule trace that
  inferred an invalid conclusion
• the expert has the opportunity to fix these
  faulty rules with respect to the considered case
• does not check any undesired side effects on
  other cases
• does not provide any performance statistics for
  the rules
• a percentage of cases a considered rule
  contributed to valid / invalid results, e.g.

• In the 80th, more sophisticated rule
  retranslation editors MORE, MOLE SALT, KNACK
• MORE, e.g., considers each pair of outputs and
  inspects a rule base for a particular
  differentiating input that is able to distinguish
  these two outputs
• Unfortunately, it has never been proven, that
  these systems control undesired side effects

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


5

1.2 Multiple Case Analysis Systems

• SEEK2 (1988)
• provides statistical performance information
  about the success rate of a considered rule with
  respect to the entire test case set
• this statistics ( meta-knowledge) is provided by
  heuristics, which support a decision on whether a
  rule should be generalized or specialized
• executable in two different modes
• automatic mode
• producing a refined rule base with a better
  performance automatically
• interactive mode
• producing several refinement suggestions that can
  be accepted / rejected by the validating expert
• unfortunately, SEEK2
• ? has an incomplete refinement dichotomy
• There is a third refinement class (besides
  generalization and specialization) called context
  refinement (cf. KELBASSA 2002)
• ? does not validate the all intermediate
  hypotheses within the reasoning chain from the
  input to the output

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


6

• RTLS (Reduced Theory Learning System) (1988)
• refines a converted (reduced) version of the
  rules that model their input/output behavior
  only, not the intermediate conclusions
• because of the reduction, there is no accurate
  retranslation into the rule language
• only suited for medium scale rule bases
  
  some modularization is
  proposed for larger ones by the developers of
  RTLS
• it is doubtful, whether the new rules that are
  introduced as a result of the retranslation
  process are acceptable from a semantic point of
  view

• KRUSTTOOL / KRUSTWORKS (2000)
• faces the problem in a three stage process
• identifying the invalid rules
• develop many alternative refinements
• select a suitable refinement by a hill-climbing
  procedure

• STALKER (1999)
• functionally similar as KRUST
• uses a Truth Maintenance System (TMS) to speed up
  the refinement process and has therefore a better
  performance than KRUST

KNAUFs Approach (2000) ...
... is also a one of this class and introduced
later
The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


7
2 Approaches to Solve the Retranslation Problem
2.1 GINSBERGs Approach

• the retranslation is based on the (meta-)
  knowledge about the inference structure
• special attention is directed to
• eigen-terms, i.e. hypotheses that support exactly
  one final conclusion,
• rule correlated theoretical terms
• theory correlated observables, i.e. inputs
• in case the same observables have different final
  conclusions, new rules are introduced, which use
  other observables and distinguish these
  conclusions
• unfortunately,
• ? there is no final validation of the obtained
  reasoning path regarding all intermediate
  conclusions
• ? this approach cant guarantee that all
  conclusions (i.e. also the intermediate ones) are
  semantically valid with respect to the technical
  and/or scientific theory behind

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


8
2.2 KNAUFs Approach

• is part of a novel entire validation framework
• produces different validity assessments according
  to their purpose (1) validities associated with
  test cases, (2) validities associated with
  outputs, (3) validities associated with rules,
  and (4) a validity of the entire system.
• the final step of this framework uses these
  assessments to restructure the rule base in a
  way, that is maps each examined test data to
  exactly the solution which obtained the maximum
  approval by the expert panel
• if some human expert obtained a solution that
  received a better rating, the rule base maps this
  test data to this so called optimal solution
  after refinement
• in a first step, KNAUF produces so called
  one-shot rules that infer the optimal solution
  directly from inputs that describe the
  corresponding (sub-) set of test data
• these rules are retranslated by just using the
  pre-defined rules, i.e. the inference structure
  of the unrefined rule base, which bags several
  disadvantages
• ? the intermediates itself are not validated and
• ? although the retranslated rule base performs a
  valid mapping from inputs to outputs, the
  produced reasoning paths still might be as
  invalid as before the retranslation

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


9
3 Validation of Rule Retranslations
To face the problem of rule retranslation, we
propose an interactive validation interface,
which performs the following functions

• approval or rejection of all final outputs
• evaluation of all conclusions, i.e. intermediate
  and final ones
• acquisition of validation knowledge from human
  domain experts by validation technologies like
  the one of KNAUF
• acquisition of reasoning faults and target rule
  trace knowledge as proposed by KELBASSA
• acceptance or rejection of rule revisions
  suggested by validation technologies
• (cf. next presentation)
• separate the identification of invalidities from
  the final determination of the effective
  refinements to enable a global optimization
• acceptance or rejection of rule refinements,
  which are suggested by a rule retranslation system

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


10
How should an interactive validation interface be
organized ?

• The validator chooses a particular final
  solution, which he/she wants to inspect a
  reasoning path for.
• This is managed by presenting a list or some
  domain related taxonomy of solutions.
• For the selected solution, the validator chooses
  a particular test case from a list.
• For each case test data , solution the
  complete rule trace has to be presented in a
  multi level reasoning evaluation mode.
• We propose to group this into (at least) three
  classes
• rules, which infer from inputs
• rules, which express knowledge at the
  intermediate conclusion level only
• rules, which infer outputs
• The validator should be able to mark conclusions
  that are wrong in his/her opinion.
• The validator should be able to zoom into the
  rules with invalid conclusions, i.e. to jump into
  a single rule validation mode to modify a
  particular rule.
• When the user jumps back to the multi level
  reasoning evaluation mode, he/she must see the
  impacts of his/her modification(s) to other cases
  with different inputs and/or outputs by listing
  the corresponding rules in their appropriate
  group, i.e. (a) input, (b) intermediate, or (c)
  output.

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


11
The multi level reasoning evaluation mode
input level
input i1 ? hypothesis h1

?
input i7 ? hypothesis h12

?
input i9 ? hypothesis h79

?
inter- mediatecon- clusion level
? hypothesis h80

?
? hypothesis h94

?
? hypothesis h121

?
? hypothesis h122

?
? hypothesis h164

?
? hypothesis h240

?
output level
? final conclusion o1

?
? final conclusion o17

?
? final conclusion o188

?
The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


12
The single rule validation mode
intermediateconclusion rule
if A and not B then Z
?
?
?
?
refined to
?
intermediateconclusion rule
if A or not B then Z
?
?
?
?
?
The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


13
4 The Proposed Two Stage Validation Process
The 1st approach delivers exactly what the 2nd
one requires. Thus, we composed both towards a
generic Two Stage Validation Process.

• KNAUFs approach provides a list of test data
  along with
• their optimal (best rated) solution
• a list of validators, who support this solution
• a refined (relaxed retranslated) knowledge base
  KB
• several kinds of validity statements
• a qualified Validation Knowledge Base VKB

• KELBASSAs approach requires
• a initial rule base (KB , for example)
• a list of test data with their optimal solutions
  and validator protocols
• a rule trace for each case
• a (list of) most competent validator(s) for each
  case
• and provides an optimal rule refinement KB .

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


14
knowledge base KB
validation criteria
VKB
Validation Refinement System I input/output
Validation Refinement cf. KNAUF
validity statements
VKB

• test cases
• test data
• best solution
• most competent validator(s)

relaxed retranslated knowledge base KB
revised rule traces
Validation Refinement System II Rule Trace
Validation Optimal Refinement cf. KELBASSA
optimally refined knowledge base KB
The two stage Validation process for Rule Bases
The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University


15
5 Summary Conclusion

• Thus, the retranslation problem can be solved in
  an optimal way by composing these approaches
  towards a Two Stage Validation Process as
  introduced here.
• The transfer of the introduced technology is
  subject of our collaboration with a commercial
  Partner. Further theoretical research is subject
  of an upcoming project.

• The validation framework of KNAUF aims at the
  assessment and improvement of the input-output
  behavior of the knowledge in a rule base.
• A little (formal, not topical) retranslation is
  performed by utilizing the intermediate
  conclusions that are part of the knowledge base
  before retranslation.
• Thus, albeit the I/O behavior gains validity by
  applying this framework, the intermediates are
  not validated and might still be wrong.
• Additionally, this approach provides
• a best solution, which gained the maximal
  approval by the experts and
• an (list of) expert(s), who supports this
  solution.

• In opposition to KNAUF, KELBASSAs approach aims
  at validating inference paths
• by selecting optimal refinements and
• including topical knowledge for fine tuning
  through a validation interface.

The Rule Retranslation Problem and the Validation
Interface Rainer Knauf, Chair of Artificial
Intelligence

School of Computer Science Automation,
Ilmenau Technical University