Validating RuleBased Systems A Complete Methodology

1
The Power of Experience On the Usefulness of
Validation Knowledge
Rainer Knauf Technical University of
Ilmenau School of Computer Science and
Automation Ilmenau, Germany
Setsuo Tsuruta Kenichi Uehara Tokyo Denki
University School of Information
Environment Tokyo, Japan
Avelino J.Gonzalez University of Central
Florida Dept. of Electrical and Computer
Engineering Orlando, FL, USA
Torsten Kurbad Knowledge Media Research Center
Tübingen, Germany
2
Content

• Motivation
• Validation Knowledge Bases (VKB)
• The Objective of a VKB
• The Content of a VKB
• The Involvement of a VKB in the TURING Test
• The Utilization of a VKB for other Purposes
• Competence estimation of experts
• Identification of an optimal solution
• The Maintenance of a VKB
• Validation Expert Software Agents (VESA)
• Dynamic Construction of a VESA
• Usage of a VESA for test case solving
• Usage of a VESA for test case rating
• Summary, conclusion, actual and upcoming research

3
1 Motivation

• Intelligent systems validity corresponds to the
  correctness of its incorporated Knowledge Base
  (KB)
• Typically, AI systems are used in application
  fields with no commonly accepted knowledge
  standard
• Human expertise is the basic source of knowledge
  for these systems

? Adjustments with human expertise is often the
only way to ensure validity
Is the model correct ?
How to adjust it ?
?
?
In our terminology,
these adjustments are called

• validation

• refinement

validity statements
4

• A basic disadvantage of this technology is the
  excessive human involvement
• The quality of the results (validity statements
  and refined system) is highly influenced by the
  quality of current human input, but ...
• Human interaction is time consuming
• Human workload is expensive
• Human experts are not always available
• Human experts may not be patient or motivated

Are there some potentials to save a little of
this extraordinary expensive, time consuming and
unstable source of knowledge?

• In fact, there was a knowledge sink within the
  technology
• Only one element of the gained knowledge is kept
  for future use The optimal (best rated) solution
  to each test case.
• Even this little piece of knowledge is not really
  available Just the refined KB maps each test
  case to this solution but we dont keep the test
  cases and their former solutions.

How to plug this sink?
By utilizing all available knowledge
exhaustively!
5

• Validation Knowledge Bases (VKB)
• 2.1 Objective of a VKB in our Validation Framework

A cut-out of our framework looks like that
TURING Test
criteria
validators
rules
test data
test case generation
test case ex-perimentation
solutions
outputs
protocol
6
Lets have closer look into this section!
test case generation
test case experimentation
expert(s)
expert panel
criteria
rate
solve
initial test case generation
test case
QuEST
ReST
reduction
solving session
rating session
solutions

• QuEST Quasi Exhaustive Set of Test Cases
• a well-designed set that ensures coverage by
  formally analyzing the input space
• ReST Reasonable Set of Test Cases
• a subset of QuEST that ensures the requirement
  efficiency by using validation criteria

7
Where is the basic source of validation knowledge?
test case generation
test case experimentation
expert(s)
expert panel
criteria
rate
solve
test case
QuEST
ReST
reduction
Initial test case generation
solving session
rating session
solutions
Unfortunately, the outcome of the validation
process is heavily addicted to the experts
availability, cooperation, patience, motivation,

• How can this instable factor be utilized more
  efficiently?
• Validity statements are based on experts
  ratings. There is a need for human responsibility
  for ratings. Their cooperation is vital to the
  rating process.
• In fact, in the solving process, human workload
  is wasted.
• Solving test cases actually is a time-consuming
  task.

8
Objectives of introducing VKB (and later, VESA)
as external knowledge

• completing human expertise by it
• substituting human expertise by it
• Additional benefits of VKB and VESA
• revealing new aspects and correlations between
  different cases
• offering redundancy and additional input without
  consulting humans
• qualifying the competence estimation for the
  involved experts
• electing an more dependable expert panel for the
  validation process
• supporting the identification of an optimal (best
  rated) solution to cases
• revealing the historical development of knowledge

9
2.2 The content of VKB

• What should VKB contain to
• fulfill these promising purposes and
• serve as a source of knowledge for the Validation
  Software Agents (VESA) ?

• All formal and informal data that can be
  collected, i.e. to each test case
• the (input) test data tj
• a list of all solvers EK
• a list of all raters EI
• associated optimal (best rated) solution solKjopt
• the ratings provided by the rating experts rIjK
• the certainties of these ratings cIjK
• a session time stamp ?S
• an informal description of the context DC

Thus, VKB is a set of 8-tuples tj , EK , EI
, solKjopt , rIjK , cIjK , ?S , DC
10
2.3 The Involvement of a VKB in the TURING Test

• To each examined test data tj ? ?1 (ReST)
• there is an actual systems solution solsys ? ?2
  (ReST) and
• there might be an 8-tuple tj , EK , EI ,
  solKjopt , rIjK , cIjK , ? S , DC in the VKB,
  i.e. tj ? ?1 (VKB) .
• In case solKjopt ? solsys , the VKB brings
  external knowledge into the process.

? solKjopt is worth to be considered in the
rating session.
11
VKB
tj ? ?1 (VKB) ? solKjopt ? solsys ?
test case generation
test case experimentation
expert panel
expert(s)
criteria
rate
solve
test case solutions
QuEST
ReST
reduction
initial test case generation
solving session
rating session
12
2.4 The Utilization of a VKB for other Purposes
2.4.1 Improving the competence estimation of
human experts

• Since a judgement of a more competent expert must
  have a stronger influence, the rating of each
  expert is weighted by a competence estimation to
  compute a final validity degree of a test case
  solution.
• Since competence is not homogeniously distributed
  in the entire problem space, this is done for
  each test case separately.

• Competence estimation cpt( ei , tj ) of an expert
  ei for a test case tj is based on
• his/her own evaluation to be competent
• his/her certainty while rating other experts'
  solutions
• his/her consistency in the solving and the rating
  process
• Does he/she give his/her own solution good marks?
• his/her stability
• Is he/she certain while rating his/her own
  solution?
• the other experts' ratings of his/her solution
• and formally defined in former publications.

13
VKB holds knowledge about the experts' competence
in previous sessions, i.e. historical
competence.
Cant we use this information to select an
appropriate expert panel ?

• From the information in VKB we derived
• a historical session competence sess_esthist( ei
  , Si' ) of an expert ei in a session Si'
• a historical competence trend trnd_esthist(ei ) ,
  which describes the development of an expert's
  competence over time
• a competence gain ? sess_esthist( ei , ?ti ) from
  one session to the next and an average competence
  gain ?i( ?ti ) over time
• a classification of experts as those with an
• increasing,
• even, and
• decreasing
• competence over time
• an average historical competence avg_esthist( ei
  ) , and finally,
• a guideline to use these concepts for an expert
  panel selection.

14
VKB
This figure needs to be further extended by
historical competence
tj ? ?1 (VKB) ? solKjopt ? solsys ?
test case generation
test case experimentation
expert panel
expert(s)
criteria
rate
solve
test case solutions
QuEST
ReST
reduction
initial test case generation
solving session
rating session
15
2.4.2 Identifying an optimal (best rated)
solution

• One of the solutions gained the maximum approval
  by the expert panel in the rating session, i.e. a
  maximum (competence weighted) average validity
  degree.
• This solution is called optimal solution.
• The system refinement technology aims at mapping
  each test data to its optimal solution.

What to do, if there are several solutions, which
enjoy the maximum approval ? Which one is the
very best ?
To solve this problem, we used the knowledge in
VKB and derived a step-by-step filtering process
that is applied until only one candidate solution
is left over.
16
Identification of the very best among best
rated candidate solutions

• There is a list of supporters ES ? EI in VKB.
• A former rater ei ? EI (? ?3 (VKB) ) is a
  supporter of a solution solKjopt ( ? ?4 (VKB) )
  if he/she provided a positive rating for it.
• In case one of the candidates received a maximum
  approval by these absent voters, it is
  qualified as the very best.
• There is a list of vetoers EV ? EI in VKB.
• A former rater ei ? EI (? ?3 (VKB) ) is a vetoer
  of a solution solKjopt ( ? ?4 (VKB) ) if he/she
  provided a negative rating for it.
• In case on of the candidates received a minimum
  resistance by these absent voters, it is
  qualified as the very best.
• If there are still several candidates, look into
  the list of supporters ES ? EI for the
  competence estimation cpt( ei , tj ) of the
  supporters.
• If one list that contains the most competent
  expert, the associated solution is qualified as
  the very best.
• Unfortunately, the very last chance is to hire
  human expertise again for a run-off session
  with the remaining candidates.

17
2.5 The maintenance of a VKB

• The contribution of VKB to a systems validity is
  as good as the knowledge in VKB.
• Therefore, we have to
• re-validate the historical knowledge in VKB and
• update the historical knowledge in VKB
• within each validation session.

• The validation of the validation knowledge is
  ensured automatically
• It is re-validated in future sessions by newly
  rating it.
• Updating, in this context, means adding new cases
  to VKB.
• The experience of a session is its (very best)
  solution solKjopt to each test data tj ? ?1
  (ReST) that was considered in the session.
• Additionally kept in VKB
• a list of solvers EK ,
• a list of raters EI along with their ratings rIjK
  and certainties cIjK
• a time stamp ? S (to compute competence trends,
  e.g.) and
• an informal context description DC

18
3 Validation Expert Software Agents (VESA)

• Objectives
• forming a model of each validators individual
  knowledge and behavior
• successive refinement of this model by
  consecutive validation sessions

• Source of VESAs knowledge
• solving and rating results
• of the associated human counterpart
• of other human validators who often have the same
  opinion as the associated human counterpart

19
3.1 Dynamic Construction of a VESA

• VESAs
• are formed just in the moment of their need and
  forgotten after their usage
• model just the required aspect of their human
  counterpart based on historical information of
  former sessions (i.e. not the current session)
• are requested in case its human counterpart is
  not available
• may be requested even if the human counterpart is
  present to validate the VESA concept itself by
  comparing the behavior of VESA with the real one
  of the human source.

• The knowledge base to dynamically form a VESA in
  case of need is simple
• For
• each human expert
• each and every solution to
• each test data and
• each and every rating of
• each and every historic session indicated by its
  time stamp.

20
3.2 The formation and usage of a VESA for test
case solving
If a validator ei is not available to solve a
present case tj the following applies
Step 1 In case ei solved tj in former sessions,
his/her provided solution with the latest time
stamp ? S will be provided by VESA.
Step 2

• All validators e', who ever delivered a solution
  to the present case tj form a set Solveri0 ,
  which is an initial dynamic agent for ei

• Select the most similar expert esim with the
  largest set of cases that have been solved by
  both ei and esim (a) with the same solution and
  (b) in the same session. esim forms a refined
  dynamic agent Solveri1 for ei

• Provide the latest solution of the expert esim to
  the present case tj , i.e. the solution with the
  latest time stamp ? S by VESA.

Step 3 If there is no such most similar expert,
provide sol unknown by VESA.
21
3.3 The formation and usage of a VESA for test
case rating
If a validator ei is not available to rate a
present case tj the following applies
Step 1 If ei rated the same test case tj
before, look at the rating with the latest time
stamp ? S and provide the same rating r and the
same certainty c by VESA.
Step 2

• All validators e' , who ever delivered a rating
  to the present case tj form a set Rateri0 , which
  is an initial dynamic agent for ei

• Select the most similar expert esim with the
  largest set of cases that have been rated by both
  ei and esim (a) with the same rating r and (b) in
  the same session. esim forms a refined dynamic
  agent Rateri1 for ei

• Provide the latest rating r along with its
  certainty c to tj of esim by VESA.
• Step 3
• If there is no most similar expert esim,
  provide r norating and c 0 .

22
4 Summary, conclusion, actual and upcoming
research

• The only source of domain knowledge are often
  human experts.
• AI system validation technologies so far wasted
  this time consuming, expensive, and not always
  reliable resource.
• The concept of VKB is the key to use this
  resource more efficiently towards valid systems.
  The VKB approach includes all aspects of
  collective historical experience that have been
  provided by previous expert panels.
• While VKB aims at modeling the human experts
  collective and most accepted (best rated)
  knowledge, the VESA concept aims at modeling the
  particular human experts itself.
• Experiments revealed that the VESA approach needs
  to be (has been) refined with respect to
• the non-deterministic nature of most problem
  domains
• Solving cases based on a previous rating is not
  appropriate
• their permanent validation
• VESAs should be applied all the time and compared
  with their human sources
• their completion towards other than (previous)
  test cases
• Under discussion compiling rules from previous
  cases to handle these cases