Dr' Mary Ann Malloy

1
XML-Native Constraint Evaluation

• presented by
• Dr. Mary Ann Malloy
• mmalloy_at_mitre.org
• XML 2004 Conference
• Washington DC
• 17 November 2004

2
Our Vocabulary

• Message Text Formats (MTFs)
• a formatted, text-oriented tactical message
  standard
• Structural Notation (SN)
• a mathematical language developed by MITRE in the
  1980s to describe constraints on MTF structure
  content, usually with some underlying semantic
  interpretation
• XML-MTF
• the web-enabled variant of MTFs
• XSN-MTF
• a specification describing the derivation of SN
  into a form that can be processed natively
  against XML-MTF using XML technologies
• it includes the declarative rules and validation
  criteria used as a basis for producing XPath
  expressions equivalent to the SN

3
Example Old School MTF
MTFs are text-oriented and hierarchically
structured like XML documents
segments
4
Related Talks, Tutorials, etc.

• The overall challenge of migrating tactical
  message processing into the 21st century is the
  topic of
• Transforming Military Command and Control
  Information Exchange (Mike Cokus, MITRE)
• Some lessons learned about designing schemas that
  are easy to understand and reuse is the topic of
• The 7 Top Lessons Learned in Designing W3C XML
  Schemas(Cheryl Conners, MITRE)

NOTE RuleML and related activities to develop
an XML Rules Standard is a current challenge on
the table, but we couldnt wait
5
MTF Validation A Three-part Problem

• Component sequencing
• The sequence of sets in the message instance is a
  realization of the theoretic sequence described
  in the specification for that type, where each
  set and segment is described as required /
  optional / repeatable.
• Element content
• The numeric / alphabetic / alphanumeric content
  of each field in the message instance is one of
  the choices allowed for that element per the
  specification.
• Structural Notation
• Additional Rules regarding the structure and
  content of the message instance that go beyond
  the general sequencing and content rules
  described above.

syntax
semantics
FOCUS
6
Example Structural Notation Rules

• 4F2 lt 10 (4F1 5F3)
• If Set 4 Field 1 contains the same numeric value
  as Set 5 Field 3, then Set 4 Field 2 must contain
  a numeric value less than 10.
• 4F3 R (4F1 !EQ /ADD/)
• If Set 4 Field 1 does not contain a value in
  which the character sequence ADD appears
  anywhere in it, then Set 4 Field 3 is required.
• 1 RP (2 _at_)
• If Set 2 is used, then the use of Set 1 is
  required otherwise the use of Set 1 is
  prohibited.
• 1F1 A /TACREP/
• Set 1 Field 1 is assigned the character string
  TACREP.

7
Old School MTF Validation
Set Sequence Field Sequence Content Validator
Message Text Format
MSGID/TACREP/CTF 124// MAROP/011800Z/1/US/SUB/ /L
M4040N01100E// OPSUP/ACTTYPASW// AIROP/020200Z/6
/US/FTR/F /ALT12000FT// OPSUP/ACTTYPDCA//
Data Tables
Structural Notation Validator
Data Tables Generator
SN Rules
(1)F1 A TACREP (1) MP (2 _at_)
8
MTF to XML Migration
Message Text Format
MSGID/TACREP/CTF 124// MAROP/011800Z/1/US/SUB/ /L
M4040N01100E// OPSUP/ACTTYPASW// AIROP/020200Z/6
/US/FTR/F /ALT12000FT// OPSUP/ACTTYPDCA//
Data Tables
SN Rules
MTF2XML XML2MTF
(1)F1 A TACREP (1) MP (2 _at_)
ltair_operations_datagt ltday_timegt 020200Z
lt/day_timegt ltquantitygt 6 lt/quantitygt
ltcountrygt US lt/countrygt ltsubject_typegt FTR
lt/subject_typegt ltaircraft_typegt F15
lt/aircraft_typegt lttrack_numbergt 401
lt/track_numbergt ... lt/air_operations_datagt
Schema Generator
???

• XSN-MTF
• Rules

• XML-MTF Schema
• structure type

XML-MTF Message
9
XML-Native Validation for MTFs
ltair_operations_datagt ltday_timegt 020200Z
lt/day_timegt ltquantitygt 6 lt/quantitygt
ltcountrygt US lt/countrygt ltsubject_typegt FTR
lt/subject_typegt ltaircraft_typegt F15
lt/aircraft_typegt lttrack_numbergt 401
lt/track_numbergt ... lt/air_operations_datagt

• XSN-MTF
• Rules

• XML-MTF Schema
• structure type

XSN-MTF Validator
XML Schema Validator
XML-MTF Message
10
XSN-MTF Overview
Parser Generator JavaCC
SN EBNF
This approach supports full XML-MTF
validation through COTS technologies
SN Parser
SN Rules
XML-SN

• XSN-MTF Specification
• Draft Version 1.3 Oct 2004

XSN Transforms
XSN-MTF XPath 2.0
Evaluator
MITRE developed reference models
for subsequent implementations

• Mapping Schema
• Agreed standards

XML-MTF instance
XML-MTF schema
Evaluator
11
Motivating Factors

• Conceptual Motivators
• Many SN Rules express relationships between
  non-sibling document elements, so they cannot be
  captured in the Schema.
• We needed XPath 2.0 to get quantification.
• Pragmatic Motivators
• For those SN Rules that could be re-expressed in
  the Schema, we were facing a 2 to 3 year
  re-implementation timeline.
• Although we initiated proposals in appropriate
  forums, in the near term we needed to support
  full XML-MTF validation to ensure community
  buy-in for the overall migration.
• This same reasoning applies to why we didnt just
  continue to do SN validation the old way by
  rendering the XML-MTF in its MTF form.

12
Take-Away Concepts

• Three-Valued Logic
• A logical system in which there is a third
  possible evaluation, null, in addition to true
  and false.
• Iteration
• The inspection of all applicable instances of
  repeated components when evaluating a Rule that
  contains references to them.
• Alignment
• The formulation of appropriate pairings of
  repeated components to evaluate a Rule that
  contains references to them.

13
The Role of a 3-Valued Logical System

• Rules must be written to apply to an arbitrary
  instance of the document type, which contains
  required, optional and/or repeatable components.
• The formulation of a Rule as if (Condition) then
  (Statement) allows constraints to be applied only
  when the instance document exhibits certain
  features, and to be gracefully ignored other
  times.
• This is critical, because documents may be
  flawed, incomplete or just not being used in a
  manner for which a given Rule applies.
• Otherwise false negatives will occur.
• We use a three-valued logical system (3VL) to
  handle this.
• We had to extend XPath 2.0 because it evaluates
  expressions involving missing data as false
  whereas we need such expressions to evaluate to
  null.

14
Logical Equivalents

• A constraint of the form
• if Statement then Condition
• is equivalently stated as the implication
• Condition ? Statement
• which is logically equivalent to
• Statement OR3 NOT3(Condition)
• which can be restated using prefix notation as
• OR3( Statement, NOT3(Condition) )

NOTE We use the 3VL functions OR3 and NOT3 so
evaluations behave
15
Three-Valued Logic (3VL) Tables
NOTE null means insufficient information to
determine truth - OR - truth assessment might
change if additional information becomes
available
16
3VL Reasoning Example

• AND3-ing two expressions produces true only if
  both expressions are known to be true.
• (true AND3 null) evaluates to null
• cannot determine if the second expression is
  true.
• AND3-ing two expressions produces false if one or
  both expressions is known to be false.
• (false AND3 null) evaluates to false
• evaluation does not depend on eventually
  determining if the second expression is true or
  false.

NOTE Similar reasoning applies to the other
logical operations. Arithmetic comparisons also
had to be extended.
17
High-Level Rule Evaluation Logic

• Given a Rule of the form
• if (Condition) then (Statement)

If Condition is false or null, then
document passes validation by default for
Rule. If Condition is true by assessment or by
default, then If Statement is true,
document passes validation for Rule. If
Statement is null, document passes validation
by default for Rule. If Statement is
false, document fails validation for Rule.
NOTE null means we could not determine the
outcome of evaluation.
18
Rule Evaluation Reasoning Example
Tells us whether the Rule applies
Condition

• If Field1 contains the same value as Field3,
• then Field 5 must contain ABC.

Statement
Rule applies?
Field5 value
Field5 exists?
Field3 value
Field3 exists?
Field1 value
Field1 exists?
Rule satisfied?
A
YES
A
YES
YES
YES
ABC
YES
A
YES
A
YES
NO
YES
XYZ
YES
YES
NO
A
YES
A
YES
???
B
YES
A
YES
NO
D/C
D/C
???
D/C
D/C
NO
A
YES
???
A
???
D/C
YES
NO
???
D/C
19
The Role of Iteration

• A Rule that references repeatable document
  components must be true for all referenced data
  instances to determine the constraint is
  satisfied.
• We found the language in which the Rule is
  expressed must expose make explicit repeatable
  components
• otherwise designers are likely to inaccurately
  encode / interpret the Rule.

• A B
• ???What does this mean???
• There is at most one A and one B, and they must
  be equal.
• - OR -
• In every case, A equals B.
• OR -
• There is a case in which some A equals some B.
• - OR -
• There is some A that equals every B.
• etc

20
The Role of Alignment

• A Rule that references repeatable document
  components must make clear for which of those
  component pairings the constraint should be
  considered.
• We found the language in which the Rule is
  expressed must expose make explicit how
  repeatable components should be paired
• otherwise designers are likely to inaccurately
  encode / interpret the Rule.

lock-step comparison
n-to-m comparison
NOTE Iteration and alignment are separate but
related issues.
21
A Simple Example Recipe documents

• Each Recipe consists of a Title, ingredients,
  preparation, possibly some Comments, and its
  Nutrition content.
• Each ingredient can be base or composite
• A BaseIngredient is composed of a Name, an Amount
  (possibly unspecified), and a Unit (unless Amount
  is dimensionless).
• A CompositeIngredient is composed of
  BaseIngredients and IngredientPreparation listed
  as Steps.
• The FinalPreparation lists the Steps needed to
  complete the Recipe.

22
Illustration of RecipeStructural View
Nutrition
Comment
FinalPreparation
Base Ingredient
Step
NameGarnish
NameFilling
BaseIngredient
BaseIngredient
BaseIngredient
BaseIngredient
IngredientPreparation
Segment
Composite Ingredient
Step
Step
Segment
Name
Sauteed
NameEgg Whites
BaseIngredient
Mushrooms
BaseIngredient
BaseIngredient
BaseIngredient
IngredientPreparation
Step
Step
Title
23
Example Rule 1

• A recipe for a beginner cook should contain the
  word Easy at the beginning of the title.
• Composite ingredients are too complicated for
  Easy recipes.
• The designer might express the Rule as
• If the recipe title starts with Easy
• then the recipe must not contain any composite
  ingredients.
• We can re-express the proposed Rule in XPath/XSL
  as follows
• OR3( EQ3-N(count(/Recipe/CompositeIngredient),0),
• NOT3(EQ3-S(/Recipe/Title,/Easy/) ) ) )

Statement
Condition
Statement
Condition
24
Illustration of Rule 1Singular Case
NameGarnish
NameFilling
BaseIngredient
BaseIngredient
BaseIngredient
BaseIngredient
IngredientPreparation
Segment
Step
Step
Segment
Name
Sauteed
NameEgg Whites
BaseIngredient
Mushrooms
BaseIngredient
BaseIngredient
BaseIngredient
IngredientPreparation
Step
Step
Easy Quiche
Title
25
Example Rule 2

• The designer might express a Rule like
• Within each composite ingredient,
• the number of base ingredients must be fewer than
• the number of ingredient preparation steps.
• We can re-express the proposed Rule in XPath/XSL
  as follows
• OR3( every i in /Recipe/CompositeIngredient
  satisfies
• LT3-N(count(i/BaseIngredient),
• count(i/Ingredient/Prepara
  tion/Step)), NOT3(true) )

No "if" means Condition is true by default
iteration
Statement
iteration
Condition
Statement
26
Illustration of Rule 2Iteration
27
Example Rule 3

• The designer might express a Rule like
• If the recipe has composite
  ingredients,
• then for each composite ingredient
  there must be
• a corresponding final preparation step
• in which its name appears.
• Let E1 GT3-N(count(/Recipe/CompositeIngredient),
  0)
• Let E2
• every i in 1 to count(/Recipe/CompositeIngredien
  t) satisfies
• EQ3-S(/Recipe/FinalPreparation/St
  epi,
• concat(/,/Recipe/Composit
  eIngredienti/Name, /))
• Then we can re-express the proposed Rule in
  XPath/XSL as
• OR3(E2, NOT3(E1) )

Condition
iteration
Statement
alignment
Condition
iteration
alignment
28
Illustration of Rule 3Alignment
NameGarnish
NameDough
BaseIngredient
BaseIngredient
BaseIngredient
NameEgg Whites
IngredientPreparation
NameFilling
Step
Step
BaseIngredient
BaseIngredient
BaseIngredient
IngredientPreparation
Segment
Step
Step
Name
Sauteed
Segment
Mushrooms
BaseIngredient
BaseIngredient
BaseIngredient
IngredientPreparation
FinalPreparation
Step
Step
Step
There must be a
corresponding FinalPreparation
Step involving each
CompositeIngredient
29
Conclusions

• Applying constraints to structured text documents
  which may be flawed or incomplete is a
  requirement not unique to military messaging.
• The principles of 3VL provide a consistent
  foundation for handling such evaluations.
• A language for expressing Rules must expose
  notions such as component repetition, iteration
  and alignment.
• This increases the likelihood that designers will
  encode and interpret Rules as intended.
• Leveraging existing languages semantics, with
  automated re-expression of Rules in XML-based
  constructs, has advantages.
• This allows Rule designers to express constraints
  using concepts and terms familiar to them.

30
Contact Information
Joined MITRE October 1987 PhD Old Dominion
University (1995) Position Lead Information
Systems Engineer Location Langley AFB, Hampton,
VA Project 4600 Information Interoperability Ta
sk XSN-MTF Structural Notation for
XML-MTF Sponsor AFC2ISRC/SCG Air Force Command,
Control, Intelligence, Surveillance and
Reconnaissance Center / XML-MTF Implementation
Mary Ann Malloy, PhD
903 Gateway Blvd Suite 200 Hampton, VA
23666 757-825-8515 (voice) 757-826-8316
(fax) mmalloy_at_mitre.org
31
(No Transcript)