M

1
MS Based System Development and Testing in a
Joint Net-Centric Environment Simulation-based
Testing of Emerging Defense Information Systems

• Bernard P. Zeigler
• Professor, Electrical and Computer Engineering
  University of ArizonaCo-Director, Arizona
  Center for Integrative Modeling and Simulation
• (ACIMS)
• and
• Joint Interoperability Test Command(JITC)
• Fort Huachuca, Arizona
• www.acims.arizona.edu

2
Overview

• Modeling and simulation methodology is attaining
  core-technology status for standards conformance
  testing of information technology-based defense
  systems
• We discuss the development of automated test case
  generators for testing new defense systems for
  conformance to military tactical data link
  standards
• DEVS discrete event simulation formalism has
  proved capable of capturing the
  information-processing complexities underlying
  the MIL-STD-6016C standard for message exchange
  and collaboration among diverse radar sensors
• DEVS is being used in distributed simulation to
  evaluate the performance of an emerging approach
  to the formation of single integrated air
  pictures (SIAP)

3
Arizona Center for Integrative Modeling and
Simulation (ACIMS)

• Mandated by the Arizona Board of Regents in 2001
• Mission Advance Modeling and Simulation
  through
• Research
• Education
• Technology Transfer
• Spans University of Arizona and Arizona State
  University
• Maintains links with graduates through research
  and technology collaborations

4
Unique ACIMS/NGIT Relationship

• A long term contractual relationship between
  Northrop Grumman IT (NGIT) with ACIMS
• Employs faculty, graduate/undergrad students,
  others
• Perform MS tasks for/at Fort Huachuca
• Challenges
• Rationalize different ways of doing business
• Deliver quality services on time and on budget
• Benefits
• NGIT Access to well-trained, creative talent
• Students gain experience with real world
  technical requirements and work environments
• Transfer ACIMS technology

5
Genesis

• Joint Interoperability Test Command (JITC) has
  mission of standards compliance and
  interoperability certification
• Traditional test methods require modernization to
  address
• increasing complexity,
• rapid change and development, and
• agility of modern C4ISR systems
• Simulation based acquisition and net-centricity
• pose challenges to the JITC and the Department of
  Defense
• must redefine the scope, thoroughness and the
  process of conformance testing

6
Response ATC-Gen

• JITC has taken the initiative to integrate
  modeling and simulation into the automation of
  the testing process
• Funded the development of Automated Test Case
  Generator (ATC-Gen) led by ACIMS
• In RD of two years, proved the feasibility and
  the general direction
• The requirements have evolved to a practical
  implementation level, with help from conventional
  testing personnel.
• ATC-Gen was deployed at the JITC in 2005 for
  testing SIAP systems and evaluated for its
  advantages over conventional methods

The ACTGen Development Team, NGIT ACIMS was
selected as the winner in the Cross-Function
category for the 2004/2005 MS Awards presented
by the National Training Systems Association
(NTSA).
7
ATC-Gen Goals and Approach

• Goals
• To increase the productivity and effectiveness
  of standards conformance testing (SCT) at Joint
  Interoperability Test Command (JITC)
• To apply systems theory, modeling and
  simulation concepts, and current software
  technology to (semi-)automate portions of
  conformance testing

Objective Automate Testing
Capture Specification as If-Then Rules in XML
Analyze Rules to Extract I/O Behavior
Synthesize DEVS Test Models
Test Driver Executes Models to Induce Testable
Behavior in System Under Test (SUT)
Interact With SUT Over Middleware
8
Link-16 The Nut-to-crack

• Joint Single Link Implementation Requirements
  Specification (JSLIRS) is an evolving standard
  (MIL-STD-6016c) for tactical data link
  information exchange and networked
  command/control of radar systems
• Presents significant challenges to automated
  conformance testing
• The specification document states requirements in
  natural language
• open to ambiguous interpretations
• The document is voluminous
• many interdependent chapters and appendixes
• labor intensive and prone to error
• potentially incomplete and inconsistent.
• Problem how to ensure that a certification test
  procedure
• is traceable back to specification
• completely covers the requirements
• can be consistently replicated across the
  numerous contexts
• military service, inter-national, and commercial
  companies

9
MIL-STD-6016C Excerpt

• Original
• 4.11.13.12 Execution of the Correlation. The
  following rules apply to the disposition of the
  Dropped TN and the retention of data from the
  Dropped TN upon origination or receipt of a J7.0
  Track Management message, ACT 0 (Drop Track
  Report), for the Dropped TN. The correlation
  shall be deemed to have failed if no J7.0 Track
  Management message, ACT 0 (Drop Track Report),
  is received for the dropped TN after a period of
  60 seconds from the transmission of the
  correlation request and all associated processing
  for the correlation shall be cancelled.
• a. Disposition of the Dropped Track Number
• (2) If own unit has R2 for the Dropped TN, a
  J7.0 Track Management message, ACT 0 (Drop
  Track Report), shall be transmitted for the
  Dropped TN. If the Dropped TN is reported by
  another IU after transmission of the J7.0 Track
  Management message, own unit shall retain the
  dropped TN as a remote track and shall not
  reattempt to correlate the Retained TN and the
  Dropped TN for a period of 60 seconds after
  transmission of the J7.0 Track Management
  message.
• XML Translation
• ltrule trans"4.11.13" stimulus"4.11.13.12"
  reference"4.11.13.12.a.2" ruleName"R2 Unit
  transmits J7.0"gt
• ltcondition txt"Check for R2 own unit"
  expression"AutoCorTrue and (CRair.TNcor.CORtest
  3 and J32.TNref.CORtest3) and
  CRair.R2held1 AND J72.MsgTxTrue"gt
• lt/conditiongt
• ltaction txt"Prepare J7.0 Drop Air Track
  message" expression"J70.TNsrcTNown
  J70.TNrefTNdrop J70.INDex0 J70.INDcu0
  J70.ACTVair0 J70.SZ0 J70.PLAT0 J70.ASchg0
  J70.ACTtrk0 J70.ENV0 MsgTx(J70)"gt
• lt/actiongt
• ltoutput txt"Transmit J7.0" outType"Message"
  outVal"J70"gtlt/outputgt
• lt/rulegt
• ltQAgt
• ltrevisit name"DHF" date"10/16/04"
  status"Open"gtneed to add timer for a period of
  60 seconds in which correlation is not
  reattemptedlt/revisitgt
• lt/QAgt

10
Discrete Event Nature of Link-16 Specification
System Theory Provides Levels of
Structure/Behavior
Level
3 Coupled System
2 I/O System
1 I/O Function
0 I/O Frame
11
ATC-Gen Top-Level Architecture
12
ATC Gen Overview

• Standard to XML Translation
• Analyst interprets the requirements text to
  extract state variables and rules, where rules
  are written in the form
• If P is true now Condition
• Then do action A later Consequence
• Unless Q occurs in the interim Exception
• Dependency Analysis Test Generation
• Dependency Analyzer (DA) determines the
  relationship between rules by identifying shared
  state variables
• Test Model Generator converts Analyst defined
  test sequences to executable simulation models
• Test Driver
• Test Driver interacts with and connects to SUT
  via HLA or Simple J interfaces to perform
  conformance testing
• Validated against legacy test tools

13
ATC Gen Tool Process
14
Rule Interpretation Example
Constraints in Appendix E.1.1.2.1 Constraints in Appendix E.1.1.2.1
If Reference TN equals 0, 77, 176, 177, 777 or 77777
Then The Host System Performs no further processing Alerts operator (CAT 4)
Translation to XML (Variables Standards) Translation to XML (Variables Standards)
Condition E1.Init True and VOI.TNref 0 or 63 or 126 or 127 or 4095 or 118783
Action E1.Init False
Output CAT 4 Alert
15
ATC Gen Tool Process
16
Dependency AnalysisAutomated Visual Display of
Rule Relations
Input Variables
Output Variables

• Output
• CAT Alert
• Message Transmission
• Operator Display

17
Test SequencesManually Derive Paths
18
Example Correlation Test Rules
19
Correlation Test Sequence Examples
Correlation was Successful
20
Correlation was Successful
21
1st Prohibition Failed Correlation Failed
22
2nd Prohibition Failed Correlation Failed
23
3rd Prohibition Failed Correlation Failed
24
Test SequencesCreate Paths Through the ATC Gen
GUI
25
Determining initial state and message field
values required to drive SUT through sequence

• Analyst
• Determine the data needed to execute a test
  sequence
• Set state variables and field values accordingly

26
Test CaseData Input Through the ATC Gen GUI
27
Test CaseGenerated XML
I/O Initial State Values
28
ATC Gen Tool Process
29
Test Driver for Controlled Testing
Coupled Test Model
Component Test Model 1
Component Test Model 2
Component Test Model 3
Jx1,data1 Jx2,data2 Jx3,data3
Jx1,data1 Jx2,data2 Jx3,data3
Jx1,data1 Jx2,data2 Jx3,data3
Jx4,data4
Jx4,data4
Jx4,data4
Middleware
SUT
30
Test Model Generation for Controlled Testing

• Mirroring (flipping) the transactions of a SUT
  model (system model behavior selected as a test
  case) allows automated creation of a test model

31
TraceabilityView Through the ATC Gen GUI
Selected Rule in XML
32
Test ModelValidation Generation
Test Case
TEST MODEL GENERATOR
GENERATED TEST CASE MIRROR (XML)
Test Model
include "hierSequence.h" include
"PPLI.h" include "RemoteTNdrop.h"   const port_t
hierSeqDigraphstart0 const port_t
hierSeqDigraphinJmsg1 const port_t
hierSeqDigraphpass2 const port_t
hierSeqDigraphoutJmsg3   hierSeqDigraphhierS
eqDigraph()staticDigraph() PPLI pp new
PPLI() add(pp) couple(this, this-gtstart,
pp, pp-gtstart) couple(pp, pp-gtoutJmsg, this,
this-gtoutJmsg) RemoteTNdrop p1 new
RemoteTNdrop() add(p1) couple(this,
this-gtstart, p1, p1-gtstart) couple(this,
this-gtinJmsg, p1, p1-gtinJmsg) couple(p1,
p1-gtoutJmsg, this, this-gtoutJmsg)
33
Test ModelExecution
Test Model (C)
include "hierSequence.h" include
"PPLI.h" include "RemoteTNdrop.h"   const port_t
hierSeqDigraphstart0 const port_t
hierSeqDigraphinJmsg1 const port_t
hierSeqDigraphpass2 const port_t
hierSeqDigraphoutJmsg3   hierSeqDigraphhierS
eqDigraph()staticDigraph() PPLI pp new
PPLI() add(pp) couple(this, this-gtstart,
pp, pp-gtstart) couple(pp, pp-gtoutJmsg, this,
this-gtoutJmsg) RemoteTNdrop p1 new
RemoteTNdrop() add(p1) couple(this,
this-gtstart, p1, p1-gtstart) couple(this,
this-gtinJmsg, p1, p1-gtinJmsg) couple(p1,
p1-gtoutJmsg, this, this-gtoutJmsg)
TEST DRIVER
SYSTEM UNDER TEST
34
SIAP/IABM Successor to Link-16

• SIAP (Single Integrated Air Picture) objective
• Improve the adequacy and fidelity of information
  to form a shared understanding of tactical
  situation
• Aegis ships would be able to target their
  missiles using information obtained by an Air
  Force Airborne Warning and Control System (AWACS)
  plane.
• All users in the battlespace will be able to
  trust that data to make decisions faster
• Integrated Architecture Behavior Model (IABM)
  requires that all sensors utilize a standard
  reference frame for conveying information about
  the location of targets.
• Navy is the lead IABM will be integrated into
  its and other services major weapons systems
• Developed by the Joint SIAP System Engineering
  Organization (JSSEO), Arlington, Va., a
  sub-office of the Assistant Secretary of the Army
  for Acquisition, Logistics and Technology.
• JITC is the test agency mandated to do test and
  evaluation initial test was Config05 end of
  last year

35
ATC Gen Test CasesMIL-STD-6016C
MIL-STD-6016C Function Remaining Completed Tested
Correlation 20 19
Decorrelation 2 1
Track Management 6 2
Reporting Responsibility 6 6
Message Handling 6 4
Track Quality 1 1
Network Management ?
TDL Alignment ?
Combat ID ?
Filters ?
Sensor Registration ?
Other System (C2) Level ?
Forwarding ?
Totals 41 33

• December Objective
• Complete 17 Link-16 test cases for IABM Config05
• Produced 28 test cases and reported results ahead
  of schedule
• Currently 29 passed, 4 failed which prompted
  corrections on new time box releases by JSSEO (8
  untested or in progress)

36
Test Manager for Opportunistic Testing

• Replace Test Models by Test Detectors
• Deploy Test Detectors in parallel, fed by the
  Observer
• Test Detector activates a test when its
  conditions are met
• Test results are sent to a Collector for further
  processing

Test Manager
Jx1,data1 Jx2,data2 Jx3,data3Jx4,data4
Test Detector 1
Results Collector
SUO
Observer
Test Detector 2
Other Federates
Test Detector 3
37
Test Detector Generation for Opportunistic Testing

• The Test Detector watches for the arrival of the
  given subsequence of messages to the SUO and then
  watches for the corresponding system output
• Define a new primitive, processDetect, that
  replaces holdSend
• Test Detector
• Tries to match the initial subsequence of
  messages received by the SUO
• When the initial subsequence is successfully
  matched, it enables waitReceive (or
  waitNotReceive) to complete the test

38
Observer System Under Observation (SUO)
System (e.g. DEVS)
inports
outports
inports
outports
Tap into inputs and outputs of SUO
Observer outports
Observer For System
Gather input/output data and forward for testing
39
Example Joint Close Air Support (JCAS) Scenario
Natural Language Specification JTAC works with
ODA! JTAC is supported by a Predator! JTAC
requests ImmediateCAS to AWACS ! AWACS passes
requestImmediateCAS to CAOC! CAOC assigns
USMCAircraft to JTAC! CAOC sends readyOrder to
USMCAircraft ! USMCAircraft sends sitBriefRequest
to AWACS ! AWACS sends sitBrief to USMCAircraft
! USMCAircraft sends requestForTAC to JTAC
! JTAC sends TACCommand to USMCAircraft
! USMCAircraft sends deconflictRequest to
UAV! USMCAircraft gets targetLocation from UAV!!
40
Observer of AWACS with JCAS
Observer is connected to SUO and monitors itsI/O
traffic
Data gathered by Observer
addObserver(USMCAircraft, JCASNUM1)
41
Test Detector PrototypeSequence Matcher
processDetect(J2.2,data1,t1)
processDetect(J3.2,data2,t2)
Sequential triggering, same as test models
waitReceive (J7.0,data3,t3)
42
Example of Effect of State AWACS

• Rules
• R1 if phase passive receive
  "ImmediateCASIn
• then output "CASResourcesSpec" state
  "doSurveillance
• R2 if state "doSurveillance receive
  "sitBriefRequestIn
• then output "sitBriefOut phase passive

i1
i2
matchsequence 1 initial state
passive processDetect(ImmediateCASIn,,1) waitRec
eive(CASResourcesSpec,)
need to know the state to enable this sequence
o1
o2
matchsequence 2 initial state
doSurveillance processDetect(sitBriefRequestIn,,
1) waitReceive(sitBriefOut,)
state doSurveillance
state passive
43
Solution make activation of matchsequence2
conditional on matchsequence1
matchsequence2 can only start when matchsequence1
has successfully been performed
44
Observation Test Of AWACS
Observer of AWACS
AWACS Test Manager
45
Problem with Fixed Set of Test Detectors

• after a test detector has been started up, a
  message may arrive that requires it to be
  re-initialized
• Parallel search and processing required by fixed
  presence of multiple test detectors under the
  test manager may limit the processing and/or
  number of monitor points
• does not allow for changing from one test focus
  to another in real-time, e.g. going from format
  testing to correlation testing once format the
  first has been satisfied

Solution

• on-demand inclusion of test detector instances
• remove detector when known to be finished
• employ DEVS variable structure capabilities
• requires intelligence to decide inclusion and
  removal

46
Dynamic Test Suite Features

• Test Detectors are inserted into Test Suite by
  Test Control
• Test Control uses table to select Detectors based
  on incoming message
• Test Control passes on just received message and
  starts up Test Detector
• Each Detector stage starts up next stage and
  removes itself from Test Suite as soon as the
  result of its test is known
• If the outcome is a pass (test is successful)
  then next stage is started up

47
Dynamic Inclusion/Removal of Test Detectors
Test Manager
Active Test Suite
Test Control
removeAncestorBrotherOf(TestControl")
message arrives
test detector subcomponent removes its enclosing
test detector when test case result is known
(either pass or fail)
add induced test detectors into test set
addModel(test detector) addCoupling2(" Test
Manager ",Jmessage",test detector", Jmessage")
48
AWACS Opportunistic Testing in JCAS
CAS Model with AWACS observation
Test Control
Initially empty Test Suite
49
AWACS Opportunistic Testing in JCAS (contd)
Test Control observes CAS request message to
AWACS
Test Control adds appropriate Test Detector and
connects it in to interface,
50
AWACS Opportunistic Testing in JCAS (contd)
First stage detector verifies request message
receipt and prepares to start up second stage
Test Control passes on start signal and request
message
51
AWACS Opportunistic Testing in JCAS (contd)
First stage detector removes self from test suite
second stage waits for expected response from
AWACS to request
52
AWACS Opportunistic Testing in JCAS (contd)
Second stage observes correct AWACS response and
removes itself and starts up second part
53
AWACS Opportunistic Testing in JCAS (contd)
At some later time, second part of Test Detector
observes situation brief request message to AWACS
First stage removes itself and starts up second
stage
54
AWACS Opportunistic Testing in JCAS (contd)
Second stage observes situation brief output from
AWACS thus passing test, It removes itself and
enclosing Test Detector
55
Expanding ATC GenAs a JITC Core Technology
Additional Test Capabilities
Service Oriented Architecture (SOA) Global
Information Grid (GIG)
Dynamic, Operational, Performance Testing
JXF, RAIDER, JIT 06-03, IABM Config07
Multiplatform Observation Opportunistic Testing

• Dimensions in which ATC Gen is/can be extended
• Additional requirements
• (e.g., standards, system-level requirements)
• Additional test capabilities
• Numbers indicate possible order of extension
  dependence on funding and customer priorities
  noted

Federation HLA Time Managed S/R
IABM Config05
IABM, ADSI v12
Platform Real Time 1-on-1 S/R
ADSI, IABM
One-on-One Stimulus/ Response
Additional Requirements
6016C, Other MIL-STDs
IABM Config05
JROC, Interoperability
Transition to Web Services
56
Summary

• Systems theory and DEVS were critical to the
  successful development of automated test case
  generator
• ATC-Gen is being integrated into JITC testing as
  core technology
• ATC-Gen methodology will be expanding to web
  services and Service Oriented Architecture over
  GIG
• Model continuity and mixed virtual/real
  environment support integrated development and
  testing process