Initial Development and Continuing Evolution of the Performance Measurement Base Object Model BOM fo

1
Initial Development and Continuing Evolution of
the Performance Measurement Base Object Model
(BOM) for the Navy Aviation Simulator Master Plan
(NASMP)
2
Purpose

• Discuss the ongoing evolution of the Performance
  Measurement Base Object Model Development for
  NASMP
• NASMP Overview
• PM/AAR Systems Overview
• PM BOM Overview
• BOM Development Process
• Data Sources
• Data Characterization
• Process Modification
• Data Source Analysis
• Data Availability
• BOM Generation
• Review, Versioning and Incorporation in the NASMP
  FOM
• Current Status of BOM
• Future Activities

3
NASMP Overview

• What is NASMP?
• The Naval Aviation Simulation Master Plan - An
  advanced networking infrastructure for
  simulator-based training systems designed to meet
  Naval Aviations tactical training needs
• Will support individual, element, and air wing
  training
• Eventually larger-scale, joint intra- and
  inter-service training
• NASMP development will encompass
• A networking architecture
• A set of simulation standards
• Common tools
• Databases which enable the sharing of software
  resources
• The production of a suite of simulation resources
  to be made available to the participating
  communities

4
PM/AAR Systems Overview

• What is Performance Measurement within NASMP?
• NASMP will support the training of mission level
  tactics across multiple Air Wing platforms
• Integrated instructional systems will be needed
  for
• Coordination of training event execution
• Evaluation of trainees operating in the
  distributed environment
• To support evaluation of training efficacy
• Performance Measurement After Action Review
  (PM/AAR) systems development challenges include
• Defining system capabilities required to support
  performance measurement of trainees within the
  distributed environment
• Defining the PM data to be used by the system to
  support trainee evaluation

5
PM BOM Overview

• Why develop a PM BOM?
• To provide a common framework to communicate PM
  data in a distributed environment
• NASMP will be implemented using the High Level
  Architecture (HLA)
• HLA is integral to supporting the execution of
  distributed training events in NASMP
• HLA is also an excellent medium for integration
  of the data collection requirements of a
  distributed PM/AAR system

6
PM BOM Overview

• What is the purpose of the PM BOM?
• The PM BOM is designed to support collection of
  targeted data over the HLA network to provide
  PM/AAR system capabilities such as
• Assess individual cockpit skills such as radar
  mechanics, and high level team or collective
  competencies such as coordination,
  decision-making, and communication
• Enable automated capabilities to display and
  share performance data during execution in near
  real time to enhance instructor awareness and
  subjective evaluation during distributed scenario
  execution
• Feed automated objective analysis of targeted
  skills to enable rapid and concise instructor-led
  debriefing of participants performance against
  defined training objectives

7
PM BOM Overview

• What systems requirements guided PM BOM
  development?
• PM BOM development was guided by PM/AAR system
  capabilities and supporting data requirements
  detailed using
• NASMP PM/AAR CONOPS
• NASMP SRD
• F-18 FOM
• NTMF

8
PM BOM Overview

• How do we define the data required in the PM BOM?
  
• Performance data will be generated at each of the
  distributed locations and shared between
  participating training sites from several sources
• Simulators
• Synthetic training environment
• PM data is being defined iteratively
• NASMP will integrate multiple platform simulators
  over time with each having distinct PM data
  collection requirements
• To support the evolving nature of the PM/AAR
  system the PM BOM process will be iterative and
  include
• NASMP Federation Object Model (FOM) integration
• Navy Training Meta FOM (NTMF) integration
• All PM data will be defined using this process
  and integrated using the FOM protocol

9
PM BOM Development Process

• What is the PM BOM development process?
• ?

10
PM BOM Development Process
Incorporate in NASMP FOM
Generate PM BOM
Analyze PM Data Sets
Modify Process
Characterize PM Data
Identify Data Sources
11
Data sources

• What sources of data were used?
• The following is a generalized list of sources
  identified
• Virtual simulator data (F/A-18C DMT first)
• CGF data
• Environmental database data (terrain, atmosphere,
  etc.)
• Mission Planning Systems (MPS) Data
• Collaboration data (alternate network data that
  includes trainee plans and brief data as well as
  instructor comments)
• Our approach had to account for programmatic
  considerations
• NASMP had already begun integration with its
  first platform simulator manufacturer which was
  executing its development schedule
• The PM/AAR team was required to identify all
  PM/AAR related data it sought from their BOM
• The F/A-18 BOM became the first concentration of
  the PM BOM development process
• As part of this effort, NASMP required the
  contractor to identify all information it would
  make available in the F/A-18 BOM

12
Performance Measurement Data Characterization

• Why was it necessary to characterize data?
• To enable analysis of trainee performance and
  support instructor lead evaluation
• Most simulation data was defined to support
  interoperability between distributed simulations
• Trainee Performance data was not adequately
  represented within the NASMP FOM
• Data characterization is needed to support PM/AAR
  system capabilities

13
Performance Measurement Data Characterization

• How was the PM data characterized?
• Using the following process
• Define tactical syllabi events and associated
  training objectives to be supported by NASMP
  training system
• Define platform specific cockpit
  tasks/sub-tasks/steps trained in syllabi events
• Identify performance measures of trainee
  sub-task/step using modified Job/Task Analysis
  (JTA) and SME provided standards for performance
• Identify simulation data to support performance
  analysis through association process using
  predetermined guidelines

14
Performance Measurement Data Characterization

• What predetermined guidelines were used in the
  association process?
• Several key criteria for evaluation of
  performance were identified for the data
  characterization process
• Task Action Initiated - Data identifies trainee
  recognition and initiation of action requirement
• Timeliness (Latency) - Data compares action to a
  time standard marked by a trigger event
• Procedural Accuracy - Data identifies actions
  taken in Sequence for completeness, accuracy and
  timeliness
• Quality of task action - Data provides level of
  compliance with a performance standard identified
  with a sub-task
• Coordination of required task interaction - Data
  supports analysis of high level coordination
  skills to assess coordination competencies

15
Modification of the Data Definition Process

• Why was the data definition process modified?
• The Process was modified to conform to the FA-18
  DMT schedule
• Examining the original process showed that
• Much of the data centered on several F/A-18
  aircraft systems
• Data sets were tied to particular phases of a
  training event
• Different training events incorporated many of
  the same system use tasks for a given phase
• The process was modified to identify data that
  would be generically applicable based on phase of
  flight (tasks and systems)
• Concentration was on defining specific aircraft
  systems used during phases
• identifying the training linkage and supporting
  data using SME support
• The change allowed the team to identify common
  training tasks associated with a common set of
  performance metrics
• Known data sets that support these metrics
  included
• TSPI (time, space, and position information)
• weapon fire, detonation results
• communications channels

16
Modification of the Data Definition Process

• What additional data sets were identified?
• Navigation The ability to compare planned vs.
  actual flight path, defined programmed sequences,
  geo-reference waypoints, and selected A/A
  waypoint or sequences
• Weapons Knowledge of initial weapons/expendables
  load-out
• Valid Shot Cue Identification a shot was fired
  within a valid shot cue/LAR
• Enhanced Set of Sensor Information Often, the
  information was specific to the sensor in use,
  some examples included
• RWR Threat indications
• Radar Mode/Sub-mode, range scale and target
  designation
• FLIR Modes FOV, look angle, target
  designation and additional states
• Instrument Panel Capturing the trainees use of
  systems located on the instrument panel
• Pilot/Trainee Identification

17
Analysis of PM Data Sets

• What analysis was performed on the identified PM
  Data?
• Navigation data provided from MPS may be
  dynamic and represented in different formats
• Pilot/Trainee identification data mission level
  assessment of trainee performance is accomplished
  at distributed sites

18
Determination of availability within the NASMP FOM

• How was availability determined?
• Straight forward process
• NASMP FOM is based on the RPR (Real time Platform
  Reference) FOM, and largely documented by the
  GRIM (Guidance, Rationale, and Interoperability
  Manual)
• Noteworthy challenges
• Inadequate documentation supporting platform
  extensions to the NASMP FOM
• Coordination between NASMP and F/A-18 teams
• PM Data identified by NASMP was only partially
  included in the F/A-18 BOM
• The instrument panel and sensor extensions were
  not represented at the time
• Lesson learned A BOM should be submitted with
  corresponding documentation, and only accepted
  for review when packaged with standardized
  GRIM-like documentation

19
PM BOM Generation

• What are the design considerations?
• Establish BOM interface standard beyond NASMP
• Data standardization
• Common, extensible framework
• Interface Definition
• Complete
• Concise
• Generic
• Considers the context of the data

20
NASMP FOM Incorporation

• How is BOM integration managed?
• Assigned a version number to reviewed BOM
• Provide a complete and validated OMT file
• Provide BOM documentation
• Run an analysis on the BOM and compared against
  the NASMP FOM
• Use tools to merge approved content of the BOM
  into the FOM

21
NASMP FOM Incorporation
BOM GRIM Submittal
Approved for Consideration
BOM Submittal
NASMP Federation WG
The Submittal Package
Rejection Notification
BOM Submittal
Incompatible
Assess for Compatibility
Compatible?
Acceptance Notification
FOM Analyst Approved
(Automated)
Assess for NASMP Functionality
BOM GRIM Submittal
Functional Problem
WG Approved
Functional?
22
Current PM BOM Content

• What are the PM BOM extensions to the NASMP FOM?
• ?

23
Current PM BOM Content
24
Future Directions
25
Future BOM Development Activities

• PM BOM development activities span the life cycle
  of NASMP development
• Integration of multiple simulators drive a
  complex incremental process
• Revisit the F/A-18 platform BOM as it matures
• Address new platform simulators as they are
  introduced
• Future activities on a programmatic level
• PMAAR System Specification will help define data
  uses
• Integration of Battle Force Tactical Team Trainer
  (BFTT) and the Air Forces Distributed Mission
  Operations (DMO) system
• Navy Training Meta-FOM Working Group coordination
• Establishment of coordination with simulation
  standards groups across services and industry

26
Summary

• The PM BOM development process was undertaken to
  insure data necessary for effective evaluation of
  trainee performance is identified and enumerated
• The PM BOM is the critical interface for the
  PMAAR system
• NASMP development is an incremental process to
  account for
• Virtual simulations
• CGF
• Environment databases other databases
• Mission Planning Systems
• Instructional collaboration data
• Future distributed training systems will benefit
  tremendously by the establishment of
  collaboration between services and industry in
  developing data standards for a generic PM BOM

27
Questions?