Irregular Warfare Modeling/Data Validation Best Practices

1
Irregular Warfare Modeling/Data Validation Best
Practices
11th Annual MOVES Research Education Summit 12
July 2011
2
Purpose and Agenda
Purpose To discuss the framework developed at
TRAC-MTRY/NPS toward conducting validation of
Methods, Models, and Tools (MMT) that inform
analysis of Irregular Warfare (IW) and to
overview ongoing development of a similar data
validation framework.

• Overview and Objectives.
• Validation Project Teams.
• Project Descriptions and Results.
• IW Model Validation Best Practices.
• IW Data Validation Best Practices.
• Summary and Way Ahead.
• Questions.

3
Overview and Objectives
Objective To develop methodologies/best
practices toward validating methods, models and
tools (MMT) and associated data that support IW
analysis, using the CG model for implementation
of these practices.

• TRAC-MTRY and Naval Postgraduate School.
• IW Modeling Validation Best Practices Guide,
  FY09-FY10.
• Identify and develop a best practices for
  verifying and validating social science theory
  and modeling required for tactical level IW MMT.
• Validating Data that Supports IW Models,
  FY10-FY11.
• Assess, at a minimum, the validation of IW data,
  to include an examination of data requirements,
  data sources, and data availability as well as
  derivation of data.

4
Validation Project Teams
Each team represents efforts that focus on
developing frameworks and methodologies toward
validating models, validating data, and making
assessments of and building documentation toward
validation of CG.

• FY09/FY10 TRAC-MTRY/NPS.
• CPT Richard F. Brown, Project Lead.
• Dr. Jeffrey Appleget, Professor, OR Department.
• Dr. Deborah Duong, Senior Analyst, Augustine
  Consulting, Inc.
• Mr. Curtis Blais, Professor, MOVES Institute.
• Mr. Timothy K. Perkins, ERDC.
• Ms. Meredith Thompson, SEED Center.
• FY10/FY11 TRAC-MTRY/NPS.
• MAJ Joe Vargas, Project Lead.
• CPT Richard F. Brown, Project Support.
• Dr. Jeffrey Appleget, Professor, OR Department.
• Mr. Curtis Blais, Professor, MOVES Institute.
• Dr. Michael Jaye, Professor, Defense Analysis
  Department.
• Dr. Eric Weisel, CEO, WernerAnderson, Inc.
• COL Robert Burks, Senior Military Advisor, OR
  Department.

5
Best Practices GuideApproach and Requirements

• Requirement 1. Provide a report that identifies
  validation best practices for IW MMT
  tactical-level representation.
• Requirement 2. Apply the BPG developed in
  Requirement 1 to both relevant MMT in order to
  demonstrate usability of the BPG framework.
• Best Practice Guide Approach.
• Conducted an extensive literature review of
  prevalent validation literature, to include the
  DOD Recommended Best Practices, MIL-STD-3022,
  SISO General Methodology for VVA, and others
  (Chapter 2).
• Synthesized literature and described the
  challenges of modeling the IW environment and
  validating the models that represent the IW
  environment (Chapter 3).
• Constructed a validation framework toward model
  validation. The framework includes detailed
  descriptions of the principal areas of the
  framework that must be identified to conduct
  proper validation (Chapter 4).
• Apply the BPG, as appropriate, to relevant MMT
  in particular, the CG model.

6
Validation Framework Concept MapThe Golden
Triangle
This effort resulted in the construction of the
following model validation and development
framework. The components of the Golden
Triangle make up the crux of model validation
and establish the critical basis of comparison to
the real world.

• User Needs
• The developer needs to obtain a succinct and
  clear statement of the problem the MS is
  expected to address.
• Requirements
• Develop specific functional or quality statements
  that can be directly and explicitly assessed to
  determine requirement.
• Acceptability Criteria
• Develop a requirements traceability matrix
  relating each specified requirement with
  acceptability criteria applicable to the intended
  use.
• Results
• The acceptability criteria identify what the
  model needs to do to satisfy or meet the set of
  respective requirements pertinent to the intended
  use.
• Data
• The greater the specificity in the data
  requirements for a model, the greater the ability
  to collect the data needed to populate the model.
  

• Intended Use.
• Obtain a clear, succinct statement of intended
  use from the user representatives.
• Simuland.
• The simuland is the real-world system of
  interest, including the objects, processes, or
  phenomena to be simulated.
• Referent.
• Identify the social science theory (or theories,
  if multiple competing theories will be
  represented in the model for comparison) that
  explains that phenomena.
• Conceptual Model.
• Develop the conceptual model using tools and
  techniques that create machine-readable
  specifications of the data and logic of the
  model.
• Executable Model.
• Design the model implementation to be as
  transparent as possible to permit analysis of
  execution paths and computed outcomes

7
IW Data Validation Best Practices
Requirement Provide a report that assesses, at a
minimum, the validation of IW data, to include an
examination of data requirements, data sources,
and data availability as well as the derivation
of data.

• WORKING OUTLINE
• Introduction.
• Literature Review.
• Review validation standard and practices from DOD
  and related sources that add value to DoD MMT.
• Data Supporting Irregular Warfare Modeling.
• Describe IW-type data and highlight the
  challenges in validating this data.
• IW Data Sources.
• Characterize sources for IW data.
• Best Practices for Validating IW Data.
• Explain the IW data validation framework, to
  include a description of methods toward
  validating IW data, what metadata must be
  included with data sets, and other significant
  insights.
• Summary and Recommendations.
• Appendix.
• Develop IW data validation framework use-case
  using relevant MMT, in particular, the CG Model.

8
Summary and Way Ahead

• Summary.
• Multi-disciplinary team developed a BPG that
  provides a useful framework that a validation
  agent could use for validating a IW model, or
  that a developer could use to properly develop an
  IW model.
• Model validation best practices incorporate
  concepts that developers are aware of, yet many
  of the concepts must be enumerated explicitly to
  ensure validation is possible.
• Validation may not be a proper term with
  respect to data. A paradigm shift may be in
  order.
• Any type of validation, whether model or data,
  must be related to a specified and/or intended
  use.
• Way Ahead.
• TRADOC Analysis Center (TRAC) Irregular Warfare
  Modeling and Simulation Summit provided a
  considerable forum for discussion and actions
  pertaining to these issues. The validation team
  will work closely with the TRAC Complex
  Operations Data Development Activity (CODDA) to
  develop and vet the data validation best
  practices, and ensure that is codified
  sufficiently for use in the analytic community.

9
Comments / Discussion
10
CG Model Assessment Requirements

• Assess the appropriateness of the CG model
  representation of the Conflict Ecosystem
  conceptual model, and whether the model referent
  provided appropriate underpinning to account for
  human socio-cultural behavior.
• Assess the appropriateness of the Social Network
  Representation within the CG Model and the
  whether this representation is generalizable to
  conflict ecosystems in any region.
• Assess the appropriateness of the representation
  of social science theory within the CG Model,
  specifically tracing the implementation of the
  Theory of Planned Behavior.
• Develop risk measures for using a model or
  simulation for a specified application.
• Develop criteria for assessing the risk of using
  a model or simulation based on consequence,
  error, and validation process.
• Characterize an approach toward validating the CG
  model based on the methodologies/insights/lessons
  learned from the tasks developed above.