Simulation Based Acquisition SBA Verification, Validation and Accreditation of Models and Simulation

1
Simulation Based Acquisition (SBA)-
Verification, Validation and Accreditation of
Models and Simulations - Key ConceptsFrom A
Practitioners Perspective 19 May 2005

• Mr. Patrick M. Cannon, P.E.
• The AEgis Technologies Group
• www.AEgisTG.com

631 Discovery Drive, Huntsville, AL 32806 Phone
[phone], Fax [phone], email
PCannon_at_AEgisTG.com
2
Outline

• SBA Background
• MS Technology to Support SBA
• VVA of MS for SBA
• VVA Background
• Cost of VVA
• VVA Procedural Recommendations
• Risk-focused Strategy
• Managed Investment Strategy
• Conclusions

3
Some SBA History

• Vice President Gores National Performance Review
  
• Cut delivery time for new systems by 25
• Department of Defense
• Stretched goal to 50 reduction of cycle time
• Set goal to reduce Total Ownership Cost
• Defense Systems Affordability Council
• Recognized potential of MS
• Set SBA as one of the top initiatives to realize
  the stretch goal

4
Definition(s) of SBA

• SBA definitions are numerous and generally
  unrevealing. Elements include
• Methodology or practice
• Supporting systems engineering
• Sometimes convolved with program or
  organizational unit
• Several aliases include
• SBA (Simulation Based Acquisition)
• SEBA (Synthetic Environments Based Acquisition)
• MSAM (Modeling and Simulation for Affordable
  Manufacturing)
• SMART (Simulation and Modeling for Analysis
  Requirements and Training)
• and variations thereof

5
Significant Attributesof SBA (as a handle)

• SBA is not
• only simulation,
• exclusively simulation-based,
• just for acquisition, or
• a program or a single organizations prerogative
• SBA is
• a concept-of-operations,
• for an entire enterprise,
• supportive of capabilities management, materiel
  systems engineering life-cycle, and collaborative
  operations, wherein
• collaborative representations support a shared
  worldview

6
SBA - Not Just Technology
Process
Iterative Acquisition Process
Culture

• Iterative Spiral Process

- Electronic Exchange of System Models
Evolved Acquisition Culture
Requirements
Functional Design
Implementation Design
- Rapid Evaluation of Multiple Options

• Enabled Integrated
• Process Teams

• Changing Roles
• and Responsibilities

Technology
Tactical
Decision
Operational
System
Req Elicitation
Need
Integration
Support
Training and
and Analysis
Ops Support
and Test
Functional
Design
Integrated Engineering Management Enterprise
and Analysis
HME / HW / SW
Implementation
and Analysis
Modification
and Upgrade
HME / HW / SW
Development
Maintenance
and
Logistics

• Collaborative
• Distributed Engineering

- Seamless Integration of Engineering
Disciplines

• Info Repository

• Integrated Design Data Schema

- User Transparent Web Style Access
Collectively, these were to facilitate --
An unprecedented quality of enterprise-wide,
collaborative decision making across the
acquisition life-cycle...
7
Outline

• SBA Background
• MS Technology to Support SBA
• VVA of MS for SBA
• VVA Background
• Cost of VVA
• VVA Procedural Recommendations
• Risk-focused Strategy
• Managed Investment Strategy
• Conclusions

8
Definitions of MS

• We propose the broadest reasonable definitions of
  MS

• Referent n. Something referenced or singled out
  for attention, a designated object, real or
  imaginary or any set of such objects
• Model n. Representation of some referent
• Simulation n. Mechanization of a models
  evolution through time

9
Scope of Consideration

• All of Modeling and Simulation
• All kinds / types of simulation
• All phases of MS life-cycle
• All domains of applications and kinds of uses
• All aspects of industrial and business
  environment
• All Interested Parties
• All Legitimate MS-Related Concerns

10
Implementation- Live, Virtual, Constructive -
Constructive Simulation Models and simulations
that involve simulated people operating simulated
systems.
Virtual Simulation A simulation involving real
people operating simulated systems.
Live Simulation A simulation involving real
people operating real systems.
Reference DoD 5000.59-P, "Modeling and
Simulation Master Plan," October 1995
11
Technology Circumstances- Net Assessment -

• MS Technology is generally available, robust,
  and relevant to SBA
• Risk accrues from application of available
  technologies in SBA enterprise collaborative
  operational environment
• Special concerns are
• Conceptual modeling
• Architecture management
• Data management
• Standards

• Composability
• Interoperability
• Reuse

12
Outline

• SBA Background
• MS Technology to Support SBA
• VVA of MS for SBA
• VVA Background
• Cost of VVA
• VVA Procedural Recommendations
• Risk-focused Strategy
• Managed Investment Strategy
• Conclusions

13
VVA Definitions

• VERIFICATION - the process of determining that a
  model or simulation implementation accurately
  represents the developers conceptual description
  and specification....Is it what was intended?
• VALIDATION - the process of determining the
  degree to which a model or simulation is an
  accurate representation the real-world from the
  perspective of the intended uses of the model or
  simulation...How well does it represent what I
  care about?
• ACCREDITATION - the official certification that a
  model or simulation is acceptable for use for a
  specific purpose...Should my organization endorse
  its use?

Reference DoDD 5000.59
14
VVA vs IVV

• Provides Independent Evaluation / Assessment of
• Are we building the product right? Verification
  
• Are we building the right product? Validation
• Verification (Are we building the product right?)
• The process of determining whether or not the
  products of a given phase of the software
  development lifecycle fulfill the requirements
  established during the previous phase
• The product is internally complete, consistent
  and correct will support the next phase
• Validation (Are we building the right product?)
• The process of evaluating software throughout its
  development process to ensure compliance with
  software requirements. This process ensures
• Expected behavior when subjected to anticipated
  events
• No unexpected behavior when subjected to
  unanticipated events
• System performs to the customer's expectations
  under all operational conditions

Source IVV Overview Briefing, NASA IVV
Facility 100 University Dr. Fairmont, WV 26554
15
In a Perfectly Engineered World
16
In a Perfectly Engineered(MS) World
Integrated Simulation

System Integration Test
Model/Sim Reqts Analysis
Source and Executable Code


Requirements Allocated to Software
Configuration Item Integration Test
Software Reqts Analysis
Source and Executable Code

Software Requirements Specification

Component Integration Test
Software Analysis
Software Design Documents
Source and Executable Code
Unit Integration Test
Software Coding
Source and Executable Code
Source and Executable Code
17
In a Perfectly Engineered(MS) World

• But Whose Requirements?
• The MS developers? System requirements,
  software requirements
• The MS Users? Accreditation requirements

18
In The Real (MS) World

• Systems Engineering is not perfect
• Requirements arent right (and, sometimes not
  even articulated)
• Consequently.

19
MS VVA Focus

• Simply put, because every model or simulation is
  an abstraction of reality, at some level they all
  are wrong.

• The VVA process should be focused on
  accreditation based on the credibility of the MS
  implementation and risks if the MS is used!

20
The Fundamental Questions
VV
How Good Is It?
Is It Good Enough?
A
21
Outline

• SBA Background
• MS Technology to Support SBA
• VVA of MS for SBA
• VVA Background
• Cost of VVA
• VVA Procedural Recommendations
• Risk-focused Strategy
• Managed Investment Strategy
• Conclusions

22
Whats the Cost?

• MS VVA activities compete with development,
  operations, and maintenance activities for scarce
  program resources
• Dollars
• Staff
• Facilities
• Equipment
• Information

23
So Answer the Question!

• I dont exactly know,
• but I know it depends!

24
What Does MS Cost in DoD?
Results of a DoD Modeling and Simulation Survey
The Workshop on Foundations for Modeling and
Simulation (MS) Verification and Validation
(VV) in the 21st Century, better known as
Foundations 02, sponsored bythe Defense
Modeling and Simulation Office, and held October
22-24, 2002 at the Johns Hopkins University
Applied Physics Laboratory in Laurel, Maryland.
25
VV Cost Factor 1
Availability of Information about the MS

• Quality of MS documentation affects the cost of
  VV (especially verification)
• e.g., no S/W design documentation means you have
  to reverse engineer the code to do verification
• Three cost drivers
• Cost of buying information about the model
• Cost of reconstructing unavailable information
• Cost difference incurred when forced to replace a
  relatively cheap VV technique with a more
  expensive VV technique

26
VV Cost Factor 2
Availability of Information about the
Referent(Validation Data)

• MS Validation requires data from dynamic
  behavior of the system being modeled
• Test costs are the biggest driver of validation
  data collection cost
• Test data may not be available for MS validation
• Insufficient instrumentation
• Test events are not generally done just for MS
  validation purposes
• Program sensitivities may preclude release of
  data
• Classification issues may get in the way
• Data collected may not be suitable for validation
• If the program doesnt need it, they wont
  measure it

27
VV Cost Factor 3
Application Complexity

• VV requirements may be difficult to separate out
  for highly integrated simulations
• Integrated live, virtual and constructive
  simulations, for example
• If the MS are only a part of the analysis
  process, VV requirements may be subjective at
  best
• For example, accuracy requirements for simulation
  federates may be difficult to quantify
• Subjective at best, political at worst

28
VV Cost Factor 4
Application Risk

• Most VVA processes used in DOD are based on risk
  assessment
• VVA activities can provide estimates of residual
  risk since no model can be completely verified
  and validated.
• Usually subjective judgments of risk based on
  expert opinion
• High risk applications require more VV resources
  than lower risk applications
• Both impact and probability of wrong answers must
  be evaluated to determine VV resource
  requirements.

29
VV Cost Factor 4
Application Risk
30
VV Cost Factor 5
Practitioner Experience

• Experience under the SMART VVA program executed
  by Joint Accreditation Support Activity
• Analyzed VVA of 5 models varied between 30,000
  and 100,000 lines of code
• VV tasking and resources expended on each was
  about the same
• This conclusion may not be applicable everywhere,
  butthe level of experience of the practitioner
  is likely to be a significant driver of VVA cost
  requirements.
• Susceptibility Model Assessment with Range Test

31
VV Cost Factor 6
Accreditation Authority Data Requirements

• Sponsor accreditation data requirements may not
  be driven by technical issues
• May not always seem logical to the MS developer,
  VVA practitioner, or anyone else for that matter
• Often driven by
• Policy
• Previous experience
• Politics
• Preconceived opinions about MS (for or against)
  on the part of the program manager
• Funding
• All of these are subjective at best and
  inherently non-measurable

32
VV Cost Function?
What do I know about the real system?
What do I know about the model?
What VV methodologies do I have?
Cost of evaluation VV Mature techniques
tools
Cost of required real system
information System KnowledgeSystem Data
Cost of requiredmodel
information Model documentation
Cost of particular V or V Enterprise
33
One Other Issue
MS Task Accounting

• How to account for MS VV costs
• Is verification counted as S/W development, but
  validation as VV?
• Is a development team peer review, with outside
  experts, an SME review and part of
  verification? Or part of development?
• Is pre-test prediction part of validation or a
  test cost?
• If post-test analysis is useful for validation,
  does it get charged as VV or as a test cost?
• For simulations of real objects that are under
  development, which costs go into the item
  development bin, the MS development bin, the
  VV bin, etc.
• When good software/simulation development
  practices are followed, the harder it is to sort
  out development costs from VV costs
• Good systems engineering produces good VV
  information

34
Outline

• SBA Background
• MS Technology to Support SBA
• VVA of MS for SBA
• VVA Background
• Cost of VVA
• VVA Procedural Recommendations
• Risk-focused Strategy
• Managed Investment Strategy
• Conclusions

35
Any SBA Activity Needs a Deliberate VVA Program

• Modeling simulation tools are CRITICAL
  RESOURCES for analysis, test, and evaluation
• Supporting key decisions (technical, cost and
  schedule)
• Risks inherent in key decisions mandate risk
  management
• The NEED exists to regularize the verification
  and validation of modeling and simulation tools
  (data, models, test beds) and activities
  (studies, exercises)
• A DELIBERATE PROGRAM of activity to establish
  confidence in the operation of models
  simulations and in the significance of modeling
  simulation results is imperative

36
Optimizing Risk VVA
RISK PROBABILITY x IMPACT
RISK ANALYSIS KEEPS the VVA EFFORTS FOCUSED ON
PROBLEM CREDIBILITY REQUIREMENTS!
Determining the Best use of all the Resources
available
37
Quantifying Risk

• Three areas typically used to identify and
  quantify risks
• Problem categories
• Impact of occurrence
• Probability of occurrence
• Use current organizational risk approaches if
  they are well defined

38
Problem Categories and Levels of Impact
Consistent with DoD Military System Safety
Standard MIL-STD-882C
39
Problem Categories and Levels of Impact
Consistent with DoD Military System Safety
Standard MIL-STD-882C
40
Probability of Occurrence
The number of Items should be specified
41
Mapping MS to Risk Areas
Probability of Occurrence
Problem Categories
Impact of Occurrence
42
VV Activities as a Function of Risk
Needed when Risk is
Typical Sources
Item Required
Medium
Low
High
SME, Scientific theory and accepted algorithms,
Laboratory tests, developmental tests, system
operational tests, engineering data, training
test results, Historical values, previously and
separately validated simulations or data
Validate MS Application
Any Three
Any Two
Any One
Audit, Desk Checking, Face Validation,
Inspection, Reviews, Turing Test, Walkthroughs
Verify MS Implementation
Required
Any Two
Either
Specifications and/or Requirements Trace Reports,
Functional Flow Diagrams,
VerifyMS Design
Any Two
Conceptual Model Review Reports
ValidateConceptual Model
Required
Required
Required
Any Two
Any One
Any Three
Simulation Support Plan, Configuration Management
Plan, VV Plan, Accreditation Plan
MS Development Process
Required
Any Two
Required
Requirements Trace Reports, Review Reports, Code
Walkthroughs, S/W Tests reports
S/W Development Results
Desired
Required
Mgmt Plans, S/W Documentation, anecdotal
S/W Mgmt Resources Desc
Any One
Any Two
CM Database, SCRs S/W Docs, CCB minutes, S/W
Design Documentation
S/W Mgmt Artifacts
Any One
Any Two
SPCR logs, test reports, verification reports,
usage history
S/W VV Results
43
Outline

• SBA Background
• MS Technology to Support SBA
• VVA of MS for SBA
• VVA Background
• Cost of VVA
• VVA Procedural Recommendations
• Risk-focused Strategy
• Managed Investment Strategy
• Conclusions

44
Putting the A in Front of VV!

• Individual MS and agency accreditation plans may
  be unique, but the MS VV activities selected
  for execution should provide essential,
  fundamental information about the simulation to
  support MS accreditation decisions.
• The VVA goal is to establish that MS produces
  realistic, unbiased, credible measurements of
  performance when operated within a specific
  domain of scenario and environmental conditions
  for it to be acceptable (accredited) for use.
• As a consequence, accreditation must be
  theprimary objective in the definition of the
  MS VV activities.

A
The planning for A focuses VV execution.
45
Requirements Flow-Down

• MS VVA programs can be defined top-down
• Accreditation-decision information needs drive
  VV data products
• Data requirements are contingent on accreditation
  scope
• Requirements are flowed down to VV activities
• ...In a perfect world...
• And, Executed from the bottom-up

46
Managing the VVA Investment

• As in MS itself, the specification of scope and
  detail of accreditation is problematic
• Managed investment addresses the problem of
  specifying the scope and detail of MS VVA
  activity
• It allows near-optimal investment in assessment
  activities and products for an economically
  constrained environment
• A managed investment means the deliberate,
  progressive, marginal investment in information
  valuable for support of accreditation decisions

While no a priori determination will provide
closed-form guidance on how much is enough, the
proposed algorithm will allow investment in VVA
activities and products to be made in a
technically complex, dynamic, time-distributed,
economically constrained environment.
47
Using A Managed Investment Strategy

• Managed Investment is the execution of a
  carefully selected subset of VV activities
• Offering the best return on investment by
  providing the essential information necessary
  from VV findings
• Providing the required evidence supporting the
  accreditation decisions of the accreditation
  authorities
• This approach considers cost as an independent
  variable during the selection and execution of
  VVA assessment activities
• An optimal subset of VVA activities can then be
  chosen based upon the
• Information and data needs of the accreditation
  authority
• Realities of the program (schedule)
• Fixed resources (budget) available for VV
  activities
• Risk tolerance of the program

48
What Is The VVA Evaluation Activity Space?

• Another concept key VVA strategy that supports
  program definition is a familiar one
• It is the systems engineer's multi-dimensional
  view of the enterprise whose dimensions exhaust
  the important attributes of the conceptual space
• The recommended evaluation spacedimensions
  consist of
• Unit-under-test (UUT)
• VV activity
• VV agent
• that yield a VV product
• The VVA Program domain-of-interest is comprised
  of the most cost-effective set of cells in the
  Evaluation Activity Space
• This explicit activity domain-of-interest assures
  complete, systematic evaluation and intelligent
  choices within each dimension (UUT, Activity,
  Agent, Product)

VV PRODUCT(S)
UNIT UNDER TEST
VV AGENT
VV ACTIVITY
49
Managed InvestmentApplied To MS VVA
50
MS VVAProgram Specification

• MS VVA program specification includes

51
Outline

• SBA Background
• MS Technology to Support SBA
• VVA of MS for SBA
• VVA Background
• Cost of VVA
• VVA Procedural Recommendations
• Risk-focused Strategy
• Managed Investment Strategy
• Conclusions

52
Lessons Learned

• Accreditation is the key to making VVA cost
  effective
• Accreditation agents are key
• Help the accreditation authority articulate
  requirements
• Communicate those requirements to the MS
  developers
• Develop the accreditation recommendations
• VV agents are key
• Help discern necessary VV activities based on
  accreditation requirements
• Focus the activities to reduce costs
• Ensure appropriate documentation is produced to
  support accreditation

53
Some Challenges Remain

• Cost and resource requirements for MS VV are
  not well understood
• Meaningful cost metrics are not widely shared
  within MS communities.
• Much more information about cost and resource
  requirements needs to be collected
• More reliable cost estimation processes are
  needed
• Data limitations required for effective VV have
  to be addressed
• Management processes should be common across
  simulation applications
• Should address required data and detailed
  characterization of associated uncertainties and
  errors, simulation/software artifacts, etc.

The Workshop on Foundations for Modeling and
Simulation (MS) Verification and Validation
(VV) in the 21st Century, better known as
Foundations 02, sponsored by the Defense
Modeling and Simulation Office, and held October
22-24, 2002 at the Johns Hopkins University
Applied Physics Laboratory in Laurel, Maryland.
54
Some Challenges Remain

• Effective communication is a problem
• There continues to be differences in the details
  about terminology, concepts, and VV paradigms
  among various MS communities
• Excessive use of acronyms makes it difficult to
  communicate easily across community boundaries
• MS VV needs to employ more formal (repeatable
  and rigorous) methods to facilitate better
  judgments about appropriateness of simulation
  capabilities for intended uses
• Advances in MS frameworks / theory can enhance
  VV capabilities, and is essential for increasing
  automated VV techniques

The Workshop on Foundations for Modeling and
Simulation (MS) Verification and Validation
(VV) in the 21st Century, better known as
Foundations 02, sponsored by the Defense
Modeling and Simulation Office, and held October
22-24, 2002 at the Johns Hopkins University
Applied Physics Laboratory in Laurel, Maryland.
55
LIVE THE VISION by Bill Waite
Imagination is more important than knowledge.
Albert Einstein

• Conceive in general
• Be liberal in imagination, because
• The big picture matters
• Execute in particular
• Everything should follow from the vision, but
• Only the real work counts

56
(No Transcript)