Measuring the Quality of Decisionmaking and Planning

1
Measuring the Quality of Decisionmaking and
Planning

• Framed in the Context of IBC Experimentation
• February 9, 2007
• Evidence Based Research, Inc.

2
Todays Ground Rules

• IBC provides the framework for the discussion on
  measuring decisionmaking.
• This is not an IBC brief.
• The briefer does not have the answers!
• The purpose is to explore how one measures the
  quality of decisionmaking in complex environments
  this is a discussion session.

3
Background Regarding IBC

• Focus
• Visualization, operational assessment, and
  planning tools for EBO.
• Multiphased effort
• Phase 0 Proof of concept using ad hoc suite of
  tools.
• Phase I Demonstrated improvement potential of
  federated tool sets developed by two contractors.
  
• Phase II Evolve tool set to expanded
  capabilities for transition to JFCOM COCOMS.

4
Objectives of Phase II Experimentation

• Establish the operational utility of the tools.
• Learn which tools can be helpful under what
  circumstances and for what purposes.
• Explore novel approaches throughout the spiral
  development process.
• Measure progress in increasing readiness of the
  technology.
• Facilitate early transition to operational users.

5
Challenges

• Complex operational problem
• Wide range of situations,
• Multiple knowledge domains, and
• Poorly understood relationships.
• Major technical challenges
• Variation in maturity of underlying models and
  tools,
• Limitations in availability of relevant data, and
• Composability of federated capability from
  continually expanding set of tools.
• Transition hurdles
• Operational risk due to high potential for
  limitations in fidelity and possible
  misapplication.

6
Option Exploration Tool (1)

• Comprehensive family of models.
• Action-to-effect and effect-to-action.
• Generates the distribution of plausible
  outcomes.
• Visualization assist leaders to understand and
  act.

7
Option Exploration Tool (2)
Modeling Paradigm
Conceptual Models
Spread Sheet
Political/Economic Model
Concept Map
Statistical Models
Social Network
Differential Equations
Influence Diagram
Political/ Religious Model
Causal Models
Petri net
Bayesian Network
Social/ Culture Model
Object Models
Economic/Infrastructure Model
Event-based Simulation
Agent Based Simulation
DIME Dimensions
Social Information Model
PMESII Dimensions
Religious Model
Military Model
Enables What-if Analysis, Answers Why
Distribution authorized only to US Government
Agencies and their contractors involved in Phase
2 of the Integrated Battle Command Program
8
Campaign Planning Tools (1)

• Supports modular development of multiple lines of
  effort.
• Automatically detects and displays
  interdependencies, assumptions, resources,
  actions, duration of effects, metrics, and next
  state.
• Ability to modify plans, actions,
  interdependencies, models, and next states based
  on measured performance.

9
Campaign Planning Tools (2)

Political/ Strategy
Security Strategy
Interdependencies
Assumptions

Political-Economic Strategy
Actions
Next State
Visualization
Resources
Measured Performance
Allows the leader to visualize complex plans
Distribution authorized only to US Government
Agencies and their contractors involved in Phase
2 of the Integrated Battle Command Program
10
Network Centric Operations
11
Decisionmaking Quality

• Two approaches
• Quality of the process
• How the decision was made, and
• Elements in the process.
• Quality of the decision
• Was it right or at least not wrong?
• Did it have the desired effect?

12
Quality of the Process

• Correct sequence of actions.
• Complete set of choices.
• Assumes an observable process.
• Repeatability.
• Outcome is irrelevant.

13
Quality of the Decision

• Applicability to the solution space.
• Compatibility with other SMEs.
• Desired outcome/effects.
• Outcome is relevant.

14
Measuring Quality

• The field needs a way to measure the quality of
  a product
• What does it mean to do good?
• What does it mean to make a good decision?
• A simple problem
• Has a correct answer.
• A complex problem
• Has no right answer,
• May have not wrong answers, and
• Will have multiple perspectives.

15
Discussion Time

• Thoughts?