Multinational Experiment 4 on Effects Based Approach to Operations A CFEC Report

1
Multinational Experiment 4 onEffects Based
Approach to OperationsA CFEC Report

• Philip S. E. Farrell, Ph.D.
• A presentation for IAMWG Meeting
• Friday 11 May 2007

2
Presentation

• Introduction Human Experimentation
• MNE 4 CFEC Report
• Discussion Can war-fighting experimentation be
  done in a controlled laboratory setting?

3
Introduction

• Not The Bear
• Now The Ball of Snakes
• New concepts need to be tested

4
Scientific Method

• High School Science Report
• Background
• Purpose
• Hypothesis (related to concept)
• Analysis
• Apparatus
• Procedures
• Results
• Discussion
• Conclusions

5
Human Experimentation
6
Human Experimentation
Job Similarity Index concept
Crew Selection Experiment (2006)
TTPs Recommendations
7
Human Experimentation in MNE 4

• Too many people?
• Too many variables?
• Too many confounds?
• Too many constraints?

8
MNE 4 Experiment

• Effects Based Approaches to Operations
• Refinement and Assessment of EBAO POT

actions
disturbances
Planning (desired effects)
Execution
Desired End State

desired actions


_
Operational Environment
World states
Assessment (effects)
Current situations
9
MNE 4 Experiment (CONOPS 0.9)
Planning
RG
Execution
Ra
Rs
Re
A
D





ESA
ED
ADRM
SPR
PREPO
CC
_
Ca
_
_
Cs

IDI action comparators
CA situation and effects comparators
Ce

EBAP
W
Analysis Function KBD, SOSA, KM
S
AAMOP
MMA
AAMOE
QCE
Assessment
10
Control Theory Model for any operation
Disturbances (adversaries actions Neutrals act
ions
Natural forces)
desired mid-level states (i.e., desired effects
)
Highest-level Desired State (i.e., desired End S
tate
Or desired situation)
Desired Actions
actions





G21
Gaa

Gan
G1s
Defence
Diplomacy
_
_
_
_
Development
Decision Makers
Executors
Planners
W
Assessors
S
H12
Hna
Hs1
Has
Feedback on Highest-level State (I.e., situatio
n)
Feedback on mid-level states (i.e., effects)
Lowest-level world states (i.e., sensory inform
ation)
Feedback on states due to Actions
David Snowden Complex acts of knowing paradox a
nd descriptive self-awareness Journal of Knowledg
e Management
Vol. 6 No. 2 2000 p100-111
11
MNE 4 Experiment
Experiment Aim
Experiment Objectives
Hypothesis
Critical Operational Issues
Measures
Data
Experiment Execution
Data Collection
12
MNE 4 Experiment

• Competing Aims!
• to explore concepts and supporting tools for
  EBAO
  
• to address the Canadian MNE 4 objectives

13
MNE 4 Experiment

• Canadian Objectives
• Refine, assess, and analyze elements of EBAO that
  Canada and the CF could apply to achieve
  integrated strategic effort in operations.
  
• Determine whether EBAO represents a means of
  applying the Canadian 3-D approach to operations
  
• Develop and refine a Multinational Knowledge
  Management Plan that can be tailored and applied
  to meet national operational requirements

14
MNE 4 Experiment

• The Hypothesis of this experiment was If an
  operational commander and staff employ effects
  based operations, then they will have the ability
  to
• Create a holistic understanding (KBD)
• Develop and refine an operational plan (EBP)
• Harmonize the full spectrum of DIME planning
  (MNIG)
  
• Coordinate, direct, and monitor operations (EBE)
• Assess actions, effects, and end state (EBA)

15
MNE 4 Experiment
16
MNE 4 Experiment

• Experimental Design

17
MNE 4 Experiment

• Types of Analyses
• Observations
• JOT (Joint Observation Tool)
• JDCAT (Joint Battle Centre Data Collection
  Analysis Tool)

18
MNE 4 Experiment

• Types of Analyses
• User surveys and network data

19
MNE 4 Experiment

• Types of Analyses
• Box Plots (KBD)

Figure C.1 Distribution of correct responses for
SA surveys, by organization
20
MNE 4 Experiment

• Types of Analyses
• Multi-Dimensional Scaling (KBD)

Figure C.4 Multi-dimensional scaling of KBD3
answers by Group
Reference Point (KBD 3)
Reported same incorrect answers
21
MNE 4 Experiment

• Types of Analyses
• Count data (KM)

22
MNE 4 Experiment

• Types of Analyses
• Whitney U (Culture)

23
MNE 4 Experiment

• Types of Analyses
• Vector Method (Common Intent)

24
MNE 4 Experiment

• Types of Analyses
• Vector Method (Common Intent)

Table 3. Angle between vector pairs in degrees
(Duncan Farrell, 2007)
25
MNE 4 Experiment

• Results, Discussion, Conclusions, and
  Recommendations are just as fuzzy as the
  objectives and hypothesis.

26
MNE 4 Experiment

• Insights related to Canadian Objectives
• The notion of planning, execution, and assessment
  in an iterative fashion until the Current State
  reaches the End State is a robust concept.
  
• The relationship between MNIG, EBAO, and the 3-D
  concepts requires further development and
  experimentation.
  
• Some of the basic KM and KBD definitions and
  concepts still have not been resolved.

27
MNE 4 Experiment

• Recommendations
• More research is required!

28
Discussion

• Can war-fighting experimentation be done in a
  laboratory controlled setting?

29
Discussion

• Military buy-in vs. Scientific reputation
• Do you need SMEs to conduct the experiment?
• Military want to know if the new concept works,
  while the scientist wants to develop a model so
  to make predictions with confidence.
  
• Need SME diversity
• Solve as much of the problem with modelling and
  simulation, and smaller experiment.
  
• It depends on the questions being asked!

30
Discussion

• Human Experimentation Ethics
• Is voluntary consent necessary?
• The international Tri-council on Human Ethics for
  experimentation says, yes!
  
• If there is any coersion in obtaining the data
  (includes volunteering as part of a job
  requirement) then the data can not be trusted.
  
• Strictly, if consent forms are not signed, then
  the results should not be published in a
  reputable journal Hey, thats science for ya!

31
Discussion

• political Considerations
• Who makes the decisions?
• Note that it is a small p political
  considerations.
  
• The issue is that the experimental design and
  ultimately the data can be very different if the
  person who holds the purse strings calls the
  shots, or if the scientist who is well versed in
  experimentation calls the shots.
• We have been trying to find some compromise, but
  the fact that were not able to generate clean
  data nor clear recommendations might indicate
  that were not succeeding.

32
Discussion

• Concept complexity
• Are there too many variables?
• The issue is not whether there are too many
  variables. The issue is are we prepared to do a
  proper Latin square design of an experiment that
  leads to an ANOVA- or MANOVA-like analysis.
• The more variables one investigates, the more
  runs are required. It is a formidable task to do
  more than one run in large warfighting
  experiments.

33
Discussion

• Experimental Design
• Whatever happened to Hypothesis-testing?
• The issue of baseline was discussed at length
  both during MNE 4 meetings as well as the IAMWG
  presentation.
  
• It is not a simple, straightforward task to
  generate a baseline, unless it occurs in
  conjunction with the actual event. But this
  becomes an expensive proposition.

34
Summary

• Human Experimentation is not new.
• MNE 4 has refined our processes for war-fighting
  experimentation.
  
• We need to pay attention to the scientific method
  if our recommendations are to be credible

35
Questions