SUCCESS AT COLLABORATION AS A FUNCTION OF KNOWLEDGE DEPTH

1
SUCCESS AT COLLABORATIONAS A FUNCTION OF
KNOWLEDGE DEPTH
Mark A. Sabol, Brooke B. Schaab, and J. Douglas
Dressel U. S. Army Research Institute for the
Behavioral and Social Sciences, Alexandria,
VA Andrea L. Rittman George Mason
University Consortium Research Fellows Program
Presented by Dr. Mark Sabol to the International
Military Testing Association Pensacola,
Florida November 5, 2003
2
Purpose of Research
Within ARI, we work in the Advanced Training
Methods Research Unit General interest in
identifying training variables that influence
how well Soldiers use computer-based/digital
systems Preliminary experiment to identify
specific issues that arise when unacquainted
Soldiers must collaborate at a distance Question
Should training for collaborative tasks
include system-wide aspects or concentrate
on a single role?
3
Overview of Presentation
Describe computer game used as experimental
environment Team tasks, performed by pairs of
players Success requires collaboration Give
experimental design One between pairs variable
(training) Two within pairs variables Present
preliminary results and speculative
interpretation Two 2-way and one 3-way
interactions Touch on just-completed follow-up
experiment
4
Context The SCUDHunt game
Played on the Internet via commercial
computer software developed and administered
by ThoughtLink Incorporated 2009 Cantata
Court Vienna, VA 22182-3313   Separated
but communicating players deploy
information-gathering assets to locate 3
missile launchers on a 5x5 grid.
Reference to and use of this product does not
constitute endorsement by the U. S. government.
5
Basics of the SCUDHunt Game
6
Air Control
Ground Control
SEALS
Satellite
UAV
HUMINT
COMINT
Manned Aircraft
Special Operations
7
Search Results
X
0
?
0
0
8
X
0 0 0 ?
9
Search Results
May have found a Scud
Nothing to report
Suspicious activity, some type of vehicle
Asset extracted (removed)
Disabled Out of action
10
UAV Intelligence Report Actual Probabilities
Situation 2
Situation 1
11

• Can move to any square
• Impervious to destruction
• Fair at detecting Scud- related messages
• May receive false reports planted by enemy

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Definitions
Collaboration actively sharing information,
knowledge, perceptions, or concepts when working
together toward a common purpose (e.g.,
find launchers).   Deep-but-Narrow (DbN) training
two identical sessions on attributes of
assets you control (aka "Own").   Broad-but-Shallo
w (BbS) training cross-training, a
session on your own assets, followed by
equivalent training on your partner's assets (aka
"All").
16
Experimental Design
Independent Variables Between   
Training Depth Deep-but-Narrow
(DbN or "Own") vs.
Broad-but-Shallow (BbS or "All")
Within    Games (1st vs. 2nd) and Turns (5)
Air controller" vs. Ground controller"
positions Dependent Variables of
launchers found with strike plans on each turn
(Also, and type of chat messages sent
not yet analyzed)
17
Results Air controllers
18
Results Ground controllers
19
Results 5th Turn only
20
Analysis of Variance
Highly significant interaction
Position (air vs. ground) x Games (1st vs.
2nd) Air controllers get worse, ground
controllers get better Marginally significant
interactions Position x Training (Own
vs. All) Air controllers better if
All trained, but on average ground
controllers same under either training
Position x Training x Games Own
All trends over games parallel for air
controllers, crossed for ground
controllers
21
Summary
Significance due to flip in performance
among ground controllers In first game, those
given Own (DbN) training did better In second
game, those given All (BbS) training did
better Meanwhile, among air controllers
Those given broad training did better on both
games.
22
Conclusions
The ground controllers seen as representing
Soldiers with a complex job, crucial to
success, but difficult to
master The air controllers seen as
representing Soldiers with a simpler job,
less crucial to success, but
easier to master INITIALLY, DbN training was
best for ground controllers, but EVENTUALLY BbS
training proved best for both positions.
23
Speculations
Deep-but-Narrow (Own or Intensive-) training
leads to a feeling of expertise, which fosters
1) a go it alone mentality 2)
inhibition of collaborative communication Broad-b
ut-Shallow (All or Cross-) training allows
meaningful attention to partners role, which
fosters 1) additional learning of partners
knowledge on-the-job 2) abundant
collaborative communications
24
Preliminary Results for Chat data
25
Further Research
Follow-up experiment, (mostly) completed Summer
2003 Explicit training on when and how to
collaborate Post-game measure of asset
knowledge, to test on-the-job learning
hypothesis All games played using chat
function to better test effects on
collaborative communication First look at
retention of asset knowledge game playing
skills learned tendency to collaborate