ELICIT Experimental Laboratory for Investigating Collaboration, Information-sharing, and Trust

1
ELICIT Experimental Laboratory for Investigating
Collaboration, Information-sharing, and Trust
2
What is ELICIT?

• ELICIT Experimental Laboratory for
  Investigating Collaboration, Information-sharing,
  and Trust
• U.S. DoD (OASD/NII) Command and Control Research
  Program (CCRP) sponsored the design and
  development of the ELICIT platform for
  experimentation focused on information,
  cognitive, and social domain phenomena
• Purpose of ELICIT-related Experimentation and
  Analysis is to investigate the cognitive and
  social impacts of C2 approach and organizational
  structure (e.g. information sharing, trust,
  shared awareness, and task performance)
• Initial applications focus on a comparison of
  traditional hierarchical and edge C2 approaches

3
ELICIT Scenario

• The goal of each set of participants is to build
  situational awareness and identify the who, what,
  when, and where of a pending attack
• Participants can share factoids directly with
  each other or post factoids to websites
• Participants build awareness by gathering and
  analyzing factoids and interacting with one
  another
• Participants receive factoids about a future
  attack
• Factoids fall into four task categories who,
  what, when, and where
• Factoids are periodically distributed to the
  participants
• No one is given sufficient information to solve
  their assigned problem without receiving
  information from others
• The receiving, sharing, and posting of factoids
  and the nature of the interactions between and
  among participants can be constrained

4
Experiment Details

• Sixty-eight (68) total factoids
• 17 when 17 where 17 who 17 what
• Factoids may be key, supportive, or nonessential
• Factoids are distributed in three waves
• 34 factoids at start
• 17 factoids at 5 minutes
• 17 factoids at 10 minutes
• C2 approach for this series of experiments were
  designated prior to the start of the run as
• Hierarchy or Edge
• 17 participants make up an organization
• Output
• Transaction log
• Scratch paper collected
• Survey conducted at the conclusion of the run

Hierarchy
Edge
5
Website Access
Hierarchy
Edge
Team Member

• The commander (gray) can access/post to all four
  websites
• All other participants can only access/post their
  groups website

• Each participant can access/post to all four
  websites

Website
Access to all websites
Where
What
When
Who
6
Illustrative Factoids and Identifies

• Who Factoid The Lion is known to work only with
  the Azur, Brown, or Violet groups (classification
  key fact)
• What Factoid Bloggers are discussing the role of
  financial institutions in oppressing the Coral,
  Violet and Chartreuse groups (classification
  supportive fact)
• When Factoid The Brown group needs time to
  regroup (classification nonessential fact)
• Where Factoid The Azur, Brown, Coral, and
  Violet groups have the capacity to operate in
  Tau, Epsilon, Chi, Psi and Omega-lands
  (classification supportive fact)
• Sample Identify "whoRed Group whatTV station
  whereB-land whenNov 8, 1000 pm
• Sample Correct Solution The Green group plans to
  attack a TV station in A-land on 25 Nov at 11AM

7
Variables of Interest
C2 Maturity Level
Allocation of Decision Rights
Patterns of Interaction
Network Characteristics Performance
Distribution of Information
Shared Information
Shared Awareness
Shared Understanding
Task Performance
Info Sharing Collaborative Behaviors
Quality of Information
Quality of Awareness
Quality of Understanding
Measures of Merit
Task Difficulty
Individual Team Characteristics
Legend
Controllable
Quality of Information Sources
Partially Controllable
Culture
8
Dependent Variables

• MOE Quality of Awareness and Shared Awareness
• Correctness (Authorized Correct IDs)
• Timeliness (Person-Minutes with Correct IDs)
• Accuracy rate (Correct IDs/Total IDs)
• Efficiency, Given Effectiveness
• Productivity (Correct IDs/Total Actions Correct
  IDs/Person-Minutes Available)
• Speed (Time of Earliest Correct ID)
• Agility
• Effectiveness over problem difficulty

9
C2 Approach Independent Variables

• Hierarchy v. Edge
• We expect Hierarchy to map to De-conflicted and
  Edge to map to a more mature level
• Each run will be mapped to a point in the C2
  Approach Space based on observed behaviors
• Rules of Interaction
• Website access
• Sharing permissions
• Initial Distribution of Factoids
• Invariant in existing runs

10
C2 Approach Intervening Variables

• Patterns of Interaction
• Characteristic path length
• Clustering coefficient
• Connectedness
• Distribution of Information
• The average number of unique facts to which each
  participant has access as a function of time

11
Measures of C2 Effectiveness (MOCE)

• Quality of Information Position
• Percentage of relevant facts for the assigned
  task that a participant can access as a function
  of time
• Percentage of key facts for the assigned task
  that a participant can access as a function of
  time
• Extent of Shared Information
• The average number of participants that have
  access to each fact as a function of time
• The average number of participants that have
  access to each key fact as a function of time

12
Intervening Behavioral Variables

• Activity over time (sharing, website posts,
  website pulls, ID attempts)
• Sharing
• Peer-to-peer sharing
• Posting
• Information Seeking
• Pulling
• Identification Attempts

13
Other Independent Variables

• IDs allowed
• ELICIT experience of the player
• Factoid set (problem difficulty)
• Translated factoids v. original
• Native Language (English v. Other)
• Communications media
• Postcards
• Chat
• Time available
• Degree of Education (Graduate, Undergraduate)
• Seniority (Rank)
• Subcultures (Military, Civilian, Special Forces,
  Civil Servants)

14
ELICIT Data Set

• Includes data from 37 ELICIT experimentation
  trials
• Venues
• Boston University (2 runs)
• Naval Postgraduate School (16 runs)
• Portugal (6 runs)
• US Military Academy (3 runs)
• Singapore (10 runs)
• Organization Types
• 18 Edge
• 19 Hierarchy
• Past excursions include
• Chat capability
• Varying the subjects (Intel, Military, PhD)

Future use planned for Germany, UK, SAS-065, NDU,
AWC
15
Extracting Data from ELICIT

• ELICIT generates output in a number of forms
• Participant survey responses and scratch paper
• Transaction logs recording all information
  exchange and identify actions
• While rich in data, transaction logs must be
  processed in order to be formulated for analysis
• Transactions of each type must be extracted and
  tabulated
• Identify attempts must be graded for correctness
• Scripts (Python, C) and macros (Visual Basic)
  were employed to facilitate data structuring for
  analysis
• Software is available for use by others
• Transactions can be parsed and organized into
  time intervals to enable observation of how
  behaviors change over time, and of the timing of
  identifies
• Whole-trial statistics also taken to characterize
  trials, but difficult to compare due to
  differences in trial duration