Tools for automated analysis of networked verbal communication

1
Tools for automated analysis of networked verbal
communication

• Peter W. Foltz
• Pearson Knowledge Technologies
• pfoltz_at_pearsonkt.com
• Research funded by ARI, ARL, DARPA, ONR, AFRL

2

• Why study networks?
• Activity and patterns reflect something
  interesting
• Why study verbal communication in networks?
• Content and patterns reflect effectiveness of
  teamwork, coordination, learning,expertise,
  knowledge, collaboration, interdependence,
  situation awareness, leadership, stress,
  workload, intent, culture, performance
  cognition.
• Need to analyze the content of what is said, by
  whom, to whom, when, in what context.

3
Communication Analysis needs

• Process content and patterns of spoken and
  written communication
• Filter, summarize, visualize
• Convert into performance metrics
• Predict
• Provide alerts
• Give feedback
• Suggest information, Connect the right people

4
Communication Analysis

• Goal Apply computational modeling approaches to
  verbal communication
• Automatically monitor and analyze communication
  streams to understand and assess team performance
  and process
• Improve collaboration within and between
  individuals, organizations and cultures
• Provide feedback and alerts to teams and leaders
  for continual improvement
• Approach
• Provide tools to monitor communication content
  and patterns of interaction
• Use Computational Linguistics/AI/Machine learning
  modeling approach
• Automatically extract measures of performance,
  situation awareness, consensus, shared interest,
  etc.
• Integrate with traditional assessment methods to
  develop objective and descriptive models of
  distributed team performance

5
Goal Communications analysis toolkit
The Communications Analysis Pipeline
Source Materials
Data Preparation
Analysis of Content Patterns
Modeling
Applications Training/Ops
Distributed Mission Communications (Voice)
Speech to Text (ASR)
AI, NLP and other Statistical Analysis of Team
Commo
Modeling of Performance (cognitive, task, Etc.)
Mission Assessment, Debriefing AARs, Training,
Feedback, Critical Incident Detection, Alarms
Distributed Mission Communications (Typed)
6
Analysis of Language Task features
Cognitive, Task and Semantic Modeling and Machine
Learning approaches
Content Latent Semantic Analysis Statistical
properties of language Syntactic features
Performance metrics Collaboration, coordination,
leadership,
Classification Speech acts planning,
uncertainty, acknowledgements Appropriate
Communication Suggested feedback,
Semantic Network patterns Connections Frequencies
Flow patterns and change over time Range of
content flow across network Cohesion/Coherence Flo
w of cdrs intent .
Search, Filtering, Summarization
Speech signal properties Speaker ID Power,
pitch, deltas,
Visuzalization
Domain/Task Properties Vectors for people,
actions, System states,
7
A quick overview of use of tools

• Knowledge Post
• Monitoring and assessing officers in online
  discussion environments
• TeamViz
• Analyzing and visualizing communication in
  multi-national exercises
• DARCAAT
• Team performance prediction and critical event
  detection in convoy operations

8
Application KnowledgePost

• Automated moderation and monitoring in
  discussion-based planning exercises
• Used at Army War College and AF academy
• Summarization and judgments of quality of
  contributions and COAs,
• Suggests information for officers that are going
  off-track
• Alerts instructors for off-topic discussion

9
Significant improved quality of discussion F (2,
113) 9.5, p lt .001
10
TeamViz

• Tools to help monitor communication
• Integrates computational tools to analyze
  communication
• Visualizations provide feedback for improved
  situation awareness of information flow,
  information gaps, information needs
• Tracked flow of commanders intent, effects of
  injects
• Enables knowledge management for large numbers of
  communications
• Exercise
• Develop and evaluate technologies to improve
  cognitive interoperability collaboration among
  joint, interagency, multinational forces
  conducting combat stability operations Two
  runs with
• 130 Singapore Armed Forces (SAF) Staff College
  Students
• US military officers (35ID JFCOM)
• Representatives of US, SN, other government
  agencies and non-government organizations

11
(No Transcript)
12
5 Singapore Signal Officers used TeamViz during
Run 2
useful in a real multinational operation
the contributions graph provides a nice overview
and makes it easy to spot problems at a glance
Can get some indication of commanders intent
we can see if both brigades are talking about the
same info and can determine if both brigades are
on the same page as the CTF.
13
DARCAAT

• Predict team performance and detect critical
  events in convoy operations
• Ambush! virtual and NTC live convoy training
• 100s of hours of communication data, event logs,
  SME ratings
• For a range of domains, tools able to accurately
  predict
• SME ratings (leadership, situation awareness,
  priorities)
• Objective performance measures (mission success,
  errors, amount of training, )

14

• Implications
• Verbal content from networked teams provides a
  window to performance and cognition
• Can convert to metrics, feedback, visualizations,
  predictions
• Challenges/New Directions
• Generalization and scaling up
• Improve language and modeling tools
• Performance metrics
• What are indicators of good/bad/critical
  communication content and patterns?
• Automatic Speech Recognition
• Better use of network modeling algorithms
• Integration/synthesis with other modalities of
  information
• Unified task, state and communication
  representation
• Application beyond Blue Forces