Building Interactive Virtual Humans for Training Environments

1
Building Interactive Virtual Humans for Training
Environments

• Patrick Kenny, Arno Hartholt, Jonathan Gratch,
• Bill Swartout, David Traum, Stacy Marsella, Diane
  Piepol
• USC Institute for Creative Technologies
• (Patrick Kenny, kenny_at_ict.usc.edu)

I/ITSEC 2007 Nov 26-29 2007, Orlando, FL
2
Outline

• Intro -1 min
• Why virtual humans -2 min
• What are virtual humans 5min
• 3 layers
• Challenges and Research issues building them
  -5min
• Vhuman areas, architecture
• Applications 10 mins
• SASO, C3IT, TacQ, ELECT,
• Evaluations 2 mins
• Conclusions 1 min

3
Introduction

• What are virtual humans and why should you care?
• Research areas in Virtual Humans
• Virtual Human Projects at ICT
• Results and Applications

4
Motivation

• Develop models and theories that try to replicate
  human thought, human behavior, and human actions
  to better understand ourselves
• Use these theories to build effective virtual
  characters that can be used in Training or
  Teaching in virtual worlds
• Integrate many multi-disciplinary research fields
  in AI to advance the state of the art in virtual
  human architectures

5
Transforming the Joint Forces Training

• Most military training simulators are
• vehicle-based
• Tanks,
• Helicopters,
• Fighters,
• Etc
• Teach
• Operation
• Strategy
• Tactics

6
Need for a Human-Oriented Approach

• But a successful mission involves many
  interpersonal issues
• Leadership
• Peacekeeping
• Cultural Awareness
• Negotiation / Coordination
• Social Skills
• Multi-Party Communications

Great need for training person-to-person skills
7
Current Approach Role-Play

• Use role players and actors in mock towns
• While this is effective, its costly and
  requires, role player training and man power
• Cant train all situations
• After Action is not automated

8
Using Virtual Humans for Interpersonal Training

• Augment this role play training with Virtual
  Humans as an alternate form of training

9
What are Virtual Humans

• Computer Generated Characters that
• Look,
• Act and
• Think like real humans
• Realistic human form
• Interact in a natural way with
• Other Humans, Agents and the Environment
• Run in Real-Time

10
Virtual Human Goals

• Research and Build Integrated Virtual Humans
  Systems with advanced AI
• Use them in training environments
• Create characters that are
• Believable Need to look, act like humans
• Responsive Need to interact with humans in
  real-time
• Interpretable Need to communicate with humans,
  humans need to understand them

11
MIND/BODY DUALISM
Mind
Body
World
12
3 Layer Architecture
World
Body
Mind
13
Information Flow
Simulation Side
Virtual Human Side
14
Disassemble a Virtual Human

• What goes on at each of the layers
• What AI research is involved
• What are the challenges and issues

15
Virtual Humans Group at USC Researching
Artificial Intelligence in fields related to
building Artificial Humans
Cognitive and Emotion Modeling
Mind
Task and Domain Reasoning
Dialog and Discourse
Theory of Mind
Natural Language Understanding
16
Virtual Humans Group at USC Researching
Artificial Intelligence in fields related to
building Artificial Humans
Cognitive and Emotion Modeling
Mind
Task and Domain Reasoning
Dialog and Discourse
Theory of Mind
Natural Language Understanding
Procedural Gestures
Body
Non-Verbal Behavior
Speech Recognition
Perception and Vision
17
Virtual Humans Group at USC Researching
Artificial Intelligence in fields related to
building Artificial Humans
Cognitive and Emotion Modeling
Mind
Task and Domain Reasoning
Dialog and Discourse
Theory of Mind
Natural Language Understanding
Simulated Worlds /Real World interaction Integrate
d Systems
World
Procedural Gestures
Body
Non-Verbal Behavior
Speech Recognition
Perception and Vision
18
Mind (Cognitive Layer)

• Supporting Research Areas
• Natural Language Understanding and Generation
• Dialogue Management
• Task and Domain Reasoning
• Emotions and theory of mind modeling

19
Agent Architecture
20
Task Model
21
Emotion Model
22
Dialog Model
23
Cognition / Emotion Model
24
Body (Virtual Human Layer)

• Supporting Research Areas
• Speech Recognition
• Speech Synthesis
• Non-verbal behavior and Gesture generation
  (Smartbody)
• Perception

25
Smartbody Procedural Animation System
26
World (Simulation Layer)

• Integration of Virtual World
• Background Characters

27
Vhuman Setups
Virtual Human
Virtual Environment
Interacts with
Virtual Environments
Virtual Human
Real Environment
Real Human
Real Environments
Real Human
28
Interaction with the Virtual Human
Sonic Speech System With Language Models
developed at USC
Agent and Emotion Model developed at ICT
NVBG Developed at ISI/ICT
SmartBody System developed at ISI/ICT
29
Putting it all together
30
Virtual Human Projects at ICT

• Show Video Montage

31
TacQ Emotional Dialog Modeling
ELECT Bi-Lateral Negotiations
Rapport Evaluation of Nonverbal Behavior Influence
32
Doctors without BordersCognition and Emotion
Models
SASO-ST Bi-Lateral Negotiations
SASO-EN Multi-Lateral Negotiations

• In the SASO system, a trainee interacts with a
  virtual human representing a doctor without
  borders doctor to negotiate the move of the
  clinic, as it is in the middle of a strategic
  operational area.

33
SASO-EN

• Show Video

"The difference between fiction and reality?
Fiction has to make sense. --Tom Clancy
34
Conclusion

• Research and development for Virtual Humans
• Applying Virtual Humans for Training

35
Acknowledgments

• This work was sponsored by the U.S. Army
  Research, Development, and Engineering Command
  (RDECOM), and the content does not necessarily
  reflect the position or the policy of the
  Government, and no official endorsement should be
  inferred.

Mission Rehearsal Exercise Virtual Human Training
Prototype