Designing and Evaluating Life-like Agents as Social Actors

1
Designing and Evaluating Life-like Agents as
Social Actors

• Helmut Prendinger
• Dept. of Information and Communication Eng.
• Graduate School of Information Science and
  Technology
• University of Tokyo
• helmut_at_miv.t.u-tokyo.ac.jp
• http//www.miv.t.u-tokyo.ac.jp/helmut/helmut.html

2
Short Bioeducation, experience

• Masters in Logic (1994)
• U. of Salzburg, Austria, Dept. of Logic and
  Philosophy of Science
• Dynamic modal logic (completeness, decidability)
• Non-degree studies in Psychology, Linguistics,
  Literature
• Ph.D. in Artificial Intelligence (1998)
• U. of Salzburg, Dept. of Logic and Philosophy of
  Science and Dept. of Computer Science U. of
  California, Irvine
• Incomplete reasoning (deduction, hypothetical
  reasoning, EBL)
• Post doctoral research
• U. of Tokyo, Ishizuka Lab
• JSPS Fellowship (4/1998-3/2000) Knowledge
  compilation, hypothetical reasoning
• Mirai Kaitaku project (since 4/2000)
  Life-like characters, affective communication
  with animated agents, markup languages for
  animated agents, emotion recognition

3
Social Computingmain objective and task
Social Computing aims to support the tendency of
humans to interact with computers as social
actors. Develop technology that reinforces human
bias towards social interaction by appropriate
feedback in order to improve the communication
between humans and computational devices.
4
Social Computingrealization
Most naturally, social computing can be
realized by using life-like characters.
5
Life-like Characters at Worksample applications
Tutoring, USC
Knowledge Sharing, ATR
Presentation, U. of Tokyo
Sales, DFKI
Entertainment, MIT
6
Life-like Charactersdesiderata

• Life-like characters should be
• emphatic and engaging as tutors
• trustworthy as sales persona
• entertaining and consistent as actors
• stimulating as match-makers
• convincing as presenters
• (in short) social actors
• and competent
• Life-like characters should enable
• effective and natural communication with humans

7
Backgroundcomputers as social actors

• Humans are biased to treat computers like real
  people
• Psychological studies show that people tend to
  treat computers as social actors (like other
  humans)
• Tendency to be nicer in face-to-face
  interactions, ...
• Animated agents may support this tendency if they
  are designed as social actors

Ref. B. Reeves and C. Nass, 1998. The Media
Equation. Cambridge University Press, Cambridge.
8
Animated Agents as Social Actorsrequirements for
life-likeness
Features of Life-like Characters
Artificial Emotional Mind
Embodiment

• Synthetic bodies
• Emotional facial display
• Communicative gestures
• Posture
• Affective voice

• Affect-based response
• Personality
• Response adjusted to social context
• social role awareness
• Adaptive behavior
• social intelligence

9
Outlinedesigning and evaluating life-like
characters

• The mind of life-like agents
• Emotion, social role awareness, attitude change
• Demo - Casino scenario
• Implementation and character behavior scripting

• Evaluating life-like characters
• Using biosignals to detect user emotions
• Experimental study with character-based quiz game

• Book project - character scripting languages and
  applications

10
SCREAM System ArchitectureSCRipting
Emotion-based Agent Minds
11
Appraisal Modulethe cognitive structure of
emotions

• Evaluates external events according to their
  emotional significance for the agent
• Outputs emotions
• joy, distress
• happy for, sorry for
• angry at
• resent, gloat
• 22 in total

Ref. A. Ortony, G. Clore, A. Collins, 1988. The
Cognitive Structure of Emotions. Cambridge
University Press, Cambridge.
12
Social Filter Moduleemotion expression
modulating factors

• Ekman and Friesens facial Display Rules (69)
• Expression and intensity of emotions is governed
  by social and cultural norms
• Brown and Levinson (87) on linguistic style
• Linguistic style is determined by social
  variables power, distance, imposition of speech
  acts

13
Agent Modelcharacter profile, affect processing

• Character Profile
• static and dynamic features
• Static features
• personality traits, standards
• Dynamic features
• goals, beliefs, attitudes
• Attitudes (liking/disliking) are an important
  source of emotions toward other agents
• an agents attitude decides whether it has a
  positive or negative emotion (toward another
  agent)
• happy for resent sorry for gloat
• an agents attitude changes as a result of
  communication
• dependent on affective interaction history

14
Signed Summary Recordcomputing attitude from
affective interaction history
winning emotional states
positive emotions
negative emotions
joy (2)
distress (1)
joy (2)
distress (1)
distress (3)
hope (2)
distress (3)
angry at (2)
good mood(1)
angry at (2)
interaction history
hope (2)
happy for (2)
gloat (1)
Attitude summary value
good mood(1)
??
?
?

gloat (1)
ltemotion, intensitygt pairs
Liking if positive Disliking if negative
happy for (2)
time
Ref. A. Ortony, 1991. Value and emotion. In W.
Kessen, A. Ortony, and F. Craik (eds.),
Memories, Thoughts, and emotions Essays in the
honor of George Mandler. Hillsdale, NJ Erlbaum,
337-353.
15
Updating Attitudeweighted update rule

• If a high-intensity emotion of opposite sign
  occurs e.g., a liked interlocutor makes the
  agent very angry
• Agent ignores inconsistent new information
• Agent updates summary value by giving greater
  weight to inconsistent information (primacy of
  recency, Anderson 65)

?w intensity of (winning) emotion ??, ?
? ,? ?h/r historical/recency weight
?
?
?(Sitn) ??(Sitn?1) ? ?h ?w? (Sitn) ? ?r
?3 (3 ? 0.25) ? (5
? 0.75)
disliking liking h-weight angry
r-weight

• Consequence for future interaction with
  interlocutor
• Momentary disliking new value is active for
  current situation
• Essential disliking new value replaces summary
  record

16
Life-like Agentsmaking them act and speak

• Realization of embodiment
• 2D animation sequences
• Synthetic affective speech
• Technology
• Microsoft Agent package (installed client-side)
• JavaScript based interface in Internet Explorer
• Microsoft Agent package
• Controls to trigger character actions
• Text-to-Speech (TTS) Engine
• Voice recognition
• Multi-modal Presentation Markup Language (MPML)
• Easy-to-use XML-style authoring tool
• Interface with SCREAM system

17
Life-like Characters in Interactionsome demos
Comics Scenario
Casino Scenario
Business Scenario
Life-like characters that change their attitude
during interaction
Animated agents that storify tacit corporate
knowledge
Animated comics actors engaging in developing
social relationships
18
Casino Scenariolife-like characters with
changing attitude

• Animated advisor (Genie)
• Emotion, personality
• Changes attitude dependent on interaction history
  with user
• Dealer (James), player (Al)
• Pre-scripted behavior

Genies Character Profile Personality
specification personality_type(genie,agreeableness
,3). personality_type(genie,extraversion,2).
Social variables specification social_power(genie,
user,0,_). social_distance(genie,user,1,_).
Goals wants(genie,user_wins_game,1,_). wants(genie
,user_follows_advice,4,_). Attitude attitude(gen
ie,user,likes,1,init).

• User in the role of player of Black Jack game

Implemented with MPML and SCREAM
19
Emotional Arcadvisors dominant emotions
depending on attitude
Round 1
Round 2
Round 3
Round 4
Round 5
advisor has agreeable personality
pos. attitude
pos. attitude
neg. attitude
pos. attitude
pos. attitude
ignores advice
ignores advice
ignores advice
follows advice
ignores advice
user looses
user looses
user looses
user looses
user wins
sorry for (4)
distress (4)
gloat (5)
sorry for (5)
good mood (5)
Internal intensity values
advisor has agreeable personality, is socially
slightly distant to user
sorry for (5)
distress (1)
gloat (2)
sorry for (5)
good mood (5)
Intensity values of expressed emotions
20
Implementation
21
Agent Scriptingsimple MPML script
lt!--Example MPML script --gt ltmpmlgt ltscene
idintroduction agentsjames,al,spaceboygt
ltseqgt ltspeak agentjamesgtDo you guys
want to play Black Jack?lt/speakgt ltspeak
agentalgtSure.lt/speakgt ltspeak
agentspaceboygtI will join too.lt/speakgt
ltpargt ltspeak agentalgtReady? You got
enough coupons?lt/speakgt ltact
agentspaceboy actapplause/gt lt/pargt
lt/seqgt lt/scenegt lt/mpmlgt
22
Mind-Body Interfaceinterface SCREAM MPML
lt!--MPML script showing interface with SCREAM
--gt ltmpmlgt ltconsult target.jamesApplet.a
skResponseComAct(james,al,5)gt lttest
valueresponse25gt ltact agentjames
actpleased/gt ltspeak agentjamesgtI am
so happy to hear that.lt/speakgt lt/testgt
lttest valueresponse26gt ltact
agentjames actdecline/gt ltspeak
agentjamesgtWe can talk about that another
time.lt/speakgt lt/testgt
lt/consultgt lt/mpmlgt
23
Alternative Viewsmart characters vs. smart
environments
infers I am happy
environment instructs agent be happy now
acts expresses happiness
perceives game state
tells available behaviors
behavior repository

• Sense-think-act cycle
• Classical AI approach
• Internet softbots search for information on the
  web, robots explore their environment
• All the intelligence is agent-side

• Annotated environments
• Shift from agent intelligence to environment
  intelligence
• Semantic web, ubiquitous computing, affordance
  theory
• Agents and environments can be developed
  independently

24
Outline revisiteddesigning and evaluating
life-like characters

• The mind of life-like agents
• Emotion, social role awareness, attitude change
• Demo - Casino scenario
• Implementation and character behavior scripting

• Evaluating life-like characters
• Using biosignals to detect user emotions
• Experimental study with character-based quiz game

• Book project - character scripting languages and
  applications

25
Affective Computingwhy should a computer
recognize user emotions?

• Human-human communication
• Based on efficient grounding mechanisms including
  the ability to recognize interlocutors emotions
  (frustration, confusion,)
• Humans may react appropriately upon detection of
  an interlocutors emotion (clarification upon
  confusion)
• Human-computer communication
• Computers typically lack ability to recognize
  user emotions
• Ignoring users emotions causes users
  frustration
• Recognizing and responding to users (often)
  negative emotions may improve users interaction
  experience

Ref. R. Picard, 1997. Affective Computing. The
MIT Press.
26
Emotion Recognitionhow can computers recognize
users emotions?

• Stereotypes
• A typical visitor of a casino wants (to win)
• Communicative modalities
• Facial display (face recognition)
• Prosody (speech analysis)
• Linguistic style (NLU)
• Gestures (gesture recognition)
• Posture (posture recognition)
• Physiological data
• Biosignals

27
Physiological Data AssessmentProComp unit
GSR
BVP

• EMG Electromyography
• EEG Electroencephalography
• EKG Electrocardiography
• BVP Blood Volume Pressure
• GSR Galvanic Skin Response
• Respiration
• Temperature

sensors
28
Inferring Emotions from Biosignals Langs
2-dimensional emotion model

• Langs two dimensions
• Valence - positive or negative dimension of
  feeling
• Arousal - degree of intensity of emotional
  response
• Biometric measures
• Skin conductivity increases with arousal (Picard
  97)
• Heart rate increases with negatively valenced
  emotions
• Note
• introverts reach a higher level of emotional
  arousal than extroverts

excited
enraged
joyful
Arousal
sad
relaxed
depressed
Valence
some named emotions in the arousal-valence space
Ref. Lang, P. 1995. The emotion probe Studies
of motivation and attention. American
Psychologist 50(5)372385.
29
Experimental Studyeffects of a character-based
interface
Junichiro Mori - Experimenter Analyser

• Aim of study
• Show that a character with affective expression
  may improve users experience ( reduce
  frustration) of a simple quiz game
• Method
• Biosignals to measure skin conductance and blood
  volume pressure (objective assessment of user
  experience)
• Questionnaire (users subjective assessment)
• Instruction
• Addition/subtraction task (short-term memory
  load)
• Solve a series of 30 quizzes correctly and as
  fast as possible
• Frustration is deliberately caused by delay (in 6
  out 30 quizzes)
• Subjects
• 20 university students (all male Japanese,
  approx. 24 years old)
• JPY 1000.- for participation, JPY 5000.- for best
  score

30
Experimental Setup
31
Instructionmathematical quiz game
timer
It is correct. (polite language)
sometimes delay here (6 14 sec.)

• Add 5 numbers and subtract the i-th number (i lt
  5)
• 1 3 8 5 4 21
• E.g. subtract the 2nd number
• Result 18

• Select the correct answer by clicking the radio
  button next to the number
• Then the character tells whether answer is
  correct

32
Two Versions of the Gameaffective vs.
non-affective (independent variables)
Affective Version Non-Affective Version
Description Description
Character expresses happiness (sorriness) for correct (wrong) answer Character shows empathy (when delay occurs) Character expresses affect both verbal and nonverbal Character does not show affective response Character ignores occurrence of delay
Hypotheses Hypotheses
Character may reduce user stress (SC) and decrease negative valence (heart rate) Character has no significant effect on user emotion (SC, heart rate)
33
Character Responsesexamples of
affective/non-affective feedback
I am sorry. It is wrong. (hyper-polite language)
I am sorry for the delay. (polite language)
Character apologizes for the delay
Hanging shoulder gesture to express sorriness
non-verbally
Non-affective feedback Wrong. No
non-verbal emotion expression.
Non-affective feedback Character ignores the
occurrence of delay.
34
Analyzing Physiological User Data
BVP could not be taken reliably
BVP
user response
agent response
DELAY segment
RESPONSE segment
GSR
delay ends
delay starts
Biograph Software (Thought Technologies)
35
Preliminary Findings9 subjects in each version
(data of 2 subjects discarded)

• Hypothesis (design) delay induces frustration in
  subjects
• All 18 subjects showed significant rise of SC in
  DELAY segment
• Corresponds to finding in behavioral psychology
  (if an individual is prohibited from attaining a
  goal, the individual experiences primary
  frustration)

• Hypothesis (main) affective agent behavior
  reduces user frustration

Non-affective version mean ?0.05 Affective
version mean 0.2 t-test (assuming
unequal variance) t(16)?2.57 p .01
DELAY segment
RESPONSE segment
mean values sf SC (BVP could not be taken
reliably)
Preliminary evaluation suggests that an animated
character expressing emotions and empathy may
undo some of the users frustration.
36
Agents Adapting to User Emotionassumes real-time
recognition of user emotions
Dynamic Decision Network (simplified)
ti1
ti
users action
evidence node
agents actions
user model
users traits
users traits
user model
learning
learning
emotional state
emotional state
U
bodily expressions
bodily expressions
QUESTION Given users state at ti,
which agent action will maximize
agents expected utility at ti1, in terms of,
e.g., users learning and emotion?
sensors
sensors
evidence nodes
37
Dynamics of User Emotions
user personality
user goals
ti1
ti
succeed by myself
provide help
agreeableness
agents action
do nothing
have fun
users emotional state at ti1
reproach
joy
shame
arousal
users emotional state at ti
bodily expressions
sensors
Ref. Conati, C. 2002. Probabilistic assessment
of users emotions in educational games.
Applied Artificial Intelligence 16(7-8)555575.
down(frowning)
high
high
38
Outline revisiteddesigning and evaluating
life-like characters

• The mind of life-like agents
• Emotion, social role awareness, attitude change
• Demo - Casino scenario
• Implementation and character behavior scripting

• Evaluating life-like characters
• Using biosignals to detect user emotions
• Experimental study with character-based quiz game

• Book project - character scripting languages and
  applications

39
Book Projectcharacter scripting languages and
applications

• Wide dissemination of life-like character
  technology requires
• standardized ways to represent the behavior of
  agents
• Book will offer state-of-the-art on XML-based
  markup languages and tools
• Scripting languages for face animation, body
  animation and gestures, emotion expression,
  synthetic speech, interaction with environment,
• Characters are already used in a wide variety of
  applications
• Book contains some of the most successful
  character-based applications
• Synopsis chapters on character design

H. Prendinger, M. Ishizuka (Eds.) Life-like
Characters. Tools, Affective Functions and
Applications Springer Hardcover (in
preparation) useful as Standard/Reference
Book State-of-the-Art in Life-like Agents Course
Book for HCI, HAI, multimedia, life-like
agent applications, scripting languages,
40
Conclusion

• Social Computing
• Human-computer interaction as social interaction
• Designing life-like characters as social actors
• Believability-enhancing agent features
• Emotion, personality, social role awareness,
  attitude change, familarity change
• Casino demo
• Future avenues smart environments (character
  annotated environments)
• Evaluating life-like characters as social actors
• Experimental study using users biosignals
• Life-like characters affective response may undo
  some of the users negative feeling
• Future avenues real-time adaptivity of agent
  behavior to users emotion, decision-theoretic
  approach to agent behavior