ECAs and HumanHuman Interaction

1
ECAs and Human-Human Interaction

• Members
• Elisabeth André
• Jean-Claude Martin
• Nuno Otero
• Matthias Rehm
• Zsofa Ruttkay (partially)

2
Design and Evaluation Paradigms

• Krahmer
• Analysis-by-Synthesis
• Analysis-by-Observation
• Cassell
• Analysis-Design-Evaluation Cycle
• Michael Kipp
• Gesture Generation by Imitation

3
Possible Sources of Information

• Literature experimental results/ models
• Corpora informal annotation/ micro-analysis/macro
  -analysis
• Motion capture

4
What Patterns of Behavior can you Extract?

• Timing and Sequencing of behaviors
• Model of the context of use of monomodal and
  multimodal behavior
• Monomodal features types/ number of gestures
• Multimodal features redundancy/ complementarity

5
What Patterns of Behavior can you Extract?

• How close to the individual ? Individual
  behaviors (most frequent gestures)
• Average/subgroups/cultural behaviors

6
Why Use a Corpus?

• Corpus
• Application and context-dependent source of
  knowledge on multimodal behaviors
• Replay function for validating / visualising
  corpus annotation
• Literature on verbal and non-verbal behaviour
  (bidirectional influence)
• Consult literature for definition of annotation
  schemes etc.
• Compare results with findings in literature
• Combine Corpus literature

7
Topics of the Working Group

• Use/design of mm human corpora for the spec of
  ECA
• Limitations due to the ECA ? drive the annotation
  level/coding scheme
• A corpus as a way to evaluate/compare the
  capabilities of several ECAs
• Unified specification language for ECAs derived
  from corpus?
• Design of a questionnaire for the Dagstuhl
  audience
• Software issues plug-in for replay
• Collection of links

8
Questions to be addressed

• What ECA-specific annotation schemes exist?
• To what theories do they correspond?
• What qualitative measurements exists?
• How can we use the results for the creation of
  ECAs
• What are the criteria for evaluating the
  resulting corpus-based ECA

9
From Questionnaire Tools to categorize users

• Standard psychological tests
• Mood/emotion
• Attitude
• Personality (Extraversion, etc.)
• Lies
• Pre-test (questionnaire)
• Familiarity with technology
• Domain knowledge
• Demographic user information like age and gender

10
Informing the Design of ECAs
1 Own observations of humans 87.1 2 Consult the
literature 87.1 3 Discussion with designers
45.2 4 Other 12.9
11
Evaluation Criteria
1 User objective/subjective response to ECA 100 2
Comparing the ECA behavior with human behavior
25.8 3 Catalogue of Criteria 54.8 4 Other 12.9
12
Use of Corpora
How many used corpus 55.6 1 Extract ECA Behavior
36.7 2 Generate ECA Behavior 23.3 3 Other 23.3
13
From Questionnaire Are Corpora Helpful for ECA
Research?

• Ground ECA behavior in empircal data
• multimodal model of interaction
  analysis/synthesis
• combined verbal and non-verbal behaviors, e.g.,
  match speech and facial expressions
• listening behaviors
• relevant idle behaviors

14
From Questionnaire Use/Sharing of corpora
helpful for ECA research?

• Evaluate ECAs
• Comparison between ECAs
• Comparison of ECA behavior with recorded data of
  human behavior
• Point/system of reference
• Automatic extraction of behaviors
• Generation of individual characteristics
• No (too specific, one person)

15
From Questionnaire Kinds of Information Source

• Recordings of users (presentation, dialogue)
• Wizard of Oz
• Movies
• TV-Shows
• News
• Recordings of Actors
• Motion tracking

16
From Questionnaire Available Corpora

• ISLE floorplan corpus (verbal and non-verbal
  deictics) (Bernsen)
• Acted emotions (Pandzic)

17
Use of Annotation Tools
18
From Questionnaire Annotation tools schemes

• Tools
• Home made (4)
• Anvil (2), Praat (1), Tasx (2), WaveSurfer (1)
• Schemes
• Home made (10)
• NITE, HamNoSys, DAMSL, FACS, DISCOUNT, Binary
  coding

19
Limitations of using corpora for ECA design

• Re-use of corpora problematic
• To specific
• Which parts of a corpus can be re-used?
• Still unclear to build a model out of the corpus
• Still unclear how to map the results of the
  corpus analysis onto ECA behaviours.
• How much HH corpus is applicable for HCI
• Cartoon exagerated behaviors
• User expectations of ECAs in an application ?
  human human behavior
•  Bad examples  of communicative behavior

20
Chances of Using Corpora for ECA Design

• Grounding the ECAs behaviour by empirical data
• Application and context-dependent source of
  knowledge on fine-grained multimodal interactive
  behaviours
• More objective research methodology
• Replay function for validating / visualising
  corpus annotation
• A corpus as a way to evaluate / compare the
  capabilities of several ECAs

21
Chances of Using Corpora for ECA Design

• Extract implicit knowledge (for example, Disney
  Designers)
• Supports the realization of individual and
  stereotype behaviours (e.g. personality-specific,
  culture-specific, role-specific etc.)
• Employ knowledge from other areas, such as speech
  processing, statistics

22
Practical Issues

• Which level of granularity of annotation?
• Can we automate part of the annotation process?
• What is the best tool for ECA?
• Corpora distribution
• Re-use of corpora

23
Burning/pending Issues

• Annotation cost
• Sharing of corpora/integration of schemes
• Evaluation of corpora/coding scheme
• Selection criteria application context 

24
Burning/pending Issues

• How to build a model out of a corpus?
• How to generate ECA behaviors from corpus?
• How to evaluate corpus-based ECA?
• Potential Approach for Cultural-Specific ECA
  Standardized staged or natural situations (e.g.
  asking for directions in different countries)