SFBMT and Tool Development for Acquisition of SuperFunctions

1
Language Understanding and Affective Information
Processing
Faculty of Engineering, The University of
Tokushima Fuji Ren
2
Outline

• Artificial Intelligence (AI) 50 Years
• Why Study Emotion? Why Research Affective
  Information Processing?
• How to Simulate the Human Brain? Mental State
  Transition Network (MSTN)
• Build a MSTN - Psychological Experiments
• Approach for Recognizing Human Emotion
• What is Emotion Energy? How to Acquire Emotion
  Energies? ( from Phonetic Information from
  Facial Expression from Speech Patterns)
• Affective Interface and Applications

3
Artificial Intelligence (AI) 50 Years

• Thanks to Drs. J. McCarthy, A. Newell, H. Simon,
  C. Shannon, M. Minsky and many others, a new
  discipline in modern science and technology,
  Artificial Intelligence, AI in brief, was born at
  Dartmouth 50 years ago.
• The major idea of AI is to use technical means
  for simulating the functions of human brain, one
  of the most complicated, powerful, and
  mysterious, systems seen in the real world so far
  and this opened up a new age in scientific
  research in human history.

4
Turing Test1950 "Computing machinery and
intelligence"
a human judge engages in a natural language
conversation with two other parties, one a human
and the other a machine if the judge cannot
reliably tell which is which, then the machine is
said to pass the test. It is assumed that both
the human and the machine try to appear human. In
order to keep the test setting simple and
universal, the conversation is usually limited to
a text-only channel such as a teletype machine as
Turing suggested
human?
human
machine
wall
?????.
5
Turing Test one viewpoint was missing
The partner beyond the wall is not seen The
partners action and emotion (gesture,
expression, etc.) are ignored
YES? NO?
Kekkodesu
human
machine
wall
it can not be judged accurately whether or not
true AI was realized.
6
Why Study Emotion? Why Research Affective
Information Processing?

• What is Information?
• Verbal Date, Message.
• Non-Verbal Emotion, Expression
• Modern information processing mainly focuses on
  verbal information and to a lesser degree deals
  with human emotions

Emotion
Human - human
Data
Human - machine
Lack of emotion
7
Information Entropy?
Based on the same point of view, persuasion by a
boy/girl friend would be much more effective than
by others. These real-life examples prove the
importance of considering emotion.
Data
Emotion
Human - machine
Human - human
e(information) f(verbal inf, non-verbal inf)
8
Affective Computing and Artificial Intelligence
Intelligent
Emotion
Inconsistent
an integrated object in a high level
9
Research and Development of an Affective
Interface which can Recognize Human Emotion and
Create Artificial Emotion
Research Representative Fuji Ren, The
University of Tokushima
Application for U-Learning
Application for Robot
Application for Medical Care
Recognize the learners emotion and provide the
study method most suitable for each learner in an
ubiquitous learning environment.
Understand the Patients facial expression to
make an improvement allowing for the most
suitable medical treatment.
Understand the emotion from the users facial
expression or voice to change the conversation
depending on the emotional state. This
application is expected to be useful for care of
the sick or the elderly.
Application for Emotion Telecommunication
Affective Interface
Emotion Recognition by Facial Expression
Application for Information Terminal
Emotion Recognition by Human
Emotion Generation by Machine
Affective Interface
Affective Interface
Respond appropriately to the users emotion by
using voice recognition terminals
Mental State Transition Network
Modern information communication focuses on
verbal information and to a lesser degree deals
with the human emotions in the message. Create
future communication which can deal with verbal
and non-verbal emotion information.
Emotion Recognition by Phonation
Emotion Recognition based on Wording
Application for Call Center
Application for Games
Understand the users emotion and provide verbal
response to the users question to realize high
quality service at unattended call centers.
Understand the users emotion status to
appropriately change the story development
realizing a new style of entertainment.
Affective Interface
10
Human Emotion Recognition Engine
11
Sensibility Language (Speech Patterns)
External DB
Ontology
Sensibility Sound Voice
Machine (quasi-human) Emotion Creation Engine
Emotion Faces Gestures
Human Emotions
Mental State Transition Network
Virtual Personality
Machine Emotion Creation Engine
12
affective Interface
13
How to Simulate Human Brain?

• A mental state transition networkof seven
  discrete emotional states
• A new method to predict the emotional
  transitional probability distributionby
  psychological experiments

This approach does not solve the mechanism of
humans brain but reasons the emotion state
situation using a black box.
14
Mental State Transition Network and Psychological
Experiments
15
Outline (MSTN)

• Abstract
• Introduction
• Affective Interface
• Mental State Transition Network
• Psychological Questionnaire Experiment
• Experiment Result Analysis
• Conclusions

16
Introduction 1

• Verbal Information
• Non-verbal Information
• Emotion recognition
• Rhythms from voice features, divide voice
  recognition dictionary.
• Natural interaction
• Semantic recognition of voice and interaction,
  interaction knowledge base and voice synthetics.
• An agent platform for constructing
  general-purpose interaction applications.

17
Introduction 2

• Aim to develop an emotion measurement model.
• Analyze information contained in the brain
  sciences etc.
• Analyze statistical data to derive a
  psychological state transition network.
• Constructing a word model
• An emotion interface, a theoretical system and a
  method of future emotion communication.

18
Mental State Transition Network

• Hypothesis
• Human emotions are placed on several states and
  they transit between several discrete states
  (Mental State)
• Mental states can transit from one state to
  another state on a certain condition
• Frequencies of transition among these states are
  not the same, however, there exists a certain
  expectation value
• Analysis of large data and human personality
  information allow building the MSTN

19
Mental State Transition Network
transition probability
External Emotion information

• MSTN based on seven basic emotional states
• ( happy, surprise, sad, fear, disgust, angry
• and neutral/calm/quiet )

20
Build a MSTN ? Psychological Questionnaire
Experiments (PQE)
How to build a MSTN, or in other words, how to
obtain the probability distribution? We performed
a psychological questionnaire experiment to
address the problem
21
PQE Experimental Condition

• We obtained the probability distribution in our
  model through an experiment using a psychological
  questionnaire experiment.
• We had about 200 participants recruited primarily
  from different high schools and universities in
  China and Japan.

22
PQE Experimental Procedures (1)

• Steps
• 1. individual information of the participant
• sex
• age
• educational level
• nationality
• occupation
• self-character assessment

23
PQE Experimental Procedures (2)

• Steps
• A table which was designed based on seven mental
  states was given.
• An example was set up to tell participants how to
  fill out the table.

24
PQE sample table
?? ?? ??? ?? ?? ?? ??
25
Normalization

• Normalization
• The original items in the table are designed to
  be able to be filled out easily and the
  normalization is necessary.
• Data Analysis
• Means are calculated to evaluate the
  transitional probability distribution of the
  model.

A(i,j) indicates the original frequency filled in
the table

• P(ai aj) is the normalized data

• ai and aj indicate the emotion states

26
Experiment Results AnalysisCPT under happy
stimulus
former
later

• The largest numbers are displayed in happy row
• The second largest numbers are almost displayed
  on calm row
• The lowest numbers occur between contradictory
  emotions

27
UNDER HAPPY STIMULUS
same as stimuli
tendency to calm
contradictory emotions
28
UNDER SAD STIMULUS
29
UNDER FEAR STIMULUS
30
UNDER ANGRY STIMULUS
31
UNDER DISGUST STIMULUS
32
UNDER SURPRISE STIMULUS
33
CPT under neutral stimulus

• The largest numbers are displayed on diagonal
• The second largest numbers are displayed on calm
  row
• The lowest numbers occur between contradictory
  emotions

34
UNDER NEUTRAL STIMULUS
35
Experiment Result Analysis

• when no external stimuli or neutral stimuli, each
  state will most likely remain in that state, but
  the Surprise state will tend to calm.
• the largest probabilities are similar to the
  external stimuli, on the other hand, next mental
  state mostly depend external stimuli.
• each mental state tends to become the calm state.

36
Model Testapproach

• Two comparative tests are carried out by another
  50 questionnaires
• Step 1 compare position
• Test Acomparison of the largest
  transitional probabilities
• Test Bcomparison of the second
  largest transitional probabilities
• Step 2 compare magnitude
• each probability (all)

37
Model Testqualitatively test result of the
Mental State Transition Network
Test A1the largest probabilities Test B2the
second largest probabilities
38
Model Testquantitatively test the model
In quantitative comparison case, the transition
probabilities in model are compared by the test
data
P indicates the probability from sate aj to state
ai in the model
Q indicates the probability from state aj to
state ai be in the test dada
In the experiment the difference between these
probabilities is used to evaluate the model
validity
39
Model Testquantitatively test the model
Difference between model and test data to
evaluate the validity
Pj evaluates the degree of the difference between
the probabilities from j emotion states.
P indicates the average degree of differences of
all states in the model.
The value of all states to evaluate the model
validity
The model validity is 0.843 by the compared test
40
Conclusions (MSTN)

• A psychological model that can be realized in
  engineering is presented
• The transition probabilities of the mental state
  transition network have been learnt and analyzed
• Several transitional constraints were drawn from
  the experiments
• The high precision rate of 0.843 is achieved in
  the model test

41
What is the Emotion Energy?
We call an appearance information an emotion
energy. At present, the emotion energies are
acquired from three parts the linguistic
information the phonetic information the
expressive information.
42
Recognizing Human Emotion
C Fuji Ren 2006
43
How to Acquire Emotion Energies?

• Emotion energy acquired from phonetic information
  
• Emotion energy acquired from expressive
  information
• Emotion energy acquired from speech patterns
• Emotion energy acquired from Brain Waves

44
Emotion energy acquired from Brain Waves and
Phonetic Information
Intonation Rhythm
Brain Waves
45
Emotion energy acquired from speech patterns
46
(No Transcript)
47
Emotion energy acquired from expressive
information
48
(No Transcript)
49
(No Transcript)
50
dynamic side of expression

• Which expression?
• delicate expression cannot be judged using only a
  momentary picture

Surprise?
Fear?
51

• Which expression?

Surprise!!
52
Research and Development of an Affective
Interface which can Recognize Human Emotion and
Create Artificial Emotion
Research Representative Fuji Ren, The
University of Tokushima
Application for U-Learning
Application for Robot
Application for Medical Care
Recognize the learners emotion and provide the
study method most suitable for each learner in an
ubiquitous learning environment.
Understand the Patients facial expression to
make an improvement allowing for the most
suitable medical treatment.
Understand the emotion from the users facial
expression or voice to change the conversation
depending on the emotional state. This
application is expected to be useful for care of
the sick or the elderly.
Application for Emotion Telecommunication
Affective Interface
Emotion Recognition by Facial Expression
Application for Information Terminal
Emotion Recognition by Human
Emotion Generation by Machine
Affective Interface
Affective Interface
Respond appropriately to the users emotion by
using voice recognition terminals
Mental State Transition Network
Modern information communication focuses on
verbal information and to a lesser degree deals
with the human emotions in the message. Create
future communication which can deal with verbal
and non-verbal emotion information.
Emotion Recognition by Phonation
Emotion Recognition based on Wording
Application for Call Center
Application for Games
Understand the users emotion and provide verbal
response to the users question to realize high
quality service at unattended call centers.
Understand the users emotion status to
appropriately change the story development
realizing a new style of entertainment.
Affective Interface