Affect Sensing in Speech: Studying Fusion of Linguistic and Acoustic Features

1
Affect Sensing in Speech Studying Fusion of
Linguistic and Acoustic Features

• Alexander Osherenko, Elisabeth André, Thurid Vogt
• University of Augsburg

2
Affect Sensing

• Acoustic information
• Linguistic information (lexical, stylometric,
  deictic)

3
Fusion

• Decision-level
• Feature-level

4
Research Questions

• Fusion
• Context
• Decision-level vs. feature-level

5
Experimental Setting

• SAL corpus, 574 turns, 5 classes
• Decision-level using majority, feature-level
  fusing features
• Data 2 stages (history 0 and history 7)
• Acoustic modality - 2 (discrete/continuous)
  acoustic datasets (A)
• Lingustic modality - 29 lexical (L), 31
  stylometric (S), 63 deictic datasets (D)

6
Results representation

• Tree
• Nodes features from particular modalities (A,
  L,S, D)
• Values
• Maximal recall value
• Maximal multimodality value
• Dotted arcs

7
Decision-level Fusion Before Discretization

• Best results 64.2 (history 7) and 44.2
  (history 0)
• Significant improvement through context
• Insignificant improvement through fusion (about
  2)
• Maximal multimodality value (76.5)

8
Decision-level Fusion After Discretization

• Best results 66.0 (history 7) and 49.0
  (history 0)
• Significant improvement through context
• Insignificant improvement through fusion (about
  2)
• Maximal multimodality value (77.8)

9
Feature-level Fusion Before Discretization

• Best results 62.8 vs. 64.2 (history 7) and
  46.7 vs. 44.2 (history 0)
• Significant improvement through context
• Insignificant improvement through fusion (about
  2)

10
Feature-level Fusion After Discretization

• Best results 67.5 vs. 64.9 (history 7) and
  52.8 vs. 45.9 (history 0)
• Significant improvement through context
• Insignificant improvement through fusion (about
  2)

11
Discussion

• Role of context
• Role of discretization
• Fusion?

12
Future work

• New modalities
• Weighting