Animation and Streaming of Virtual Assistants for Mobile Applications

1
Animation and Streaming of Virtual
Assistantsfor Mobile Applications

• Markus Kampmann
• Ericsson Eurolab Deutschland GmbH
• Herzogenrath/Aachen

2
Background

• Introduction of 2.5G (GPRS) and 3G (UMTS) mobile
• communication systems
• Mobile services
• Voice services
• Data services
• WAP
• Multimedia services
• MMS (multimedia messaging)
• Multimedia streaming

3
Background

• Mobile data services (mobile Internet)
• Until now only limited success
• Not suitable user interfaces ???
• Data services in the Internet
• News, online banking etc.
• User interface
• Keyboard, mouse
• Large Display
• Mobile terminals
• limited display size
• Keyboard or mouse are difficult to use

4
Background

• New interfaces for mobile users
• First step Usage of voice
• Voice recognition
• Voice portals
• Voice markup language VoiceXML (standardized by
  W3C)
• Next steps
• Usage of graphical capabilities of mobile
  terminals
• Multimodal interfaces
• Standards SALT, W3C (in future)
• Our approach Virtual assistant (Avatar)

5
Virtual assistant

• Animated face/person
• synthesized facial expressions/body movements
  (animation)
• talks to the user (speech synthesis)
• understands what the user says (speech
  recognition)

6
Virtual assistant
Mobile Services
Virtual Assistant
Mobile User

Animation
Control Unit
Speech Synthesis
Speech Recognition
Location of the virtual assistant Terminal or
Network
7
Terminal solution

• Located at the terminal
• Animation
• Speech synthesis
• Speech recognition
• High demands regarding
• Processing power
• Memory size
• Low bit rate transmission of
• Text/phonemes
• Animation parameter

8
Terminal solution

• Prototype on PC
• Facial Animation
• MPEG-4 Facial Animation Standard
• Speech recognition
• Ericsson recognizer
• Microsoft recognizer
• Speech synthesis
• ATT synthetic voices
• Applications
• Medical expert
• Quiz master
• Conversational partner

Demo
9
Terminal solution

• Extension to animated persons
• Body Animation
• Reaction to voice commands

Demo (Body animation pre-recorded)
10
Network solution

• Located in the network
• Animation gt video signal
• Speech synthesis gt audio signal
• Speech recognition gt audio signal
• Higher bit rate streaming of audio/video to the
  terminal
• Low demands for the terminal
• Standard compliant streaming client

11
Streaming over live GPRS
bluetooth
GPRS 3 slots
Vodafone
iPAQ
mobilephone
Video/Audio
I n t e r n e t
Streaming Server
Renderer
12
Network solution

• Two examples (pre-recorded, without interaction)

• News show with virtual news speakers

13
Network solution

• 2. Virtual humans in a personalized story (soap
  opera)
• Group of people sends their facial images to the
  operator
• Avatar of each group member is generated
• Story with animated avatars
• Streaming of the story to the group members

14
Demo Streaming over live GPRS
bluetooth
GPRS 3 slots
Vodafone
iPAQ
phone
Streaming Server
Video/Audio 20 kbps
I n t e r n e t

• Virtual news speaker
• Virtual humans in a personalized story

15
Conclusions

• Virtual assistant
• Facial/body animation
• Speech synthesis/recognition
• New interface to mobile services
• Components located in the terminal/network
• Bit rates
• Demands for the terminal