Seminar Multimodale Rume

1
The FAME Project

• Acronym Facilitating Agent for Multicultural
  Exchange
• Partners
• Universität Karlsruhe, INPG Grenoble, UJF
  Grenoble, ITC-irst Trento, UPC
  Bacelona, SONY Europe, Stuttgart ATLAS
  Barcelona
• Project volume 5.5 M Euro
• Duration 40 months, started October 2001
• More info http//www.fame-project.org

2
The FAME Projekt Facilitating Agent for
Multicultural Exchange

• Volume
• Duration 40 months (since October 2001)
• Financial Volume ? 5,5 Mio.
• currently approx. 30 scientists
• Partners
• Uni Karlsruhe , INPG Grenoble
  , UJF Grenoble,UPC Barcelona , ITC-irst
  Trento , Stuttgart,
  Barcelona

www.fame-project.org
3
Project Goals

• long term vision facilitate communication
  between humans
• reduce the workload on the users of technical
  equipment
• observe humans and their activities in an
  intelligent room and serve as a context-aware
  information butler
• FAME project goal provide and integrate core
  technologies (video and speech perception,
  augmented reality, translation, information
  retrieval) to show feasibility of the concept
• demonstrate system at fair
• scenario 1 (lecture scenario) one person is
  giving a talk or lecture or presentation
• scenario 2 (meeting scenario) several people
  are discussing / working on a common task

4
The FAME Showcases
scenario 1 (presentation)
scenario 2 (meeting)

• use A/V equipment
• intelligent cameraman
• presentation tracking
• summarisation archiving
• translation, crosslingual IR

• augmented reality
• video-based activity tracking
• topic spotting
• information butler
• service planning of fair visit

5
The FAME Demonstrator(at Barcelona Fair Forum
of Cultures 2004)
FAME outside view
reception by FAME-guy
room gives information about spotted topics
meeting inside
people mention topics
6
The FAME Demonstrator(at Barcelona Fair Forum
of Cultures 2004)
at the phicon wall
gestures
the projection table
borrow a camera for photographs of the visit
multimodal input on table output also on the wall
interactive visit planning
7
The FAME Demonstrator(at Barcelona Fair Forum
of Cultures 2004)
Back from the visit in the FAME room dowload ...
... and look at photos
select, print, save photos using phicon
interaction
intelligent cameraman, presentation tracker
take home photos and information about FAME
record testimony
8
Important Components

• multimodal environment
• context-aware intelligent camera-man automatical
  ly track people and their activities
• augmented reality environment move physical
  icons (phicons) on table/wall, and interact with
  projection on table/wall
• spontaneous speech recognition (with distant
  microphones)
• translation and crosslingual information
  retrieval in European-English, Catalan, and
  Spanish
• dialog and context model

9
Multimodal Environment at UKA
Livingroomg
Microphon-Array (Speaker Lokalization)
Smartboard as Projection Wall
Audio Signals
IR-Remote Control
X-10
Illumination
Loudspeakers
Microphones
Several Beamers
TV/Video
4 Cameras
10
Augmented Reality Table

• project virtual reality on real table
• move around physical icons (multiple users)
• interact with projection
• select, move, rotate, resize, delete, change
  color
• write on table, pass notes to others, point to
  items

11
Intelligent Camera Man

• follow speaker while talking and moving
  around
• detect interaction from audience
• zoom on area of interest e.g. when pointing
  somewhere or showing something

12
Lecture Supporter

• track lecture or presentation
• operate FAME room equipment by speech
  commands
• automatically switch slides during presentation
• automatically create transcript of lecture
• create summary, translate to other languages
• record and store all lectures in searchable
  database
• retrieve and browse through previously recorded
  lectures

13
Adaptation Overview
estimate
trigramm- model P(wnwn-1,wn-2)
wichtige Wörter
tf-idf
wichtige Wörter
wichtige Wörter
most frequent 40k words least frequent 20k words
important words
72 classes P(Cnwn-1,wn-2) P(wnCn)
60k vocabulary HUB-4 corpus
presentation- slides
100 Links
add to classes
100 Links
20 fewer errors
100 links
2 contexts
scores
CLASS 32 CLASS 14 CLASS 57 CLASS 6 CLASS 70
top n
perplexity
14
Welcome in Barcelona in Summer 2004