Showcase1 Scenario

1
Enhancing Usability and Performance of NESPOLE!
a Speech-to-Speech Translation System for
E-commerce Applications
Alon Lavie Florian Metze Fabio
Pianesi Carnegie Mellon University Universität
Karlsruhe ITC-irst USA Germany Italy
Showcase-1 Scenario
Multi-modal Components and User Interface

• Interface display consists of four windows
• Explorer Web Browser that allows shared web pages
• NetMeeting video-conferencing application
• AeWhiteboard shared whiteboard for maps and
  images, supporting gestures
• Nespole! Monitor for visually monitoring status
  of conversation/translation
• NESPOLE! Monitor displays
• Transcript of users spoken utterance (output of
  ASR)
• Paraphrase translation (back generation from
  interlingua)
• Translated text of utterance spoken by the other
  party

Multi-modality Experiment

• Goal evaluate effect of multi-modality on
  communication and usability
• Experiment collected 16 dialogues 8 with
  multi-modality and 8 without
• Repeated for English/Italian and for
  German/Italian (16 dialogues each)
• MM dialogues gestures preceded by verbal cues,
  very few deictic expressions
• With MM fewer repeated turns, smoother
  progression, fewer ambiguities

Network Traffic Impact
General Architecture of NESPOLE!

• Experiment to evaluate how does packet loss
  affect ASR performance?
• Fed same audio through system from CMU and UKA
  multiple days/times
• Packet loss ratios range from 0.1 to 5.2
• ASR WARs ranged between 59-63, without a clear
  correlation to packet loss

End-to-End Evaluation

• Multiple graders (at least 3 per set), averaged
  results
• Monolingual (paraphrase) and crosslingual
  translation
• Three-scale grade Perfect/OK/Bad.
  AcceptablePerfectOK
• Percent of SDUs Acceptable shown below