Speech and multimodal

1
Speech and multimodal

• Jesse Cirimele

2
papers

• Multimodal interaction Sharon Oviatt
• Designing SpeechActs Yankelovich et al

3
Why multimodal?

• More transparent, flexible, efficient, and
  powerfully expressive means of HCI

4
flexiblility

• Modality choice for different situations
• Modality choice for different functions
• Broader range of users
• Broader range of environments

5
Users prefer multimodal

• For example, 95 to 100 of users preferred to
  interact multimodally when they were free to use
  either speech or pen input in a map-based spatial
  domain (Oviatt, 1997).

6
What do you gain?

• Some speed and efficiency
• Improved error handling
• Simpler language used leads to less recognition
  errors
• Mutual disambiguation of different input modes

7
When do people use multimodal?

• Manipulation spatial information
• High task difficulty
• Communicative complexity

8
Complementary vs redundancy

• Very little redundancy of information
• Cant rely on duplicate information from other
  modalities, but rather use the strengths of some
  modes to overcome the weaknesses of others

9
Multimodal language

• Is often linguistically simpler than spoken
  language
• hard to process disfluent language has been
  observed to decrease by 50 during multimodal
  interaction with a map.
• Often different word ordering different
• LOC-S-V-O instead of S-C-O-LOC

10
GUI vs multimodal

• GUI
• Serial and discrete
• Multimodal
• Parallel and probabalistic

11
SpeechActs

• user-study style paper
• Speech only interface that controls mail,
  calendar, weather, stock quotes, for traveling
  professionals.

12
The study

• 22 tasks accomplished via telephone in a room set
  up to look like a hotel room
• Users tested were traveling professionals (same
  users that would use end system)

13
results

• Users found speechacts promising as a concept and
  eagerly awaited improvements

14
What would the improve?

• In order for Voice User Interfaces (VUI) to be
  successful they need to create a conversation
  with the user.
• This can be accomplished through
• Shared context
• When is the right time to input into the system?
• Conversation pacing
• How can information be shared or skipped at the
  right speed?

15
GUI to SUI?

• No. it doesnt make sense to directly translate a
  GUI experience into a SUI experience.
• Instead, take information orgainization and
  information flow of GUI and build SUI from ground
  up to accomplish the tasks that the users want to
  accomplish

16
Recognition errors

• Rejection errors
• Find creative ways to get users to repeat input
  without getting mad
• Substitution errors
• Confirm some commands
• Insertion errors
• Turn off mic, same as above

17
New User Skills

• SUIs have different challenges than GUIs
• Users need to have different skills
• Short term memory
• Mental model of system state
• Visualizing the organization of information

18
Conclusions SUIs

• Adhere to principles of conversation
• Information must be delivered in a dense fashion
  for audio output to be fast enough
• Immediate and informative feedback on input
• Dont directly translate a GUI into a SUI

19
Questions multimodal

• Oviatts paper gives a lot of benefits to
  multimodal interaction, why dont we see many
  multimodal systems in commercial production
• Or do we?

20
SpeechActs

• Does SpeechActs still make sense 10 years later?
• do traveling professionals use these kind of
  systems now?
• Who might benefit from these kinds of systems?