Interaction through audio

1
Interaction through audio
2
Agenda

• Questions
• Audio (Khai)
• Video (Gabriel Brostow)

3
What makes interaction through audio interesting?

• Natural
• Off-the desktop
• hands-free
• display not necessary
• effective for a short distance

4
How sound is created

• Sound is created when air is disturbed, usually
  by vibrating object(s) causing ripples of varying
  air pressure propagated by the collision of air
  molecules.

5
How sound is perceived

• Characteristics of sound
• Volume
• Pitch
• Features of sound wave
• Amplitude
• Volume amplitude of a sound wave
• Frequency
• Pitch frequency of a single sound wave

6
Example
7
Example
8
When multiple sounds occur
9
Sampling sound

• Note sampling rate affects accurate
  representation of the sound wave

10
Acquiring audio
11
Generating audio
12
Java sound architecture
13
Issue

• Perfect recognition of speech (or semantic
  understanding of any kind of audio) is difficult
  to achieve
• Challenge How would you begin?
• Segmentation
• Syntax

14
Suede

• Toolkit for prototyping speech interface
• http//guir.berkeley.edu/projects/suede/

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
Issue (continued)

• It is still difficult to
• grab
• chunk (segment)
• store
• playback
• Challenge provide support for handling audio in
  manner similar to text

19
INCA

• Toolkit for capture and access
• Contains library of common capture access
  components, including basic audio support
• More on this toolkit in Fridays lecture

20
Interesting features in speech

• Pauses between phrases as well

21
Personal Audio Loop

• Application which continuously buffers users
  last 15 minutes of audio
• What were we talking about?
• What was that phone number I heard?
• Features above are used to speed up audio
  playback when skimming for point of access
• compressed or discarded in some cases
• More on the application in Fridays lecture

22
The Family Intercom

• Use location sensing in context-aware environment
  to connect people in different places in a
  conversation

23
The Family Intercom (Ubicomp 2001)
24
The Family Intercom (Ubicomp 2001)
25
Issue (continued)

• Use of open air microphones speakers can result
  in undesired audio
• ambient noise
• audio feedback
• Challenge allow developers to easily add/use
  functions in their applications
• Noise reduction
• Enhance audio quality
• Echo cancellation

26
Noise reduction

• Random noise is hard to predict

Noise Filter
f (t)
f(t)
27
Echo cancellation

• Software hardware exist, but are hard for
  developers to easily add to application
• Random noise is hard to predict, but echoes are
  not so random

Echo Canceller
f(t)
f (t)
28
Other interesting audio work

• Auditory icons or earcons
• sound that identifies objects actions
• Automatic transformation of GUI interfaces to
  auditory interfaces
• Mercator (Mynatt, 1995)
• Sound localization sound cones