Contextual Awareness and Messaging in Nomadic Environments Design for Wearable Audio Computing

1
Contextual Awareness and Messaging in Nomadic
Environments Design for Wearable Audio Computing
Nitin Sawhney nitin_at_media.mit.edu Ericsson
Eurolabs Deutschland GmbH August 24th,
1998 Speech Interface Group MIT Media
Laboratory, Cambridge, USA
2
Outline of Presentation

• Personal Augmentation Landscape (80s, 90s,
  future)
• Prior work in Speech Group Research Challenges
• Wearable Audio Computing Design of Nomadic
  Radio
• Voice Navigation, Ambient Awareness and Spatial
  Audio
• Contextual and Scaleable Notification
• Ongoing Research and Future Work

3
Personal Augmentation Landscape (early 80s)
High
Airplane Cockpit
Situational Awareness
Automobiles
Pagers
Walkman
Watch
Desktop PCs
Pen-paper
Eye-glasses
Low
Active
Passive
Demand on Users Attention Resources
4
Personal Augmentation Landscape (early 90s)
High
Airplane Cockpit
HMD Wearables
Automobiles
Situational Awareness
Newton
Personal Organizers
Mobile-Phones
Pagers
Virtual Reality
Walkman
Watch
Network PCs
Pen-paper
Eye-glasses
Low
Active
Passive
Demand on Users Attention Resources
5
Telephone-based Messaging Systems

• Phoneshell - Chris Schmandt 93
• Chatter - Eric Ly 94
• NewsPhone - 96
• MailCall - Matt Marx 96
• Industry Products WildFire, ATT PocketNet,
  Nokia Communiator, General Magic Porticio

6
Mobile Audio Devices

• Voice Notes - Lisa Stifelman 93
• NewsComm - Deb Roy 95
• Audio Notebook - Lisa Stifelman 96
• Dynomite - Sawhney FXPAL 96
• Industry Digital Voice Recorders Internet
  Audio Players, IBM CrossPad and
  Telematics

7
Wearable (audio) Computing

• Ubiquitous Talker - Sony 95
• Speech Wear - CMU 96
• Audio Aura - Xerox PARC 97
• Remembrance Agent - Thad Starner and Brad Rhodes
  96 (MIT Media Lab)
• DIE AUDIO GRUPPE - electro-acoustic clothes 83-98

8
Personal Augmentation Landscape (1997-future)
Airplane Cockpit
High
COMRIS
Nomadic Radio
HMD Wearables
Telematics
Automobiles
Audio-Aura
Situational Awareness
Newton
Mobile Audio
Speech Wear
Smart-Phones
Palm-Pilot
Pagers
Virtual Reality
Walkman
Watch
Desktop PCs
Pen-paper
Eye-glasses
Low
Active
Passive
Demand on Users Attention Resources
9
Research Challenges for Nomadic Augmentation

• Unified Access to Heterogeneous Information
• Unobtrusive Peripheral Interfaces
• Knowing when and How to Interrupt the User
• Sensing Environmental Context
• Adaptive Learning of Usage Patterns

10
Nomadic Radio Design Overview
Nomadic Radio
Integrated Audio Interface for Personal Messaging
on a Wearable Platform
Audio Cues for
Spatial Audio for
Notification
Simultaneous
Listening
Ambient Sound
for Awareness
"New short personal
message from Walter about
Where do we go for
Lunch?"
Voice Recognition
for Navigation and
Synthetic Speech
Control
for Feedback
11
Unifying Information Services via Dynamic Views
and Modeless Interaction
12
System Architecture
System Architecture
13
Wired Wearable vs. Distributed Wireless Mode
Spatial Audio Speech Synthesis PC
Wireless 3D Audio
Networked Speech Rec. PC
Wireless Speech Input
Nortel Soundbeam Neckset
MIT Media Lab Lizzy
Wavelan Tactile Input
Toshiba Libretto
VIA FlexiPC
14
Radio Vest Alternative Wearable Design
Radio Vest
Alternative Wearable Audio Design
15
Listening and Speaking on the Run A Scenario
16
Speech I/O and Voice Navigation

• Always on, Listening and Easily Woken-up
• Networked Implementation of ATT Watson Speech
  API
• Push-to-Talk vs. Continuous Monitoring
• Navigation Browsing vs. Scanning

Temporal Target Windows
17
Awareness Notification via Auditory Cues

• Priority

• Feedback

• Awareness

• Identification

18
Simultaneous and Spatial Listening

• Spatial Memory
• Stream Segregation
• Mapping Time to Space

Localization using an HRTF Model
Chronological 12-hour Spatial Layout
19
Message Swatch Mapping Time to Space
Message Swatch
Spatial Scanning
20
Notifications in Everyday Environments

• Binary Notification Cues
• No Awareness of User and Environment
• No Adaptive Learning from Prior Interactions
• No Unified Notification strategy between devices
• Solution Scaleable Contextual Notification

Cell-phone
PDA
Pager
21
Scaleable Presentation and Dynamic Operational
Modes
22
Contextual Notification Model
23
Ongoing Research and Future Vision

• Environmental Audio Classification
• Multi-user Awareness and Communication Protocols
• Attention-Cognition Research Evaluation
• Affective Attention via EEG Sensing Analysis
• Vision Ubiquitous Autonomous Entities (Devices
  and Agents) with Cooperative and Evolving
  Context-Attention Models

24
Contextual Awareness and Messaging in Nomadic
Environments Design for Wearable Audio Computing
Nitin Sawhney nitin_at_media.mit.edu Speech
Interface Group http//www.media.mit.edu/nitin/N
omadicRadio/