earPod: Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback

1
earPod Eyes-free Menu Selection with Touch
Input and Reactive Audio Feedback
Shengdong Zhao
Pierre Dragicevic
Mark Chignell
Ravin Balakrishnan
Patrick Baudisch
2
(No Transcript)
3
(No Transcript)
4
Menu Selection on Mobile Devices Often
Requires Visual Feedback
5
Why Eyes-free?
6
Why Eyes-free?
7
Eyes-free capable
8
Eyes-free capable
Marking Menus
9
Eyes-free capable
Marking Menus (Expert Use)
10
Eyes-free capable
Marking Menus (Expert Use)
11
True eyes-free

• TouchPlayer (Pirhonen et al, 2002)

12
True eyes-free

• TouchPlayer (Pirhonen et al, 2002)
• Head gesture-based auditory pie menu (Brewster et
  al, 2003)

13
earPod
13
14
earPod
14
15
earPod Design
15
16
earPod Walkthrough
16
17
Browsing
18
Fast browsing
18
19
Fast browsing
20
Very fast browsing and cancel
20
21
Very fast browsing and cancel
22
Expert use
23
Expert use
24
Interactive Voice Response System
24
25
Interactive Voice Response System
Slow Painful Touchtone Hell (Yin and Zhai,
2006)
25
26
Visual vs. auditory menu
Visual Linear Menu
26
27
Visual vs. auditory menu
Visual Linear Menu
IVR System
27
28
Problems of IVR system

• Speech is serial and slow (Slowiaczek and
  Nusbaum 1985)

29
Problems of IVR system

• Synchronous vs. asynchronous communication

Synchronous
30
Problems of IVR system

• Synchronous vs. asynchronous communication

Asynchronous
Synchronous
31
Problems of IVR system

• Transition to expert use

Skill level
Novice
(Voice menu recognition)
Skill development
32
Problems of IVR system

• Transition to expert use

Expert
(direct code access recall)
Skill level
Novice
(Voice menu recognition)
Skill development
33
Problems of IVR system

• Transition to expert use

Expert
(direct code access recall)
Skill level
Novice
(Voice menu recognition)
Skill development
34
earPods approach

• Problem speech is serial and slow

35
earPods approach

• Problem speech is serial and slow
• By-pass audio

36
earPods approach

• Problem speech is serial and slow
• By-pass audio
• Use interruptible audio feedback

37
earPods approach

• Problem speech is serial and slow
• By-pass audio
• Use interruptible audio feedback
• Use non-speech audio

38
earPods approach

• Problem asynchronous communication

39
earPods approach

• Problem asynchronous communication
• Synchronous communication (reactive audio
  feedback)

40
earPods approach

• Problem steep novice to expert transition

41
Smooth transition

• Learning menu structure

42
Smooth transition

• Learning menu structure
• Learning expert gesture

Target Item
43
Smooth transition

• Learning menu structure
• Learning expert gesture

Target Item
Novice
44
Smooth transition

• Learning menu structure
• Learning expert gesture

Target Item
Novice
Novice/Expert
45
Smooth transition

• Learning menu structure
• Learning expert gesture

Target Item
Novice
Novice/Expert
Expert
46
Expert
(direct tapping recall)
Skill level
Novice
(glide browsing recognition)
46
Skill development
47
Support Single-hand Usage
(Karlson et al, 2007)
47
48
Haptic guidance
49
Haptic guidance
50
earPod Design Summary

• Touch Input

50
51
earPod Design Summary

• Touch Input
• Reactive Interruptible Audio

51
52
earPod Design Summary

• Touch Input
• Reactive Interruptible Audio
• Non-speech Audio

52
53
earPod Design Summary

• Touch Input
• Reactive Interruptible Audio
• Non-speech Audio
• Input/Output Mapping

53
54
earPod Design Summary

• Touch Input
• Reactive Interruptible Audio
• Non-speech Audio
• Input/Output Mapping
• Direct Item Access

54
55
earPod Design Summary

• Touch Input
• Reactive Interruptible Audio
• Non-speech Audio
• Input/Output Mapping
• Direct Item Access
• Transition to Expert Use

55
56
Experiment

• earPod vs. iPod-like linear menu

56
57
Experiment visual condition
57
58
Experiment audio condition
58
59
Experimental design

• Within-subject, counter-balanced
• 12 participants x
• 2 techniques (audio and visual) x
• (4080) items for the 2 menu configurations (8
  and 8x8) x
• 4 blocks
• 11,520 menu selections in total.

59
60
Results
Block number
Block number
60
61
Results
Block number
Block number
61
62
Results
Block number
Block number
62
63
Results
Block number
Block number
63
64
Results
Block number
Block number
64
65
Results

• Detailed Log Analysis
• 9/12 reached expert use (1 level menu)
• 8/12 reached expert use (2 level menu)

65
66
Results

• Detailed Log Analysis
• 9/12 reached expert use (1 level menu)
• 8/12 reached expert use (2 level menu)
• Subjective Feedback
• 4 preferred earPod
• 4 preferred iPod-like visual menu
• 4 had no preferences
• 11/12 suggest combine audio visual

66
67
earPod limitations

• Work with structured menu of limited breadth

67
68
earPod limitations

• Work with structured menu of limited breadth

68
69
Conclusion

• earPod eyes-free menu selection for structured
  menu of limited breadth
• Performance comparable to iPod-like linear menus
• Support single-handed interaction
• Design for mobile use

69
70
Future Work

• Long/dynamic menus

70
71
Future Work

• Long/dynamic menus
• Dual channel menu

71
72
Future Work

• Long/dynamic menus
• Dual channel menu
• Dichotic listening (Ranjan et al., 2006)

72
73
Future Work

• Long/dynamic menus
• Dual channel menu
• Dichotic listening (Ranjan et al., 2006)
• IVR on the phone

73
74
Acknowledgement
Danielle Lottridge, Tomer Moscovich, Mike
McGuffin, Jingnan Yang, Heather Thorne, Alvin
Chin, Derek Nowrouzezahrai, Yongchang Zhao, Lisa
Min, Andrew Chignell, Paula Gilverson, Michelle
Qu, Andrew McGutman, Noah Lockwood, John
Chattoe, Annie Xu, Silvia Gonzaloz, Anna
Malandrino Anonymous reviewers
75
Mobile Usage Design

• Stability issues

75
76
earPod Novice to Expert Transition
1
7
0
2
6
3
5
4