Mobile Devices for Control

1
Mobile Devices for Control
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University

• Brad A. Myers
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles

2
Mobile Devices

• Mobile Devices for communication
• As a Personal Information Manager
• As a Personal Digital Assistant
• As a PC replacement
• PocketPC
• For Remote Control of other devices
• Remote Control of PCs
• Remote Control of appliances

3
Problem

• Appliances are too complex

4
Problem

• Too many remotes

5
Problem

• User Interfaces of Mobile Devices themselves are
  too hard to use

6
Problem

• PCs have only a keyboard and mouse as input
  devices

7
Approach

• Use a personal Mobile Device as an Interface to
  the PC and to the Appliances

8
History

• ParcTab continuous communication
• Early organizers no communication
• Newton extra cost for synchronization
• Palm HotSync once a day
• WindowsCE/PocketPC ActiveSync once a day

9
Now, Handhelds will be Communicating
Toshiba e740

• 802.11 (Wi-Fi)
• BlueTooth
• Cell-phone network (G3)
• (Infrared)
• (Wires or cradle)

HP iPaq 3870
Microsoft SmartPhone
10
Premises of our Research

• With the coming wireless technologies,
  connecting the PCs and PDAs together will no
  longer be an occasional event for
  synchronization. Instead, the devices will
  frequently be inclose, interactive
  communication.

• Brad Myers, Using Hand-Held Devices andPCs
  Together, Comm. ACM,Vol. 44, No. 11. Nov.,
  2001. pp. 34 - 41.

11
Pebbles Project

• Performed as part of the Pebbles Project
• Overall goal Use of multiple devices at the same
  time
• Palm and desktop PC
• Mobile Phone and Smart Home
• Multiple handhelds in a meeting
• Pocket PC and appliances
• Multiple users with their devices
• Single user with multiple devices

12
Pebbles is
DAs for ntry of oth ytes and ocations
from xternal ources.

• P
• E
• B
• B
• L
• E
• S

http//www.pebbles.hcii.edu/
13
Controlling a PC

• Remote Commander
• Slide Show Commander
• Semantic Snarfing
• Scrolling
• Shortcutter

http//www.pebbles.hcii.edu/
14
Important Work by Others

• PARCTab Want 95, Weiser, CACM93
• Abowd eClass using Handhelds CHI98
• Greenberg public and private dataPersonal
  Technologies, 3(1), 1999
• Jun Rekimoto UIST97, CHI98, CHI99,
• Pick and drophandhelds withbig displays

15
Remote Commander

• Allow PDAs to control a PC
• Can be used with any application
• Uses the standard (single) cursor
• Dont have to jump up andgrab mouse
• All mouse and keyboard functions
• Use PDA like touchpad
• Graffiti or our own pop-upkeyboard
• Word prediction and completion

16
For People with Muscular Disabilities

• Using handhelds as interface to PCs
• People with Muscular Dystrophyhave fine-motor
  control but losegross motor control
• Difficulties with mouse and keyboard,but stylus
  OK
• So use Remote Commanderas PCs keyboard and mouse

17
PocketPC version

• Get PCs screen onto PocketPC
• Full view, or one-to-one zooming
• Scroll with iPaqs buttons
• Control or scribble

18
Power Point Control

• Use PC to give the presentation
• Use hand-held to control the PC
• Two-way communication
• Hand-held shows picture of slide,notes, timer

19
SlideShow Commander, cont.

• See list of slide titles
• Scribble
• Tap on on-screen buttonsand links
• Preview otherslides
• Controldemonstrations

20
Control at a Distance

• Controlling a PC fromacross the room
• Meetings, etc.
• Use a laser pointer?
• Studies
• Difficult due to
• Jitter of hand (/- 10 pixels)
• No button on the laser pointer
• People not accurate at wherebeam will go on and
  off

21
IdeaSemantic Snarfing

• Grab contents to handheld
• Picture
• Semantic because reformat based on the meaning
  of the contents
• Menus
• Text
• Re-visualization

22
Command Post of the Future

• Large DARPA funded project
• Make commanders more effective
• Private drill-down of public information

23
Classroom Investigations

• Investigate improving large lectures with
  frequent in-class concept tests
• Provide immediate feedback to instructor
• Easy for instructor

24
Scrolling Desktop Applications

• Scrolling using the non-dominant hand
• Parallel and efficient use of both hands
• Generates Windowsscrolling events
• Study showed can befaster for some tasks

25
Shortcutter

• User-created panels of controls
• Create custom interfaces and extensions to PC
  applications
• And then take them with you
• Direct manipulationfor edit, then setproperties
• PocketPC or Palm

26
Shortcutter Widgets

• Buttons
• Virtual Joy stick
• Virtual Knob
• Sliders
• Mouse pad
• Gesture panel

27
Shortcutter Actions

• Send any keyboard key, mouse button, scrolling
  action or string to PC
• Open a file or URL
• Run an application
• Invoke any PC menu or button
• Windows message
• Recorded
• Switch to a different Shortcutter panel
• Control the Mouse

28
Shortcutter Actions, cont.

• Control external devicesthrough PCs serial port
• Directly (e.g., projectors)
• X-10 for electrical devices
• Macro
• Can be multi-application
• Application-specific
• Same button, different messages
• Useful for application setsbrowsers, compilers

29
More Scenarios of Use

• Lean-back mail reading
• Controlling media players
• and many others

30
Controlling Appliances

• PhD research of Jeffrey Nichols

http//www.pebbles.hcii.edu/puc/
31
Problem

• Too many complex devices, eachwith its own
  idiosyncratic interface
• Stereo system
• Telephones
• ATM
• Fax machine
• Photocopier
• Hotel alarm clock
• Increasingly computerized
• Low usability

32
Problem
33
Existing universal controls

• Pre-programmed at the factory with a subset,
• Or, Laboriously hand-programmed by the user

34
Important Work by Others

• Xweb Olsen Jr. 2000
• INCITS V2 standardization effort Alternative
  Interface Access Protocol (AIAP) Zimmermann,
  CHI02
• Stanford iRoom, iCrafterPonnekanti, UBICOMP
  2001

35
Our Approach

• Handheld is a Personal Universal Controller
  (PUC)
• Two-way communication
• Appliance describes its functions
• Handheld PUC
• Automatically creates interface
• Controls the appliance
• Displays feedback about appliance status

Specifications
Control
Feedback
36
Automatic Generation of UIs

• Benefits
• All interfaces consistent for a user
• With conventions of the handheld
• Even from multiple manufacturers
• Addresses hotel alarm clock problem
• Can take into account user preferences
• Multiple modalities (GUI Speech UI)
• A Hard Problem
• Previous automatic systems have not generated
  high quality interfaces

37
Research Challenges

• Automatic Design of Graphical User Interfaces
• Automatic Design of Speech User Interfaces
• Connection with real devices
• Through various protocols
• X-10, AV/C, HAVi, UPnP, etc.
• Also, custom hardware and software

38
Hand-Generated Graphical Interfaces

• First, Hand-Designed PocketPC interfaces
• AIWA Shelf Stereo (Tape,CD,Tuner)
• ATT Telephone/AnsweringMachine
• Used Embedded Visual Basic
• Ensured quality with heuristicanalysis and
  think-aloud studies
• Compared with manufacturersinterfaces

39
Results of Comparison

• Using PUC, users took 50 less time made 50
  fewer errors
• All differences are significant (p lt 0.05)

40
Discussion of Comparison

• Our hand designed interfaces succeeded for
  several reasons
• Good organization
• Each button has one function
• Good labels
• Only available functions are active
• Others, hidden on tabs or grayed out
• Better feedback and error messages

41
Current PUC Specification Language

• XML
• Full documentation for the specification language
  and protocol
• http//www.pebbles.hcii.cmu.edu/puc/
• Contains sample specification for a stereo

42
Properties of PUC Language

• State variables commands
• Each can have multiple labels
• Useful when not enough room
• Typed variables
• Base types Boolean, string,enumerated,
  integers,fixed-point, floating-point, etc.
• Optional labels for values
• Hierarchical Structure
• Groups

43
Dependency Information

• Crucial for high-quality interfaces
• Expressed as ltactive-ifgt clauses
• Operations
• Equals, Less-Than,Greater-Than
• Combined Logically
• AND, OR
• Used for
• Dynamic graying out
• Layout
• Widget selection

44
Specifications

• Have working specifications for
• Audiophase stereo
• X-10 lights control
• Sony CamCorder
• Windows Media Player
• Audio ReQuest hardware MP3 player
• WinAmp Media Player
• Elevator

45
Examples of Generated GUIs

• Stereo and X-10

46
More Examples

• Elevator

47
Generating Speech Interfaces

• Universal Speech Interface (USI) project
• Prof. Roni Rosenfeld of CMU
• http//www.cs.cmu.edu/usi
• Creates grammar, language model and pronunciation
  dictionary from PUC specification
• Pronunciation from labels using phonetic rules
• Can provide other pronunciations as labels for
  fine-tuning
• Will use dependency information to help with
  disambiguation and explanation
• Supports queries and spoken feedback
• Paraphrases as confirmation

48
PUC Architecture
49
Adaptors

• Adaptors provide the interface to existing (and
  future) appliances
• If do not support specification language directly
• Custom hardware
• Audiophase Stereo
• Custom software
• ARQ MP3 player
• SIMA MP3 player (future)
• X-10
• Light switches, etc.

• AV/C (standard protocol)
• Sony CamCorder
• PlayStation, etc. (future)
• HAVi (current work)
• Mitsubishi HDTV 65 TV
• Mitsubishi HDTV VCR
• UPnP (future)
• ??

50
Adaptor using Custom Hardware for Stereo

• Pretends to send IR codes
• Reads LED panel signals to decode state
• Created by Pittsburgh company Maya Design

51
Adaptor for AV/C
PUC Java Proxy
PUC Java Library
AutomaticGUIgeneration
AutomaticSpeech UIgeneration
Hand-written spec of AV/C devices
Our Java device code
Our C code
Multiple devicecontrollers
Microsoft DirectShowlibrary
AV/C protocol
Firewire cable(IEEE 1394)
Digital VideoCamera or VCR
52
Demonstration

• X-10
• Camcorder through AV/C protocol forIEEE 1394
  (Firewire)
• Two way communication
• When state changes from appliance, GUI is updated
  and can query state with speech

53
For More Information

• Many papers on the Pebbles web site
• On PUC and other work
• Most programs available for free downloading
• SlideShow Commander available commercially

http//www.cs.cmu.edu/pebbles
http//www.slideshowcommander.com
54
Conclusions

• Study The Big Picture
• How mobile devices fit into users entire
  information and control space
• As more and more electronics are computerized,
  mobiles devices can help control them
• Mobile devices can improve the user interfaces of
  everything else
• Not just be another gadget to be learned

55
Thanks to Our Sponsors!

• The Pebbles research project is supported by
  grants from
• DARPA
• Microsoft
• NSF
• Pittsburgh Digital Greenhouse
• General Motors
• And equipment grants from
• Hewlett Packard
• Lucent Technologies
• Mitsubishi
• Palm Computing
• Symbol Technologies

• IBM
• SMART Technologies
• VividLogic
• Synergy Solutions
• Handango

56
Mobile Devices for Control
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University

• Brad A. Myers
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles

Thank you!