Mobile Devices for Control - PowerPoint PPT Presentation

About This Presentation
Title:

Mobile Devices for Control

Description:

– PowerPoint PPT presentation

Number of Views:624
Avg rating:3.0/5.0
Slides: 57
Provided by: bradm4
Learn more at: http://www.cs.cmu.edu
Category:

less

Transcript and Presenter's Notes

Title: Mobile Devices for Control


1
Mobile Devices for Control
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University
  • Brad A. Myers
  • bam_at_cs.cmu.edu
  • http//www.cs.cmu.edu/pebbles

2
Mobile Devices
  • Mobile Devices for communication
  • As a Personal Information Manager
  • As a Personal Digital Assistant
  • As a PC replacement
  • PocketPC
  • For Remote Control of other devices
  • Remote Control of PCs
  • Remote Control of appliances

3
Problem
  • Appliances are too complex

4
Problem
  • Too many remotes

5
Problem
  • User Interfaces of Mobile Devices themselves are
    too hard to use

6
Problem
  • PCs have only a keyboard and mouse as input
    devices

7
Approach
  • Use a personal Mobile Device as an Interface to
    the PC and to the Appliances

8
History
  • ParcTab continuous communication
  • Early organizers no communication
  • Newton extra cost for synchronization
  • Palm HotSync once a day
  • WindowsCE/PocketPC ActiveSync once a day

9
Now, Handhelds will be Communicating
Toshiba e740
  • 802.11 (Wi-Fi)
  • BlueTooth
  • Cell-phone network (G3)
  • (Infrared)
  • (Wires or cradle)

HP iPaq 3870
Microsoft SmartPhone
10
Premises of our Research
  • With the coming wireless technologies,
    connecting the PCs and PDAs together will no
    longer be an occasional event for
    synchronization. Instead, the devices will
    frequently be inclose, interactive
    communication.
  • Brad Myers, Using Hand-Held Devices andPCs
    Together, Comm. ACM,Vol. 44, No. 11. Nov.,
    2001. pp. 34 - 41.

11
Pebbles Project
  • Performed as part of the Pebbles Project
  • Overall goal Use of multiple devices at the same
    time
  • Palm and desktop PC
  • Mobile Phone and Smart Home
  • Multiple handhelds in a meeting
  • Pocket PC and appliances
  • Multiple users with their devices
  • Single user with multiple devices

12
Pebbles is
DAs for ntry of oth ytes and ocations
from xternal ources.
  • P
  • E
  • B
  • B
  • L
  • E
  • S

http//www.pebbles.hcii.edu/
13
Controlling a PC
  • Remote Commander
  • Slide Show Commander
  • Semantic Snarfing
  • Scrolling
  • Shortcutter

http//www.pebbles.hcii.edu/
14
Important Work by Others
  • PARCTab Want 95, Weiser, CACM93
  • Abowd eClass using Handhelds CHI98
  • Greenberg public and private dataPersonal
    Technologies, 3(1), 1999
  • Jun Rekimoto UIST97, CHI98, CHI99,
  • Pick and drophandhelds withbig displays

15
Remote Commander
  • Allow PDAs to control a PC
  • Can be used with any application
  • Uses the standard (single) cursor
  • Dont have to jump up andgrab mouse
  • All mouse and keyboard functions
  • Use PDA like touchpad
  • Graffiti or our own pop-upkeyboard
  • Word prediction and completion

16
For People with Muscular Disabilities
  • Using handhelds as interface to PCs
  • People with Muscular Dystrophyhave fine-motor
    control but losegross motor control
  • Difficulties with mouse and keyboard,but stylus
    OK
  • So use Remote Commanderas PCs keyboard and mouse

17
PocketPC version
  • Get PCs screen onto PocketPC
  • Full view, or one-to-one zooming
  • Scroll with iPaqs buttons
  • Control or scribble

18
Power Point Control
  • Use PC to give the presentation
  • Use hand-held to control the PC
  • Two-way communication
  • Hand-held shows picture of slide,notes, timer

19
SlideShow Commander, cont.
  • See list of slide titles
  • Scribble
  • Tap on on-screen buttonsand links
  • Preview otherslides
  • Controldemonstrations

20
Control at a Distance
  • Controlling a PC fromacross the room
  • Meetings, etc.
  • Use a laser pointer?
  • Studies
  • Difficult due to
  • Jitter of hand (/- 10 pixels)
  • No button on the laser pointer
  • People not accurate at wherebeam will go on and
    off

21
IdeaSemantic Snarfing
  • Grab contents to handheld
  • Picture
  • Semantic because reformat based on the meaning
    of the contents
  • Menus
  • Text
  • Re-visualization

22
Command Post of the Future
  • Large DARPA funded project
  • Make commanders more effective
  • Private drill-down of public information

23
Classroom Investigations
  • Investigate improving large lectures with
    frequent in-class concept tests
  • Provide immediate feedback to instructor
  • Easy for instructor

24
Scrolling Desktop Applications
  • Scrolling using the non-dominant hand
  • Parallel and efficient use of both hands
  • Generates Windowsscrolling events
  • Study showed can befaster for some tasks

25
Shortcutter
  • User-created panels of controls
  • Create custom interfaces and extensions to PC
    applications
  • And then take them with you
  • Direct manipulationfor edit, then setproperties
  • PocketPC or Palm

26
Shortcutter Widgets
  • Buttons
  • Virtual Joy stick
  • Virtual Knob
  • Sliders
  • Mouse pad
  • Gesture panel

27
Shortcutter Actions
  • Send any keyboard key, mouse button, scrolling
    action or string to PC
  • Open a file or URL
  • Run an application
  • Invoke any PC menu or button
  • Windows message
  • Recorded
  • Switch to a different Shortcutter panel
  • Control the Mouse

28
Shortcutter Actions, cont.
  • Control external devicesthrough PCs serial port
  • Directly (e.g., projectors)
  • X-10 for electrical devices
  • Macro
  • Can be multi-application
  • Application-specific
  • Same button, different messages
  • Useful for application setsbrowsers, compilers

29
More Scenarios of Use
  • Lean-back mail reading
  • Controlling media players
  • and many others

30
Controlling Appliances
  • PhD research of Jeffrey Nichols

http//www.pebbles.hcii.edu/puc/
31
Problem
  • Too many complex devices, eachwith its own
    idiosyncratic interface
  • Stereo system
  • Telephones
  • ATM
  • Fax machine
  • Photocopier
  • Hotel alarm clock
  • Increasingly computerized
  • Low usability

32
Problem
33
Existing universal controls
  • Pre-programmed at the factory with a subset,
  • Or, Laboriously hand-programmed by the user

34
Important Work by Others
  • Xweb Olsen Jr. 2000
  • INCITS V2 standardization effort Alternative
    Interface Access Protocol (AIAP) Zimmermann,
    CHI02
  • Stanford iRoom, iCrafterPonnekanti, UBICOMP
    2001

35
Our Approach
  • Handheld is a Personal Universal Controller
    (PUC)
  • Two-way communication
  • Appliance describes its functions
  • Handheld PUC
  • Automatically creates interface
  • Controls the appliance
  • Displays feedback about appliance status

Specifications
Control
Feedback
36
Automatic Generation of UIs
  • Benefits
  • All interfaces consistent for a user
  • With conventions of the handheld
  • Even from multiple manufacturers
  • Addresses hotel alarm clock problem
  • Can take into account user preferences
  • Multiple modalities (GUI Speech UI)
  • A Hard Problem
  • Previous automatic systems have not generated
    high quality interfaces

37
Research Challenges
  • Automatic Design of Graphical User Interfaces
  • Automatic Design of Speech User Interfaces
  • Connection with real devices
  • Through various protocols
  • X-10, AV/C, HAVi, UPnP, etc.
  • Also, custom hardware and software

38
Hand-Generated Graphical Interfaces
  • First, Hand-Designed PocketPC interfaces
  • AIWA Shelf Stereo (Tape,CD,Tuner)
  • ATT Telephone/AnsweringMachine
  • Used Embedded Visual Basic
  • Ensured quality with heuristicanalysis and
    think-aloud studies
  • Compared with manufacturersinterfaces

39
Results of Comparison
  • Using PUC, users took 50 less time made 50
    fewer errors
  • All differences are significant (p lt 0.05)

40
Discussion of Comparison
  • Our hand designed interfaces succeeded for
    several reasons
  • Good organization
  • Each button has one function
  • Good labels
  • Only available functions are active
  • Others, hidden on tabs or grayed out
  • Better feedback and error messages

41
Current PUC Specification Language
  • XML
  • Full documentation for the specification language
    and protocol
  • http//www.pebbles.hcii.cmu.edu/puc/
  • Contains sample specification for a stereo

42
Properties of PUC Language
  • State variables commands
  • Each can have multiple labels
  • Useful when not enough room
  • Typed variables
  • Base types Boolean, string,enumerated,
    integers,fixed-point, floating-point, etc.
  • Optional labels for values
  • Hierarchical Structure
  • Groups

43
Dependency Information
  • Crucial for high-quality interfaces
  • Expressed as ltactive-ifgt clauses
  • Operations
  • Equals, Less-Than,Greater-Than
  • Combined Logically
  • AND, OR
  • Used for
  • Dynamic graying out
  • Layout
  • Widget selection

44
Specifications
  • Have working specifications for
  • Audiophase stereo
  • X-10 lights control
  • Sony CamCorder
  • Windows Media Player
  • Audio ReQuest hardware MP3 player
  • WinAmp Media Player
  • Elevator

45
Examples of Generated GUIs
  • Stereo and X-10

46
More Examples
  • Elevator

47
Generating Speech Interfaces
  • Universal Speech Interface (USI) project
  • Prof. Roni Rosenfeld of CMU
  • http//www.cs.cmu.edu/usi
  • Creates grammar, language model and pronunciation
    dictionary from PUC specification
  • Pronunciation from labels using phonetic rules
  • Can provide other pronunciations as labels for
    fine-tuning
  • Will use dependency information to help with
    disambiguation and explanation
  • Supports queries and spoken feedback
  • Paraphrases as confirmation

48
PUC Architecture
49
Adaptors
  • Adaptors provide the interface to existing (and
    future) appliances
  • If do not support specification language directly
  • Custom hardware
  • Audiophase Stereo
  • Custom software
  • ARQ MP3 player
  • SIMA MP3 player (future)
  • X-10
  • Light switches, etc.
  • AV/C (standard protocol)
  • Sony CamCorder
  • PlayStation, etc. (future)
  • HAVi (current work)
  • Mitsubishi HDTV 65 TV
  • Mitsubishi HDTV VCR
  • UPnP (future)
  • ??

50
Adaptor using Custom Hardware for Stereo
  • Pretends to send IR codes
  • Reads LED panel signals to decode state
  • Created by Pittsburgh company Maya Design

51
Adaptor for AV/C
PUC Java Proxy
PUC Java Library
AutomaticGUIgeneration
AutomaticSpeech UIgeneration
Hand-written spec of AV/C devices
Our Java device code
Our C code
Multiple devicecontrollers
Microsoft DirectShowlibrary
AV/C protocol
Firewire cable(IEEE 1394)
Digital VideoCamera or VCR
52
Demonstration
  • X-10
  • Camcorder through AV/C protocol forIEEE 1394
    (Firewire)
  • Two way communication
  • When state changes from appliance, GUI is updated
    and can query state with speech

53
For More Information
  • Many papers on the Pebbles web site
  • On PUC and other work
  • Most programs available for free downloading
  • SlideShow Commander available commercially

http//www.cs.cmu.edu/pebbles
http//www.slideshowcommander.com
54
Conclusions
  • Study The Big Picture
  • How mobile devices fit into users entire
    information and control space
  • As more and more electronics are computerized,
    mobiles devices can help control them
  • Mobile devices can improve the user interfaces of
    everything else
  • Not just be another gadget to be learned

55
Thanks to Our Sponsors!
  • The Pebbles research project is supported by
    grants from
  • DARPA
  • Microsoft
  • NSF
  • Pittsburgh Digital Greenhouse
  • General Motors
  • And equipment grants from
  • Hewlett Packard
  • Lucent Technologies
  • Mitsubishi
  • Palm Computing
  • Symbol Technologies
  • IBM
  • SMART Technologies
  • VividLogic
  • Synergy Solutions
  • Handango

56
Mobile Devices for Control
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University
  • Brad A. Myers
  • bam_at_cs.cmu.edu
  • http//www.cs.cmu.edu/pebbles

Thank you!
Write a Comment
User Comments (0)
About PowerShow.com