Jeffrey Nichols and Brad A' Myers

1
Generating Remote Control Interfaces for Complex
Appliances
The Fifteenth Annual Symposium on User Interface
Software Technology (UIST) 2002 Paris, France

• Jeffrey Nichols and Brad A. Myers
• Carnegie Mellon University
• October 30, 2002

2
The Problem

• Appliances are too complex

3
The Problem, cont.

• Each complex appliances has its own idiosyncratic
  interface!
• Stereo systems
• Telephones
• VCRs
• Alarm Clocks
• Increasingly Computerized
• Low Usability

4
Our Solution

• Separate the interface from the appliance!
• Handheld becomes personal universal controller
  (PUC)
• Key Features
• Interface-independent appliance specification
• Automatic generation of GUI and speech interfaces

5
Automatic Generation of UIs

• Benefits
• All interfaces consistent for the user
• With conventions of handheld
• Even from multiple manufacturers
• Addresses hotel alarm clock problem!
• Multiple modalities (GUI Speech UI)
• Can take into account user preferences
• Will work on special purpose devices (for
  disabled)

6
Important Work By Others

• INCITS V2 Standardization Effort
• Alternative Interface Access Protocol (AIAP)
• Zimmermann, CHI 2002
• User Interface Modeling Language (UIML)
• http/www.uiml.org/
• Xweb (now ICE)
• Olsen Jr., UIST 2000
• Stanford iRoom, iCrafter
• Ponnekanti, Ubicomp 2001
• Speakeasy
• Newman, UIST 2002

7
Architecture
- Comm. Protocol
- Interface Generators
- Specification Lang.
- Appliance Adaptors
XML-based
8
Language Design

• Approach
• Create reference interfaces
• AIWA Shelf Stereo
• ATT Telephone/Answering Machine
• Test interfaces with subjects
• Users twice as fast and made half the errors with
  reference interfaces as compared to
  manufacturers interfaces
• Analyze interfaces for functional information

9
Language Elements

• State Variables and Commands
• Represent functions of appliance
• State variables have types
• Boolean, Enumeration, Integer, String, etc.
• Variables sufficient for most functions but not
  all
• seek button on a Radio
• Label Information
• One label not suitable everywhere
• The optimal label length changes with screen size
• Speech interfaces may benefit from pronunciation
  and text-to-speech information

10
Language Elements, cont.

• Group Tree
• Specify organization of functions
• We use n-ary tree with variables or commands at
  leaves

11
Language Elements, cont.

• Dependency Information
• Formulas that specify when a variable or command
  is active in terms of other state variables
• Equals, Greater Than, Less Than
• Linked with logical operators (AND, OR)
• For example,
• ltandgt ltequals statePowerStategttruelt/equa
  lsgt ltequals stateRadioBandgtAMlt/equalsgtlt/and
  gt

12
Interface Generators

• Generators for Two Modalities
• Graphical
• Implemented for PocketPC in Java 1.1
• Uses dependency information to generate panel
  structure of interface
• Speech
• Implemented using Universal Speech Interface
  (USI) techniques Rosenfeld 2001
• Uses dependency information to disambiguate
  shortcut words (e.g. play) and resolve
  pre-conditions for a requested function (e.g.
  play CD)

13
Graphical Interface Generator

• Focuses on panel structure of user interface
• Small groups of controls have basic layouts
• Complexity comes from structure of groups
• Structure can be inferred from dependency info!

14
Inferring Structure

• Find sets of variables that are mutually
  exclusive
• Every variable in a set will never be active at
  the same time as a variable in another set
• Create structure with sets, using overlapping
  panels

15
Choosing Panel Types
a)
b)
c)
full screen
partial screen
tabbed
16
Making the Interface Concrete

• Finish conceptual layout
• Choose controls (decision tree)
• Choose row layouts
• (one column, two column, etc.)
• Allocate space
• Examine panel contents and choose sizes
• Instantiate and place controls

17
Generating Speech Interfaces

• Automatically build USI tree from dependencies
• Allows verbal navigation of functional groups
• Automatically generate grammar for parser
• Phrases for query and control
• What is playmode?
• Set playmode to play
• play
• Automatically generate language model and
  pronunciation for recognizer

18
Controlling Appliances

• We have built adaptors for many actual appliances
• Sony Digital Camcorder
• X10 Lighting
• Audiophase Shelf Stereo
• AudioReQuest MP3 player
• Software Media Players (WinAmp, WMP)
• Written specifications for others
• Elevator
• Telephone/Answering Machine

19
Demonstration
20
Future Work

• Improve quality of generated user interfaces
• Back-tracking, etc.
• Specification and generation of list interfaces
• Necessary for many appliances
• Adaptive interface generation
• New interfaces modeled on past interfaces
• General purpose appliance adaptor for HAVi and
  UPnP protocols

21
Acknowledgements

• PUC Project Members
• Brad A. Myers
• Michael Higgins
• Joseph Hughes
• Thomas K. Harris
• Roni Rosenfeld
• Mathilde Pignol
• Stefanie Shriver
• Peter Lucas
• Kevin Litwack

• Funding
• National Science Foundation
• Pittsburgh Digital Greenhouse
• Microsoft
• Equipment Grants
• Mitsubishi (MERL)
• VividLogic
• Symbol Technologies
• Hewlett-Packard
• Lucent

22
Thanks!
http//www.cs.cmu.edu/pebbles/puc/ http//www.cs.
cmu.edu/jeffreyn/

• User Interface Software Technology (UIST)
• October 30, 2002

23
(No Transcript)
24
Determining Panel Structure

• Find states that others depend upon
• Search for the group that contains this state
• Determine mutually exclusive sets within group
  children
• Assign sets to panels
• Based upon types, tree properties

FMStation
Mode Any
FMPresets
Power
AMStation
Mode
AMPresets
Band
Radio
Mode Radio
PlayState
CD
CDTrack
Mode CD
25
Choosing Panel Types

• Rules for Choosing Panels
• If
• state has boolean type
• all controls are one set
• Create
• Two full-screen overlapping panels.

26
Choosing Panel Types, cont.

• Rules for Choosing Panels
• If
• state has enumerated type
• there is a set for every value of the state
• Create
• a tabbed panel

27
Choosing Panel Types, cont.

• Rules for Choosing Panels
• If
• there are mutually exclusive sets of controls
• Create
• overlapping panels
• a control for the state not on the overlapping
  panel