User Interfaces for Post-Desktop Environments

1
User Interfaces for Post-Desktop Environments

• Jan BorchersStanford Interactivity
  Labhttp//www.stanford.edu/borchers/
• Advanced Systems Development LabTalk Series,
  PARC, 6/3/2002

2
Outline

• The Stanford iRoom A Ubicomp Environment
• User Interface Challenges for Ubicomp
  environments
• iStuff Supporting Research in Ubicomp UIs

3
The Shift to Ubiquitous Computing

• Desktop-centric
• Generic machines
• Client-server
• Monolithic apps
• One person, one application, one set of hardware

• Invisible/implicit interaction
• Dedicated appliances
• Universal glue-like connectivity
• Infrastructure-centric services
• Multi-person, multi-application, multi-device
  integration

4
What Is an Interactive Room?

• An augmented dedicated space
• e.g. conference room
• can be used with teleconnected spaces
• Task-oriented scenarios
• e.g. brainstorming meeting, design review
• Multi-device, multi-user, multi-application
• Easy to (re)configure
• Movement of information across multiple devices,
  and from personal to/from multi-user devices
• Integration augmentation of legacy apps

5
The Stanford iRoom
6
The Stanford iRoom
7
Whats in the iRoom Today

• 3 video cameras
• Projector Control System
• Speakers for each Smart Board
• X-10 Lighting Control
• iPaq PDA
• Printer, scanner
• Anoto pen, Wacom tablets
• Other custom hardware
• Behind the Scenes
• 5 Windows 2000 Boxes
• 5 Windows NT 4.0 Boxes
• 2 Windows 98 Boxes
• 32 processor Linux Cluster
• 1 Mac OS X Box
• Lightning 2 Digital Video Switch
• IEEE 802.11 Wireless LAN
• 1000s of feet of cable!
• more as time goes on...

• Accessible Infrastructure
• 3 4x5 Touch Sensitive Smart Boards
• Back Projected 4x6 Interactive Mural with 12
  tiled projectors (9Mpix)
• Interactive Table with 3x4 Screen and Pen Input
• Control Panel Machine with Wacom LCD Tablet
  Screen
• PointRight Mouse and Keyboard Redirection System
• Laptop video drop switched to all displays
• Cluster Control Machine with display to Smart
  Boards

8
iRoom Video

• http//mediastream.stanford.edu8080/video/iwork_s
  cenario/iwork_scenario_dv.ram

9
Application Domains
3-D Medical Imaging (Med School/Radiology)
Design Brainstorming (Center for Design Resarch,
IDEO)
Construction project management (CivEng Center
for Integrated Facilities Engineering)
10
Ubicomp UI Challenges

• Multiple users, multiple applications, multiple
  displays
• No simple 2-D "desktop"
• Window? Focus? Pointing? Selection?
• Post-Desktop equivalent of mouse keyboard?

11
Some iRoom Technology Solutions

• Mural cluster High-resolution interactive wall
  ("Chromium")
• Event Heap "Event queue for the room"
• MultiBrowse Sending URLs around the room
• Data Heap Storing ( converting) information
  in the room
• PointRight One mouse keyboard for the room
• Room Controller Web pad to control room lights,
  projectors, ....
• Interface Crafter Generating interfaces to room
  services
• Barehands Gesture control for Smart Boards

12
iStuff

• Prototyping desktop GUIs has become easy
• VisualBasic, Interface Builder, ...
• Lesson learned Rapid ubicomp UI prototyping is
  hard
• Example "Add a feedback light to a tablet-based
  UI"
• -gt Run wires, solder adapters, write serial
  drivers,...
• Idea A toolkit for building ubicomp UIs?
• Simplify experimentation with ubicomp UI
  scenarios designs
• Basic input/output "phidgets" Greenberg01
• But Wireless, design for augmented environment
  (iRoom)

13
iStuff We've Built

• Input devices
• iButton
• iSlider
• Output devices
• iLight
• iBuzzer
• iSpeaker
• Others in the works (sensors, MIDI controllers,
  microphones, servos, ...)

14
Taking Advantage of Infrastructure
iSpeaker
iSlider
Radio Transmit
Radio Receiver
Application
PC daemon
PC proxy
eheap.jar
eheap.jar
eheap.jar
Event Heap

• Any Event Heap based application can talk to
  iStuff
• iStuff device wireless device PC proxy/daemon
• Lightweight, cheap, low-power, ... devices

15
Event Heap app to speak text
import iwork.eheap2. class speaktext
// Connects to
event heap in static void main(String args)
// arg0, and sends
AudioEvent
// with the text in arg1.
try EventHeap theHeapnew
EventHeap(args0) // Connect to the Event
Heap Event myEventnew
Event("AudioEvent") // Create an event
myEvent.setPostValue("AudioCommand",
"Read") // Set its fields
myEvent.setPostValue("Text", args1)
theHeap.putEvent(myEvent) //
Put event into Event Heap
catch(Exception e)
e.printStackTrace()
gtspeaktext iw-room2.stanford.edu "Hello World"
16
Initial Evaluation

• Successful sample uses
• Start The Room
• iPen
• iPong
• Findings and next steps
• Allows rapid integration of ubicomp UI components
  into apps
• Spend time designing the software and interface,
  not soldering
• Advantage of collaboration with Product Design
  ME students
• Abstraction layers device - normalized -
  application specific
• Device Classes and IDs, PatchPanel
• Applications need to be written differently

17
Great Unified Input Theory

• iStuff wireless hardware layer can be RF, X-10,
  Bluetooth, 802.11b, Infrared, Cellular,...
• Offer iStuff as abstraction layer of room devices
  above this level
• Adopters can
• Use our hardware and software designs and
  replicate devices
• Use our software and use off-the-shelf components
  as iStuff
• Implement the "plugins" for new hardware
  technologies
• Make applications listen only to Event Heap Events

18
Summary

• The Stanford iRoom
• Ubicomp requires re-thinking user interface
  metaphors and architecture
• iStuff as a tool to facilitate experimentation in
  ubicomp interfaces
• Not the answer to "post-desktop mouse
  keyboard", but may help finding it...
• http//iwork.stanford.edu/
• http//www.stanford.edu/borchers/iStuff/

19
Bonus Material
20
PointRight

• A system for pointer/keyboard redirection among
  multiple displays and machines (Brad Johanson,
  Greg Hutchins)

video
21
BareHands

• Gesture-based interaction with a wall-mounted
  display (Merrie Ringel)