Ubiquitous Computing Toolkits

1
Ubiquitous Computing Toolkits

• Tara Matthews
• Advance User Interface Software Fall 2004

2
Goals of Ubicomp

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity

3
Toolkits for Goals

• Scalable interfaces
• multiple devices Ubicomp Infrastructure
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity

4
Toolkits for Goals

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users CSCW (not covered)
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity

5
Toolkits for Goals

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop Physical UIs /
  Mobile
• augmenting surrounding environment (other
  lectures)
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity

6
Toolkits for Goals

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
  Context-Aware
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity

7
Toolkits for Goals

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users Peripheral
  Displays
• aware of users interruptiblity

8
Toolkits for Goals

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity
  Interruptibility (not covered)

9
Toolkits for Goals

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity
• Goals accomplished? Evaluation

10
Toolkit Categories

• Ubicomp infrastructure
• Context-aware
• Peripheral displays
• Evaluation
• Not covering
• CSCW (separate field)
• Interruptibility (no toolkits yet)
• Physical UIs (covered in Jacks lecture)
• Mobile Devices (covered in an unclaimed lecture)

11
Ubicomp Infrastructure

• iROS
• Stanford Interactive Room Operating System
• iStuff
• Physical application toolkit build on iROS
• Aura
• CMU Distraction-Free Ubiquitous Computing
  project
• Distributed services infrastructure
• Gaia
• University Of Illinois at Urbana-Champaign
• Infrastructure for Active Spaces

12
iROS

• Interactive Room (iRoom) Operating System
• Definition A ubiquitous computing environment
  where groups come together to collaborate on
  solving problems (not a toolkit)
• Contains
• Embedded devices large display screens,
  table-top display, other output devices
• Mobile devices mobile devices brought into space
  can be used with embedded facilities

13
iROS Goals

• Facilitate common Interactive Workspace usages
  observed in iRoom prototype workspace
• Move data between devices
• Control devices apps from other devices
• Coordinate apps, including ones not written to
  work together
• Additional goals
• Provide for heterogeneity of devices in
  workspace, new devices being added over time
• Allow easy integration of legacy devices
• Robust to transient failures
• Portable across installations
• Actions appear instantaneous to users

14
iROS

• 3 sub-systems
• Event Heap stores and forwards events
• Data Heap facilitates data movement in an
  interactive workspace
• iCrafter provides a system for service
  advertisement and invocation, along with a user
  interface generator for services.

15
Event Heap

• Tuple space model
• Associated memory content is addressable using a
  pattern
• Distributed systems based on blackboard model are
  easily created in tuple space
• Decouples apps in space time
• Added semantics
• Apps not designed to work together, so need
• Self-describing tuples, Flexible typing, Typed
  tuples
• Apps can snoop or interpose
• Tuple sequencing
• Data shared can build up
• Tuple expiration
• Benefits
• Anonymous coordination
• Interposability
• Snooping

16
Event Heap
Figure Example operation showing placed events,
and using template events for matching
17
Data Heap

• Provides type location independent storage
• Machine independent
• Dont need explicit location of data
• Type independent
• If a set of type converters are available, system
  automatically transforms the data

Figure Shows automatic retrieval of a Word
document in PostScript form
18
iCrafter

• Universal control system control anything from
  anywhere
• Services advertise how they can be controlled
• System returns appropriate interface per device
• Interfaces can range from fully custom to
  dynamically generated

19
iCrafter
20
iCrafter
21
iCrafter
22
iCrafter
23
iStuff

• Toolkit to prototype physical UIs
• Flexible, lightweight components
• Protocol independence
• Platform independence with cross-platform
  capabilities
• Ease of integration with existing applications
• Support for multiple simultaneous users
• Built on iROS
• iStuff wireless devices (e.g., iButton, iSlider,
  iLight) send/receive events to/from Event Heap
  via a proxy
• PatchPanel translates events (e.g., iButton
  light on light off)
• Apps get events from devices via PatchPanel
  Event Heap

24
iStuff Architecture
25
iStuff Devices
Input
Output
26
Aura

• CMU Distraction-Free Ubiquitous Computing
  theme
• Mobility allow users to move computational tasks
  from one place to another
• Maximize use of available resources
• Minimize user distraction drains on attention
• Personal information aura spans wearable,
  handheld, desktop, infrastructure devices
• Requires new system design hardware, network,
  OS, middleware, UI support

27
Aura Research Framework

• Research Examples
• UI / Agents / Speech
• Task-driven computing
• Rapid service configuration
• Energy-aware adaptation
• Multi-fidelity computation
• Nomadic data access
• Intelligent networking
• Power-aware devices
• Wearable computers

• Users
• Tasks
• Services
• OS / Network
• Physical Devices

28
Aura Example Scenario

• Fred prepares presentation is late
• Fred gets PDA Aura transfers slides to it
• Finishes slides on PDA walking to meeting
• Aura finds location of meeting from calendar
  uploads slides to projection machine
• Aura recognizes unfamiliar meeting attendees
  faces skips budget slides

29
Aura Architecture

• User tasks are represented as set of distributed
  services
• Database query interface to access distributed
  services
• Easily search mixed environments for needed
  services
• A key service is People Location Service
• Environments self-monitor re-negotiate task
  support given runtime variation of capabilities
  resources
• Can detect when task requirements (e.g., min
  response time) arent met, search deploy
  alternative configurations to support task
• Uses software architecture models (graph of
  components connectors) for runtime adaptability

30
Aura Architecture

• Task Manager tracks user context, environment,
  task changes
• Context Observer provides info on physical
  context reports relevant events Task
  Environment Managers
• Environment Manager handles access to
  distributed services
• Suppliers provide services for tasks (text
  editing, video playing,...)

31
Privacy in Aura

• Aura services depend on user location info
• Introduces privacy concerns
• Location policies define who gets location info,
  where, when, and at what granularity
• e.g., Bob is allowed to know Alices location
  when she is in NSH
• Similar to Bell Labs Houdini systems rules
• Not based on how users actually disclose location

32
GAIA

• Infrastructure for Active Spaces, UIUC
• Like Aura
• Distributed services architecture accessed via
  structured queries
• Main difference
• Focus on user customization of apps based on
  resources in current space

33
Toolkit Categories

• Ubicomp infrastructure
• Context-aware
• Peripheral displays
• Evaluation

34
Context

• Context is key to ubicomp environment that reacts
  correctly to users current situation
• Definition Info used to characterize the
  situation of a person, place or object relevant
  to the interaction between a human a
  computational service
• Primary context types
• Identity (who), Location (where), Time (when),
  Activity (what)
• Secondary context types
• e.g., for identity email address, phone number,
  birthdate, etc

35
Context-Aware Applications

• Definition Uses context to provide relevant info
  and/or services to the user, where relevancy
  depends on the users task
• 3 categories
• Present information services to users
• e.g., nearby wireless networks, directions
• Automatically execute services
• e.g., phone vibrates when in meeting
• Tag info with context for later retrieval
• e.g., class lectures and notes

36
Toolkits for Context

• 3 main approaches
• Widget model
• Context Toolkit
• Dey, Abowd, Salber, 2001
• Based on architecture of GUIs
• Distributed services model
• Context Fabric
• Hong Landay, 2001
• Based on client-server dialog
• Blackboard model
• iRoom
• Winograd, 2001
• Based on artificial intelligence apps (Engelmore,
  1988)

37
Widget Model Context Toolkit

• Operating system view
• Abstraction to hardware (sensors)data is
  secondary
• Similar to GUI input handling uses widget
  abstractions for handling context input
• Uses 3 abstractions
• widgets, interpreters aggregators

38
Benefits

• Separates semantics from low-level input sensor
  handling
• Apps can focus on context, not sensors or
  analysis of sensor input
• Allows re-use of context widgets, interpreters
  aggregators

39
Drawbacks

• Tight coupling less robust to component failure
• Single-manager (Discoverer) control less
  flexible

40
Context Toolkit Architecture
41
Widgets

• Encapsulate info about a single piece of context
  (e.g., location or activity)
• Hide details of underlying sensors from apps
• Provide customizable reusable building blocks
  of context sensing
• Provide historical context info

42
Widgets

• Include optional services, or actuation of output
  in environment
• Like Interactors responsible for input output
• e.g., light widget could sense light level
  adjust lights accordingly
• Get context by subscription or polling
• Callback methods used to notify subscribers

43
Aggregators

• Widgets ability to aggregate context info
• Collect sensory info about an entity (person,
  place, thing, etc.) from multiple sources into
  one widget
• Hide more complexity about context-sensing
  mechanisms by combining multiple sensors
• Enable maintainability and efficiency

44
Interpreters

• Abstract sensor info
• Use lower level sensory information to deduce
  higher level info
• e.g., identity location sound sensors ?
  is there a meeting?

45
Putting an App Together

• Discoverer enables apps to locate and subscribe
  to context components
• Distributed infrastructure uses p2p comm.

46
Distributed Services Model

• Database view (vs. OS view of CTK)
• Data is primary, hardware is secondary
• Focus on how data is modeled, distributed,
  protected, used
• Middleware technologies that can be accessed
  through a network
• Example infrastructureInternet serviceDNS
• Any kind of device or application can use context
  gathering processing services by adhering to
  predefined data formats network protocols

47
Benefits

• Interoperable independent of hardware platform,
  OS, programming language
• Decoupling Sensors, services, apps can be
  changed w/o affecting each other
• Simpler devices sensors, processing power, data,
  services w/in infrastructure shared

48
Drawbacks

• Applications must poll for data (proactively)
• Servers are specialized per sensor

49
Context Fabric

• Includes 3 pieces
• Distributed data model of people, places, things
• each device has a space private data goes to
  local space
• multiple views of context (mine, yours, theirs)
• Context Specification Language (like SQL)
• apps specify what context info they need in
  uniform way
• e.g., "What are the nearby movie theaters?
• Context service as API (per device)
• interpret CSL queries and provide context info

50
Privacy in Context Fabric

• Decentralized architecture allows it to capture,
  store, process, share in privacy-sensitive
  manner
• Capture, store, process data on users device
• Provides mechanisms for end-user control of
  sharing
• Plausible deniability built-in
• e.g., if request for context info denied, system
  can reply unknown

51
Aura

• Also follows distributed services model
• Uses SQL-like interface to access a set of
  distributed contextual services
• Handles privacy through user-set policies

52
Blackboard Model

• Communication goes through a centralized server
• Components
• post messages shared blackboard
• subscribe to get posted messages matching
  specified pattern
• Route by matching message content to subscriber's
  pattern
• Pattern matching uses AI (often apply
  sophisticated inference procedures to logical
  representations)
• Direct communication between components simulated
  by including an identifier for path endpoints as
  a field in message
• Announce-listen style
• Components that provide services periodically
  post events
• Component that uses the data listens to these
  events, if one does not appear within the
  expected time can initiate restart operations

53
Benefits

• Ease of configuration
• Robustness to component failure
• When component fails, only communication link
  broken is from it to blackboard
• Dependent components can detect failure and call
  for restart
• Simplicity provided by a uniform communications
  path (to the blackboard)
• No complex protocol for finding ports or
  resources, establishing connections

54
Drawbacks

• Communication less efficient
• Each communication requires 2 hops
• General message structure not optimized for
  particular data or interaction protocol

55
iRoom iStuff

• Event Heap the blackboard central
  communication server

56
Review of Context Approaches

• Widget Model CTK
• Widget architecture where Discoverer acts as a
  manager of context widgets
• Distributed Infrastructure Model ConFab, Aura,
  Gaia
• Client-server architecture where client apps
  communicate directly w/ services using structured
  language
• Blackboard Model iRoom, iStuff
• Central server architecture where apps
  communicate only w/ server events are routed
  using subscriber pattern matching

57
Toolkit Categories

• Ubicomp infrastructure
• Context-aware
• Peripheral displays
• Evaluation

58
Peripheral Displays

• Provide awareness with min attention
• Separate from primary task
• Important to ubicomp vision of many devices to 1
  user
• Non-focal display not used unless providing
  peripheral info
• Enable you to monitor many info sources while
  maintaining a calm environment
• But, only calm if designed to manage attention
  they attract

59
Why is creating PDs hard?

• Need to abstract info to be glance-able
• Need mechanisms for dynamically managing
  attention PDs attract
• Deciding attention levels to attract(notification
  levels)
• Displaying info appropriately (transitions)
• Peripheral Display Toolkit (PTK)
• Supports these 3 key issues in PD creation

60
Example PTK Applications

• Remote Activity
• Social Guitar
• Audio Monitor
• Motion Monitor
• Remote Awareness Display
• Bus Displays
• Bus Mobile
• Bus LED
• Instant Messenger Status

Orb showing remote activity
Bus LED BusMobile
IM Picture Frame
Social Guitar
61
PTK Architecture

1. Support for managing impact on human attention
   using abstraction, notification levels, and
   transitions
2. Simplified code design and code re-use
3. Library of common PD components

Input
Abstractor
Transition
NotificationMap
Output
62
Architecture Diagram
63
Library Components

• Input
• audio, camera, Phidgets, Context Toolkit, online
  calendars, news, stocks, weather, Web page
  parser, serial port

64
Library Components

• Input
• audio, camera, Phidgets, Context Toolkit, online
  calendars, news, stocks, weather, Web page
  parser, serial port
• Output
• ticker text, Ambient Orb, Phidgets

65
Library Components

• Input
• audio, camera, Phidgets, Context Toolkit, online
  calendars, news, stocks, weather, Web page
  parser, serial port
• Output
• ticker text, Ambient Orb, Phidgets
• Abstractors
• motion, people counting, voices, phone ringing

66
Library Components

• Input
• audio, camera, Phidgets, Context Toolkit, online
  calendars, news, stocks, weather, Web page
  parser, serial port
• Output
• ticker text, Ambient Orb, Phidgets
• Abstractors
• motion, people counting, voices, phone ringing
• Notification
• exact match, threshold, contains, degree of
  change

67
Toolkit Categories

• Ubicomp infrastructure
• Context-aware
• Peripheral displays
• Evaluation

68
Evaluation

• 3 tools for evaluating people in natural settings
  from Intille et al.
• Context-aware experience sampling based on GPS or
  other sensors
• e.g., ask questions when user goes to store
• Ubiquitous sensing system that detects
  environmental changes
• e.g., tape-on touch, proximity, light sensors
• Image-based experience sampling system
• i.e., take a picture ask user about it later

69
Summary Ubicomp Goals

• Scalable interfaces
• multiple devices
• inter-device communication
• multiple users
• Natural interaction
• getting away from the desktop
• augmenting surrounding environment
• provide context-appropriate services
• Calmness
• more info w/o overwhelming users
• aware of users interruptiblity

70
Summary Ubicomp Toolkits

• Ubicomp infrastructure
• iROS, iStuff, Aura, Gaia
• Context-aware
• Context Toolkit, Context Fabric, iRoom
• Peripheral displays
• Peripheral Display Toolkit
• Evaluation
• Intilles toolkit

71
Thanks!
72
CSCW

• Groupware Architectures
• Phillips, W.G., 1999. Architectures for
  Synchronous Groupware, Tech. Rep..
  http//phillips.rmc.ca/greg/pub/
• Greenberg, S. and Roseman, M., 1999. Groupware
  Toolkits for Synchronous Work. In
  Beaudouin-Lafon, M. (Ed.), Trends In CSCW'99, No.
  7 in Trends in Software, John Wiley Sons, New
  York, NY, USA, ch. 6, pp. 135168.
• Roseman, M. and Greenberg, S., 1992. GROUPKIT a
  groupware toolkit for building real-time
  conferencing applications. In Proceedings of the
  conference on Computer-supported cooperative
  work, ACM Press, pp. 4350. http//doi.acm.org/10.
  1145/143457.143460