saving user attention in ubiquitous computing

1
saving user attention in ubiquitous computing

• Scaling Task Managementin Space and Time

committee David Garlan, chair Mahadev
Satyanarayanan Reid Simmons Gregory Abowd,
Georgia Tech
João Pedro Sousa March 28, 2005
2
the problemusers work hard
when moving tasks from one location to another

• save files
• stop applications

• remember task
• transfer data
• find restart apps
• reconstruct state

• worse with
• different platforms
• resource variation

3
ideallyall migration and configuration handled
automatically
when moving tasks from one location to another

• leave

• select task to resume

4
reality
solutions

• desktop managers(Rooms, 1987)

5
saving user attention in ubiquitous computing

• Scaling Task Managementin Space and Time

committee David Garlan, chair Mahadev
Satyanarayanan Reid Simmons Gregory Abowd,
Georgia Tech
João Pedro Sousa March 28, 2005
6
outline

• the problem
• how existing solutions fall short
• new approach
• summary of research
• validation
• related future work
• contributions
• discussion

7
Goal
support mobile users in their daily tasks

• Enable users to
• carry out their tasks in diverse environments
• take full advantage ofthe available capabilities
  and resources

• Reduce user overhead in managing
• heterogeneity
• resource variability

tension
computers and info everywhereheterogeneous
devices and applications
8
arent existing solutions good enough?
support mobile users in their daily tasks

• desktop managers
• remote computing
• mobile computing
• code migration
• internet suspend/resume

9
desktop managers
10
remote computing
thin clients
11
mobile wearable computing
people try to be self-sufficient by carrying
personal devices
12
code migration
personalized applications andtheir data follow
the user around Ex Gaia, ICrafter, Kimura,
Labscape
(new apps)
13
internet suspend/resume
migrate contentsof the virtual memory
(no changes to apps)
14
arent existing solutions good enough?
good job at some of the requirementsbut a poor
job at others
reducedoverhead
tasksupport
take fulladvantage
usermobility

desktop managers

remote computing
tension

mobile computing

code migration

internet S/R
15
new Approach
support mobile users in office-like tasks by
migrating high-level models of user tasks
infrastructure
infrastructure
16
new Approachhighlights
17
outline

• the problem
• new approach
• summary of research
• thesis
• challenges
• validation
• related future work
• contributions
• discussion

18
terminology

• servicewhat users need
• suppliersoftware/device offering services
• resourceconsumed by supplierswhile delivering
  services
• materialinformation asset

print, edit text, browse the web
MS Word, Emacs
CPU, battery, bandwidth
files, data streams
19
thesis statement
High-level models of user tasks can be used to
address the scalability of task management

• in space, across heterogeneous environments
• in time, enabling users to recover past tasks

while simultaneously

• enabling users to take full advantageof the
  services and resources of computing environments
• reducing user overhead associated with
  configuring and managing those environments

20
challenges
infrastructure
infrastructure
21
challengesmodeling user tasks
model
22
challengesdescribing operating on tasks
user-interaction mechanisms
23
challengesinfrastructure for task management
configure environment
24
describing operating on taskstask
suspend/resume
user-interaction mechanisms

• describe tasks
• associate services materials
• dynamic updates
• incremental benefit for incr. effort
• clear assumptions easy to correct
• browse tasks
• operate on tasks
• show likely tasks
• indicate feasibility
• trigger environment configuration

25
describing operating on tasksenable users to
find past tasks
user-interaction mechanisms

• describe tasks
• associate services materials
• dynamic updates
• incremental benefit for incr. effort
• clear assumptions easy to correct
• browse tasks
• operate on tasks
• show likely tasks
• indicate feasibility
• trigger environment configuration

26
describing operating on tasksdescribe user
needs
27
modeling user tasks
model

• services materialswhat users need,not which
  applications
• user-perceived statewindow layout, cursors
• persistent semantic identitygoals, time place,
  collaborators
• user preferences

28
modeling user tasksindependently of specific apps
ltauraTask id"34"gt ltpreferences /gt ltservice
/gt ltmaterial /gt ltconfiguration name"skip
web" weight"0.8"gt ltservice
id"1"gt ltuses materialId"87"/gt ltuses
materialId"45"/gt lt/servicegt ltservice
id"3"gt ltuses materialId"102"/gt
lt/servicegt lt/configurationgt ltconfiguration
name"paper only" /gt ltconfiguration name"full"
/gt lt/auraTaskgt
29
modeling user tasksindependently of specific apps
ltauraTask id"34"gt ltpreferences /gt ltservice
type"edit Text" id"1"gt ltsettingsgt ltformat
overtype"0"/gt lt/settingsgt lt/servicegt
ltmaterial id"87"gt ltstategt ltcursor
position"31510"/gt ltscroll horizontal"0"
vertical"7"/gt ltzoom value"140"/gt
ltspellchecking enabled"1" language"English"/gt
ltwindow height"500" width"600"
xpos"20" ypos"100" mode"min"/gt lt/stategt
lt/materialgt ltconfiguration /gt lt/auraTaskgt
30
modeling user tasksenable users to find past
tasks
lttask state"pending" id"34"gt ltdescriptiongt
ltnamegtconference paperlt/namegt ltnotesgtmain
result scalability of task management
in space and timelt/notesgt
ltcollabsgtDavidVahelt/collabsgt lt/descriptiongt
lthistory due"2/07/04" created"30/06/04 951
PM"gt ltaccessed at"home"
stop"30/06/04 1032 PM"
start"30/06/04 953 PM"/gt ltaccessed /gt
lt/historygt ltlinksgt ltlink label"previous
conference" tId"14"/gt lt/linksgt lt/taskgt
31
modeling user tasksheterogeneity change
32
infrastructure for task managementexploits these
models
configure environment

• find the best match to user needs
• resource allocation across applications
• utility-theoreticcomparison of alternatives
• activate/deactivate suppliers
• reconstruct user-perceived state
• monitor adapt to changes
• framework for adaptation
• coordinate adaptation policies

ltauraTask id"34"gt ltpreferences /gt ltservice
type"edit Text" id"1"gt ltsettingsgt ltformat
overtype"0"/gt lt/settingsgt lt/servicegt
ltmaterial id"87"gt ltstategt ltcursor
position"31510"/gt ltscroll horizontal"0"
vertical"7"/gt ltzoom value"140"/gt
ltspellchecking enabled"1" language"English"/gt
ltwindow height"500" width"600"
xpos"20" ypos"100" mode"min"/gt lt/stategt
lt/materialgt ltconfiguration /gt lt/auraTaskgt
33
infrastructure for task managementseparate
concerns
34
infrastructure for task managementseparate
concerns
35
infrastructure for task managementuses models to
configure environment
Task Management
Task Management
Managed Environment
Managed Environment
36
infrastructure for task managementarchitectural
framework
37
infrastructure for task managementarchitectural
framework
38
infrastructure for task management implementation
Prism
User
Speakeasy
Task Manager
Distributed File System
Task
Dashboard
Lamp
Java Swing
Identity
Task
Focus
Models
XML/TCP
39
infrastructure for task management implementation

• Suppliers (several students)
• MS Word
• PowerPoint
• Excel
• Internet Explorer
• Media Player
• Babel Fish, Festival, Emacs, Pangloss, Sphinx,
  Xanim
• EM (Vahe Poladian)

UI
Prism
Context
Observer
UI
service
Environment
Supplier
Manager
App
Operating System
40
challenges
usable on adaily basis
infrastructure
infrastructure
41
are delays introduced by the infrastructurea
source of overhead?
measure latency of

• operational Dashboardafter user authentication
• read task directory
• search pending tasks
• evaluate task feasibility
• search tasks using Lamp
• suspend/resume tasks

experiments on a IBM ThinkPad 30 Windows XP, 1.6
GHz, 512 MB
42
latency introduced by infrastructureblends in
with starting/stopping apps

• operational Dashboardafter user authentication
• read task directory
• search pending tasks
• evaluate task feasibility
• search tasks using Lamp
• suspend/resume tasks

3sfor 10 tasks
lt 1s
lt 1s
43
latency introduced by infrastructureblends in
with starting/stopping apps

• operational Dashboardafter user authentication
• read task directory
• search pending tasks
• evaluate task feasibility
• search tasks using Lamp
• resume tasks
• suspend tasks

avg. 200ms (dev. 50ms)between 4-24 combinations

• avg. 700ms (dev. 200ms)
• avg. 170ms (dev. 20ms)

44
outline

• the problem
• new approach
• summary of research
• thesis validation
• cost-benefit analysis
• related future work
• contributions
• discussion

45
thesis validation
scalability of task management

• in space, across heterogeneous environments
• in time, enabling users to recover past tasks

while simultaneously

• enabling users to take full advantage
• reducing user overhead

?
46
thesis validationreducing user overhead

• compare overheadwith vs. without the
  infrastructure
• focus on c-tasks
• suspend/resume tasks
• describe tasks
• find tasks
• general model of overhead(independent of
  specific apps/OS)
• count of user operations(coarse interactions
  start apps, open files...)

47
reducing user overheadmodel overhead of each
c-task
starting a task of preparing a presentation
without ops with ops overhead
open app (slide editor) look for relevant papers open app (text viewer) on the papers look for web pages 1 n m k new task new service (slides) resume task look for relevant papers add the papers to the task save task look for web pages add pages to the task save task 1 1 1 n m 1 k n. pages 1 1 1 1 n. pages 1
() For opening an application, users issue a command, click on some icon, etc. for adding the slide editing service to the new task, users click one button and make a list selection. () For adding the papers to the new task, users drag the papers from some file view in the operating system and drop them into the more area in the Dashboard. () For opening an application, users issue a command, click on some icon, etc. for adding the slide editing service to the new task, users click one button and make a list selection. () For adding the papers to the new task, users drag the papers from some file view in the operating system and drop them into the more area in the Dashboard. () For opening an application, users issue a command, click on some icon, etc. for adding the slide editing service to the new task, users click one button and make a list selection. () For adding the papers to the new task, users drag the papers from some file view in the operating system and drop them into the more area in the Dashboard. () For opening an application, users issue a command, click on some icon, etc. for adding the slide editing service to the new task, users click one button and make a list selection. () For adding the papers to the new task, users drag the papers from some file view in the operating system and drop them into the more area in the Dashboard. () For opening an application, users issue a command, click on some icon, etc. for adding the slide editing service to the new task, users click one button and make a list selection. () For adding the papers to the new task, users drag the papers from some file view in the operating system and drop them into the more area in the Dashboard.
48
reducing user overheadmodel overhead of each
c-task
resuming a task
without ops with ops benefit
find files and links open files/links with apps recover state of apps 1 n. files/pages n. files/pages find task in Dashboard resume 1 1 n. files/pages 1 n. files/pages
() Assuming that users have fresh on their minds where all the files are stored. () Assuming recovering the user-perceived state of each service/file takes a single operation.For instance, recovering the editing position on a text file. () Assuming that users have fresh on their minds where all the files are stored. () Assuming recovering the user-perceived state of each service/file takes a single operation.For instance, recovering the editing position on a text file. () Assuming that users have fresh on their minds where all the files are stored. () Assuming recovering the user-perceived state of each service/file takes a single operation.For instance, recovering the editing position on a text file. () Assuming that users have fresh on their minds where all the files are stored. () Assuming recovering the user-perceived state of each service/file takes a single operation.For instance, recovering the editing position on a text file. () Assuming that users have fresh on their minds where all the files are stored. () Assuming recovering the user-perceived state of each service/file takes a single operation.For instance, recovering the editing position on a text file.
49
benefit of using the infrastructureadds up over
time
accumulated benefit of using the infrastructure
time(c-task)
(disregarding scalability in space and time)
50
benefit of using the infrastructureincreases
with user mobility

• environment
• heterogeneity
• dynamic change

unassisted configuration much harder depending on
user expertise

• finding past tasks
• no significant differencewhen compared, e.g.,
  with Google Desktop Search

51
outline

• the problem
• new approach
• summary of research
• validation
• related future work
• contributions
• discussion

52
related work
task management
task modeling
assistance with tasks

• desktop managersone machine
• Rooms (1987),
• teleport desktopsthin clients
• X windows (1994)
• MS Easy Living
• TM in smart spacesmigrate (new) apps
• Kimura, ICrafter, Gaia, Labscape
• user mobilitymigrate virtual memory
• ISR

• plans to be carried outby automated systems
• robotics, agents, distributed systems
• plans to be carried outby humans
• business processes
• usage models
• user needsin a computing environment
• UI adaptation
• task management

• guide usersin complex tasks
• Adtranz (1998)
• Smart Homeselderly/health care
• carry out taskson users behalf
• autonomous agents
• RADAR

(future work)
53
related workadaptive systems
service awareness
context awareness
resource awareness

• fault tolerancemodel availability
• load balancingmodel QoS
• model-based adaptationknown components
• autonomic computing
• Rainbow
• service ontologyweb services
• DAML/OWL, UDDI
• scoping discoveryis an open problem

• context sensinglocation, activity
• CIS
• context-aware appsone applicationlocal rules
• context-awaretask managementis an open problem

• fidelity-aware appsone applicationlocal
  policies
• Odyssey/Chroma, Puppeteer

(future work)
54
future workcomplementary areas
incorporate results from

• architectural economic model-based
  adaptatione.g., Shang-Wen Cheng, Vahe Poladian
• fidelity/resource-aware applicationse.g., Rajesh
  Balan
• context awareness
• distributed file systems
• natural human-computer interfaces

55
future worktask management

• delegation of services/parts of tasks to humans
  framework to combine automatic configuration
  with human participation
• proactive supplyforecast service demands,
  customize service offering to demand
• scoping the environmentcombine demand
  preferences (familiar/desired suppliers)with
  supply offering, across administrative boundaries
• describing task modelsframework that
  accommodates a range of domains
• guidance on tasksreconcile guidance with
  freedom/serenity in carrying out tasks

56
contributions
approach
infrastructure
framework

• task management
• placeholder fortask knowledge
• environment management
• utility-theoreticframework forfinding best
  match
• coordinatesadaptation at 3 levels
• user tasks
• environment capabilities
• resources

• demonstrates
• feasibility of approach
• usabilityby non-experts
• adequate performancefor daily use
• foundation forfurther research
• well documented design
• affordable to integrate legacy/new components

57
contributions
approach

• uses task models toscale task management
• in space, across heterogeneous environments
• in time, enabling users to find past tasks
• control over which tasks to carry out
• accounts for user preferences
• reconciles
• reducing user overhead
• taking full advantage

58
requirement taking full advantageartifact of
todays limitations?
mobile computing
59
the next big thingcomputation gets physical
support mobile users in their daily tasks

• tomorrow, actuate
• workplace
• home
• medicine

• today
• process information

models of user needs preferences
60
reality
solutions

• desktop managers(Rooms, 1987)

• devices becoming
• more powerful
• cheaper
• interconnected

• expect computingto be availableeverywherejust
  like furniture

• tasks
• last long time
• recur

• need to find recoverpast tasks

61
outline

• the problem
• new approach
• related work
• summary of research
• validation
• future work
• contributions
• discussion

62
limitations assumptions

• migrating/accessingtask models materials
• dependent on apps
• APIs
• good behavior wrt resource management
• shared vocabulariesfor describing services
• data format compatibility

63
modeling user tasksheterogeneity change
64
combining utilities by multiplicationreflects
and semantics
and
overall utility supplier utility x QoS utility
available suppliers
real number in 0,1
U
achievable QoS
supplier utility supplier 1 ut. x supplier 2
ut. x
QoS utility QoS dim. 1 x QoS dim. x 2
65
model supplier preferencesby enumeration
supplier utility supplier 1 ut. x supplier 2
ut. x
prefer text editing supplier
66
model QoS tradeoffsby good and bad thresholds
QoS utility QoS dim. 1 x QoS dim. x 2
high
accuracy
medium
low
67
model QoS tradeoffsby good and bad thresholds
QoS utility QoS dim. 1 x QoS dim. x 2
high
accuracy
medium
low
68
model QoS tradeoffsby good and bad thresholds
QoS utility QoS dim. 1 x QoS dim. x 2
high
accuracy
medium
low
1
20
latency (s)
69
users understand controlthreshold-based QoS
tradeoffs
user study 10 students
all 10

• completed experiments
• recognized expected adaptation behavior(95
  statistical significance)

joint work with Rajesh Balan Vahe Poladian ?
paper submission