Joe Tullio

1
Model-based UI Tools

• Joe Tullio
• CS4470/6456
• November 22, 2002

2
Intelligent User Interfaces

• Intelligent user interfaces are human-machine
  interfaces that aim to improve the efficiency,
  effectiveness, and naturalness of human-machine
  interaction by representing, reasoning, and
  acting on models of the user, domain, task,
  discourse, and media (e.g., graphics, natural
  language, gesture). Intelligent user interfaces
  are multifaceted, in purpose and nature, and
  include capabilities for multimedia input
  analysis, multimedia presentation generation,
  model-based interfaces, agent-based interfaces,
  and the use of user, discourse, and task models
  to personalize and enhance interaction. Mark
  T. Maybury

3
What is a model?

• Abstraction of some real-world process (natural
  language recognition, cognition, etc.)
• Often expressed through some concrete
  representation

4
Example model Making Tea(Hierarchical Task
Model)
5
IUI Examples SATIN(Hong Landay, UIST 2000)

• Multimodal input models
• User models
• Agents
• Discourse models

6
IUI Examples Lumiere(Horvitz et al, UAI 1998)

• Multimodal input models
• User models
• Agents
• Discourse models

7
IUI ExamplesOffice Assistant

• Multimodal input models
• User models
• Agents
• Discourse models

8
IUI Examples(Maes, CACM 37(7), 1994)

• Multimodal input models
• User models
• Agents
• Discourse models

9
IUI Examples REA(Cassell et al, CHI 1999)

• Multimodal input models
• User models
• Agents
• Discourse models

10
Model-based UI

• User constructs a declarative model of how the
  interface should look and behave (presentation,
  dialog, platform, etc.)
• Run-time component executes the model
• Construct display for the application
• Interpret input according to model specification
• Can view as a knowledge base that generates the
  appropriate UI given context of use

11
Rest of Lecture Overview

• Rest of Lecture Overview
• Model-based UI Old school
• UIMS Definition and examples
• Discussion MASTERMIND
• Task models
• UIMS Why it died
• Model-based UI New school
• UIMS Why it came back
• Scalable UI for mobile devices
• MIMIC
• UIML

12
UIMS

• Motivation
• Standard PLs not suited to what UI designers
  want to express
• Old GUI builders limited in flexibility of
  behavior, display
• So
• Allow designers to specify application as a model
• Provide abstractions for input/output
• Automatically generate the UI based on this model

13
UIMS Examples UIDE

• Sukaviriya, P., Foley, J., and Griffith, T. A
  Second Generation User Interface Design
  Environment The Model and the Runtime
  Architecture. In Proc. Of InterCHI93, pp.
  375-382.
• Application model
• Data objects
• Actions on those objects
• Pre/post-conditions
• Parameters and constraints
• Interaction techniques and objects

14
UIDE continued

• Advantages
• Automatic GUI layout
• Animated help
• Generation of text help
• Disadvantages
• Modeling done in C
• Rapid prototyping - iteration/design alternatives?

15
UIMS Examples HUMANOID

• Szekely, P., Luo, P., Neches, R. Beyond
  Interface Builders Model-based Interface Tools.
  In Proc. Of InterCHI93.
• Similar application model to UIDE
• Designers interface
• Up-to-date previews of generated UI
• Users can edit those examples
• Sophisticated presentation model
• Action side effects

16
HUMANOID continued

• Advantages
• More intuitive UI for designers
• Specify side effects
• Supports iteration
• Disadvantages
• Lacks many of the features of UIDE (automatic
  layout, help generation, explanation, evaluation)

17
MASTERMIND A Task Model-based approach

• Robert Neches, James D. Foley, Pedro A. Szekely,
  Piyawadee Noi Sukaviriya, Ping Luo, Srdjan
  Kovacevic, Scott E. Hudson Knowledgeable
  development environments using shared design
  models. Intl. Workshop on Intelligent User
  Interfaces (IUI), 1993, pp. 63-70.
• Key contribution - biggest acronym ever
• Models Allowing Shared Tools and Explicit
  Representations to Make Interfaces Natural to
  Develop

18
MASTERMIND continued

• Contributions
• Unification of benefits of HUMANOID, UIDE
• Permits testing of partial designs
• Exploration of alternatives
• Run-time services (context-sensitive help)
• Toward a knowledge base of interface concept
  models used in progressive stages of the design
  process

19
MASTERMIND vision

• Task is modeled by designer in initial phases of
  problem definition
• Design abstractions and policies are defined, and
  alternative designs are explored
• Conceptual design is mapped to I/O media,
  low-level user input is mapped to semantics
• Each stage uses the knowledge created in previous
  phases
• Common modeling scheme allows the creation of a
  knowledge base

20
Task Models

• Obtained through observation, interviews,
  questionnaires, etc.
• Hierarchical Task Analysis (HTA)
• User task is broken down into subtasks, subtasks
  into more subtasks, etc.
• Tea-making example shown earlier
• GOMS (Goals,Operators,Methods,Selection)
• Goals specified hierarchically, achieved through
  operators and subgoals (methods)
• Rules exist to select one method over another
• Can be formative and evaluative

21
Quick GOMS example

• Goal Access ATM
• Goal Enable Access
• Insert card
• Insert password
• Goal Take cash
• Select withdraw
• Select dollar amount
• Press ok
• Take money
• Verify amount of money

22
UIMS Why it died

• Moving target problem. See
• Myers et al. Past, present and future of user
  interface software tools. ACM TOCHI. Vol. 7,
  Issue 1, March 2001, pp. 3-28.
• Poor UIs generated
• Interface elements standardized
• Designer control over look feel
• Improvement of GUI builders better integration
  of DM and high-level coding

23
Model-based UI New life

• Proliferation of computing devices
• Display colors/resolutions/sizes
• Different input methods
• Mobile or stationary?
• Different power/memory constraints
• Prohibitive to create/iterate custom UIs for
  each possible device

24
Multiple-platform UIs using a Task Model

• Farooq Ali, M. and Perez-Quinones, M.
  Multi-Platform User Interface Development from a
  Single Task Model. In ACM Symposium on User
  Interface Software and Technology Conference
  Companion, 2002, pp. 17-18.
• Uses the Concurrent Task Tree (CTT) notation for
  task model
• User tasks cognitive/perceptual
• Interaction tasks user interacting with system
• Application Tasks performed by system
• Abstraction tasks complex tasks
• Temporal operators between adjacent tasks

25
UIML

• User Interface Modeling Language
• Generic language for describing user interfaces
• Can be rendered to a platform (e.g., Java, HTML)
• Tasks in CTT descriptions mapped to UIML
• Tasks mapped to UIML interactors (parts),
  developer can customize
• Subtasks define containment/grouping
• Navigation style must be defined
• Developer must provide this information for all
  platforms

26
MIMIC

• J. Eisenstein, J. Vanderdonckt, and A. Puerta.
  Applying model-based techniques to the
  development of user interfaces for mobile
  computers. In Proc. ACM Conference on Intelligent
  User Interfaces IUI'2001, pages 69-76.

27
MIMIC continued

• Platform model
• Includes constraints on UI by each device
• Used at design time or preferably run-time
• Presentation Model
• Widget hierarchy
• Abstract and concrete interaction objects
• Task model
• Hierarchical
• Goals, preconditions, postconditions (MASTERMIND)
• Mappings between models

28
MIMIC Automation potential

• Using platform model, apply rules for
  shrinking/substituting interactors
• Compare display needs of layout/grouping choices
  by comparing with what each platform offers
• Use task model to map particular tasks with
  particular devices (can even switch at runtime)
• Presentation can then be tuned to most important
  tasks for that platform

29
Discussion

• New model-based tools for mobile devices are
  largely untested
• Are the problems that doomed UIMS still present?