The subArctic Input System and Extensions for Handling Inputs with Ambiguity

1
The subArctic Input System and Extensions
for Handling Inputs with Ambiguity
2
subArctic

• A Java-based GUI toolkit that I (along with Ian
  Smith) built and distributed in 1996-97
• Goal highly extensible allowing support for lots
  of cool new interaction techniques
• Emphasis on making new and strange widgets /
  components / interactors easy to create
• High ceiling

3
Parties involved with a toolkit

• Toolkit designer (me)
• Interactor designer
• Interface programmer
• User

4
Parties involved with a toolkit

• Toolkit designer (me)
• Interactor designer
• Interface programmer
• User

Most toolkits target support here
5
Parties involved with a toolkit
By moving work up (into reusable library)

• Toolkit designer (me)
• Interactor designer
• Interface programmer
• User

6
Parties involved with a toolkit

• Toolkit designer (me)
• Interactor designer
• Interface programmer
• User

But typically dont help much here (assume a
fixed library)
7
subArctic

• Toolkit designer (me)
• Interactor designer
• Interface programmer
• User

SA tries to move work for many kinds of
interactors into toolkit infrastructure
8
subArctic

• Toolkit designer (me)
• Interactor designer
• Interface programmer
• User

Input system is a big part of that
SA tries to move work for many kinds of
interactors into toolkit infrastructure
9
Schema for pretty much all GUIs

• init()
• for ()
• evt wait_for_next_event()
• dispatch(evt)
• if ( damage_exists() ) redraw()

10
Schema of a GUI
Event Record recording of the relevant facts
about some occurrence of interest (i.e., user has
manipulated an input device)

• init()
• for ()
• evt wait_for_next_event()
• dispatch(evt)
• if ( damage_exists() ) redraw()

11
Schema of a GUI
Send (dispatch) the event to the object(s) that
want it and/or know how to respond to it (e.g.,
widget/component/interactor)

• init()
• for ()
• evt wait_for_next_event()
• dispatch(evt)
• if ( damage_exists() ) redraw()

12
Event dispatch

• All the work happens here
• Typically delegated to interactors
• E.g., buttons know how to respond to press and
  release like buttons should
• Each object keeps track of its own state
• ... but which interactor gets it
• Toolkit event dispatch process

13
Event dispatch policies

• Two primary ways to decide which interactor gets
  an event
• What are they?

14
Event dispatch policies

• Two primary ways to decide which interactor gets
  an event
• Positional dispatch
• Based on where mouse is pointing
• Examples
• Focus-based dispatch
• Designated object always gets input
• Examples

15
Pop quiz

• Should input for dragging be dispatched via
  positional or focus?

16
Pop quiz

• Should input for dragging be dispatched via
  positional or focus?
• Answer No! (both)

17
subArctic input policies

• subArctic encapsulates these ways of dispatching
  inputs in dispatch policy objects
• Manages bookkeeping (e.g., picking)
• Extensible set
• Turns out there are other useful policies (e.g.,
  for modal dialogs)

18
When interactors get events

• they typically respond to them with the
  equivalent of a simple finite state machine

19
subArctic has lib of common FSMs

• Move a lot of input handling work typically done
  by interactor programmer up into the toolkit
• One (highly parameterized) FSM for all
• Brads interactor model (awful terminology
  -)
• Many customized FSM (extensible set)
• subArctic input model

20
FSMs moved to toolkit object

• Dispatch agent
• Translates low level input into higher level terms

21
Dispatch agent example move_drag

• Translated to calls in input protocol
• drag_start()
• drag_feedback()
• drag_end()
• With useful parameters (e.g. new pos)

22
Dispatch agent example move_drag

• Translated to calls in input protocol
• drag_start()
• drag_feedback()
• drag_end()
• With useful parameters (e.g. new pos)

23
Set of dispatch agents is extensible

• E.g., can subclass for specialized kinds of drag
  such as drag_within_box or snap_drag
• Can create custom for one interface
• Once created can reuse

24
How it all goes together
25
How does interactor indicate it wants / can
handle some type of input?

• implements input_protocol
• Where input_protocol is interface with calls
  like drag_start(), etc.
• For positional thats it!
• For focus-based must also ask for focus

26
Example Hypertext for all

• User (Ken Anderson) wanted to add hyperlinks to
  all objects
• Hold down the control key and click
• His external hyperlink database would take over
  and map interactor id to hyperlink target
• But how do you change every interactor to do
  this?

27
Example Hypertext for all

• In Swing, Motif, etc. this is essentially
  impossible
• In SA, just insert a new subclass of the click
  dispatch agent that checks for the control key
  down
• About 15 lines of code
• Works for interactors written later!

28

• Questions about the SA input system?

29
Providing Toolkit Level Support for Handling
Ambiguity in Recognition-Based Input

• See http//doi.acm.org/10.1145/354401.354407and
  http//doi.acm.org/10.1145/332040.332459
• Jennifer Mankoff, Gregory Abowd
• Georgia Institute of Technology
• Scott HudsonCarnegie Mellon University

30
Motivation

• Recognition-based input offers the promise of
  naturalistic input modalities, BUT

31
Motivation

• Recognition-based input offers the promise of
  naturalistic input modalities, BUT
• Recognizers are imperfect
• affects users
• breaks current system models
• New interfaces mechanisms

32
Example Interaction

• SILK
• Hand-sketched interactors

33
Example Interaction

• SILK
• Interface developer can replace interactors with
  best recognition result

A button
34
Example Interaction

• Correction Dialog (mediator)

test
35
Example Interaction

• Correction interactor (mediator)

test
36
Example Interaction

• Problems with dialog
• Not reusable or customizable
• Hard to grow your own
• Basically we dont have toolkit support for
  recognition based UI

37
Motivation (cont.)

• At much the same stage we were at for GUIs in
  1983
• No common model for input
• No re-use
• Infrastructure
• widget library

38
An alternative Burlap
VIDEO
39
Goals of This Work

• Robust, reusable infrastructure
• Reusable library
• Integrate with convent. toolkit
• Dont throw out the baby with the bathwater

40
Talk Roadmap

• Requirements for handling uncertain input
• Extending toolkits to handle it
• Interaction techniques for ambiguity
• Implementation

41
Invoking Application Actions

• Action often done by callbacks
• Direct procedure call to application
• Hierarchical events are alternate approach
• Delivered to app as well as toolkit

42
Hierarchical Events

• Low-level events contribute to production of
  higher-level events
• Green TOG 86 Myers Kosbie CHI 96

43
Implicit Assumption of Certainty

• Implicit in all this is the assumption that the
  events really happened as reported
• Problems arise when this isnt true
• E.g., brittle dialogs

44
Needed to Handle Uncertainty

• Allow for (and explicitly model) multiple
  alternatives
• alternative higher level events
• in recognition context interpretations
• Detect conflicting interpretations
• Mediation of conflicts

45
Needed to Handle Uncertainty

• Lexical feedback about uncertain events
• split feedback from action
• Library of mediators

46
How do we do this...
47
Extended Event Model

• Uncertainty results in multiple interpretations
• ? interpretation graph

48
Toolkit Extensions

• Toolkits job is still to deliver events to
  objects
• Now delivered to recognizers, interactors, and
  application objects

49
Toolkit Extensions

• Toolkits job is still to deliver events to
  objects
• Objects initially only produce (reversible)
  feedback, no actions

50
Another Change

• Events dispatched to all who might use it

51
Details Arranging for Mediation

• Identify any conflicts
• Look for a mediators
• Pluggable list of them in toolkit
• Mediator chosen by meta-mediator
• Mediator can
• Pass, Pause, Accept

52
Doing Mediation

• ExampleUser selects interpretation

circle
circle
box
53
Doing Mediation (cont.)

• Mediator prunes interpretation graph to tree
• App informed of accept reject

54
Mediation Strategies

• Many mediation strategies
• e.g., Automatic vs. user involvement
• Toolkit is fully pluggable (!)
• Library of mediators provided, but
• Can extend/build new ones as needed
• Research goalFinding new ones

55
Providing a Library of Mediators
56
Providing a Library of mediators

• Survey of existing techniques Abowd Mankoff
  GVU Tech Report 99
• Three major classes
• Explored additional techniques

57
Providing a Library of mediators

• Survey of existing techniques
• Three major classes
• Repetition

58
Providing a Library of mediators

• Survey of existing techniques
• Three major classes
• Repetition
• Choice Ripe for toolkit support
• Presentation form
• Instantiation time
• Contextual information
• Interaction
• Feedback

59
Providing a Library of mediators

• Survey of existing techniques
• Three major classes
• Repetition
• Choice
• Automatic
• Thresholds
• Confusion matrix
• Plug in machine learning?

60
Providing a Library of mediators

• Survey of existing techniques
• Three major classes
• Explored additional techniques

61
Example Target Ambiguity

• Problem There may be multiple targets of a user
  action
• Example clicking
• Kabbash (95)
• Worden (97)

62
Example Target Ambiguity

• Problem There may be multiple targets of a user
  action
• Example Clicking
• Solution Give the user a choice of all of the
  targets

63
Example Target Ambiguity

• Problem There may be multiple targets of a user
  action
• Example Clicking
• Solution Give the user a choice of all of the
  targets

• Other applications
• Any interface involving mouse press/release
• Requires separation of concerns
• Works with all interactors

64
Implementation

• Added support for mediation ambiguity to
  subArctic toolkit
• Reusable
• Fully pluggable
• Full existing library still works as is (!)
• Small library of mediators
• Also working on non-GUI toolkit
  (http//dx.doi.org/10.1145/572003)

65
Experience

• Major example Burlap
• Smaller version of SILK Landay
• For sketching UI designs and turning them into
  functioning interfaces

66
Conclusions

• Reusable infrastructure to support ambiguous
  input
• Reduces difficulty of creating UIs
• Easier to explore new design space
• Done by modifying a toolkit, not a separate
  mechanism
• Integrated with conventional input
• Other support from toolkit still useful

67
(No Transcript)