Toolkits for Supporting Gestures in Applications

1
Toolkits for Supporting Gestures in Applications

• Justin Weisz
• 05-830 UI Software
• Nov. 16, 2004

2
What is a gesture?

• A single stroke indicates the operation (move
  text), the operand (the text to be moved), and
  additional parameters (the new location of the
  text).
• -- Rubine

3
What is a gesture?

• A single stroke indicates the operation (move
  text), the operand (the text to be moved), and
  additional parameters (the new location of the
  text).
• -- Rubine

4
Uses of gestures
5
Uses of gestures
6
Applications of gesturing - Lightpens
7
Applications of gesturing - Tablets
8
Applications of gesturing - PDAs
9
Applications of gesturing - Video games
PowerGlove in The Wizard
10
Applications of gesturing - Video games
Black and White - 2001
11
Rubine 1991

• Gesture
• Recognizers
• Automated (in a)
• Novel
• Direct
• Manipulation
• Architecture

12
Rubine 1991
13
Rubine 1991
14
Rubine 1991
For the line gesture
recog Seq handler mousetoolLineCursor
view createLine
setEndpoint0
xltstart Xgt yltstart Ygt
manip recog setEndpoint1
xltcurrent Xgt
yltcurrent Ygt done nil
15
Rubine 1991
BUT, how are gestures actually represented and
recognized?
Assumptions Gestures are 2D, single strokes
Start and end of a gesture is clearly defined
Representation
set of P sample points
position timestamp, preprocessed to remove
jitter
16
Rubine 1991
Feature vector extracted from G
17
Rubine 1991
BUT...
The aforementioned feature set was empirically
determined by the author to work well on a number
of different gesture sets -- Rubine
18
Rubine 1991
Classification
Each gesture class represented by a weight vector
To classify gesture G
weight of feature i
score
bias(?)
feature i of gesture G
Take the highest score
19
Rubine 1991
Training
Optimal classifier
20
Rubine 1991
Rejection
gt 0.95?
ACCEPT
Gesture G
Pr(G matches i)
Classification i
REJECT
g1
g2
g3
mean(i)
21
Rubine 1991
Evaluation
22
Rubine 1991
Evaluation
23
Rubine 1991
Evaluation
24
Aside Agate - Landay, Myers 1993
25
gdt - Long et al. 1999
26
gdt - Long et al. 1999
Newton and Palm users reported Gestures are
powerful, efficient and convenient Want more
commands to have gestures Want to define new
gestures Recognition accuracy is not good enough
27
gdt - Long et al. 1999
Oh Agate, I will make you beautiful!
28
gdt - Long et al. 1999
29
gdt - Long et al. 1999
Distance matrix
30
gdt - Long et al. 1999
Classification matrix
31
gdt - Long et al. 1999
Experiment - Hypotheses
Participants could use gdt to improve their
gesture sets. The tables gdt provided would
aid designers. PDA users and non-PDA users
would perform differently.
32
gdt - Long et al. 1999
Experiment - Procedure
(pay no attention to the man behind the
curtain...)
33
gdt - Long et al. 1999
Experiment - Results
34
gdt - Long et al. 1999
Experiment - Problems with gdt
35
gdt - Long et al. 1999
Experiment - Problems with gdt
36
gdt - Long et al. 1999
Lessons learned
GDT helpful, but participants averaged a 95.4
recognition rate Tables too confusing, didnt
help performance (better Gesture class A is too
similar to gesture class B) Should be able to
create a test set of gestures and run it against
a different gesture class
37
Break time!
Muchas gracias to my officemate for the
suggestion. Smiling babies make people happy.
BE HAPPY!
38
GT2k - Westeyn et al. 2003
39
GT2k - Westeyn et al. 2003
GT2k system components
Data generator
Results interpreter
40
Aside Hidden Markov Models
41
Aside Hidden Markov Models
42
Aside Hidden Markov Models

• Evaluation problem
• Given HMM and Oo1,...,oT, compute Pr(OHMM)
• Forward algorithm
• Decoding problem
• Given O, compute most likely state sequence that
  produced O
• Viterbi algorithm
• Learning problem
• Given O, compute transition probs. to maximize
  likelihood of observing O
• Forward-Backward algorithm (aka. Baum-Welch)

43
GT2k - Westeyn et al. 2003
Grammars
MoveForward Advance Slow_Down Halt MoveBackward
Reverse Slow_Down Halt command Attention
ltMoveForward MoveBackwardgt
44
GT2k - Westeyn et al. 2003
Converting raw sensor data to feature vectors
1 56 Attention 57 175 Advance 176 235
Slow_Down 236 250 Halt
45
GT2k - Westeyn et al. 2003
Training
Training and validation procedure
46
GT2k - Westeyn et al. 2003
Accuracy
A accuracy N number of examples S
substitution errors (misclassification) D
deletion errors (failed to recognize a gesture) I
insertion errors (system hallucinates a
gesture)
47
GT2k - Westeyn et al. 2003
Applications - Gesture Panel
gesture up down left right up-left
up-right down-left down-right
Result 99.20 accuracy on 251 examples (2
substitution errors)
48
GT2k - Westeyn et al. 2003
Applications - Prescott
blinkprint person_1 person_2 person_3
Result 89.6 accuracy on 48 examples (5
substitution errors, not good!)
49
GT2k - Westeyn et al. 2003
Applications - TeleSign
word my computer helps me talk sentence
( calibrate word word word word word exit )
Result 90.48 accuracy on 72 examples
50
GT2k - Westeyn et al. 2003
Applications - Workshop Activity Recognition
gesture hammer file sand saw screw
vise drill clap use_drawer grind
Result 93.33 accuracy on 10 examples per
activity
51
GT2k - Westeyn et al. 2003
Major conclusions
HMMs can learn from arbitrary types of data
Domain-specific knowledge may be needed to
construct proper HMM topologies Shouldnt
assume that gestures are only applicable to 2D
strokes with a mouse Wearing all that gear just
to speak 5 sign language words is kind of
ridiculous
52
BONUS SLIDES What are the neat gesturing apps?
gestures used for
handwriting issuing commands
system-wide commands, interacting with UI widgets
http//www.bitart.com/
http//www.xstroke.org/
53
BONUS SLIDES What are the neat gesturing apps?
gestures used for
issuing commands
(several gestures plugins available)
(gesturing built in)
54
BONUS SLIDES What are the neat gesturing apps?

• SwingGestures
• Java Swing gesture recognizer
• http//sourceforge.net/projects/swinggestures/
• Jestur
• Python gesture recognizer
• http//sourceforge.net/projects/jestur/
• Quill
• Java gesture creation toolkit
• http//sourceforge.net/projects/quill/

55
BONUS SLIDES What are the neat gesturing apps?

• Quill