Context Sensitive Magic Words for Programming by Voice

1
Context Sensitive Magic Words for Programming by
Voice

• Alain Désilets
• Institute for Information Technology
• National Research Council of Canada
• Canada
• alain.desilets _at_iit.nrc.ca

2
HCI Issues for PBV

• Main PBV Issues are
• Programming language constructs (loops,
  conditionals, etc)
• Symbols (variable, function names, etc)
• Navigation

3
PBV Issue 1 Programming language constructs

• Syntax of programming languages
• has lots of punctuation
• increases vocal load
• is unnatural to pronounce
• increases cognitive load
• Example
• for (cp newlb.buffer cp lt lp-1 cp)

4
PBV Issue 2 Symbols

• Symbols
• contain abbreviations
• require spelling with alpha-bravo alphabet
• mix different capitalisation styles
• require uttering discrete commands for
  capitalisation
• Examples
• A_CONSTANT
• aFuncName

5
PBV Issue 3 Navigation

• Two types of navigation
• local (within page)
• global (jump between pages and files)
• Programming vs Word Processing
• more navigation in general
• more global navigation in particular
• cannot use Select and Say because target is
  unrecognisable

6
PBV Issue 3 Local navigation

• For ( )

Test.c
Client_Record a_client a_client.name Alain
Desilets a_client.email alain.desilets_at_nrc.ca

Move from here
To here
7
PBV Issue 3 Global navigation

• For ( )

Test.c
Client_Record a_client a_client.name Alain
Desilets a_client.email alain.desilets_at_nrc.ca

Move from here
For ( )
client_rec.c
Class Client_Record
To here
8
Current PBV approaches

• Lots and lots of discrete macros for
• entering template code
• moving between slots of the templates
• formatting symbols
• navigation
• Example want to enter
• for (loopvar 0 loopvar lt 100 loopvar)

9
Current PBV approaches Example
10
Current PBV approaches Example

• For ( )

Test.c
for (loop variable )
loop variable
11
Current PBV approaches Example

• For ( )

Test.c
for (loopvar )
format as symbol
Cache Pad
(1) loopvar
12
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 )
equals zero
Cache Pad
(1) loopvar
13
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 )
next slot
Cache Pad
(1) loopvar
14
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 loopvar )
symbol 1
Cache Pad
(1) loopvar
15
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 loopvar lt 100 )
less or equal to 100
Cache Pad
(1) loopvar
16
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 loopvar lt 100 )
next slot
Cache Pad
(1) loopvar
17
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 loopvar lt 100 loopvar)
symbol 1
Cache Pad
(1) loopvar
18
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 loopvar lt 100 loopvar)
plus plus
Cache Pad
(1) loopvar
19
Current PBV approaches Example

• For ( )

Test.c
for (loopvar 0 loopvar lt 100 loopvar)
go to body
Cache Pad
(1) loopvar
20
Current PBV approaches Problems

• Problems
• too many pauses (11 in the example)
• lengthy
• mixes continuous with discrete speech
• problems if pauses at the wrong place
• hundreds of voice macros
• hard to remember out of the blue
• speaking in terms of programming syntax is hard
• interference with speech process

21
Typing code vs Dictation
22
VoiceCode HCI principles

• Do what I mean, not what I say
• Flexible wording
• Continuous speech, pause wherever

23
Do what I mean Language constructs
For ( )
Test.c
for (x 0 x lt 100 x)
for x-ray up to 100 do
24
Do what I mean Symbols
For ( )
Test.c
int app_status app_status 0
application status equals 0
25
Do what I mean Navigation
For ( )
Test.c
int check_status (app an_application) some
lines of code here
26
Do what I mean Navigation
For ( )
Test.c
float check_status (app an_application) some
lines of code here
change return type to float
27
Do what I mean Navigation
For ( )
Test.c
float check_status (app an_application) some
lines of code here
go back
28
Flexible wording
For ( )
Test.c
for (x 0 x lt 100 x)
for x up to 100 do
for x stopping at 100 do
for x equals 0 up to 100 do
etc...
29
Continuous speech, pause wherever
For ( )
Test.c
for (x 0 x lt 100 x)
for x up to 100 do
for x ... up to 100 do
for ... x up to ... 100 ... do
etc...
30
Isnt this Natural Language Understanding?

• Yes, but within a domain that is
• very restricted
• highly structured
• easily parsable
• So, can get away with less than full NLU

31
Context Sensitive Magic Words (CSMWs)

• A CSMW is a word which
• fires a particular action when uttered
• but only if a particular context applies
• action is applied taking into account that
  context

32
CSMW Example Language constructs
For ( )
Test.c
for (x 0 x lt 100 x)
for x up to 100 do
33
CSMW Example Language constructs
34
CSMW Example Language constructs
For ( )
Test.c
Magic Word yes

for
Context blank line

• Action
• insert template for

35
CSMW Example Language constructs
For ( )
Test.c
for ( 0 lt )
Magic Word no
x
Context none

• Action
• insert as is

36
CSMW Example Language constructs
For ( )
Test.c
Magic Word yes
for (x 0 lt )

• Context for loop
• loop var x
• start bound 0
• end bound undef
• direction increasing
• step size 1

up to

• Action
• reprint context
• move to end bound

37
CSMW Example Language constructs
For ( )
Test.c
for (x 0 x lt x)
Magic Word no
100
Context none

• Action
• print as is

38
CSMW Example Language constructs
Magic Word yes
For ( )
Test.c

• Context for loop
• loop var x
• start bound 0
• end bound 100
• direction increasing
• step size 1

for (x 0 x lt 100 x)
do

• Action
• move to body

39
CSMW Example Language constructs
For ( )
Test.c
for (x 0 x lt 100 x)
moves here
40
Do what I mean Navigation
For ( )
Test.c
int check_status (app an_application) some
lines of code here
change return type to float go back
41
CSMW Example Navigation
42
CSMW Example Navigation
Magic Word yes
For ( )
Test.c

• Context function
• name check_status
• return type int
• etc...

int check_status (app an_application) some
lines of code here
change return type to

• Action
• drop breadcrumb
• move to ret. type

43
CSMW Example Navigation
For ( )
Test.c
check_status (app an_application) some
lines of code here
Magic Word no
float
Context none
breadcrumb inserted here

• Action
• insert as is

44
CSMW Example Navigation
For ( )
Test.c
float check_status (app an_application) some
lines of code here
Magic Word yes
go back
Context any

• Action
• move to last breadcrumb

45
CSMW Example Navigation
For ( )
Test.c
float check_status (app an_application) some
lines of code here
returned to last breadcrumb
46
CSWMs Status

• Prototype infrastructure for interpreting CSMWs
  in Perl
• CSMWs implemented for few C constructs
• for loop
• function

47
Challenges open questions

• Robust context parsing
• partially correct code
• mix of native and pseudo code
• context must only parse when applicable
• Developing good CSMWs for a given language
• Correction process
• strings recognised by SR system do not get typed
  as is
• Correct that erases the wrong things

48
Challenges open questions

• Customisability
• Change of wording (easy)
• Change of actions (moderate)
• Addition of new CSMWs (hard)
• using high level configuration tool
• Retraining SR system
• To recognise programmese
• instead of English

49
Conclusion

• Context Sensitive Magic Words is a powerful
  technique which
• supports advanced continuous Natural Language
  dictation and commands for PBV
• while avoiding the complexities of full NLU
• but its still not trivial