Programming By Voice

1
Programming By Voice

• Andrew Begel
• Advisor Prof. Susan L. Graham
• University of California, Berkeley
• VL/HCC 2004

2
Programming by Voice
for int i equals zero i less than ten i plus plus
1
3
Programming by Voice
for int i equals zero i less than ten i plus plus

• for (int i 0 i lt 10 i )

1
4
Ambiguities
Spelling of ID?
KW or ID?
KW or ?
4 int eye equals 0 aye less then ten i plus plus

• for (int i 0 i lt 10 i )

2
5
Sometimes its hard!
for times ate equals file two load equals one
3
6
Many Valid Interpretations!
for times ate equals file two load equals one
for (times ate file to().load 1)
3
7
Many Valid Interpretations!
for times ate equals file two load equals one
for (times ate file to().load 1)

• 4 8 file toload won

3
8
Many Valid Interpretations!
for times ate equals file two load equals one
for (times ate file to().load 1)

• 4 8 file toload won

fore.times(8).equalsFile(2, load 1)
3
9
Disambiguating filetoload
file to load
file 2 load
file toload
filetoload
4
10
Disambiguating filetoload
file to load
file 2 load
file toload
filetoload

1. filetoload()

1. filetoload

4
11
Disambiguating filetoload
file to load
file 2 load
file toload
filetoload

1. filetoload()

1. filetoload

1. No immediately adjacent identifiers

4
12
Disambiguating filetoload
file to load
file 2 load
file toload
filetoload

1. filetoload()

1. filetoload

1. No immediately adjacent identifiers

1. No variables named file

4
13
Disambiguating filetoload
file to load
file 2 load
file toload
filetoload

1. filetoload()

1. filetoload

1. No immediately adjacent identifiers

1. No variables named file

1. No methods named file

4
14
Disambiguating filetoload
file to load
file 2 load
file toload
filetoload

1. filetoload()

1. filetoload

1. No immediately adjacent identifiers

1. No variables named file

1. No methods named file

1. No method named filetoload

4
15
Disambiguating filetoload
file to load
file 2 load
file toload
filetoload

• filetoload

1. No immediately adjacent identifiers

1. No variables named file

1. No methods named file

1. No method named filetoload

4
16
The Vision

• Naturally Verbalized Programs
• Spoken Java Language
• Navigation and Editing Command Language
• Analyses that Resolve Ambiguities
• Prototype SPED SPeech EDitor

5
17
Implementation

• Speech Recognition IBM ViaVoice
• Eclipse IDE
• Harmonia program analysis toolkit
• Generalized LR parsing with input stream
  ambiguities
• Persistent, incremental semantics

6
18
Evaluation

• Hypothesis
• Programmers can learn to use SPED efficiently for
  many programming tasks
• User Study
• Train users on Spoken Java and command language
• Edit an existing program
• Create some new code
• Metrics
• Speed, vocabulary mistakes, grammatical mistakes,
  system understanding errors, subjective
  impressions

7
19
Contributions

• Enabling programming by voice
• New methods for handling input ambiguities
• Exploiting syntax and semantics of programming
  domain
• Analyses for mixed command and programming
  languages
• Interface to commercial speech recognition tools

Andrew Begel abegel_at_cs.berkeley.edu
8