The SPACE project: Speech Algorithms for Clinical and Educational Applications

1
The SPACE projectSpeech Algorithms for Clinical
and Educational Applications

• Hugo Van hamme
• SPACE symposium
• Antwerp

2
Outline

• partners
• What is it about ?
• Why does it make sense ?
• The educational component
• The clinical component
• Challenges examples
• The technologies foreground and background
• The first 6 months

3
Partners

• K.U.Leuven - ESAT coordinator - speech
  recognitionProf. Hugo Van hamme
• R.U.Gent ELIS speech recognitionProf.
  Jean-Pierre Martens
• V.U.Brussel ETRO text-to-speechProf. Werner
  Verhelst
• K.U.Leuven ORTHO disability, special needs
  education and child careProf. Pol Ghesquière
• U.Antwerpen communication disordersProf. Marc
  De Bodt

4
In touch with the field

• user group
• Technology providers
• ScanSoft
• Technology users
• Technology Integratie
• Artec
• eXplio
• Interest groups
• Stichting Integratie Gehandicapten (SIG)
• Modem
• this symposium

5
What ?

• Speech technology
• automatic speech recognition (ASR)
• speech synthesis (TTS)
• Clinical and educational
• Speech therapy related.
• Speech assessment
• Adapt technology
• To suit requirements of the applications
• Demonstrate usefulness of technology
• Automation of existing methods
• New methods enabled by the technology
• Interdisciplinary

6
Why ?

• spoken interaction with the computer comes
  naturally
• Unlike many other applications of ASR/TTS
• Similar characteristics language learning
• pre-assessment in 2003
• social relevance
• role of universities
• large group of beneficiaries
• persons with dyslexia
• reading skill development of all primary school
  pupils
• deaf, communication disorders

7
Why ? (2)

• other applications possible
• language learning and language proficiency
  assessment
• training of professional speakers
• pronunciation training and stutter therapy
• E-learning
• technology improvements applicable in other
  areas
• HMI with voice mode
• entertainment

8
Some background

• Project sponsor IWT
• Instituut voor de aanmoediging van innovatie door
  Wetenschap Technologie in Vlaanderen
• SBO Strategisch BasisOnderzoek
• 4 years March 1, 2005 February 28, 2009
• 28 person-years total effort
• This symposium is co-sponsored by the Nederlandse
  Taalunie

9
Domain of interest 1 Automated reading
assessment and remedial practice

• reading tutor
• replace human supervision in current diagnostic
  practice and in therapy
• make assessment objective and repeatable
• explore new strategies for diagnosis and remedy,
  enabled by speech technology
• use
• automate diagnosis of dyslexia gt early detection
• a program that helps you develop your reading
  skill
• increase intensity (and effectiveness) of therapy
• AVI reading tests in primary schools

10
Domain of interest 2 Clinical applications for
speech assessment

• clinical practice
• perceptual evaluation
• subjective tests of articulation
• interrater and intrarater disagreements
• use articulatory speech analysis
• compare to human judgement
• reference database
• determine type and degree of error

11
The challenge - examples

• reading tutor
• mis-pronunciation
• Immediate auditive feedback (cues)
• assessment mis-articulation

12
Hestiations, unwanted speech

• Joep rijdt op zijn fiets door de straat. Het is
  een mooie gele fiets.Die heeft hij voor zijn
  verjaardag gekregen. Er zit een grote glimmende
  bel op.

13
The technology

• background
• large vocabulary speech recognizer (ESAT)
• voice assessment, pronunciation modelling (ELIS)
• text-to-speech and voice modification (ETRO)
• requirements
• accurate assessment of utterance
• acceptance/rejection
• Fine-grained analysis/feedback
• speech representations that give articulatory
  insight
• modelling of imperfect speech
• mis-articulations
• mis-pronunciations
• at phoneme, word or sentence level
• feedback and guidance through TTS

14
Approaches acoustics

• optimize acoustic models for children
• model the disfluencies
• non-phonemes
• articulatory analysis of speech
• voicing, high/low, lip rounding,
• estimated from wave form
• relevant for articulation assessment
• accurate phonetic classification
• phonetic hypotheses generated in phoneme lattice
• phoneme-specific features and tests added

15
Approaches miscues

• lexical mispronunciation models
• exploit prior knowledge on reading mistakes
• orthography
• frequency rare words substituted by common
• semantics read-by-guessing strategy
• data driven at word level or by transformation
  rules
• sentence level misreading models
• hestitations, restarts

16
Approaches TTS

• TTS for
• providing pronunciation examples
• providing reading cues
• synchronised reading
• special reading mode speech synthesis
• spelling mode (letter/phoneme)
• syllable mode (isolated/lengthened)
• extremely slow speech
• special stress patterns

17
Where are we ?

• articulatory speech analysis
• data collection
• dyslalia, dysarthria, hearing loss
• reading exercises content, tools
• TTS public domain software analysis
• reading tutor prototype
• childrens acoustic model
• track reading progress
• model for word skips and restarts
• model for unintended speech
• model for lexical errors swap of letters,
  phoneme substitution

18
conclusion

• the SPACE project
• has challenging objectives
• interdisciplinary
• will deepen insights in new speech modelling
  approaches
• will develop prototypes in both application areas
• has mainly a social relevance, also
• economic spin-off activities possible
• improvement in accuracy and robustness of ASR
• additional speaking modes and synchronisation in
  TTS