Natural Language Processing in Augmentative and Alternative Communication

1
Natural Language Processing in Augmentative and
Alternative Communication

• Yael Netzer

2
Outline

• Natural Language Processing
• Augmentative and Alternative Communication
• My work - Generation of messages
• How does the process look like
• What is needed
• NLP in AAC
• Word prediction
• Message generation
• IR methods

3
Natural Language Processing

• Applications/models with usage of linguistic
  knowledge, or that provide linguistic knowledge
  (POS taggers, parsers etc.)
• Language applications
• Machine translation
• Text summarization
• Information retrieval/extraction
• Human Computer interface.

4
Alternative and Augmentative Communication

• AAC Users
• Congenital diseases e.g. cerebral palsy
• Progressive diseases e.g. ALS Amyotrophic Lateral
  Sclerosis (Lou Gehrig's Disease)
• Trauma e.g. head injury
• Cognitive disabilities vs. physical disabilities
  (each requires different methods and
  assumptions).
• Slow rate of conversation
• Speech rate 150-200 wpm, skilled typist 60 wpm
• Speech prosthesis users 10-15 wpm
• Each key stroke may consume a lot of energy.
• Trade off between conversation rate and cohesion
  of utterances.

5
AAC Techniques

• Simple pointing on boards and letter charts
• Portable keyboard devices
• Computer-based systems using single-switch
  access for severely impaired subjects.
• Symbols or letters
• Various symbol systems (Blissymbolics) /sets
  (PCS).
• Pre-stored phrases accessible via grid or iconic
  buttons.

6
(No Transcript)
7
AAC and NLP

• Common issues
• Text generation
• Speech recognition
• Text to speech synthesis
• Information retrieval.
• 3 workshops, 1 special edition in journal
  (Natural Language Engineering).

8
Our framework

• Natural language generation
• Content planning
• Surface realization
• Lexical choice
• Syntactic realization
• Morphological processing.
• FUF/SURGE, HUGG
• Lexicon (Jing et al.)

9
Desired Scenario
???
lexicalization
Content planning
Syntactic realization
vocalization
10
Examples

• ME / TO SEE / CAT / TO EAT
• I saw the cat eating.
• CAT / TO EAT / TO SEE / ME
• The cat ate and I saw it
• The cat that ate saw me.

11
Blissymbolics

• Invented by Charles Bliss 1965 as a written
  universal language.
• Adapted by Canadian speech therapists in the
  early 70s. successful alternative to verbal
  comm.
• Consists of approx. 100 basic symbols.
• Language now consists of more than 2000 complex
  words.
• Not so easy to learn but
• Enables good novel/personal expressions
• Good basis for literacy
• Adults like to use it too.

12
Bliss Example

• Afraid sad future

13
Example - Minspeak /PCS lexicon
apple, food, eat
house food grocery
rainbow apple red
14
Me See Cat Eat
15
Cat Eat See Me (Heb. Ver.)
16
Syntax Ambiguity

• ME / SEE / CAT / TO EAT
• I saw the cat eating.
• CAT / TO EAT / SEE / ME
• The cat ate and I saw it
• The cat that ate saw me.
• However, users of AAC dont usually obey the word
  order of spoken language
• gogirlhouse or girlhousego or housegogirl
• Twobedsleepboyonegirlwhitebedbrownbed
  (the boy and the girl are sleeping in two bed,
  one in a white bed and the other in a brown bed).

17
Pragmatics

• where situation is taking place,
• whos the hearer,
• Good morning vs. Hi
• Open the window vs. Can you please open the
  window?
• Gestures (facial, body)

18
Pragmatics Ambiguity
I want to eat.
Do you want to eat?
19
Contextual Resources

• What is the context of the things that are said,
  following what was already said before,
  referential expressions.
• In a restaurant you can talk about the menu
• In front of a computer, the menu is a set of
  commands.

20
Textual Context Previous Utterances

• someone
• a person
• a woman
• a mother
• a female parent
• Lucy
• she
• etc

I met Lucy. She looks great
?
21
Generating from Symbols Issues

• Syntactic ambiguity
• Contextual ambiguity
• No strict rules for use of symbols Individual
  codes, conventions, abbreviations.
• Textual how one word affects the choice of
  another, ordering words, fluency.
• Practical Enhancing communication rate w/o
  limiting expressing abilities.
• (efficient keyboard setup, word prediction,
  structure prediction).

22
An overview on Architecture

???
Task Parsing Requirements world
knowledge Lexical information Output well formed
input for syntactic realizer
Tools Valliants conceptual graphs
parsing Lexicon for verbs (Jing et al.) Bliss
Lexicon
lexicalization
Content planning
Syntactic realization
Task generating well formed sentences
Tools SURGE/HUGG
vocalization
23
Lexicon

• Mapping concepts - symbols to word
• Compositional vs. non-compositional
• Organization of symbols for efficient retrieval.
• (POS, semantic connections)
• Available lexical knowledge
• Syntactic structure, irregularities etc.

24
Methodology

• Test interaction of different aspects
• Word/symbol/ structure prediction
• With more specific questions
• Concepts to words
• Referential expression generation
• Pragmatic considerations.

25
Semantic Network (Valliant)
26
Blissymbols Grid
27
Word -gt Symbols  Symbols -gt Symbol Symbol near
Symbols  Symbol -gt FeaturedComponent
SymbolFeaturedComponent  FeaturedComponent -gt
(Atomic)PositionSizeDirection Atomic -gt
Pictographs Arbitrary  Pictographs -gt
protection, house, circle, plus, pointer, arrow,
room, body, legs, chair, water, wheel,
feeling  Arbitrary -gt Articles Numerals
Math-sign Bliss-arbitrary  Position -gt Vertical
and Horizontal   Vertical-gtVerticalPosition
Spacing _at_ VerticalSigner VerticalPosition -gt
right, left, centralized VerticalSigner -gt
skyline, midline, earthline   Horizontal -gt
HorizontalPosition Spacing _at_ HorizontalSigner Hori
zontalPosition -gt above, under,
centeralized  HorizontalSigner -gt leftline,
middleline, rightline   Spacing -gt zero one
two the distance between the
constituents  Size -gt full, half,
quarter   Direction -gt as-is, horizontal,
vertical, left, right, upside-down
Direction-Direction  Bliss-arbitrary -gt action ,
enclosure, multiplication, evaluation, nature,
horizontal-line, vertical-line,
28
SURGE
29
Work left to do

• Integration
• Evaluation
• symbols to utterances corpus
• keystrokes savings

30
Previous work of NLP-AAC

• Word prediction
• Message Generation
• Text simplification

31
Word Prediction

• Simple non-linguistic methods - possibly up to
  50 savings of keystrokes.
• Required improvement,
• Including syntactic/semantic knowledge in the
  prediction process, using machine learning
  methods, based on corpus analysis
• Methods
• Frequency-based models (bi/tri-grams)
• Grammatical and conceptual modeling to predict
  well formed utterances such as the use of POS
  tags.

32
Word Prediction

• KOMBE project hand written syntactic rules.
• Carlberger
• Different languages?

33
Message Generation

• Language generation from reduced input
• Telegraphic text
• Cushler Badman Demasco and McCoy
• think red hammer break John gt
• I think that the red hammer was broken by John.
• Cogeneration Copestake
• Construction of full sentences from templates.
• PVI
• Main assumption order of word choice implies
  topicalization and should be considered.

34
The common architecture
Iconic/telegraphic input

Semantic parser Identification of
predicator Unification of Arguments.
Lexical choice
Syntactic realization Closed-words
selection Linearization morphology
35
Cogeneration approach

• Situation-based approach.
• A set of pre-defined templates
• Topic of discussion ltgt Participants ltgt
• Time of discussion ltgt (optional)
• You know ltparticipantsgt talking about lttopicgt
• Prefer
• You know we were talking at breakfast about
  buying a desk lamp.
• On ambiguous
• You know we were talking about buying a desk lamp
  at breakfast.
• Templates W/o cogeneration
• You know us talking about buy desk lamp breakfast.

36
PVI

• Paradigmatic dimension icons organized in
  taxemes, further grouped in samantic domains.
• Syntagmatic dimension build a casual structure
  of predicative concepts.
• Meaning of an icon the features that distinguish
  it from the other icons.
• Semantic analysis reconstructing the meaning of
  the icon sequence building a semantic network.
• Lexical choice assuming there is no bijection
  mapping of icons/words.
• Generation

37
Message Selection Systems

• Discourse structure
• TalkAbout univ. of Dundee
• A user uses pre-stored sentences.
• The sentences are indexed using rhetorical
  structure assumptions.

38
Language Simplification and Language Understanding

• PSET project Carroll et al.
• Intended for aphasic readers with lexical or
  syntactic impairments.
• Syntactic simplification
• Passive to active
• Lexical simplification lookup for synonyms, use
  most frequent.

39
To sum..

• NLP can be naturally and effectively integrated
  into AAC systems.
• Relaxations user feedback is available on the
  spot.
• Data collection IS an issue here.
• The aim make more flexible, expressive tools,
  with enhanced rate.
• Possibly, combined approaches.