Kathy McCoy

1
Kathy McCoy

• Artificial Intelligence
• Natural Language Processing
• Applications for People with Disabilities

2
Primary Research Areas

• Natural Language Generation problem of choice.
• Deep Generation --- structure and content of
  coherent text
• Surface Generation particularly using TAG
  (multi-lingual generation and machine
  translation)
• Discourse Processing
• Second Language Acquisition
• Applications for people with disabilities
  affecting their ability to communicate

3
Projects

• Augmentative Communication
• Word Prediction and Contextual Information (Keith
  Trnka)
• Using prestored text (Jan Bedrosian, Linda
  Hoag, Tim Walsh)
• General Interfaces (Stephen Steward)
• ICICLE CALL system for teaching English as a
  second language to ASL natives (Rashida Davis)
• Text Skimming for someone who is blind to find
  an answer to a question (Debbie Yarrington)
• Generating Textual Summaries of Graphs (Sandee
  Carberry, Seniz Demir)
• Generating Appropriate Referring Expressions
  (Charlie Greenbacker)

4
Developing Intelligent Communication Aids for
People with Disabilities

• Kathleen F. McCoy

Computer and Information Sciences Center for
Applied Science and Engineering in Rehabilitation
University of Delaware
5
Augmentative Communication

• Intervention that gives non-speaking person an
  alternative means to communicate
• User Population
• May have severe motor impairments
• Unable to speak
• Unable to write
• Cannot use sign language
• May have cognitive impairments and/or
  developmental disabilities
• Our focus here adults with no cognitive
  impairments and very good literacy skills

6
Row-Column Scanning
7
Row-Column Scanning II
8
Can we be faster?
9
Language Representation Words
10
Still Need to Spell!
11
Predicting Fringe Vocabulary

• Word Prediction of Spelled Words (infrequent
  context-specific words)
• Methods
• Statistical NLP Methods
• Learning from the context of the individual
• Other Contextual Clues
• Geographic Location, Time of Day, Conversational
  Partner, Topic of Conversation, Style of the
  Document

12
Prediction Example
13
(No Transcript)
14
(No Transcript)
15
Trigram Model P(wh)P(ww-2 w-1)
16
Can we do better??

• Intuitively all possible words do not occur with
  equal likelyhood during a conversation.
• The topic of the conversation affects the words
  that will occur.
• E.g., when talking about baseball ball, bases,
  pitcher, bat, triple.
• How often do these same words occur in your
  algorithms class?

17
Topic Modeling

• Goal Automatically identify the topic of the
  conversation and increase the probability of
  related words and decrease probability of
  unrelated words.
• Questions
• Topic Representation
• Topic Identification
• Topic Application
• Topic Language Model Use

18
Topic Modeling Approach
19
Topic Identification
20
Topic Identification
21
Topic Application

• How do we use those similarity scores?
• Essentially weight the contribution of each topic
  by the amount of similarity that topic has with
  the current conversation.

22
(No Transcript)
23
Results Using Topics
24
Current Work

• Other kinds of tuning to the user can we do
• Recency
• Style
• What about using a much larger corpora?
• Does keystroke savings translate into
  communication rate enhancement?

25
Text Skimming

• Debra Yarrington, Kathleen McCoy

26
Problem

• Blind and dyslexic individuals cannot skim text
• Example Whats the syntax for calling a
  function with template parameters? (skimming
  through code)
• Why was Ayers Rock renamed?
• What type of tree produces leaves with three
  distinct shapes?
• Where can I find more information about
  Portugal?
• People who cannot read text rely on
• screen readers (Jaws, Window-Eyes)
• braille output
• more difficult to come by
• extremely bulky to carry around

27
Example of Jaws Output at 400 wpm

• Link
• What psychological and philosophical
  significance should we attach to recent efforts
  at computer simulations of human cognitive
  capacities? In answering this question, I find it
  useful to distinguish what I will call "strong"
  AI from "weak" or "cautious" AI (Artificial
  Intelligence). According to weak AI, the
  principal value of the computer in the study of
  the mind is that it gives us a very powerful
  tool. For example, it enables us to formulate and
  test hypotheses in a more rigorous and precise
  fashion. But according to strong AI, the computer
  is not merely a tool in the study of the mind
  rather, the appropriately programmed computer
  really is a mind, in the sense that computers
  given the right programs can be literally said to
  understand and have other cognitive states. In
  strong AI, because the programmed computer has
  cognitive states, the programs are not mere tools
  that enable us to test psychological
  explanations rather, the programs are themselves
  the explanations.
• I have no objection to the claims of weak AI,
  at least as far as this article is concerned. My
  discussion here will be directed at the claims I
  have defined as those of strong AI, specifically
  the claim that the appropriately programmed
  computer literally has cognitive states and that
  the programs thereby explain human cognition.
  When I hereafter refer to AI, I have in mind the
  strong version, as expressed by these two claims.
• I will consider the work of Roger Schank and
  his colleagues at Yale (Schank Abelson 1977),
  because I am more familiar with it than I am with
  any other similar claims, and because it provides
  a very clear example of the sort of work I wish
  to examine. But nothing that follows depends upon
  the details of Schank's programs. The same
  arguments would apply to Winograd's SHRDLU
  (Winograd 1973), Weizenbaum's ELIZA (Weizenbaum
  1965), and indeed any Turing machine simulation
  of human mental phenomena.

28
Proposed Solution

• A system that takes a question and a document or
  a few documents, and returns a small set of text
  links where potential answers to the question
  might be found
• In order to accomplish this, we will potentially
  use
• Data collected from skimming text with an eye
  tracking device
• Techniques used in existing Question Answering
  systems

29
Example
30
Gaze Plot

• link

31
Hot Spots

32
What Art Middle infused purpose with also served people believed writing does who read Sculpture. The mission as well as decorate Biblical tales lessons to were church sculpture animals life Green man peering carefully wrought forth Romanesque era classical conventions of figures Romanesque At the beginning era the style of architecture that was in vogue Known as Romanesque because it copied the pattern proportion of the architecture the Roman Empire chief characteristics of the Romanesque style were vaults, round arches, and few windows The easiest point to look for is the rounded arch, seen in door openings windows In general churches were heavy Carrying about them an air solemnity and These early tapestries or look closely were France called it gothic was a reference Ransacked Rome twilight architectural Romanesque vaults incorporated of window The easiest point of arch doors. Also later Gothic very especially the the churches outdo each of For the construction, througt The architect same place
33
Current Directions

• Have collected eye-tracking data from close to
  100 people (on several documents each)
• Analysis quite interesting enough data to find
  patterns in where the skimmers are looking.
• Looks like standard methods used in NLP to
  identify similar words not enough
• Implemented a hack into Google search to get
  more words.
• How to present this to the user?

34
Feature Selection for Reference Generation

• Charlie Greenbacker
• Kathleen F. McCoy

35
Project at a Glance

• Decide which type of referring expression to
  produce based on context
• Designed to help develop rules for producing
  referring expressions in natural language
  generation (NLG) applications

36
Background

• Big picture generating referring expressions
• Deciding when to use a pronoun vs a name, etc.
• Examine human output in order to design NLG
  systems that make similar decisions
• Generation of Reference in Context (GREC)
• Shared task challenge for NLG
• Select appropriate references to an entity in a
  document from a list of alternatives
• Corpus introductory sections of Wikipedia
  articles with instances of referring expressions
  replaced by a list of possible references of
  different types

37
Background (cont...)

• GREC data format
• Example article with tagged references to main
  subject (Mount Greylock) in bold/underlined
• Mount Greylock is a mountain of 3,491 feet
  (1,064 m)in elevation, located in northwestern
  Massachusetts. It is the highest point in the
  state.

38
Background (cont...)

• GREC data format
• Snippet of example XML file containing list of
  alternate referring expressions for each
  reference

39
Background (cont...)

• Though beneficial, GREC task differs from
  traditional referring expression generation
• Standard generation information is not available
• Must work from surface context no access to
  underlying data used in conventional NLG tasks
• Identification of interfering antecedents must be
  derived without object attributes, etc.
• Discourse segment information also missing
• The challenge of correctly extracting this vital
  information makes the overall task of producing
  referring expressions that much more difficult

40
Approach

• Intuition findings in psycholinguistic research
  could be utilized to inform the selection of
  features upon which to train a classifier system
  to determine proper referring expression types
• Process
• Survey psycholinguistic literature to identify
  potential features
• Build rapid prototyping system for determining
  preliminary efficacy of features
• Iteratively review results to refine features and
  theorize new features patterns
• Train a decision tree based on the selected
  features

41
Approach (cont...)

• Psycholinguistic research indicates several
  factors influence interpretation of pronouns
• Subjecthood whether the entity is in subject
  position (or was in most recent mention)
• Parallelism whether the entity is in the same
  grammatical role as the previous mention
• Recency how recently has the entity been
  mentioned in the discourse
• Ambiguity whether any interfering antecedents
  exist which may confuse the listener
• Discourse Structure sentences, segments, etc.

42
(No Transcript)
43
Results Analysis

• Accuracy of each decision tree computed via
  ten-fold cross-validation on training set
• Surprisingly, highest performing decision tree
  did not use full feature set, but a rather
  limited subset!

44
Results Analysis (cont...)

• Comparison of our best classifiers (in bold) to
  GREC '08 submissions on type accuracy
• Scored higher than all except two variants from
  best GREC '08 team

45
Results Analysis (cont...)

• Initially troubling that we were outperformed by
  best traditional statistics-based system
• Possibly explained by use of very basic means of
  NP-chunking, named entity recognition, sentence
  segmentation, etc.
• Especially when another decision tree trained on
  only the features used by the best GREC '08
  system yielded a type accuracy of only 57.89!
• Access to more complete data produced during the
  generation process would render these additional
  steps obsolete should increase performance

46
Partial representation of DT_2 decision tree
47
Conclusions

• Findings in psycholinguistic research regarding
  the production of referring expressions have been
  validated as useful in determining proper feature
  selection for the task of selecting appropriate
  reference types
• With more time, further psycholinguistic
  literature review, and the incorporation of
  deeper generation knowledge, even better results
  might possibly be attained