SIMS 256: Applied Natural Language Processing

1
SIMS 256 Applied Natural Language Processing
Marti Hearst November 27, 2006  
2
Outline

• Discourse Processing
• Going beyond the sentence
• Characteristics
• Issues
• Segmentation
• Linear
• Hierarchical
• Co-reference / anaphora resolution
• Dialogue Processing

3
What makes a text/dialogue coherent?

• Consider, for example, the difference between
  passages (18.71) and (18.72). Almost certainly
  not. The reason is that these utterances, when
  juxtaposed, will not exhibit coherence. Do you
  have a discourse? Assume that you have collected
  an arbitrary set of well-formed and independently
  interpretable utterances, for instance, by
  randomly selecting one sentence from each of the
  previous chapters of this book.
• vs.

4
What makes a text/dialogue coherent?

• Assume that you have collected an arbitrary
  set of well-formed and independently
  interpretable utterances, for instance, by
  randomly selecting one sentence from each of the
  previous chapters of this book. Do you have a
  discourse? Almost certainly not. The reason is
  that these utterances, when juxtaposed, will not
  exhibit coherence. Consider, for example, the
  difference between passages (18.71) and (18.72).
  (JM695)

5
What makes a text coherent?

• Discourse/topic structure
• Appropriate sequencing of subparts of the
  discourse
• Rhetorical structure
• Appropriate use of coherence relations between
  subparts of the discourse
• Referring expressions
• Words or phrases, the semantic interpretation of
  which is a discourse entity

6
Information Status

• Contrast
• John wanted a poodle but Becky preferred a corgi.
• Topic/comment
• The corgi they bought turned out to have fleas.
• Theme/rheme
• The corgi they bought turned out to have fleas.
• Focus/presupposition
• It was Becky who took him to the vet.
• Given/new
• Some wildcats bite, but this wildcat turned out
  to be a sweetheart.
• Contrast Speaker (S) and Hearer (H)

7
Determining Given vs. New

• Entities when first introduced are new
• Brand-new (H must create a new entity)
• I saw a dinosaur today.
• Unused (H already knows of this entity)
• I saw your mother today.
• Evoked entities are old -- already in the
  discourse
• Textually evoked
• The dinosaur was scaley and gray.
• Situationally evoked
• The light was red when you went through it.
• Inferrables
• Containing
• I bought a carton of eggs. One of them was
  broken.
• Non-containing
• A bus pulled up beside me. The driver was a
  monkey.

8
Given/New and Definiteness/Indefiniteness

• Subject NPs tend to be syntactically definite and
  old
• Object NPs tend to be indefinite and new
• I saw a black cat yesterday. The cat looked
  hungry.
• Definite articles, demonstratives, possessives,
  personal pronouns, proper nouns, quantifiers like
  all, every
• Indefinite articles, quantifiers like some, any,
  one signal indefinitenessbut.
• This guy came into the room

9
Discourse/Topic Structure

• Text Segmentation
• Linear
• TextTiling
• Look for changes in content words
• Hierarchical
• Grosz Sidners Centering theory
• Morris Hirsts algorithm
• Lexical chaining through Rogets thesaurus
• Hierarchical Relations
• Mann et al.s Rhetorical Structure Theory
• Marcus algorithm

10
TextTiling (Hearst 94)

• Goal find multi-paragraph topics
• Example 21 paragraph article called Stargazers

11
TextTiling (Hearst 94)

• Goal find multi-paragraph topics
• But its difficult to define topic (Brown
  Yule)
• Focus instead on topic shift or change
• Change in content, by contrast with setting,
  scene, characters
• Mechanism
• compare adjacent blocks of text
• look for shifts in vocabulary

12
Intuition behind TextTiling
13
TextTiling Algorithm

• Tokenization
• Lexical Score Determination
• Blocks
• Vocabulary Introductions
• Chains
• Boundary Identification

14
Tokenization

• Convert text stream into terms (words)
• Remove stop words
• Reduce to root (inflectional morphology)
• Subdivide into token-sequences
• (substitute for sentences)
• Find potential boundary points
• (paragraphs breaks)

15
Determining Scores

• Compute a score at each token-sequence gap
• Score based on lexical occurrences
• Block algorithm

16
(No Transcript)
17
Boundary Identification

• Smooth the plot (average smoothing)
• Assign depth score at each token-sequence gap
• Deeper valleys score higher
• Order boundaries by depth score
• Choose boundary cut off (avg-sd/2)

18
Evaluation

• Data
• Twelve news articles from Dialog
• Seven human judges per article
• major boundaries chosen by gt 3 judges
• Avg number of paragraphs 26.75
• Avg number of boundaries 10 (39)
• Results
• Between upper and lower bounds
• Upper bound judges averages
• Lower bound reasonable simple algorithm

19
Assessing Agreement Among Judges

• KAPPA Coefficient
• Measures pairwise agreement
• Takes expected chance agreement into account
• P(A) proportion of times judges agree
• P(E) proportion expected chance agreement
• .43 to .68 (Isard Carletta 95, boundaries)
• .65 to .90 (Rose 95, sentence segmentation)
• Here, k .647

20
TextTiling Conclusions

• First computational investigation into
  multi-paragraph discourse units
• Simple Discourse Cue position-sensitive term
  repetition
• Acceptable performance for some tasks
• Has been reproduced/used by many researchers
• Multi-lingual
• (applied by others to French, German, Arabic)

21
What Can Hierarchical Structure Tell Us?

• Welcome to word processing.
• Thats using a computer to type letters and
  reports. Make a typo?
• No problem.
• Just back up, type over the mistake, and its
  gone.
• ?And, it eliminates retyping.
• ?And, it eliminates retyping.

22
Centering Theory of Discourse Structure (Grosz
Sidner 86)

• A prominent theory of discourse structure
• Provides for multiple levels of analysis Ss
  purpose as well as content of utterances and S
  and Hs attentional state
• Identifies only a few, general relations that
  hold among intentions
• Often leads to a hierarchical structure
• Three components
• Linguistic structure
• Intentional structure
• Attentional structure

23
Example of Hierarchical Analysis(Morris and
Hirst 91)
24
(No Transcript)
25
Rhetorical Structure Theory (Mann, Matthiessen,
and Thompson 89)

• One theory of discourse structure, based on
  identifying relations between parts of the text
• Identify meaningful units and the relations
  between them
• Clauses and clause-like units that are
  unequivocally the nucleus or satellite of a
  rhetorical relation.
• Only the midday sun at tropical latitudes is warm
  enough to thaw ice on occasions, but any
  liquid water formed in this way would evaporate
  almost instantly because of the low atmospheric
  pressure.
• Nucleus/satellite notion encodes asymmetry

26
Rhetorical Structure Theory

• Some rhetorical relations
• Elaboration (set/member,class/instance/whole/part
  )
• Contrast multinuclear
• Condition Sat presents precondition for N
• Purpose Sat presents goal of the activity in N
• Sequence multinuclear
• Result N results from something presented in Sat
• Evidence Sat provides evidence for something
  claimed in N

27
Determining high-level relations
Smart cards are not a new phenomenon.1 They
have been in development since the late 1970s and
have found major applications in Europe, with
more than a quarter of a billion cards made so
far.2 The vast majority of chips have gone into
prepaid, disposable telephone cards, but even so
the experience gained has reduced manufacturing
costs, improved reliability and proved the
viability of smart cards.3 International and
national standards for smart cards are well under
development to ensure that cards, readers and the
software for the many different applications that
may reside on them can work together seamlessly
and securely.4 Standards set by the
International Organization for Standardization
(ISO), for example, govern the placement of
contacts on the face of a smart card so that any
card and reader will be able to connect.5
28
Representing implicit relations
Smart cards are becoming more attractive2 as
the price of microcomputing power and storage
continues to drop.3 They have two main
advantages over magnetic-stripe cards.4 First,
they can carry 10 or even 100 times as much
information5 - and hold it much
more robustly.6 Second, they can execute
complex tasks in conjunction with a terminal.7
29
Whats the Rhetorical Structure?

1. System Hello. How may I help you?
2. User I would like to find out why I was charged
   for a call?
3. System What call would you like to inquire
   about?
4. User My bill says I made a call to Syncamaloo,
   Texas, but Ive never even heard of this town.
5. System May I have the date of the call that
   appears on your bill?

30
Issues for RST

• Many variations in expression
• I have not read this book. It was written by
  Bertrand Russell.
• I have not read this book, which was written
  by Bertrand Russell.
• I have not read this book written by Bertrand
  Russell.
• I have not read this Bertrand Russell book.
• Rhetorical relations are ambiguous
• He caught a bad fever while he was in Africa.
• Circumstance gt Temporal-Same-Time
• With its distant orbit, Mars experiences frigid
  weather conditions. Surface temperatures
  typically average about 60 degrees Celsius at
  the equator and can dip to 123 degrees C near
  the poles.
• Evidence gt Elaboration

31
Identifying RS Automatically (Marcu 99)

• Train a parser on a discourse treebank
• 90 RS trees, hand-annotated for rhetorical
  relations
• Elementary discourse units (edus) linked by RR
• Parser learns to identify N and S and their RR
• Features Wordnet-based similarity, lexical,
  structural
• Uses discourse segmenter to identify discourse
  units
• Trained to segment on hand-labeled corpus (C4.5)
• Features 5-word POS window, presence of
  discourse markers, punctuation, seen a verb?,
• Eval 96-8 accuracy

32
Identifying RS Automatically (Marcu 99)

• Evaluation of parser
• Id edus Recall 75, Precision 97
• Id hierarchical structure (2 edus related)
  Recall 71, Precision 84
• Id nucleus/satellite labels Recall 58,
  Precision 69
• Id RR Recall 38, Precision 45
• Later errors due mostly to edu mis-identification
• Id of hierarchical structure and n/s status
  comparable to human when hand-labeled edus used
• Hierarchical structure is easier to id than RR

33
Some Problems with RST (cf. Moore Pollack 92)

• How many Rhetorical Relations are there?
• How can we use RST in dialogue as well as
  monologue?
• RST does not allow for multiple relations holding
  between parts of a discourse
• RST does not model overall structure of the
  discourse

34
Referring Expressions

• Referring expressions are words or phrases, the
  semantic interpretation of which is a discourse
  entity
• (also called referent)
• Discourse entities are semantic objects .
• Can have multiple syntactic realizations within
  a text
• Discourse entities exist in the domain D, in
  which a text is interpreted

35
Referring Expressions Example

• A pretty woman entered the restaurant. She sat at
  the table next to mine and only then I recognized
  her. This was Amy Garcia, my next door neighbor
  from 10 years ago. The woman has totally changed!
  Amy was at the time shy

36
Pronouns vs. Full NP

• A pretty woman entered the restaurant. She sat at
  the table next to mine and only then I recognized
  her. This was Amy Garcia, my next door neighbor
  from 10 years ago. The woman has totally changed!
  Amy was at the time shy

37
Definite vs. Indefinite NPs

• A pretty woman entered the restaurant. She sat at
  the table next to mine and only then I recognized
  her. This was Amy Garcia, my next door neighbor
  from 10 years ago. The woman has totally changed!
  Amy was at the time shy

38
Common Noun vs. Proper Noun

• A pretty woman entered the restaurant. She sat at
  the table next to mine and only then I recognized
  her. This was Amy Garcia, my next door neighbor
  from 10 years ago. The woman has totally changed!
  Amy was at the time shy

39
Modified vs. Bare head NP

• A pretty woman entered the restaurant. She sat at
  the table next to mine and only then I recognized
  her. This was Amy Garcia, my next door neighbor
  from 10 years ago. The woman has totally changed!
  Amy was at the time shy

40
Premodified vs. Postmodified

• A pretty woman entered the restaurant. She sat at
  the table next to mine and only then I recognized
  her. This was Amy Garcia, my next door neighbor
  from 10 years ago. The woman has totally changed!
  Amy was at the time shy

41
Anaphora resolution

• Finding in a text all the referring expressions
  that have one and the same denotation
• Pronominal anaphora resolution
• Anaphora resolution between named entities
• Full noun phrase anaphora resolution

42
Anaphora Resolution

• A pretty woman entered the restaurant. She sat at
  the table next to mine and only then I recognized
  her. This was Amy Garcia, my next door neighbor
  from 10 years ago. The woman has totally changed!
  Amy was at the time shy

43
Pronominal anaphora resolution

• Rule-based vs statistical
• (Ken 1996), (Lap 1994) vs (Ge 1998)
• Performed on full syntactic parse vs on shallow
  syntactic parse
• (Lap 1994), (Ge 1998) vs (Ken 1996)
• Type of text used for the evaluation
• (Lap 1994) computer manual texts (86 accuracy)
• (Ge 1998) WSJ articles (83 accuracy)
• (Ken 1996) different genres (75 accuracy)

44
Pronominal anaphora resolution

• Generic vs specific reference
• 1. The Vice-President of the United States is
  also President of the Senate.
• 2. Historically, he is the Presidents key person
  in negotiations with Congress
• 3a. He is required to be 35 years old.
• 3b. As Ambassador to China, he handled many
  tricky negotiations, so he is well prepared for
  the job

45
Talking to a Machine.and (often) Getting an
Answer

• Todays spoken dialogue systems make it possible
  to accomplish real tasks without talking to a
  person
• Key advances
• Stick to goal-directed interactions in a limited
  domain
• Prime users to adopt the vocabulary you can
  recognize
• Partition the interaction into manageable stages
• Judicious use of system vs. mixed initiative

46
Acoustic and Prosodic Cues to Discourse Structure

• Intuition
• Speakers vary acoustic and prosodic cues to
  convey variation in discourse structure
• Systematic? In read or spontaneous speech?
• Evidence
• Observations from recorded corpora
• Laboratory experiments
• Machine learning of discourse structure from
  acoustic/prosodic features

47
Boston Directions Corpus (Hirschberg Nakatani
96)

• Experimental Design
• 12 speakers 4 used
• Spontaneous and read versions of 9
  direction-giving tasks
• Corpus 50m read 67m spon
• Labeling
• Prosodic ToBI intonational labeling
• Discourse Grosz Sidner

48
Boston Directions Corpus Describe how to get to
MIT from Harvard

• ds1 step 1, enter and get token
• first
• enter the Harvard Square T stop
• and buy a token
• ds2 inbound on red line
• then
• proceed to get on the
• inbound
• um
• Red Line
• uh subway

49

• ds3 take subway from hs, to cs to ks
• and
• take the subway
• from Harvard Square
• to Central Square
• and then to Kendall Square
• ds4 describe ks station
• youll see a music sculpture there
• which will tell you its Kendall Square
• its very nice
• ds5 get off T.
• then get off the T

50
Dialogue vs. Monologue

• Monologue and dialogue both involve interpreting
• Information status
• Coherence issues
• Reference resolution
• Speech acts, implicature, intentionality
• Dialogue involves managing
• Turn-taking
• Grounding and repairing misunderstandings
• Initiative and confirmation strategies

51
Segmenting Speech into Utterances

• What is an utterance?
• Why is EOU detection harder than EOS?
• How does speech differ from text?
• Single syntactic sentence may span several turns
• A We've got you on USAir flight 99
• B Yep
• A leaving on December 1.
• Multiple syntactic sentences may occur in single
  turn
• A We've got you on USAir flight 99 leaving on
  December. Do you need a rental car?
• Intonational definitions intonational phrase,
  breath group, intonation unit

52
Turns and Utterances

• Dialogue is characterized by turn-taking who
  should talk next, and when they should talk
• How do we identify turns in recorded speech?
• Little speaker overlap (around 5 in English
  --although depends on domain)
• But little silence between turns either
• How do we know when a speaker is giving up or
  taking a turn? Holding the floor? How do we
  know when a speaker is interruptable?

53
Simplified Turn-Taking Rule (Sacks et al)

• At each transition-relevance place (TRP) of each
  turn
• If current speaker has selected A as next
  speaker, then A must speak next
• If current speaker does not select next speaker,
  any other speaker may take next turn
• If no one else takes next turn, the current
  speaker may take next turn
• TRPs are where the structure of the language
  allows speaker shifts to occur

54

• Adjacency pairs set up next speaker expectations
• GREETING/GREETING
• QUESTION/ANSWER
• COMPLIMENT/DOWNPLAYER
• REQUEST/GRANT
• Significant silence is dispreferred
• A Is there something bothering you or not?
  (1.0s)
• A Yes or no? (1.5s)
• A Eh?
• B No.

55
Turntaking and Initiative Strategies

• System Initiative
• S Please give me your arrival city name.
• U Baltimore.
• S Please give me your departure city name.
• User Initiative
• S How may I help you?
• U I want to go from Boston to Baltimore on
  November 8.
• Mixed initiative
• S How may I help you?
• U I want to go to Boston.
• S What day do you want to go to Boston?

56
Grounding (Clark Shaefer 89)

• Conversational participants dont just take turns
  speaking.they try to establish common ground (or
  mutual belief)
• H must ground a S's utterances by making it clear
  whether or not understanding has occurred
• How do hearers do this?
• Several different mechanisms

57
Grounding Mechanisms(Clark Shaefer 89)

• S I can upgrade you to an SUV at that rate.
• Continued attention
• (U gazes appreciatively at S)
• Relevant next contribution
• U Do you have a RAV4 available?
• Acknowledgement/backchannel
• U Ok/Mhmmm/Great!
• Demonstration/paraphrase
• U An SUV.
• Display/repetition
• U You can upgrade me to an SUV at the same rate?
• Request for repair
• U I beg your pardon?

58
How do we evaluate Dialogue Systems?

• PARADISE framework (Walker et al 00)
• Performance of a dialogue system is affected
  both by what gets accomplished by the user and
  the dialogue agent and how it gets accomplished
• Efficiency of the InteractionUser Turns, System
  Turns, Elapsed Time
• Quality of the Interaction ASR rejections, Time
  Out Prompts, Help Requests, Barge-Ins, Mean
  Recognition Score (concept accuracy),
  Cancellation Requests
• User Satisfaction
• Task Success perceived completion, information
  extracted

59
Identifying Misrecognitions and User Corrections
Automatically (Hirschberg, Litman Swerts)

• Collect corpus from interactive voice response
  system
• Identify speaker turns
• incorrectly recognized
• where speakers first aware of error
• that correct misrecognitions
• Identify prosodic features of turns in each
  category and compare to other turns
• Use Machine Learning techniques to train a
  classifier to make these distinctions
  automatically

60
Turn Types
TOOT Hi. This is ATT Amtrak Schedule System.
This is TOOT. How may I help you? User Hello.
I would like trains from Philadelphia to New York
leaving on Sunday at ten thirty in the evening.
TOOT Which city do you want to go to? User
New York.
misrecognition
correction
aware site
61
Results

• Reduced error in predicting misrecognized turns
  to 8.64
• Error in predicting awares (12)
• Error in predicting corrections (18-21)

62
Dialogue Conclusions

• Spoken dialogue systems presents new problems --
  but also new possibilities
• Recognizing speech introduces a new source of
  errors
• Additional information provided in the speech
  stream offers new information about users
  intended meanings, emotional state (grounding of
  information, speech acts, reaction to system
  errors)
• Why spoken dialogue systems rather than web-based
  interfaces?