Exploiting a Verb Lexicon in Automatic Semantic Role Labelling

1
Exploiting a Verb Lexicon in Automatic Semantic
Role Labelling

• Robert S. SwierSuzanne Stevenson
• Department of Computer ScienceUniversity of
  Toronto

The support of NSERC of Canada is gratefully
acknowledged.
2
Task Description

• Goal Find a full semantic representation of
  text.
• Intermediate Goal Label each constituent with
  the role it plays in the event
• John gave a book to Mary.

JohnltAgentgt gave a bookltThemegt to MaryltRecipientgt.
3
Supervised Semantic Role Labelling

• Many SRL systems have appeared recently.Gildea
  Jurafsky, 02 Gildea Palmer, 02 Fleischman
  et al., 03 Thompson et al., 03 Hacioglu et
  al., 04 Pradhan et al., 03, 04, 05a,b and 3
  shared tasks Senseval 04 CoNLL 04, 05
• Good results, but use supervised learning.
• Require an annotated corpus with verb arguments
  identified and labelled with semantic roles.
• Expensive, needs manual development.
• Corpus may have limited use for other tasks.
• Suitability of corpus is an issue.

4
Unsupervised Semantic Role Labelling

• Our previous work (Swier Stevenson, 04)
• Combined direct application of lexical knowledge
  with iterative bootstrapping.
• Applied knowledge of possible role assignment in
  a predicate lexicon to syntactic chunks extracted
  from sentences.
• Showed promise, but was limited in several ways

5
Limitations of Unsupervised Approach

• Predicate lexicon (VerbNet) has no associated
  labelled corpus, so testing was limited.
• Training data not needed needs testing data.
• Solution Create a VN-labelled corpus by mapping
  from the roles used in an existing corpus.
• Iterative bootstrapping model
• Used a large amount of unannotated data.
• Was very slow. (Many iterations.)
• Solution Employ a more streamlined probability
  model.

6
Role Mapping

• Goal Create a VerbNet-labelled corpus by
  adapting an existing role labelled corpus.
• Choose the FrameNet Corpus.
• Uses sentences from BNC.
• Uses finer-grained roles than VerbNet.
• Intuition Fine-to-coarse is easier than
  coarse-to-fine.

7
Role Mapping Four Steps

• Used an existing mapping from FrameNet to an
  intermediate set of 39 more general roles.
  (provided by Roxana Girju, UIUC)
• Mapped from intermediate set to set of 22 roles
  used by VerbNet.
• Map from VerbNet to a more general set of 16
  roles used by our system.
• Correct the mappings on constituents which result
  in inconsistency with VerbNet.

8
Second Mapping Step, in More Detail

• Mapping from intermediate set to VerbNet roles.
• Required several sorts of mappings.
• One to one IMCause ? VNCause
• Merger IMDegree, Measure ? VNAmount
• Split IMExperiencer ? VNAgent, Experiencer
• Splits are obvious on a class-by-class basis
• NoRole IMManner, Means, ? VNNoRole
• Used for FrameNet roles that have no subsuming
  VerbNet role (such as those for adjuncts).

9
Third Step, in More Detail

• Mapping from VerbNet to our systems roles.
• We use a slightly coarser version of the VerbNet
  roles.
• Merge roles appearing difficult to distinguish.
• VNTheme, Topic ? SSTheme
• VNAsset, Amount ? SSAmount
• Step four is performed by the Frame Matcher,
  described next.

10
Frame Matcher

• The Frame Matcher is the core of our system.
• Matches thematic frames against extracted
  arguments (NPs) to find likely role
  possibilities.
• VN thematic frame Agent V Theme Prep Instrument
• Potential arguments are extracted from automatic
  parse trees via a set of hand-coded heuristics.
• Some arguments only have one role possibility.
• This primary labelled data is used for
  bootstrapping.

11
Frame Matching And Role Initialization

Jane cut the bread SUBJ(jane), OBJ(bread)
12
Frame Matching And Role Initialization

Jane cut the bread SUBJ(jane), OBJ(bread)
Best Only SUBJ Ag, Inst, OBJ Th
13
Fourth Mapping Step, in More Detail

• The role mapping sometimes produces test data
  that is inconsistent with VerbNet.
• Example brawl with NP
• FNSide2 ? SSTheme
• VerbNet (and SS) do not accept Theme as object
  of with for the verb brawl.
• Inconsistent labels in evaluation data are
  converted to NoRole by the Frame Matcher.

14
Limitations of the NoRole Conversion

• The Frame Matchers NoRole conversions have some
  limitations
• Only gold-standard labels on properly extracted
  arguments can be evaluated for consistency.
• Arguments that we fail to extract cannot be
  considered by the Frame Matcher.
• Sometimes, a role is converted to NoRole when
  there is another role that VerbNet would have
  assigned.

15
Probability Models

• We train a probability model on the primary
  labelled data to help resolve the ambiguous
  assignments.
• Investigate four simple models
• P(role verb, slot)
• P(role slot)
• P(role slot class)
• Default Assignment
• Subj/Obj/IObj/PP-Obj ? Agent/Theme/Recipient/Locat
  ion

16
Experimental Data

• Evaluated over 1159 target verbs.
• Corpus
• 30 of FrameNet for development.
• 30 for unseen testing.
• Remainder reserved for future experimentation.
• If the annotated verb is not a target verb, then
  the sentence is ignored.
• About 50 of FN sentences contain a target verb.

17
Evaluation

• Report F-measures for three tasks
• Identification of argument heads.
• Labelling of correctly identified argument heads.
• Identification Labelling (combined task).
• Compare to an informed baseline
• Uses same set of extracted arguments as system.
• Labels all arguments with default assignment.
• Uses no input from Frame Matcher.

18
Model Comparison
Task
System
19
Model Comparison
Task
System
20
Model Comparison
Task
System
21
Model Comparison
Task
System
22
Model Comparison
Task
System
23
Evaluation of Frame Choice

• System benefits slightly from reduced ambiguity
  of the Best frames method

24
Heads vs. Full Phrases

• Lower performance with full phrases, due to
  greater parsing difficulty.
• 80 of argument heads correctly extracted.
• 74 of full phrase arguments correctly extracted.

25
Comparison to Other Work

• Several differences prevent a direct comparison
• E.g., different roles, different set of
  constituents.
• Best supervised performance is .80
  F-measure.(Pradhan et al., 2005)
• Our system obtains .61 using full phrases.
• Performance lower, but system uses no labelled
  training data.

26
Conclusion

• Use an expensive, but highly reusable lexical
  resource for role labelling.
• Frame Matcher greatly reduces role ambiguity.
• Allows for a simple probability model.
• Existing labelled corpora can be effectively
  adapted to role sets of different lexicons using
  a mapping.