Recognising Textual Entailment Johan Bos School of Informatics University of Edinburgh Scotland,UK

1
Recognising Textual EntailmentJohan BosSchool
of InformaticsUniversity of EdinburghScotland,UK
2
Textual Entailment

• Example 1 (TRUE)Text His family has
  steadfastily denied the charges.
• Hypothesis The charges were denied by his
  family.

3
Textual Entailment

• Example 2 (TRUE)
• Text In 1998, the general Assembly of the
  Nippon Sei Ko Kai (Anglican Church in Japan)
  voted to accept female priests.
• Hypothesis The Anglican church in Japan
  approved the ordination of women.

4
Textual Entailment

• Example 3 (FALSE)
• Text The city Tenochtitlan grew rapidly and
  was the center of the Aztecs great empire.
• Hypothesis Tenochtitlan quickly spread over
  the island, marshes, and swamps.

5
Textual Entailment

• Example 4 (FALSE)
• Text Clintons new book is not big seller
  here.
• Hypothesis Clintons book is a big seller.

6
Approach in a Nutshell

• Compute semantic representations for Text and
  Hypothesis
• Use logical inference (theorem proving) to
  determiner whether T entails H
• Compare this with a shallow approach (word
  overlap)
• Use Machine Learning to combine logical inference
  with shallow word overlap

7
Talk Outline

• Compositional Semantics
• Discourse Representation Theory
• Combinatorial Categorial Grammar
• Lambda Calculus as glue
• Inference
• Theorem Proving
• Model Building
• Approximating Entailment
• Evaluation

8
Evaluation

• Use dataset (training and test set) of the RTE
  challenge organised by PASCAL network
• Baseline is 50
• Dataset is based on different tasks
• CD (Comparable documents)
• QA (Question answering)
• IE (Information extraction)
• MR (machine translation
• RC (reading comprehension
• PP (paraphrase acquisition
• IR (information retrieval)

9
Machine Learning

• Each entailment example pair is expressed as a
  feature vector
• Train a decision tree for classification into
  TRUE and FALSE
• WEKA (Witten Frank 2000)
• Test on the test set
• Evaluation measures
• Accuracy ( correct judgements)
• CWS (confidence-weighted score)

10
Features

• Shallow features
• Word overlap between text and hypothesis
• Length of text and hypothesis
• Deep semantic features
• uninformative/inconsistent (theorem prover)
• Domainsize, modelsize, and absolute and relative
  differences between text and hypothesis (model
  builder)

11
Results
Accuracy CWS
Shallow 0.569 0.624
Deep 0.562 0.608
Hybrid (SD) 0.577 0.632
HybridTask 0.612 0.646
12
Conclusions

• Hybrid approach combines shallow analysis with
  both theorem proving and model building and
  achieves high accuracy scores (compared to other
  systems)
• Need more work on computing appropriate
  background knowledge
• Future work also includes task-based evaluation,
  for instance in a QA system

13
Acknowledgements

• Joined work with
• James Curran (Sydney University)
• Steve Clark (University of Oxford)
• Katja Markert (University of Leeds)
• Patrick Blackburn (LORIA, Nancy)