Hang Cui Renxu Sun Keya Li Min-Yen Kan Tat-Seng Chua Department of Computer Science National University of Singapore

1
Question Answering Passage Retrieval Using
Dependency Parsing Hang CuiRenxu SunKeya
LiMin-Yen KanTat-Seng ChuaDepartment of
Computer ScienceNational University of Singapore
2
Passage Retrieval in Question Answering
Document Retrieval
QA System

• To narrow down the search scope
• Can answer questions with more context

Passage Retrieval

• Lexical density based
• Distance between question words

Answer Extraction
3
Density Based Passage Retrieval Method

• However, density based can err when

ltQuestiongt What percent of the nation's cheese
does Wisconsin produce? Incorrect the number
of consumers who mention California when asked
about cheese has risen by 14 percent, while the
number specifying Wisconsin has dropped 16
percent. Incorrect The wry It's the Cheese
ads, which attribute California's allure to its
cheese _ and indulge in an occasional dig at the
Wisconsin stuff'' sales of cheese in
California grew three times as fast as sales in
the nation as a whole 3.7 percent compared to 1.2
percent, Incorrect Awareness of the Real
California Cheese logo, which appears on about 95
percent of California cheeses, has also made
strides. Correct In Wisconsin, where farmers
produce roughly 28 percent of the nation's
cheese, the outrage is palpable.
Relationships between matched words differ
4
Our Solution

• Examine the relationship between words
• Dependency relations
• Exact match of relations for answer extraction
• Has low recall because same relations are often
  phrased differently
• Fuzzy match of dependency relationship
• Statistical similarity of relations

5
Measuring Sentence Similarity
Sim (Sent1, Sent2) ?
Sentence 1
Sentence 2
Matched words
Lexical matching
Similarity of relations between matched words

Similarity of individual relations
6
Outline

• Extracting and Paring Relation Paths
• Measuring Path Match Scores
• Learning Relation Mapping Scores
• Evaluations
• Conclusions

7
Outline

• Extracting and Paring Relation Paths
• Measuring Path Match Scores
• Learning Relation Mapping Scores
• Evaluations
• Conclusions

8
What Dependency Parsing is Like

• Minipar (Lin, 1998) for dependency parsing
• Dependency tree
• Nodes words/chunks in the sentence
• Edges (ignoring the direction) labeled by
  relation types

What percent of the nation's cheese does
Wisconsin produce?
9
Extracting Relation Paths

• Relation path
• Vector of relations between two nodes in the tree

produce lt P1 subj gt Wisconsin
percent lt P2 prep pcomp-n gt cheese

• Two constraints for relation paths
• Path length (less than 7 relations)
• Ignore those between two words that
• are within a chunk, e.g. New York.

10
Paired Paths from Question and Answer
In Wisconsin, where farmers produce roughly 28
percent of the nation's cheese, the outrage is
palpable.
What percent of the nation's cheese does
Wisconsin produce?
lt P1(Q) subj gt
lt P1(Sent) pcomp-n mod i gt
Paired Relation Paths
SimRel (Q, Sent) ?i,j Sim (Pi (Q), Pj(Sent))
11
Outline

• Extracting and Paring Relation Paths
• Measuring Path Match Scores
• Learning Relation Mapping Scores
• Evaluations
• Conclusions

12
Measuring Path Match Degree

• Employ a variation of IBM Translation Model 1
• Path match degree (similarity) as translation
  probability
• MatchScore (PQ, PS) ? Prob (PS PQ )
• Relations as words
• Why IBM Model 1?
• No word order bag of undirected relations
• No need to estimate target sentence length
• Relation paths are determined by the parsing tree

13
Calculating Translation Probability (Similarity)
of Paths
Given two relation paths from the question and a
candidate sentence
Considering the most probable alignment
(finding the most probable mapped relations)
Take logarithm and ignore the constants (for all
sentences, question path length is a constant)
MatchScores of paths are combined to give the
sentences relevance to the question.
?
14
Outline

• Extracting and Paring Relation Paths
• Measuring Path Match Scores
• Learning Relation Mapping Scores
• Evaluations
• Conclusions

15
Training and Testing
Testing
Training

1. Mutual information (MI) based
2. Expectation Maximization (EM) based

Sim ( Q, Sent ) ?
Q - A pairs
Similarity between relation vectors
Prob ( PSent PQ ) ?
Paired Relation Paths
Similarity between individual relations
P ( Rel (Sent) Rel (Q) ) ?
Relation Mapping Scores
Relation Mapping Model
16
Approach 1 MI Based

• Measures bipartite co-occurrences in training
  path pairs
• Accounts for path length (penalize those long
  paths)
• Uses frequencies to approximate mutual
  information

17
Approach 2 EM Based

• Employ the training method from IBM Model 1
• Relation mapping scores word translation
  probability
• Utilize GIZA to accomplish training
• Iteratively boosting the precision of relation
  translation probability
• Initialization assign 1 to identical relations
  and a small constant otherwise

18
Outline

• Extracting and Paring Relation Paths
• Measuring Path Match Scores
• Learning Relation Mapping Scores
• Evaluations
• Can relation matching help?
• Can fuzzy match perform better than exact match?
• Can long questions benefit more?
• Can relation matching work on top of query
  expansion?
• Conclusions

19
Evaluation Setup

• Training data
• 3k corresponding path pairs from 10k QA pairs
  (TREC-8, 9)
• Test data
• 324 factoid questions from TREC-12 QA task
• Passage retrieval on top 200 relevant documents
  by TREC

20
Comparison Systems

• MITRE baseline
• Stemmed word overlapping
• Baseline in previous work on passage retrieval
  evaluation
• SiteQ top performing density based method
• using 3 sentence window
• NUS
• Similar to SiteQ, but using sentences as passages
• Strict Matching of Relations
• Simulate strict matching in previous work for
  answer selection
• Counting the number of exactly matched paths
• Relation matching are applied on top of MITRE and
  NUS

21
Evaluation Metrics

• Mean reciprocal rank (MRR)
• Measure the mean rank position of the correct
  answer in the returned rank list
• On the top 20 returned passages
• Percentage of questions with incorrect answers
• Precision at the top one passage

22
Performance Evaluation

• All improvements are statistically significant
  (plt0.001)
• MI and EM do not make much difference given our
  training data
• EM needs more training data
• MI is more susceptible to noise, so may not scale
  well

Fuzzy matching outperforms strict matching
significantly.
Passage retrieval systems MITRE SiteQ NUS Rel_Strict(MITRE) Rel_Strict(NUS) Rel_MI (MITRE) Rel_EM (MITRE) Rel_MI (NUS) Rel_EM (NUS)
MRR 0.2000 0.2765 0.2677 0.2990 0.3625 0.4161 0.4218 0.4756 0.4761
MRR improvement over MITRESiteQNUS N/AN/AN/A 38.26N/AN/A 33.88N/AN/A 49.508.1411.69 81.2531.1035.41 108.0950.5055.43 110.9452.5757.56 137.8572.0377.66 138.0872.1977.83
Incorrect 45.68 37.65 33.02 41.96 32.41 29.63 29.32 24.69 24.07
Precision at top one passage 0.1235 0.1975 0.1759 0.2253 0.2716 0.3364 0.3457 0.3889 0.3889
23
Performance Variation to Question Length

• Long questions, with more paired paths, tend to
  improve more
• Using the number of non-trivial question terms to
  approximate question length

24
Error Analysis

• Mismatch of question terms
• e.g. In which city is the River Seine
• Introduce question analysis
• Paraphrasing between the question and the answer
  sentence
• e.g. write the book ? be the author of the book
• Most of current techniques fail to handle it
• Finding paraphrasing via dependency parsing (Lin
  and Pantel)

25
Performance on Top of Query Expansion

• On top of query expansion, fuzzy relation
  matching brings a further 50 improvement
• However
• query expansion doesnt help much on a fuzzy
  relation matching system
• Expansion terms do not help in paring relation
  paths

Passage Retrieval Systems NUS(baseline) NUSQE Rel_MI (NUSQE) Rel_EM (NUSQE)
MRR( improvement over baseline) 0.2677 0.3293(23.00) 0.4924(83.94) 0.4935(84.35)
MRR improvement over NUSQE N/A N/A 49.54 49.86
Incorrect 33.02 28.40 22.22 22.22
Precision at top one passage 0.1759 0.2315 0.4074 0.4074
Rel_EM (NUS) 0.4761
26
Outline

• Extracting and Paring Relation Paths
• Measuring Path Match Scores
• Learning Relation Mapping Scores
• Evaluations
• Conclusions

27
Conclusions

• Proposed a novel fuzzy relation matching method
  for factoid QA passage retrieval
• Brings dramatic 70 improvement over the
  state-of-the-art systems
• Brings further 50 improvement over query
  expansion
• Future QA systems should bring in relations
  between words for better performance
• Query expansion should be integrated to relation
  matching seamlessly

28
Q A
Thanks!?