Answering List Questions using Cooccurrence and Clustering

1
Answering List Questions using Co-occurrence and
Clustering

• Majid Razmara and Leila Kosseim
• Concordia University
• m_razma_at_cs.concordia.ca

2
Introduction

• Question Answering
• TREC QA track
• Question Series
• Corpora

• Target American Girl dolls
• FACTOID In what year were American Girl dolls
  first introduced?
• LIST Name the historical dolls.
• LIST Which American Girl dolls have had TV
  movies made about them?
• FACTOID How much does an American Girl doll
  cost?
• FACTOID How many American Girl dolls have been
  sold?
• FACTOID What is the name of the American Girl
  store in New York?
• FACTOID What corporation owns the American Girl
  company?
• OTHER Other

3
Hypothesis

• Answer Instances
• Have the same semantic entity class
• Co-occur within sentences, or
• Occur in different sentences sharing similar
  context
• Based on Distributional Hypothesis Words
  occurring in the same contexts tend to have
  similar meanings Harris, 1954.

4
Target 232 "Dulles Airport Question
232.6 "Which airlines use Dulles

• Ltw_Eng_20050712.0032 (AQUAINT-2)
• United, which operates a hub at Dulles, has six
  luggage screening machines in its basement and
  several upstairs in the ticket counter area.
• Delta, Northwest, American, British Airways and
  KLM share four screening machines in the
  basement.
• Ltw_Eng_20060102.0106 (AQUAINT-2)
• Independence said its last flight Thursday will
  leave White Plains, N.Y., bound for Dulles
  Airport.
• Flyi suffered from rising jet fuel costs and the
  aggressive response of competitors, led by United
  and US Airways.
• New York Times (Web)
• Continental Airlines sued United Airlines and
  the committee that oversees operations at
  Washington Dulles International Airport
  yesterday, contending that recently installed
  baggage-sizing templates inhibited competition.
• Wikipedia (Web)
• At its peak of 600 flights daily, Independence,
  combined with service from JetBlue and AirTran,
  briefly made Dulles the largest low-cost hub in
  the United States.

4
5
Our Approach

• Create an initial candidate list
• Answer Type Recognition
• Document Retrieval
• Candidate Answer Extraction
• It may also be imported from an external source
  (e.g. Factoid QA)
• Extract co-occurrence information
• Cluster candidates based on their co-occurrence

6
Answer Type Recognition

• 9 Types
• Person, Country, Organization, Job, Movie,
  Nationality, City, State, and Other
• Lexical Patterns
• (Name List What Which) (persons people
  men women players contestants artists
  opponents students) ? PERSON
• (Name List What Which) (countries
  nations) ? COUNTRY
• Syntagmatic Patterns for Other types
• (WDT WP VB NN) (DT JJ) (NNS NNP NN
  JJ ) (NNS NNP NN NNPS) (VBN VBD
  VBZ WP )
• (WDT WP VB NN) (VBD VBP) (DT JJ JJR
  PRP IN) (NNS NNP NN ) (NNS NNP
  NN)
• Type Resolution

7
Type Resolution

• Resolves the answer subtype to one of the main
  types
• List previous conductors of the Boston Pops.
• Type OTHER Sub Type Conductor ? PERSON
• WordNet's Hypernym Hierarchy

8
Document Retrieval

• Document Collection
• Source Document Collection
• Few documents
• To extract candidates
• Domain Document Collection
• Many documents
• To extract co-occurrence information
• Query Generation
• Google Query on Web
• Lucene Query on Corpora

9
Candidate Answer Extraction

• Term Extraction
• Extract all terms that conform to the expected
  answer type
• Person, Organization, Job
• Intersection of several NE taggers LingPipe,
  Stanford tagger Gate NE
• To get a better precision
• Country, State, City, Nationality
• Gazetteer
• To get a better precision
• Movie, Other
• Capitalized and quoted terms
• Verification of Movie
• Verification of Other

numHits(GoogleQuery intitleTerm
sitewww.imdb.com)
10
Co-occurrence Information Extraction

• Domain Collection Documents are split into
  sentences
• Each sentence is checked as to whether it
  contains candidate answers

11
Hierarchical Agglomerative Clustering

• Steps
• Put each candidate term ti in a separate cluster
  Ci
• Compute the similarity between each pair of
  clusters
• Average Linkage
• Merge two clusters with highest inter-cluster
  similarity
• Update all relations between this new cluster and
  other clusters
• Go to step 3 until
• There are only N clusters, or
• The similarity is less than a threshold

12
The Similarity Measure

• Similarity between each pair of candidates
• Based on co-occurrence within sentences
• Using chi-square (??2)
• Shortcoming

13
Pinpointing the Right Cluster

• Question and target keywords are used as spies
• Spies are
• Inserted into the list of candidate answers
• Are treated as candidate answers, hence
• their similarity to one another and similarity to
  candidate answers are computed
• they are clustered along with candidate answers
• The cluster with the most number of spies is
  returned
• The spies are removed
• Other approaches

14
Target 269 Pakistan earthquakes of October
2005 Question 269.2 What countries were affected
by this earthquake?
Cluster-31
oman
pakistan, 2005, afghanistan, octob, u.s, india,
affect, earthquak
pakistan, 2005, afghanistan, octob, u.s, india,
affect, earthquak
pakistan, 2005, afghanistan, octob, u.s, india,
affect, earthquak
Recall 2/3
Precision 2/3
F-score 2/3
14
15
Results in TREC 2007
F14.5
16
Evaluation of Clustering

• Baseline
• List of candidate answers prior to clustering
• Our Approach
• List of candidate answers filtered by the
  clustering
• Theoretical Maximum
• The best possible output of clustering based on
  the initial list

17
Evaluation of each Question Type
18
Future Work

• Developing a module that verifies whether each
  candidate is a member of the answer type
• In case of Movie and Other types
• Using co-occurrence at the paragraph level rather
  than the sentence level
• Anaphora Resolution can be used
• Another method for similarity measure
• ?2 does not work well with sparse data
• for example, using Yates correction for
  continuity (Yates ?2)
• Using different clustering approaches
• Using different similarity measures
• Mutual Information

19

• Questions?