NRRC Summer 02 MiniWorkshop Report Reuse

1

• NRRC Summer 02Mini-Workshop Report Reuse

Abraham Ittycheriah, Andy Latto, Marc Light,
Nancy McCracken
June 6th, 2002
MITRE
2
Reuse what is it?
An advanced QA system should be able to
accumulate questions, answers, and other
auxiliary information. This information could
then be reused to enable the system to better
answer future questions. In this way, a system
could duplicate a humans ability to gain
knowledge and proficiency in an area as she or he
answers questions.

• More than just same question, different wording
  (TREC-9)
• What dates are the Sydney Olympics on? vs.
• What are the dates of the Summer Olympics in
  Sydney?
• More than just context processing (TREC-10
  Context)
• What type of vessel was the Varyag?, In what
  facility was it constructed?, In what country is
  this facility located?

3
Examples

• When did the Brady bill come into effect?
  (February 1994)
• What president passed the bill?
  (Bill Clinton)
• How much calcium should an adult woman get in her
  diet?
• How much Vitamin D do you need to absorb the
  recommended amount of calcium?
• Whats the name of the Nike golf ball Tiger Woods
  uses?
• What golf ball did Tiger Woods use in the US
  open?
• Who did Billy Jean King beat in battle of the
  sexes?
• When tennis champion Billie Jean King accepted
  the half-humorous challenge Bobby Riggs threw
  down in 1973, ...
• How many titles did King win?
• Golfer Betsy King or tennis player King ...

4
What We Proposed To Do

• Explicate find questions and answers such as the
  previous examples from a small number of domains
  (e.g., epidemiology, terrorism, nuclear
  proliferation, etc.)
• Categorize find and classify different types of
  possible re-use
• Describe write short document discussing the
  categories, their characteristics, and their
  distributions
• Recommend discuss possible future work in the
  form of a full workshop

5
The Reuse Corpus

• Single-user Question-Answer sets (sequences)
• Researchers and grad students generated questions
• Instructions research a topic of interest and
  emulate a QA system by using a web search engine
• 221 questions, 29 topics, one sequence each, 8
  qs ave.
• Multi-user sets
• Web queries from summer of 2000, grepped out 1
  Million questions (John Henderson is responsible
  for corpus)
• 2130 questions, 13 topics, 154 on average (large
  variance)
• Annotations
• Researchers annotated 100 reuse possibilities
  during two day workshop at MITRE
• Reuse Corpus will be openly available but only a
  fraction of the multi-user questions will be
  derived so that they can be distributed

6
Topics

• IBM Challenger Space Craft, Choosing between a
  macintosh and a pc, Columbine High School
  shooting, Dave Koresh and Waco, Death Penalty,
  Ebay business model, End of Apartheid in 1993,
  Global warming, Human cloning, Martha Stewart
  Corporation, NAFTA, Pathfinder lands on Mars,
  Terrorism in the 21st Century, the Brady bill,
  The collapse of the Soviet Union, The fall of
  Enron, The Israeli-Palestinian conflict, Timothy
  McVeigh, Who are the Taliban, World Trade Center
  - Structural integrity, World Wide Web
• Syracuse heel pain, calcium, ice cream, poetry,
  Saskatoon, special education, Washington DC
• MITRE all of Syracuses Tiger Woods, Tupac
  Shakur, The Oscars, the Stanley Cup, the 2000
  Summer Olympics, Digital cameras

7
Reuse Categories
8
Examples
sameQuestions

• Which team won the Stanley Cup in 1994?
• Who won the Stanley Cup playoffs in 1994?
• When does the summer 2000 Olympics start?
• When are the Summer 2000 Olympics?
• How much calcium should an adult woman get in her
  diet?
• How much Vitamin D do you need to absorb the
  recommended amount of calcium?
• How many awards did Gone with the Wind win?
• Who starred in gone with the wind?
• Who poses the biggest threat to the United
  States?
• What countries are developing weapons of mass
  destruction?

subSuperQuestions
embeddedQs
questionFacts
questionClarifications
9
More Examples
sameDocuments

• What is achilles tendonitis?
• What are the causes of achilles tendonitis?
• How can achilles tendonitis be treated?
• What irons does tiger woods use?
  (Titleist)
• What loft one wood does tiger woods use?
  (Titleist)
• What putter does Tiger Woods use?
  (Titleist)
• When did the Brady bill come into effect?
  (February 1994)
• What president passed the bill?
  (Bill Clinton)

InductiveAnswers
answerLimits
10
Even More Examples
sameFacts

• What medicines cure anthrax? (Cipro,and other
  antibiotics)
• What does Cipro cure? (anthrax)
• Who did Billy Jean King beat in battle of the
  sexes?
• When tennis champion Billie Jean King accepted
  the half-humorous challenge Bobby Riggs threw
  down in 1973, ...
• How many titles did King win?
• Golfer Betsy King or tennis player King ...
• Is Tupac still alive?
• How Tupac really died, where can I find a picture
  of tupacs death, what happened between tupac and
  suge knight, when did tupac die?, where can I
  find the newspaper entitled is tupac alive, who
  killed tupac shackur

topicCharacterizations
evidences
11
Please, No More Examples!

• How much calcium should an adult woman get in her
  diet?
• Adult women need 1000-1500mg per day ...
• What are the factors that increase calcium
  excretion?
• High protein diet, caffeine, alcohol, sodium, low
  exercise. Note that the minimum daily
  requirements seem to assume some of these, so a
  lifestyle without these factors may require fewer
  mgs of calcium per day.

answerRevisions
12
Strategies for Implementing Reuse

• Keep a Cache of previous questions and answers
• Build topic detection and tracking for question
  sets
• Build a classifiers that take the current
  question and a previous question on the same
  topic and assigns a reuse categories to the pair
  (likely the NULL category)
• For example, a classifier could be built to
  identitify sameQuestions
• Use the User to help
• Assess the reuse capabilities by evaluating QA
  system with and without them turned on

13
Recommendations for Full Workshop

• Collect and annotate a larger Reuse Corpus
• Extend a functioning QA system (maybe even work
  with 2 systems that differ in approach)
• Build QA TDT system
• Build question-answer set classifiers for reuse
  categories
• Mine the user for information relevant to
  identifying reuse categories and for vetting
  information to be reused
• Evaluate the systems with and without the reuse
  modules

14
Conclusion

• Introduce and demarcate the topic of reuse
• Collected Reuse Corpus and started annotating
• Developed Categories of reuse
• Discussed strategies for implementing reuse
• Wrote 18 page report
• The basis for a solid proposal for next years
  workshop