Question Generation: Taxonomies and Data

1
Question Generation Taxonomies and Data

• Corina FORASCUa, b, and Iuliana DRAGHICIa
• a Faculty of Computer Science, University Al. I.
  Cuza of Iasi, Romania
• b Research Institute in Artificial Intelligence,
  Romanian Academy, Bucharest
• corinfor_at_info.uaic.ro
• idraghici_at_info.uaic.ro

2
Outline

1. Theoretical approaches
2. Socrates
3. Bloom
4. Lehnert Graesser
5. Practical settings
6. LSAT tests
7. QA_at_TREC
8. QA_at_CLEF
9. Conclusions

3
A prototype of a QG system

• Formalizations and models from question
  taxonomies model for question creation and
  extraction
• Collection of texts a specific purpose gt a
  selected taxonomy of questions used to feed the
  system
• Questions are generated from the minimally
  processed text corpus using a lexicalized model
  of the taxonomy and a set of pattern-rules,
  previously developed either manually or
  automatically
• Benefits research-oriented tasks such as QA,
  automatic creation of questionnaires, quizzes and
  test questions.

4
Theoretical approaches in question taxonomies

• The taxonomy of Socratic questions
• Bloom's Taxonomy
• The taxonomy of Lehnert and Graesser

5
The taxonomy of Socratic questions

• based on the practice of thoughtful questioning,
  essential for critical thinking
• deep questions
• used in teaching, (medical) examination,
  engineering and trial questioning
• used in the LSAT (Law School Admission Test)
  multiple-choice tests, mainly as sub-objectives
  to evaluate the critical thinking abilities

6
The taxonomy of Socratic questions

• Clarification - Can you give me an example of a
  citric fruit?
• Probing assumptions - What would happen if people
  would stop using plastic bottles?
• Probing Reasons and Evidence - How do you know
  that the Earth revolves around the Sun?
• Viewpoint and Perspectives - Why is recycling
  necessary?
• Probing Implications and Consequences - How does
  globalization affect the culture and traditions
  of every country?
• Questions about Questions - Why do you think I
  asked you the previous question?

7
Bloom's Taxonomy

• expresses qualitatively different levels of
  thinking
• contains six levels, arranged from the lowest
  level of complexity (shallow questions) to the
  highest (deep questions)
• assigns a set of keywords to each question type
• starting point for many students knowledge
  evaluation developers

8
Bloom's Taxonomy

• Knowledge - What is the biggest river in Europe?
• (know, who, define, what, where, list, when)
• Comprehension - If you draw lines between 3
  points, what geometric figure do you get?
• (describe, use your own words, outline, explain,
  discuss, and compare)
• Application - How would you use the stars to
  locate north?
• (apply, manipulate, put to use, employ,
  dramatize, demonstrate, interpret, and choose)
• Analysis - What are the factors that cause
  amnesia?
• (analyze, why, take apart, diagram, draw
  conclusions, simplify, distinguish, and survey)
• Synthesis - How would you rearrange these
  letters to form a new word
• (compose, construct, design, revise, create,
  formulate, produce, and plan)
• Evaluation - Why do you think George Washington
  was elected president?
• ( judge, assess, evaluate, value, criticize,
  compare)

9
The taxonomy of Lehnert and Graesser

• basis for the QA_at_TREC questions
• used in the subsequent interpretation and memory
  search

10
The taxonomy of Lehnert and Graesser

• Lehnert 13 question types
• Shallow (Verification, Disjunctive, Concept
  completion)
• Intermediate (Feature specification,
  Quantification)
• Deep (Causal antecedent consequent, Goal
  orientation, Enablement, Instrumental/procedural,
  Expectation, Judgmental)
• Graesser and Person four more categories
• Example - What is an example of X?
• Definition - What is meant by X?
• Comparison - How is X different from Y?
• Interpretation - What is the significance of X?

11
Question taxonomies in practical settings

• The LSAT data and taxonomy
• Taxonomies in Question-Answering evaluation
  campaigns

12
The LSAT data and taxonomy

• used for admission in law schools in US, Canada
  and other countries
• measures reading and verbal reasoning skills in a
  standardized way, without any need of knowledge
  of the law
• three categories, with other subtypes, each of
  them based on specially selected and prepared
  texts

13
The LSAT data and taxonomy

• Reading Comprehension - measure the ability to
  read, understand and reason based on lengthy,
  complex materials (eight question types)
• Logical Reasoning - evaluate the ability to
  understand, analyze, criticize, and complete a
  variety of arguments contained in short passages
  of text (seven question types)
• Analytical Reasoning - measure the ability to
  understand and to draw logical conclusions about
  the structure of relationships. Based on
  specially created texts and specific subtypes.

14
LSAT Reading Comprehension

• Overall nature of the passage main idea or
  primary purpose of the passage
• Information explicitly stated what the passage
  literally says and doesnt say
• Meaning or purpose clarifications about the
  non-typical meaning of a word or phrase from
  context
• Authors attitude authors purpose in mentioning
  or citing something, as it is revealed in a
  passage tone or the language used
• Organization or structure measures the
  understanding of the way the passage is
  structured or organized can overlap with the
  first subtype
• Inference draw inferences about the
  information, about the views of the author or a
  person the author discusses, or about the
  authors attitude
• Application relate the material in the passage
  to information not in the passage (new
  contexts/new evidence)
• Comparative reading similar to any of the
  previous types, except that they are based on two
  shorter passages instead of one longer passage

15
LSAT Logical Reasoning

• Point or issue identify the conclusion or main
  point of an argument or a dispute (dialogue)
• Assumption determine which assumptions are
  either necessary or sufficient to properly draw
  the conclusion in a chain of reasoning
• Inference (deductively or non-deductively) draw
  reasonable conclusions from given evidence or
  premises the text can be an incomplete argument
• Principles identify general rules and
  principles involved in reasoning and to
  understand their use
• Structure identify the technique of reasoning or
  the role played by a sentence or phrase an
  argument or chain of reasoning
• Additional evidence determine how additional
  evidence or argumentation strengthens or weakens
  an argument or conclusion
• Identifying explanations and recognizing
  resolutions of conflicting facts or arguments

16
QA_at_TREC

• using an ad-hoc English text collection (LA
  Times, Financial Times, Federal Register and
  FBIS)
• questions following the Lehnert taxonomy,
  manually developed
• TREC 2003
• List Name U.S. entertainers that later became
  politicians.
• Definition What is a golden parachute?
• Factoid How many time zones are there in the
  world?
• since 2004 questions grouped into series,
  focusing on a target

17
QA_at_CLEF

• Since 2003, following the TREC model
• Uses comparable corpora of news in three
  languages (Dutch, Italian, Spanish)
• 450 questions in four languages (including
  English)
• 3 subtypes Factoid, Definition, List
• 2007-2008 use of the Wikipedia corpus, in 11
  European languages
• Topic-related questions , with no indications
  about the topic or the relations between
  topic-related questions.

18
QA_at_CLEF

• 2009 campaign the Acquis Communautaire corpus
  a parallel collection of the European laws,
  available in 22 languages.
• Taxonomy of (unrelated ) questions
• Factoid What rights do performers have with
  respect to the reproduction in phonograms?
• Definition What are import duties and taxes?
• Purpose What is the purpose of quick-freezing?
• Reason Why can some alcohols or essential oils
  help in the preservation of cosmetic products?
• Procedure How must the fuel tanks be installed?

19
Conclusions

• Question taxonomies basis for QG campaigns
• Existing taxonomies (LSAT, TREC, CLEF)
• wide overlap,
• freely available data,
• various domains,
• multilingual setting