TERQAS: Time and Event Recognition for Question Answering Systems

1
TERQASTime and Event Recognition for Question
Answering Systems

• James Pustejovsky
• September 30, 2002

2
Question Answering SystemsThe Good News

• Retrieval of structured and unstructured data
• Querying over Names and Locations
• 75 of all questions can be answered

3
What QA Systems Cant Do WellAnswer Questions
about events and their ordering

• Is Gates currently CEO of Microsoft?
• Did the Enron merger with Dynegy take place?
• How long did the hostage situation in Beirut
  last?
• Were there any meetings between the terrorist
  hijackers and Iraq before the WTC event?
• Who was Secretary of Defense during the Gulf War?

4
Workshop Goals

• TimeML Define and Design a Metadata Standard for
  Markup of events, their temporal anchoring, and
  how they are related to each other in News
  articles.
• TIMEBANK Given the specification of TimeML,
  create a gold standard corpus of 300 articles
  marked up for temporal expressions, events, and
  basic temporal relations.

5
TERQAS Participants

• James Pustejovsky, Brandeis U.
• Rob Gaizauskas, Sheffield U.
• Graham Katz, Osnabruck U.
• Bob Ingria, LingoMotors
• José Castaño, Brandeis U.
• Inderjeet Mani, MITRE
• Antonio Sanfilippo, SRA
• Dragomir Radev, U. Michigan
• Patrick Hanks, Brandeis U.
• Marc Verhagen, Brandeis U.
• Beth Sundheim, SPAWAR
• Andrea Setzer, Sheffield U.

• Jerry Hobbs, SRI, ISI
• Bran Boguraev, IBM
• Andy Latto, Metacarta
• John Frank, Metacarta
• Lisa Ferro, MITRE
• Marcia Lazo, MITRE
• Roser Saurí, Brandeis U.
• Anna Rumshisky, Brandeis
• David Day, MITRE
• Luc Belanger, U. Montreal
• Harry Wu, U. Michigan
• Andrew See, Brandeis U.

Supported by
6
TERQAS2002 Workshop Schedule

• January 30-31 Kick-off Meeting Setting Agenda
• March 11-15 Corpus Selection, Query Studies,
  TimeML
• April 22-26 TimeML Specification, Corpus Work
• May 8-15 Annotation Fest
• June 10-20 Algorithm Specification, Annotation
• July 15-22 Wrap-up and Evaluation
• Aug-Sept 30 Prepare Final Report

time2002.org
7
TERQAS Accomplishments

• Creation of a robust markup language for temporal
  expressions, event expressions, and the anchoring
  and ordering between them (TimeML 1.0)
• Creation of a Gold standard annotated against
  this language (TIMEBANK)
• Creation of Algorithms for recognizing
• Temporal Expressions
• Event Expressions
• Times associated with Events
• Ordering between Events and Times
• Development of a Text Segmented Closure Algorithm

8

• Creation of a Semi-graphical Annotation Tool
• Creation of Guidelines for Annotation
• Query Database Creation Tool and Guidelines
• Scoring and Inter-annotator Evaluation Tool
• Standards Impact
• Integration with ISO
• Adopted by European and Japanese Projects
• Dissemination of Knowledge
• CSLI /U. Chicago Press Book on TimeML
• Special Issue of Journal on TERQAS
• AAAI Spring Symposium 2003
• ESSLLI 2003 Course on TimeML

9
What Does it Do?

• Recognizes events in texts
• Recognizes temporal expressions in texts
• Time stamps the events
• Converts the temporal expressions to ISO std.
• Puts the events in the right order
• Infers the temporal orderings not mentioned

10
Recognizing Events and Time in Text

• AP-NR-08-16-90 2041EDT
• The Persian Gulf showdown between Iraq and
  the United States took a more personal turn
  Thursday when Iraq's Saddam Hussein called
  President Bush a liar and said the outbreak of
  holy war could bring thousands of Americans home
  in coffins.
• Bush, commenting on the two-week-old gulf
  crisis from his vacation home in Maine, said he
  saw little reason to be optimistic about a
  settlement of the dispute, which stems from
  Iraq's invasion of oil-wealthy Kuwait and its
  subsequent military buildup on the border of
  Saudi Arabia.
• After a two-hour meeting at his
  Kennebunkport home with King Hussein of Jordan,
  Bush said, I did not come away with any feeling
  of hope'' that Iraq would withdraw its army from
  Kuwait.
• Bush also said Thursday that King Hussein
  assured him Jordan would close the last remaining
  free port to most Iraqi trade as the economic
  embargo on materials to Iraq continued unabated.
• Foodstuffs are among the goods being blocked
  from entry Iraq imports about three-quarters of
  its food.
• Pentagon sources in Washington meanwhile said
  the Bush administration plans to deploy 45,000
  Marines to the region to back up the thousands of
  Army, Navy and Air Force troops already in place
  in the gulf and the Saudi desert.
• At a news conference, Secretary of State
  James A. Baker III said Jordan is seeking some
  guidance'' about a provision in the U.N.-backed
  trade embargo that allows food for humanitarian
  purposes.
• Worries however grew about the safety of
  Americans and other Westerners trapped in Kuwait.
  Iraqi military authorities ordered all Americans
  and Britons in Kuwait to assemble at a hotel,
  officials said.
• Very few'' of the 2,500 Americans in
  occupied Kuwait complied with the order, a senior
  U.S. official told The Associated Press.
• Iraq said the roundup was to protect them
  from unspecified threats British Foreign Office
  minister William Waldegrave called the order
  grave and sinister.''
• What we fear is that they will be
  interned somewhere, most likely in Iraq,''
  Waldegrave said. What we fear is that they
  will be interned somewhere, most likely in
  Iraq,'' Waldegrave said.
• A total of about 3,000 Americans, 3,000
  Britons and more than 450 Japanese are in Iraq
  and Kuwait. Overall, more than 2 million
  foreigners are in both countries. Iraq has
  called them restrictees.''
• In addition to the estimated 45,000 Marines
  to ultimately be part of Operation Desert Shield,
  Stealth fighter planes and the aircraft carrier
  John F. Kennedy are also headed to Saudi Arabia
  to protect it from Iraqi expansionism.
• In Washington, Pentagon spokesman Pete
  Williams said Iraq has continued to increase its
  armed forces in Kuwait and they now number about
  160,000.

Saddam has been under international quarantine
since his Aug. 2 power-grab, or what he calls an
eternal merger'' with Kuwait. In a long
verbal attack read on Iraqi television Thursday,
Saddam repeatedly called Bush a liar'' and said
a shooting war could produce body bags courtesy
of Baghdad. We continue to pray and pray hard
to God so that there will be no confrontation
whereby you will receive thousands of Americans
wrapped in sad coffins after you had pushed them
into a dark tunnel,'' Saddam said. He called
U.S. soldiers massing in Saudi Arabia the real
occupiers in the Persian Gulf. Replied State
Department deputy spokesman Richard Boucher, We
have n't really analyzed the statement in detail
but it appears to be just another example of his
outlandish rhetoric and his attempts to distort
the truth. We believe that his words cannot
distract the world from the facts of Iraqi
aggression.'' An international land, sea and
air force has mobilized since Iraq's invasion,
which was sparked by disputes over oil, land and
repayment of war loans. In the largest U.S.
military operation since Vietnam, an estimated
20,000 American GIs have already massed to
defend Saudi Arabia. We do n't just
arrive,'' said four-star Gen. John Dailey,
assistant commandant of the U.S. Marine Corps.
We're there to stay for a fairly lengthy
period.'' Egypt, Syria, Morocco and
Bangladesh also committed ground troops, to a
much lesser degree. The U.S. Navy has 27
ships in the maritime barricade of Iraq. They
are aided by Britain, West Germany, Australia,
Canada, the Netherlands and Belgium. Bush was
expected to authorize naval commanders to use
the minimum force necessary'' to interdict
shipments to and from Iraq, a U.S. official said.
That could include firing across the bow to
halt a ship. In the air, U.S. Air Force
fliers say they have engaged in a little cat
and mouse'' with Iraqi warplanes, which have
retreated when weapons radar locks onto them.
They do n't want to play with us,'' one U.S.
crew chief said. In Kuwait, the Iraqis have
rimmed the capital city with an air-defense
system, according to a U.S. official who spoke on
the condition of anonymity. He declined to say if
the weapons included missiles, but the Iraqis
have them in their arsenal. The Iraqis also
possess chemical weapons. The combined
operations are designed to isolate and strangle
Iraq until it retreats from Kuwait. The
quarantine hopes to staunch the flow of Iraqi
oil, which is Iraq's economic lifeblood,
and clamp down on food and supplies going in.
Iraq now controls 20 percent of the world's oil
reserves with its conquest of Kuwait. Only Saudi
Arabia has more oil reserves. (...)
11
Complete Time-Event Graph
12
Graph of Major Events
13
User Friendly Graph
2 weeks
1990-8-02
1990-8-16
crisis
military buildup
Iraqs invasion of Kuwait
meeting Bush-Hussein (Jordan)
disputes
international forces mobilized
international forces mobilized
2 hours
international quarantine
14
Algorithm Overview

• Pre-processing
• Time and Event Recognition modules
• Linking of Events with Times
• Order Events
• Resolve unknown Times
• Infer hidden relations between events

15
Time-Event Closure

• Reading between the Lines
• Infers temporal ordering of events in a text that
  are not explicitly given
• Infers events in a text not explicitly given

16
Israel will ask the United States to delay a
military strike against Iraq until the Jewish
state is fully prepared for a possible Iraqi
attack. Israel will ltEVENT eid"e1"
class"I_ACTION"gt ask lt/EVENTgt the United States
to ltEVENT eid"e2" class"I_ACTION"gt
delay lt/EVENTgt a military ltEVENT eid"e3"
class"OCCURRENCE"gt strike lt/EVENTgt against
Iraq until the Jewish state is fully ltEVENT
eid"e4" class"I_STATE"gt prepared lt/EVENTgt for a
possible Iraqi ltEVENT eid"e5" class"OCCURRENCE"gt
attack lt/EVENTgt
17
Deliverables

• TimeML
• Markup language for time and events
• TIMEBANK
• Gold standard annotated against this language
• Time/Event Algorithms
• For recognizing events and times in text
• Closure Algorithm
• For inferring hidden temporal relations between
  events
• Dissemination of Knowledge
• Book, journal articles, symposium, and courses on
  TimeML

18
www.time2002.org
19
More slides
20
How TimeML Differs from Previous Markups

• Extends TIMEX2 annotation
• Temporal Functions three years ago
• Anchors to events and other temporal expressions
  
• Identifies signals determining interpretation of
  temporal expressions
• Temporal Prepositions for, during, on, at
• Temporal Connectives before, after, while.
• Identifies event expressions
• tensed verbs has left, was captured, will
  resign
• stative adjectives sunken, stalled, on board
• event nominals merger, Military Operation, Gulf
  War
• Creates dependencies between events and times
• Anchoring John left on Monday.
• Orderings The party happened after midnight.
• Embedding John said Mary left.

21
TimeML 1.0

• 1.0 Spec Released July 20, 2002
• TimeML1.0.html
• 1.0 Guideline Released July 21, 2002
• TimeMLGuideline1.0.pdf
• DTD created
• TimeML.dtd
• Schema created
• TimeML.xsd

time2002.org
22
Creating TIMEBANK

• Corpus Collection (corpus tools)
• Extensions to MITREs Alembic Workbench
• Creation of Additional Annotation Tools
• Alembic ? TimeML Converter
• Annotation Visualization Tools
• Preprocessing Algorithms

23
TIMEX3 Parser Objects (T3PO)

• Extends TIDES TIMEX2 annotation
• Broader Coverage of temporal expressions
• Larger lexicon of temporal triggers
• Delays Computation of Temporal Math
• Annotation with Temporal Functions
• Import Hobbs Semantic Web Temporal System
• Distinct Cascaded Processes
• TIMEX3 and signal recognizer
• Event Predicate recognizer
• LINK creation transducer.