Dialogue matters

1
Dialogue matters

• Spoken Dialogue Systems

2
Challenges in Open-Domain Question Answering

• Johan Bos
• University of Rome "La Sapienza
• Dipartimento di Informatica

3
Question Answering

• What is open domain QA?
• Questions in natural language, not queries!
• Answers, not documents!

Q
Where did Olof Palme die?
A
In Stockholm.
4
Why do we need QA?

• Information overload problem
• Accessing information using traditional methods
  such as IR and IE are limited
• QA increasingly important because
• Size of available information grows
• There is duplicate information
• There is false information
• More and more computer illiterates accessing
  electronically stored information

5
Information Avalanche

• Available information is growing
• 1999 250MB pp for each person on earth
• 2002 800MB pp for each person on earth
• People want specific information
• source M.de Rijke
  2005

6
Information Pinpointing

• Information required Average number of car
  accidents per year in Sweden.
• Two ways of getting this information
• Ask Google or a similar search engine (good
  luck!)
• Ask a QA system the questionWhats the rate of
  car accidents in Sweden?

7
People ask Questions
source M. de Rijke 2005
8
Natural search

• Google gets about 200 million searches per day
• That would mean about 2 million natural language
  searches

9
Overview of this talk

• Open domain QA
• Why QA is hard
• Three methods
• Typical architecture of a QA system
• Scaling up
• Unsolved problems
• Integration of dialogue

10
Open-Domain Question Answering

• TREC
• Acquint corpus
• Factoid and Definition questions
• TAC
• Blog06 corpus
• List and opinion questions

11
State of the art

• QA is hard
• Best systems use extensive answer typing and
  semantics
• Systems complex

12
Why QA is hard (1/5)
Q
When was NATO established?
A
NATO launched its first attack against Yugoslavia
on March 24. WordNetlaunchestablish
13
Why QA is hard (2/5)
Q
Bing Crosby. What was his profession?
A
Crosby sang her first song, Starlight. She
scribbled notes for the song on the back of a
menu at a New York jazz bar in 1931. . WordNet
bar hyponym of profession
14
Why QA is hard (3/5)
Where did Ricky Williams, American football
player, grow up?
Q
A
Texas running back Ricky Williams is from
California. Those who follow the Longhorns don't
like Williams any less because he didn't grow up
in Texas.
15
Why QA is hard (4/5)

• Variation in questions
• What is the population of Rome?
• How many people live in Rome?
• Whats the size of Rome?
• How many inhabitants does Rome have?
• Variation in answers
• is estimated at 2.5 million residents
• current population of Rome is 2817000
• Rome housed over 1 million inhabitants

16
Why QA is hard (5/5)

• Answers can be spread across different documents
• Which European countries produce wine?Document
  A contains information about Italy, and document
  B about France
• What does Bill Clintons wife do for a
  living?Document A explains that Bill Clintons
  wife is Hillary Clinton, and Document B tells us
  that shes a politician

17
Methods in QA

• Viewed from a distance, roughly three different
  methods can be distinguished
• Guessing
• Googling
• GOFAI

18
Guessing

• Limit the amount of deep NLP tools
• Simple pattern matching
• Use shallow features, such as frequency, to guess
  an answer

19
Guessing example
Q
What country is Berlin in?
Ties between Iran and Germany strained after a
Berlin court verdict ruled on April 10 that
Iran's top leaders were behind the assassination
of four exiled Kurdish opposition leaders in a
Berlin restaurant.
A
20
Guessing example
Q
What country is Berlin in?
Ties between Iran and Germany strained after a
Berlin court verdict ruled on April 10 that
Iran's top leaders were behind the assassination
of four exiled Kurdish opposition leaders in a
Berlin restaurant.
A
21
Another guessing example
Q
Port Arthur Massacre.What was the killer's
nationality?
A nation asks why, the portrait of a lone gunman,
Martin Bryant, should have no reason to be a
killer. The man arrested after the Port Arthur
massacre should be a wealthy man, only four years
ago he inherited more than 500,000 Australian
dollars 375,000 US dollars from one of the heirs
of George Adams's great Tattersalls fortune.
A
22
Googling

• Use other, larger, corpora to find the answer.
  i.e., the internet.
• Use Google tricks.

23
Googling example
Q
What is the state bird of Alaska?
Google search "The state bird of Alaska is the
"
A
24
Googling example
Q
What is the state bird of Alaska?
Google results The state bird of Alaska is the
ptarmigan. The state bird of Alaska is the willow
ptarmigan.
A
25
Googling example
Q
What is the state bird of Alaska?
Google results The state bird of Alaska is the
ptarmigan. The state bird of Alaska is the willow
ptarmigan. The state bird of Alaska is the
mosquito. The state bird of Alaska is the
Mosquito.
A
26
GOFAI

• Using background knowledge to give support for an
  answer
• Use logical inference when possible

27
GOFAI example
Q
Where did Olof Palme die?
A
Stockholm.
28
GOFAI example
Q
Where did Olof Palme die?
A
. given clause 13 (wt2) 69 hyper,47,16
vehicle(c3). given clause 14 (wt2) 71
hyper,50,16 vehicle(c4). given clause 15
(wt2) 74 hyper,52,18 building(c5). given
clause 16 (wt3) 51 have(vincent,c4). given
clause 17 (wt2) 76 hyper,55,18
building(c6). given clause 18 (wt2) 78
hyper,61,16 vehicle(c6). given clause 19
(wt2) 80 hyper,65,10 organism(c1). given
clause 20 (wt2) 84 hyper,67,10
organism(c2). given clause 21 (wt3) 53
die(palme,c5). given clause 22 (wt2) 86
hyper,69,15 instrument(c3). given clause 23
(wt2) 88 hyper,71,15 instrument(c4). given
clause 24 (wt2) 90 hyper,74,7
artifact(c5). given clause 25 (wt2) 94
hyper,76,7 artifact(c6). given clause 26
(wt3) 56 c7c6. given clause 27 (wt2) 96
hyper,78,15 stockholm(c6). ----- EMPTY
CLAUSE at 0.01 sec ---- 113 hyper,96,24,76
F.
29
GOFAI example
Q
Where did Olof Palme die?
A
1. You are looking for location. 2. I know that
Stockholm is a city. 3. Every city is a
location. 4. If x is shot to death then x
died. 5. I found the following evidence in
document APW20000227.0124 "In 1986,
Swedish Prime Minister Olof Palme was
shot to death in central Stockholm. 6. Hence
Stockholm is the answer.
30
Overview of this talk

• Open domain QA
• Why QA is hard
• Three methods
• Typical architecture of a QA system
• Scaling up
• Unsolved problems
• Integration of dialogue

31
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
32
Parsing

• Providing a syntactic analysis of the question
• Includes tokenisation and NER
• Based on CCG, using CC tools
• Issues
• Hard, because no treebanks available for
  questions
• Robustness

33
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
34
Boxing

• Providing a semantic analysis on the basis of the
  CCG derivation
• DRT, using Boxer
• Issues
• Pronoun resolution
• Ellipsis

35
Boxer

• Works on output of a CCG parser
• Input CCG syntax trees
• Output DRT semantic representations
• The CC Parser for CCG
• Statistical, robust, wide-coverage
• Clark Curran (ACL 2004)
• Grammar derived from CCGbank
• 409 different categories
• Hockenmaier Steedman (ACL 2002)

36
Example derivation
37
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
38
Knowledge

• Manually constructed
• WordNet
• CYC
• FrameNet
• NomLex
• CIA Factbook
• Automatically constructed
• Instances and Hyponyms e.g. Hearst
• Paraphrases e.g. Lin Pantel

39
Hyponyms (X such-as Y)
TREC 20.2 (Concorde) What airlines have Concorde
in their fleets?

• WordNet has no instances of airlines.

40
Hyponyms (X such as Y)
TREC 20.2 (Concorde) What airlines have Concorde
in their fleets?

• Search for X such as Y patterns in large
  corpora.
• airlines such as Continental and United now fly

41
Hyponyms (X such as Y)
TREC 20.2 (Concorde) What airlines have Concorde
in their fleets?

• Knowledge (Acquaint corpus)Air Asia, Air
  Canada, Air France, Air Mandalay, Air Zimbabwe,
  Alaska, Aloha, American Airlines, Angel Airlines,
  Ansett, Asiana, Bangkok Airways, Belgian Carrier
  Sabena, British Airways, Canadian, Cathay
  Pacific, China Eastern Airlines, China Xinhua
  Airlines, Continental, Garuda, Japan Airlines,
  Korean Air, Lai, Lao Aviation, Lufthansa,
  Malaysia Airlines, Maylasian Airlines, Midway,
  Northwest, Orient Thai Airlines, Qantas, Seage
  Air, Shanghai Airlines, Singapore Airlines,
  Skymark Airlines Co., South Africa, Swiss Air, US
  Airways, United, Virgin, Yangon Airways

42
Paraphrases
TREC 4.2 (James Dean) When did James Dean
die? ---- APW19990929.0165 In 1955, actor James
Dean was killed in a two-car collision near
Cholame, Calif.

• Knowledge ?x?t(?e(kill(e)theme(e,x)in(e,t))
  ? ?e' (die(e')agent(e',x)in(e',t)))

43
World Knowledge
TREC 14.4 (Horus) Who was his father? ---- XIE1999
0713.004 It also hosted statues of Amons
wife, Mut, the goddess Isis, her husband, Osiris,
and their son Horus.

• Knowledge?x(husband(x)?male(x))
  ?x?y(son(x,y)male(y) ? father(y,x))

44
Automatically harvested knowledge

• Good at simple relations
• hyponyms, instances, synonyms
• Not so good at paraphrases
• DIRT paraphrases Lin and Pantel
• No improvement on QA and RTE
• Problems
• Typically low precision, high recall
• No Word Sense Disambiguation
• No common sense knowledge

45
Knowledge in QA

• We need knowledge for QA
• We need a lot of knowledge for QA
• Current knowledge resources are not sufficient
• Automatically extracted knowledge
• needs to be accurate
• Word sense disambiguated
• Deal with more complex relations

46
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
47
Answer Typing

• Providing information on the expected answer type
• Syntactic type of answer
• Semantic type sortal ontology
• Answer cardinality
• Issues
• Ambiguities
• Vagueness
• Classification problems

48
Answer typing examples

• Simple rules
• Who ? person
• Where ? geo-location
• When ? calendar time
• How many ? number
• often fail
• Who launched the iPod?
• Where in the human body is the liver?
• When is it time to go to bed?

49
Answer typing examples

• AmbiguitiesHow long ? distance or duration
• Vague wh-wordsWhat do pinguins eat? What is the
  length of a football pitch?
• Ontology gapsWhich alien race featured in Star
  Trek?What is the cultural capital of Italy?

50
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
51
Document retrieval

• Document selection
• Resize documents
• A document can be two sentence sliding window
• Index documents
• Semantic indexing
• Preparse documents
• Query
• Expansion, using knowledge
• Multiple queries

52
Query example
TREC 44.6 (Sacajawea) How much is the Sacajawea
coin worth?

• Query sacajawea Returns only five documents
• New query sacajawea OR sagajaweaReturns two
  hundred documents

53
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
54
Answer extraction

• Analyse documents
• Match answer with question
• DRS for question
• DRS for each possible document
• Score for amount of overlap
• Inference or matching
• Use knowledge

55
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
56
Answer selection

• Rank answer
• Group duplicates
• Syntactically or semantically equivalent
• Sort on frequency
• How specific should an answer be?
• Semantic relations between answers
• Hyponyms, synonyms
• Answer modelling Dalmas 2007
• Answer cardinality

57
Answer selection example 1

• Where did Franz Kafka die?
• In his bed
• In a sanatorium
• In Kierling
• Near Vienna
• In Austria

58
Answer selection example 2

• Where is 3M based?
• In Maplewood
• In Maplewood, Minn.
• In Minnesota
• In the U.S.
• In Maplewood, Minn., USA

59
Architecture of PRONTO
question
parsing
answer reranking
answer
ccg
boxing
WordNet NomLex
answer selection
drs
knowledge
answer extraction
Indri
answer typing
query
Indexed Documents
60
Answer reranking

• Using additional features to rerank selected
  answers
• Validation
• Inference check
• Googling
• Tie breaking

61
Overview of this talk

• Open domain QA
• Why QA is hard
• Three methods
• Typical architecture of a QA system
• Scaling up
• Unsolved problems
• Integration of dialogue

62
Unsolved problems

• Yes/No-questionsCan you give me the names of Mr
  Jones?Are debit cards expensive?Are growing
  stock piles an indication of trouble in a company?

63
Unsolved problems

• Conditional questionsCan I get a mortgage if
  the house needs repairs?
• Why-questions
• How-questions

64
Unsolved problems

• Comparison-questionsIs X better than Y?Of the
  two governments involved over Kinmen, which has
  air superiority?What is the difference between
  X and Y?

65
Unsolved problems

• Questions with negationWhich European countries
  do not charge VAT tax on gold?Are there
  hearing aids that do not have to go inside your
  ear?My computer is set up to reject cookies.
  What will I not be able to do on your website?

66
Unsolved problems

• Multiple questionsWhere and when was Franz
  Kafka born?

67
Unsolved problems

• Questions asking for opinionsWhat do people
  like about IKEA?What do young people like about
  IKEA?What do women like about IKEA?

68
The scaling up challenge

• Why not add dialogue to QA?
• Clarification dialogue will solve all the
  problems
• Whether this is a good idea or not depends on the
  application
• Techno push

69
Example 1

• U Is there a UniCredit bank in London?
• S Do you mean a river bank or a financial
  institute?
• U Financial institute please.
• S OK, please wait.
• U Sure.
• S Sorry. There are no UniCredit banks in
  London.

70
Example 2

• U When was William Crozier born?
• S There are four William Croziers. Do you
  mean the artillerist, the Scottish artist,
  the Irish artist, or the cricketer?
• U I actually mean the Irish artist.
• S He was born in 1930.

71
Example 3

• U When was William Crozier born?
• S William Crozier (artillerist) was born in
  1855.
• William Crozier (Scottish artist) was born in
  1893.
• William Crozier (Irish artist) was born in
  1930.
• William Crozier (cricketer) was born in 1873.

72
Open domain QA and dialogue

• Does not always seems to be a good idea
• Killer app might be mobile devices without the
  usual input and output modalities
• Speech