Title: Generation-Heavy Hybrid Machine Translation Nizar Habash Postdoctoral Researcher Center for Computational Learning Systems Columbia University
1Generation-Heavy Hybrid Machine Translation
Nizar Habash Postdoctoral ResearcherCenter for
Computational Learning SystemsColumbia University
- Columbia University
- NLP Colloquium
October 28, 2004
2The IntuitionGeneration-Heavy Machine Translation
Español ????
Dictionary
English
3IntroductionResearch Contributions
- A general reusable and extensible Machine
Translation (MT) model that transcends the need
for large amounts of deep symmetric knowledge - Development of reusable large-scale resources
for English - A large-scale Spanish-English MT system
Matador Matador is more robust across genre and
produce more grammatical output than simple
statistical or symbolic techniques
4Roadmap
- Introduction
- Generation-Heavy Machine Translation
- Evaluation
- Conclusion
- Future Work
5IntroductionMT Pyramid
Source meaning
Target meaning
Source syntax
Target syntax
Source word
Target word
Analysis
Generation
6IntroductionMT Pyramid
Source meaning
Target meaning
Source syntax
Target syntax
Source word
Target word
Analysis
Generation
7IntroductionMT Pyramid
Source meaning
Target meaning
Transfer
Source syntax
Target syntax
Gisting
Source word
Target word
8IntroductionWhy gisting is not enough
Sobre la base de dichas experiencias se
estableció en 1988 una metodología.
Envelope her basis out speak experiences them
settle at 1988 one methodology.
On the basis of these experiences, a methodology
was arrived at in 1988.
9IntroductionTranslation Divergences
- 35 of sentences in TREC El Norte Corpus (Dorr et
al 2002) - Divergence Types
- Categorial (X tener hambre ? X be hungry)
- Conflational (X dar puñaladas a Z ? X stab Z)
- Structural (X entrar en Y ? X enter Y)
- Head Swapping (X cruzar Y nadando ? X swim across
Y) - Thematic (X gustar a Y ? Y like X)
10Roadmap
- Introduction
- Generation-Heavy Machine Translation
- Evaluation
- Conclusion
- Future Work
11Generation-Heavy Hybrid Machine Translation
- Problem asymmetric resources
- High quality, broad coverage, semantic resources
for target language - Low quality resources for source language
- Low quality (many-to-many) translation lexicon
- Thesis we can approximate interlingual MT
without the use of symmetric interlingual
resources
12Relevant Background Work
- Hybrid Natural Language Generation
- Constrained Overgeneration ? Statistical Ranking
- Nitrogen (Langkilde and Knight 1998), Halogen
(Langkilde 2002) - FERGUS (Rambow and Bangalore 2000)
- Lexical Conceptual Structure (LCS) based MT
- (Jackendoff 1983), (Dorr 1993)
13LCS-based MTExample
(Dorr, 1993)
14Generation-Heavy HybridMachine Translation
Generation
Analysis
Translation
15MatadorSpanish-English GHMT
Spanish
Analysis
Translation
English
16GHMTAnalysis
- Source language syntactic dependency
- Example Yo le di puñaladas a Juan.
- Features of representation
- Approximation of predicate-argument structure
- Long-distance dependencies
17GHMTTranslation
- Lexical transfer but NO structural change
- Translation Lexicon
- (tener V) ? ((have V) (own V) (possess V) (be
V))(deber V) ? ((owe V) (should AUX) (must
AUX))(soler V) ? ((tend V) (usually AV))
?
18GHMTThematic Linking
- Syntactic Dependency ? Thematic Dependency
- Which divergence
19GHMTThematic Linking Resources
- Word Class Lexicon
- NUMBER "V.13.1.a.ii" NAME "Give - No Exchange
POS V - THETA_ROLES (((ag obl) (th obl) (goal obl to))
- ((ag obl) (goal obl) (th obl)))
- LCS_PRIMS (cause go)
- WORDS (feed give pass pay peddle refund render
repay serve)) - Syntactic-Thematic Linking Map
- (subj ? ag instr th exp loc src goal perc
mod-poss poss) - (obj2 ? goal src th perc ben)
- (across ? goal loc)
- (in ? loc mod-poss perc goal poss prop)
- (to ? prop goal ben info th exp perc pred loc
time)
20GHMTThematic Linking
- Syntactic Dependency ? Thematic Dependency
((ADMINISTER V.13.2 ((AG OBL) (TH OBL) (GOAL OPT
TO))) (CONFER V.37.6.b ((EXP OBL))) (DELIVER
V.11.1 ((AG OBL) (GOAL OBL) (TH OBL) (SRC OPT
FROM))) (EXTEND V.47.1 ((TH OBL) (MOD-LOC OPT .
T))) (EXTEND V.13.3 ((AG OBL) (TH OBL) (GOAL OPT
TO))) (EXTEND V.13.3 ((AG OBL) (GOAL OBL) (TH
OBL))) (EXTEND V.13.2 ((AG OBL) (TH OBL) (GOAL
OPT TO))) (GIVE V.13.1.a.ii ((AG OBL) (TH OBL)
(GOAL OBL TO))) (GIVE V.13.1.a.ii ((AG OBL)
(GOAL OBL) (TH OBL))) (GRANT V.29.5.e ((AG OBL)
(INFO OBL THAT))) (GRANT V.29.5.d ((AG OBL) (TH
OBL) (PROP OBL TO))) (GRANT V.13.3 ((AG OBL) (TH
OBL) (GOAL OPT TO))) (GRANT V.13.3 ((AG OBL)
(GOAL OBL) (TH OBL))) (HAND V.11.1 ((AG OBL) (TH
OBL) (GOAL OPT TO) (SRC OPT FROM))) (HAND V.11.1
((AG OBL) (GOAL OBL) (TH OBL) (SRC OPT FROM)))
(LAND V.9.10 ((AG OBL) (TH OBL))) (RENDER
V.13.1.a.ii ((AG OBL) (TH OBL) (GOAL OBL TO)))
(RENDER V.13.1.a.ii ((AG OBL) (GOAL OBL) (TH
OBL))) (RENDER V.10.6.a ((AG OBL) (TH OBL)
(MOD-POSS OPT OF))) (RENDER V.10.6.a.LOCATIVE
((AG OPT) (SRC OBL) (TH OPT OF))))
21GHMTThematic Linking
- Syntactic Dependency ? Thematic Dependency
((ADMINISTER V.13.2 ((AG OBL) (TH OBL) (GOAL OPT
TO))) (CONFER V.37.6.b ((EXP OBL))) (DELIVER
V.11.1 ((AG OBL) (GOAL OBL) (TH OBL) (SRC OPT
FROM))) (EXTEND V.47.1 ((TH OBL) (MOD-LOC OPT .
T))) (EXTEND V.13.3 ((AG OBL) (TH OBL) (GOAL OPT
TO))) (EXTEND V.13.3 ((AG OBL) (GOAL OBL) (TH
OBL))) (EXTEND V.13.2 ((AG OBL) (TH OBL) (GOAL
OPT TO))) (GIVE V.13.1.a.ii ((AG OBL) (TH OBL)
(GOAL OBL TO))) (GIVE V.13.1.a.ii ((AG OBL)
(GOAL OBL) (TH OBL))) (GRANT V.29.5.e ((AG OBL)
(INFO OBL THAT))) (GRANT V.29.5.d ((AG OBL) (TH
OBL) (PROP OBL TO))) (GRANT V.13.3 ((AG OBL) (TH
OBL) (GOAL OPT TO))) (GRANT V.13.3 ((AG OBL)
(GOAL OBL) (TH OBL))) (HAND V.11.1 ((AG OBL) (TH
OBL) (GOAL OPT TO) (SRC OPT FROM))) (HAND V.11.1
((AG OBL) (GOAL OBL) (TH OBL) (SRC OPT FROM)))
(LAND V.9.10 ((AG OBL) (TH OBL))) (RENDER
V.13.1.a.ii ((AG OBL) (TH OBL) (GOAL OBL TO)))
(RENDER V.13.1.a.ii ((AG OBL) (GOAL OBL) (TH
OBL))) (RENDER V.10.6.a ((AG OBL) (TH OBL)
(MOD-POSS OPT OF))) (RENDER V.10.6.a.LOCATIVE
((AG OPT) (SRC OBL) (TH OPT OF))))
22GHMTThematic Linking
- Syntactic Dependency ? Thematic Dependency
23Interlingua Approximationthrough Expansion
Operations
RelationConflation / Inflation
Relation Variation
??
??
24Interlingua Approximation2nd Degree Expansion
cross
go
mod
mod
subj
subj
obj
across
John
swimming
John
river
river
swimming
Relation Inflation
swim
across
subj
John
river
Node Conflation
25GHMTStructural Expansion
,
26GHMTStructural Expansion
- Conflation and Inflation
- Structural Expansion Resources
- Word Class Lexicon
- NUMBER "V.42.2" NAME Poison Verbs POS V
- THETA_ROLES (((ag obl)(goal obl)))
- LCS_PRIMS (cause go)
- WORDS (crucify electrocute garrotte hang knife
poison shoot smother stab strangle) - Categorial Variation Database (Habash and Dorr
2003) - (V (hunger) N (hunger hungriness) AJ
(hungry)) - (V (validate) N (validation validity) AJ
(valid)) - (V (cross) N (crossing cross) P (across))
- (V (stab) N (stab))
27GHMTStructural Expansion
28GHMTStructural Expansion
29GHMTStructural Expansion
,
30GHMT Syntactic Assignment
- Thematic ? Syntactic Mapping
31GHMT Structural N-gram Pruning
- Statistical lexical selection
32GHMTTarget Statistical Resources
- Structural N-gram Model
- Long-distance
- Lexemes
- Surface N-gram Model
- Local
- Surface-forms
33GHMTLinearization Ranking
- Oxygen Linearization (Habash 2000)
- Halogen Statistical Ranking (Langkilde 2002)
- --------------------------------------------------
------- - I stabbed John . -1.670270
- I gave a stab at John . -2.175831
- I gave the stab at John . -3.969686
- I gave an stab at John . -4.489933
- I gave a stab by John . -4.803054
- I gave a stab to John . -5.045810
- I gave a stab into John . -5.810673
- I gave a stab through John . -5.836419
- I gave a knife wound by John . -6.041891
34Roadmap
- Introduction
- Generation-Heavy Machine Translation
- Evaluation
- Overall Evaluation
- Component Evaluation
- Conclusion
- Future Work
35Overall EvaluationSystems
Gisting(GIST) Systran(SYST) IBM Model 4(IBM4) Matador(MTDR)
Approach SymbolicWord-based SymbolicTransfer-based StatisticalWord-based HybridGeneration-Heavy
TranslationModel 400Ksurface-lexeme pairs 120Klexeme-lexemepairsandlarge transferlexicon Model 4Giza Trained50K UN sentence pairs 50Klexeme-lexemepairs
LanguageModel UnigramsBrown Corpus1M words 120Klexeme-lexemepairsandlarge transferlexicon Bigrams3M words (UN) Bigrams3M words (UN)andStructural Bigrams1.5M words (UN)
DevelopmentTime 1 person-month Hundreds ofperson-years 1 person-month 1 person-year
(Brown et al 1990)(Al-Onaizan et al
1999)(Germann and Marcu 2000)
(Resnik 1997)
36Overall EvaluationBleu Metric
- Bleu
- BiLingual Evaluation Understudy (Papineni et al
2001) - Modified n-gram precision with length penalty
- Quick, inexpensive and language independent
- Correlates highly with human evaluation
- Bias against synonyms and inflectional variations
37Overall EvaluationTest Sets
UN FBIS Bible
Genre United Nations documents News broadcast Religious
Spanish-EnglishSentence pairs 2,000 2,000 1,000
Sentence Length(words) 15.39 19.27 16.38
38Overall EvaluationResults
39Overall EvaluationResults
- Systran is overall best
- Gist is overall worst
- Matador is more robust than IBM4
- Matador is more grammatical than IBM4
- Matador has less information loss than IBM4
40Overall Evaluation Grammaticality
- Example
- SP Ademàs dijo que solamente una inyecciòn
masiva de capital extranjero ... - EN Further, he said that only a massive
injection of foreign capital ... - IBM4 further stated that only a massive
inyecciòn of capital abroad ... - MTDR Also he spoke only a massive injection of
foreign capital ... - Parsed all sentences (Spanish, English reference
and English output) - Can we find main verb?
- Pro Drop Restoration
41Overall Evaluation Grammaticality Verb
Determination
42Overall Evaluation Grammaticality Subject
Realization
43Overall Evaluation Loss of Information
- Example
- SP El daño causado al pueblo de Sudáfrica jamás
debe subestimarse. - EN The damage caused to the people of his
country should never be underestimated. - IBM4 the damage the people of south must
never underestimated . - MTDR Never the causado damage to the people of
South Africa should be underestimated.
Gisting(GIST) Systran(SYST) IBM Model 4(IBM4) Matador(MTDR)
Reference length 109 109 94 104
44Component Evaluation
- Conducted several component evaluations
- Parser
- 75 correct (labeled dependency links)
- Categorial Variation Database
- 81 Precision-Recall
- Structural Expansion
- Structural N-grams
45Component EvaluationStructural Expansion
- Insignificant increase in Bleu score
- 40 of divergences pragmatic
- LCS lexicon coverage issues
- Minimal handling of nominal divergences
- Over-expansion
- Además, destruyó totalmente sus cultivos de
subsistencia - EN It had totally destroyed Samoa's staple crops
... - MTDR Furthermore, it totaled their cultivations
of subsistence - SP Dicha adición se publica sólo en años
impares. - EN That addendum is issued in odd-numbered years
only. - MTDR concerned addendum is excluded in odd years.
46Component EvaluationStructural N-grams
- 60 speed-up with no effect on quality
47Roadmap
- Introduction
- Generation-Heavy Machine Translation
- Evaluation
- Conclusion
- Future Work
48ConclusionResearch Contributions
- A general reusable and extensible MT model that
transcends the need for large amounts of
symmetric knowledge - A systematic non-interlingual/non-transfer
framework for handling translation divergences - Extending the concept of symbolic overgeneration
to include conflation and head-swapping of
structural variations. - A model for language-independent
syntactic-to-thematic linking
49ConclusionResearch Contributions
- Development of reusable large-scale modules and
resources Exerge, Categorial Variation Database,
etc. - A large-scale Spanish-English GHMT
implementation - An evaluation of Matador against four models of
machine translation found it to be robust across
genre and to produce more grammatical output.
50Ongoing Work
- Retargetability to new languages
- Chinese, Arabic
- Extending system to use bi-texts
- Phrase dictionary
- Weighted translation pairs
- Generation-Heavy parsing
- Small dependency grammar for foreign language
- English structural n-grams to rank parses
- Extending system with new optional modules
- Cross-lingual headline generation
- DepTrimmer (work with Bonnie Dorr) extending
Trimmer (Dorr, et al. 2003) to dependency
representation
51Future Work
- Categorial Variation Database
- Improving word-cluster correctness
- Structural Expansion
- Extending to nominal divergences
- Improving thematic linking with a statistical
model - Structural N-grams
- Enriching with syntactic/thematic relations
52 53Overall EvaluationBleu Metric
- Test Sentence
- colorless green ideas sleep furiously
Gold Standard References all dull jade ideas
sleep irately drab emerald concepts sleep
furiously colorless immature thoughts nap angrily
54Overall EvaluationBleu Metric
- Test Sentence
- colorless green ideas sleep furiously
Gold Standard References all dull jade ideas
sleep irately drab emerald concepts sleep
furiously colorless immature thoughts nap angrily
Unigram precision 4/5
55Overall EvaluationBleu Metric
- Test Sentence
- colorless green ideas sleep furiously
- colorless green ideas sleep furiously
- colorless green ideas sleep furiously
- colorless green ideas sleep furiously
Gold Standard References all dull jade ideas
sleep irately drab emerald concepts sleep
furiously colorless immature thoughts nap angrily
Unigram precision 4 / 5 0.8 Bigram precision
2 / 4 0.5 Bleu Score (a1 a2 an)1/n
(0.8 ? 0.5)½ 0.6325 ? 63.25
56Overall Evaluation
- Investigating BLEUs bias towards inflectional
variants - SP Los programas de ajuste estructural se han
aplicado rigurosamente. - EN Structural adjustment programmes had been
rigorously implemented. - IBM4 structural adjustment programmes have been
applied strictly. - MTDR programmes of structural adjustment have
been added rigurosament.
57Overall Evaluation Inflectional Normalization