Automatic Translation of Human Languages

1
Automatic Translation of Human Languages

• Kevin Knight

USC/Information Sciences Institute USC/Computer
Science Department
2
Machine Translation (MT)
?????????????????????????????????????,????????????
???????,??????????
?
The U.S. island of Guam is maintaining a high
state of alert after the Guam airport and its
offices both received an e-mail from someone
calling himself the Saudi Arabian Osama bin Laden
and threatening a biological/chemical attack
against public places such as the airport.
3
Why People Get Into This Field

• Passion about understanding how human language
  works
• What makes one sequence of words grammatical, and
  another not?
• Interest in foreign languages
• Whats the difference between English and
  Chinese?
• Desire to change the world
• How will the world be different when the language
  barrier disappears?

4
Why Its Challenging

• Each word has tons of meanings
• Ill get a cup of coffee ?
• I didnt get that joke ?
• I get up at 8am ?
• I get nervous ?
• Yeah, I get around ?
• Each word has zillions of contexts
• Word order is very different

5
Why Its Challenging

• Output must be a grammatical, sensible,
  never-before-uttered sentence!
• Computers consume lots of human language
• Google, Yahoo, Altavista
• Speech recognizers
• More challenging to also produce human language
• What makes one sequence of words grammatical, and
  another not?

6
Recent Progress
2002
2003

• insistent Wednesday may recurred her trips to
  Libya tomorrow for flying
• Cairo 6-4 ( AFP ) - An official announced
  today in the Egyptian lines company for flying
  Tuesday is a company "insistent for flying" may
  resumed a consideration of a day Wednesday
  tomorrow her trips to Libya of Security Council
  decision trace international the imposed ban
  comment.

• Egyptair Has Tomorrow to Resume Its Flights to
  Libya
• Cairo 4-6 (AFP) - Said an official at the
  Egyptian Aviation Company today that the company
  egyptair may resume as of tomorrow, Wednesday its
  flights to Libya after the International Security
  Council resolution to the suspension of the
  embargo imposed on Libya.

7
2005
news broadcast
foreign language speech recognition
English translation
searchable archive
8
Statistical Machine Translation
Hmm, every time he sees banco, he either types
bank or bench but if he sees banco
de, he always types bank, never bench
Man, this is so boring.
Translated documents
9
Things are Consistently Improving
Annual evaluation of Arabic-to-English MT systems
Translation quality
70
60
50
40
30
20
Exceeded commercial-grade translation here.
10
2004
2005
2006
2002
2003
10
Progress Driven by Experiments!
Translation quality
35
30
25
20
USC/ISI Syntax-Based MT System. Chinese/English NI
ST 2002 Test Set
15
Mar 1
Apr 1
May 1
2005
11
Warren Weaver (1947)
ingcmpnqsnwf cv fpn owoktvcv hu ihgzsnwfv
rqcffnw cw owgcnwf kowazoanv ...
12
Warren Weaver (1947)
e e e e ingcmpnqsnwf cv fpn
owoktvcv e e e hu
ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv
...
13
Warren Weaver (1947)
e e e the ingcmpnqsnwf cv fpn
owoktvcv e e e hu
ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv
...
14
Warren Weaver (1947)
e he e the ingcmpnqsnwf cv fpn
owoktvcv e e e t hu
ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv
...
15
Warren Weaver (1947)
e he e of the ingcmpnqsnwf cv fpn
owoktvcv e e e t hu
ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv
...
16
Warren Weaver (1947)
e he e of the fof ingcmpnqsnwf cv fpn
owoktvcv e f o e o oe t hu
ihgzsnwfv rqcffnw cw owgcnwf ef kowazoanv
...
17
Warren Weaver (1947)
e he e of the ingcmpnqsnwf cv fpn owoktvcv
e e e t hu ihgzsnwfv
rqcffnw cw owgcnwf e kowazoanv ...
18
Warren Weaver (1947)
e he e is the sis ingcmpnqsnwf cv fpn
owoktvcv e s i e i ie t hu
ihgzsnwfv rqcffnw cw owgcnwf es kowazoanv
...
19
Warren Weaver (1947)
decipherment is the analysis ingcmpnqsnwf cv fpn
owoktvcv of documents written in ancient hu
ihgzsnwfv rqcffnw cw owgcnwf languages
... kowazoanv ...
20
Warren Weaver (1947)
When I look at an article in Russian, I say to
myself This is really written in English, but it
has been coded in some strange symbols. I will
now proceed to decode.
21
Spanish/English text
 
22
Spanish/English text
Translate Clients do not sell pharmaceuticals
in Europe.
 
23
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
24
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
25
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
26
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
???
27
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
28
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
29
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
30
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
???
31
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
32
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
process of elimination
33
Centauri/Arcturan Knight, 1997
Your assignment, translate this to Arcturan
farok crrrok hihok yorok clok kantok ok-yurp
cognate?
34
Centauri/Arcturan Knight, 1997
Your assignment, put these words in order
jjat, arrat, mat, bat, oloat, at-yurp
zero fertility
35
Bilingual Training Data
Millions of words (English side)
1m-20m words for many language pairs
(Data stripped of formatting, in sentence-pair
format, available from the Linguistic Data
Consortium at UPenn).
36
Sample Learning Curves
Swedish/English French/English German/English Finn
ish/English
BLEU score
of sentence pairs used in training
Experiments by Philipp Koehn
37
MT Evaluation

• Traditionally difficult because there is no
  right answer
• 20 human translators will translate the same
  sentence 20 different ways.

38
New Evaluation Metric (BLEU) (Papineni et al,
ACL-2002)
Reference (human) translation The U.S. island
of Guam is maintaining a high state of alert
after the Guam airport and its offices both
received an e-mail from someone calling himself
the Saudi Arabian Osama bin Laden and threatening
a biological/chemical attack against public
places such as the airport .

• N-gram precision (score is between 0 1)
• What percentage of machine n-grams can be found
  in the reference translation?
• An n-gram is an sequence of n words
• Not allowed to use same portion of reference
  translation twice (cant cheat by typing out the
  the the the the)
• Brevity penalty
• Cant just type out single word the (precision
  1.0!)
• Amazingly hard to game the system (i.e.,
  find a way to change machine output so that BLEU
  goes up, but quality doesnt)

Machine translation The American ?
international airport and its the office all
receives one calls self the sand Arab rich
business ? and so on electronic mail , which
sends out The threat will be able after public
place and so on the airport to start the
biochemistry attack , ? highly alerts after the
maintenance.
39
Multiple Reference Translations
40
BLEU Tends to Predict Human Judgments
(variant of BLEU)
slide from G. Doddington (NIST)
41
BLEU in Action
???????? (Foreign Original) the gunman was
shot to death by the police . (Reference
Translation) the gunman was police kill .
1wounded police jaya of 2the gunman
was shot dead by the police . 3the gunman
arrested by police kill . 4the gunmen were
killed . 5the gunman was shot to death by
the police . 6 gunmen were killed by police
?SUB0 ?SUB0 7 al by the police . 8the
ringer is killed by the police . 9police
killed the gunman . 10
42
BLEU in Action
???????? (Foreign Original) the gunman was
shot to death by the police . (Reference
Translation) the gunman was police kill .
1wounded police jaya of 2the gunman
was shot dead by the police . 3the gunman
arrested by police kill . 4the gunmen were
killed . 5the gunman was shot to death by
the police . 6 gunmen were killed by police
?SUB0 ?SUB0 7 al by the police . 8the
ringer is killed by the police . 9police
killed the gunman . 10
green 4-gram match (good!) red word not
matched (bad!)
43

• Word-Based Statistical MT

44
Statistical MT Systems
Spanish/English Bilingual Text
English Text
Statistical Analysis
Statistical Analysis
Broken English
Spanish
English
What hunger have I, Hungry I am so, I am so
hungry, Have I that hunger
Que hambre tengo yo
I am so hungry
45
Statistical MT Systems
Spanish/English Bilingual Text
English Text
Statistical Analysis
Statistical Analysis
Broken English
Spanish
English
Translation Model P(se)
Language Model P(e)
Que hambre tengo yo
I am so hungry
Decoding algorithm argmax P(e) P(se) e
46
Bayes Rule
Broken English
Spanish
English
Translation Model P(se)
Language Model P(e)
Que hambre tengo yo
I am so hungry
Decoding algorithm argmax P(e) P(se) e
Given a source sentence s, the decoder should
consider many possible translations and return
the target string e that maximizes P(e s) By
Bayes Rule, we can also write this as P(e) x
P(s e) / P(s) and maximize that instead. P(s)
never changes while we compare different es, so
we can equivalently maximize this P(e) x P(s
e)
47
Three Problems for Statistical MT

• Language model
• Given an English string e, assigns P(e) by
  formula
• good English string - high P(e)
• random word sequence - low P(e)
• Translation model
• Given a pair of strings , assigns P(f e)
  by formula
• look like translations - high P(f e)
• dont look like translations - low P(f
  e)
• Decoding algorithm
• Given a language model, a translation model, and
  a new sentence f find translation e maximizing
  P(e) P(f e)

48
The Classic Language ModelWord N-Grams

• Goal of the language model
• He is on the soccer field
• He is in the soccer field
• Is table the on cup the
• The cup is on the table
• Rice shrine
• American shrine
• Rice company
• American company

49
The Classic Language ModelWord N-Grams

• Generative story
• w1 START
• repeat until END is generated
• produce word w2 according to a big table P(w2
  w1)
• w1 w2
• P(I saw water on the table)
• P(I START)
• P(saw I)
• P(water saw)
• P(on water)
• P(the on)
• P(table the)
• P(END table)

Probabilities can be learned from online English
text.
50
Translation Model?
Generative story
Mary did not slap the green witch
Source-language morphological analysis Source
parse tree Semantic representation Generate
target structure
Maria no dió una botefada a la bruja verde
51
Translation Model?
Generative story
Mary did not slap the green witch
Source-language morphological analysis Source
parse tree Semantic representation Generate
target structure
What are all the possible moves and their
associated probability tables?
Maria no dió una botefada a la bruja verde
52
The Classic Translation ModelWord
Substitution/Permutation IBM Model 3, Brown et
al., 1993
Generative story
Mary did not slap the green witch
n(3slap)
Mary not slap slap slap the green witch
P-Null
Mary not slap slap slap NULL the green witch
t(lathe)
Maria no dió una botefada a la verde bruja
d(ji)
Maria no dió una botefada a la bruja verde
Probabilities can be learned from raw bilingual
text.
53
Word Alignment
la maison la maison bleue la fleur
the house the blue house the flower
All word alignments equally likely All
P(french-word english-word) equally likely
54
Word Alignment
la maison la maison bleue la fleur
the house the blue house the flower
la and the observed to co-occur
frequently, so P(la the) is increased.
55
Word Alignment
la maison la maison bleue la fleur
the house the blue house the flower
house co-occurs with both la and maison,
but P(maison house) can be raised without
limit, to 1.0, while P(la house) is limited
because of the (pigeonhole principle)
56
Word Alignment
la maison la maison bleue la fleur
the house the blue house the flower
settling down after another iteration
57
Word Alignment
la maison la maison bleue la fleur
the house the blue house the flower

• Inherent hidden structure revealed by EM
  training!
• For details, see
• A Statistical MT Tutorial Workbook (Knight,
  1999).
• The Mathematics of Statistical Machine
  Translation (Brown et al, 1993)
• Software GIZA

58
Word Alignment
la maison la maison bleue la fleur
the house the blue house the flower
P(juste fair) 0.411 P(juste correct)
0.027 P(juste right) 0.020
Possible English translations, to be rescored by
language model
new French sentence
59
Decoding
Actual process of translating a new
sentence. Given foreign sentence f, find English
sentence e that maximizes P(e) x P(f e)
Que hambre tengo yo what hunger have I that h
ungry am me so make where
60
Decoding
Actual process of translating a new
sentence. Given foreign sentence f, find English
sentence e that maximizes P(e) x P(f e)
Que hambre tengo yo what hunger have I that h
ungry am me so make where
61
Decoding
Actual process of translating a new
sentence. Given foreign sentence f, find English
sentence e that maximizes P(e) x P(f e)
Que hambre tengo yo what hunger have I that h
ungry am me so make where
62
Decoding
Actual process of translating a new
sentence. Given foreign sentence f, find English
sentence e that maximizes P(e) x P(f e)
Que hambre tengo yo what hunger have I that h
ungry am me so make where
63
Decoding
Actual process of translating a new
sentence. Given foreign sentence f, find English
sentence e that maximizes P(e) x P(f e)
Que hambre tengo yo what hunger have I that h
ungry am me so make where
64
Decoder Actually Translates New Sentences
1st target word
2nd target word
3rd target word
4th target word
start
end
all source words covered
Each partial translation hypothesis contains
- Last English word chosen source words covered
by it - Next-to-last English word chosen -
Entire coverage vector (so far) of source
sentence - Language model and translation model
scores (so far)
Jelinek, 1969 Brown et al, 1996 US
Patent (Och, Ueffing, and Ney, 2001
65
Dynamic Programming Beam Search
1st target word
2nd target word
3rd target word
4th target word
best predecessor link
start
end
all source words covered
Each partial translation hypothesis contains
- Last English word chosen source words covered
by it - Next-to-last English word chosen -
Entire coverage vector (so far) of source
sentence - Language model and translation model
scores (so far)
Jelinek, 1969 Brown et al, 1996 US
Patent (Och, Ueffing, and Ney, 2001
66
The Classic Results

• la politique de la haine . (Foreign Original)
• politics of hate . (Reference Translation)
• the policy of the hatred . (IBM4N-gramsStack)
  
• nous avons signé le protocole . (Foreign
  Original)
• we did sign the memorandum of agreement .
  (Reference Translation)
• we have signed the protocol . (IBM4N-gramsSta
  ck)
• où était le plan solide ? (Foreign Original)
• but where was the solid plan ? (Reference
  Translation)
• where was the economic base ? (IBM4N-gramsStac
  k)

the Ministry of Foreign Trade and Economic
Cooperation, including foreign direct investment
40.007 billion US dollars today provide data
include that year to November china actually
using foreign 46.959 billion US dollars and
67
Flaws of Word-Based MT

• Multiple English words for one French word
• IBM models can do one-to-many (fertility) but not
  many-to-one
• Phrasal Translation
• real estate, note that, interest in
• Syntactic Transformations
• Verb at the beginning in Arabic
• Translation model penalizes any proposed
  re-ordering
• Language model not strong enough to force the
  verb to move to the right place

68

• Phrase-Based Statistical MT

69
Phrase-Based Statistical MT
Morgen
fliege
ich
nach Kanada
zur Konferenz
Tomorrow
I
will fly
to the conference
In Canada

• Foreign input segmented in to phrases
• phrase is any sequence of words
• Each phrase is probabilistically translated into
  English
• P(to the conference zur Konferenz)
• P(into the meeting zur Konferenz)
  HUGE TABLE!!
• Phrases are probabilistically re-ordered
• See Koehn et al, 2003 for an intro.
• This is state-of-the-art

70
Advantages of Phrase-Based

• Many-to-many mappings can handle
  non-compositional phrases (e.g., real estate)
• Local context is very useful for disambiguating
• Interest rate ?
• Interest in ?
• The more data, the longer the learned phrases
• Sometimes whole sentences

71
How to Learn the Phrase Translation Table?

• One method alignment templates (Och et al,
  1999)
• Start with word alignment, build phrases from
  that.

Maria no dió una bofetada a
la bruja verde
This word-to-word alignment is a by-product of
training a translation model like
IBM-Model-3. This is the best (or Viterbi)
alignment.
Mary did not slap the green witch
72
How to Learn the Phrase Translation Table?

• One method alignment templates (Och et al,
  1999)
• Start with word alignment, build phrases from
  that.

Maria no dió una bofetada a
la bruja verde
This word-to-word alignment is a by-product of
training a translation model like
IBM-Model-3. This is the best (or Viterbi)
alignment.
Mary did not slap the green witch
73
IBM Models are 1-to-Many

• Run IBM-style aligner both directions, then merge

E?F best alignment
MERGE
F?E best alignment
Union or Intersection
74
How to Learn the Phrase Translation Table?

• Collect all phrase pairs that are consistent with
  the word alignment

Maria no dió una bofetada a la
bruja verde
Mary did not slap the green witch
one example phrase pair
75
Consistent with Word Alignment
Maria no dió
Maria no dió
Maria no dió
Mary did not slap
Mary did not slap
Mary did not slap
consistent
inconsistent
inconsistent
Phrase alignment must contain all alignment
points for all the words in both phrases!
76
Word Alignment Induced Phrases
Maria no dió una bofetada a
la bruja verde
Mary did not slap the green witch
(Maria, Mary) (no, did not) (slap, dió una
bofetada) (la, the) (bruja, witch) (verde, green)
77
Word Alignment Induced Phrases
Maria no dió una bofetada a
la bruja verde
Mary did not slap the green witch
(Maria, Mary) (no, did not) (slap, dió una
bofetada) (la, the) (bruja, witch) (verde,
green) (a la, the) (dió una bofetada a, slap the)
78
Word Alignment Induced Phrases
Maria no dió una bofetada a
la bruja verde
Mary did not slap the green witch
(Maria, Mary) (no, did not) (slap, dió una
bofetada) (la, the) (bruja, witch) (verde, green)
(a la, the) (dió una bofetada a, slap
the) (Maria no, Mary did not) (no dió una
bofetada, did not slap), (dió una bofetada a la,
slap the) (bruja verde, green witch)
79
Word Alignment Induced Phrases
Maria no dió una bofetada a
la bruja verde
Mary did not slap the green witch
(Maria, Mary) (no, did not) (slap, dió una
bofetada) (la, the) (bruja, witch) (verde, green)
(a la, the) (dió una bofetada a, slap
the) (Maria no, Mary did not) (no dió una
bofetada, did not slap), (dió una bofetada a la,
slap the) (bruja verde, green witch) (Maria no
dió una bofetada, Mary did not slap) (a la bruja
verde, the green witch)
80
Word Alignment Induced Phrases
Maria no dió una bofetada a
la bruja verde
Mary did not slap the green witch
(Maria, Mary) (no, did not) (slap, dió una
bofetada) (la, the) (bruja, witch) (verde, green)
(a la, the) (dió una bofetada a, slap
the) (Maria no, Mary did not) (no dió una
bofetada, did not slap), (dió una bofetada a la,
slap the) (bruja verde, green witch) (Maria no
dió una bofetada, Mary did not slap) (a la bruja
verde, the green witch) (Maria no dió una
bofetada a la bruja verde, Mary did not slap the
green witch)
81
Phrase Pair Probabilities

• A certain phrase pair (f-f-f, e-e-e) may appear
  many times across the bilingual corpus.
• We hope so!
• We can calculate phrase substitution
  probabilities P(f-f-f e-e-e)
• We can use these in decoding
• Much better results than word-based translation!

82
Syntax and Semanticsin Statistical MT
83
MT Pyramid
interlingua
semantics
semantics
syntax
syntax
phrases
phrases
words
words
SOURCE
TARGET
84
Why Syntax?

• Need much more grammatical output
• Need accurate control over re-ordering
• Need accurate insertion of function words
• Word translations need to depend on
  grammatically-related words

85
Linguistic Transformations using Tree Automata
Original input
Transformation
S
S
NP
VP
NP
VP
PRO
VBZ
NP
PRO
VBZ
NP
he
enjoys
SBAR
he
enjoys
SBAR
VBG
VP
VBG
VP
listening
P
NP
listening
P
NP
to
music
to
music
86
Linguistic Transformations using Tree Automata
Original input
Transformation
S
S
NP
VP
NP
VP
PRO
VBZ
NP
PRO
VBZ
NP
he
enjoys
SBAR
he
enjoys
SBAR
VBG
VP
VBG
VP
listening
P
NP
listening
P
NP
to
music
to
music
87
Linguistic Transformations using Tree Automata
Original input
Transformation
S
NP
VP
PRO
VBZ
NP
VBZ
NP
NP
,
,
,
,
o
wa
he
enjoys
SBAR
enjoys
SBAR
PRO
VBG
VP
he
VBG
VP
listening
P
NP
listening
P
NP
to
music
to
music
88
Linguistic Transformations using Tree Automata
Original input
Transformation
S
NP
VP
PRO
VBZ
NP
VBZ
NP
,
,
,
,
kare
wa
o
he
enjoys
SBAR
enjoys
SBAR
VBG
VP
VBG
VP
listening
P
NP
listening
P
NP
to
music
to
music
89
Linguistic Transformations using Tree Automata
Original input
Final output
S
NP
VP
PRO
VBZ
NP
,
,
,
,
,
,
,
,
kare
kiku
ongaku
o
wa
daisuki
desu
ga
no
he
enjoys
SBAR
VBG
VP
listening
P
NP
to
music
90
Automata Linguistics Learning
MT
Applications
Automata Theory
Tree Automata (Rounds 70)
91
Automata Linguistics Learning
Transformational Grammar (Chomsky 57)
MT
Applications
Linguistic Theory
Automata Theory
Tree Automata (Rounds 70)
92
Automata Linguistics Learning
Transformational Grammar (Chomsky 57)
MT (05)
Compression (01)
QA (03)
Applications
Linguistic Theory
Generation (00)
Automata Theory
Tree Automata (Rounds 70)
93
Automata Linguistics Learning
Transformational Grammar (Chomsky 57)
MT (05)
Compression (01)
QA (03)
Applications
Linguistic Theory
Generation (00)
Algorithms
Automata Theory
Efficient Automata Algorithms
Tree Automata (Rounds 70)
Generic Toolkits
94
Making Good Progress

• Algorithms Data Evaluation Computers
• Interdisciplinary work
• Natural language processing
• Machine learning
• Linguistics
• Automata theory
• Lots of room for improvement!

95
Future PhD Theses?

• Syntax-based Language Models for Improving
  Statistical MT
• Discriminative Training of Millions of Features
  for MT
• Semantic Representations Induced from
  Multilingual EU and UN Data
• What Makes One Language Pair More Difficult to
  Translate Than Another
• A State-of-the-Art MT System Based on Syntactic
  Transformations
• New Training Methods for High-Quality Word
  Alignment
• many unpredictable ones

96

• Thank you
• if you are interested in getting
• research experience in this area,
• and are a very good programmer
• contact -- knight_at_isi.edu