Parts of Speech - PowerPoint PPT Presentation

Loading...

PPT – Parts of Speech PowerPoint presentation | free to view - id: 1e5412-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Parts of Speech

Description:

Parts of Speech – PowerPoint PPT presentation

Number of Views:36
Avg rating:3.0/5.0
Slides: 127
Provided by: DanJur1
Category:
Tags: channel | com | parts | speech | weather

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Parts of Speech


1
Parts of Speech
  • Sudeshna Sarkar
  • 7 Aug 2008

2
Why Do We Care about Parts of Speech?
  • Pronunciation
  • Hand me the lead pipe.
  • Predicting what words can be expected next
  • Personal pronoun (e.g., I, she) ____________
  • Stemming
  • -s means singular for verbs, plural for nouns
  • As the basis for syntactic parsing and then
    meaning extraction
  • I will lead the group into the lead smelter.
  • Machine translation
  • (E) content N ? (F) contenu N
  • (E) content Adj ? (F) content Adj or
    satisfait Adj

3
What is a Part of Speech?
Is this a semantic distinction? For example,
maybe Noun is the class of words for people,
places and things. Maybe Adjective is the class
of words for properties of nouns. Consider green
book book is a Noun green is an
Adjective Now consider book worm This green
is very soothing.
4
How Many Parts of Speech Are There?
  • A first cut at the easy distinctions
  • Open classes
  • nouns, verbs, adjectives, adverbs
  • Closed classes function words
  • conjunctions and, or, but
  • pronounts I, she, him
  • prepositions with, on
  • determiners the, a, an

5
Part of speech tagging
  • 8 (ish) traditional parts of speech
  • Noun, verb, adjective, preposition, adverb,
    article, interjection, pronoun, conjunction, etc
  • This idea has been around for over 2000 years
    (Dionysius Thrax of Alexandria, c. 100 B.C.)
  • Called parts-of-speech, lexical category, word
    classes, morphological classes, lexical tags, POS
  • Well use POS most frequently
  • Ill assume that you all know what these are

6
POS examples
  • N noun chair, bandwidth, pacing
  • V verb study, debate, munch
  • ADJ adj purple, tall, ridiculous
  • ADV adverb unfortunately, slowly,
  • P preposition of, by, to
  • PRO pronoun I, me, mine
  • DET determiner the, a, that, those

7
Tagsets
Brown corpus tagset (87 tags)
http//www.scs.leeds.ac.uk/amalgam/tagsets/brown.h
tml Penn Treebank tagset (45 tags)
http//www.cs.colorado.edu/martin/SLP/Figures/
(8.6) C7 tagset (146 tags) http//www.comp.lancs
.ac.uk/ucrel/claws7tags.html
8
POS Tagging Definition
  • The process of assigning a part-of-speech or
    lexical class marker to each word in a corpus

9
POS Tagging example
  • WORD tag
  • the DET
  • koala N
  • put V
  • the DET
  • keys N
  • on P
  • the DET
  • table N

10
POS tagging Choosing a tagset
  • There are so many parts of speech, potential
    distinctions we can draw
  • To do POS tagging, need to choose a standard set
    of tags to work with
  • Could pick very coarse tagets
  • N, V, Adj, Adv.
  • More commonly used set is finer grained, the
    UPenn TreeBank tagset, 45 tags
  • PRP, WRB, WP, VBG
  • Even more fine-grained tagsets exist

11
Penn TreeBank POS Tag set
12
Using the UPenn tagset
  • The/DT grand/JJ jury/NN commmented/VBD on/IN a/DT
    number/NN of/IN other/JJ topics/NNS ./.
  • Prepositions and subordinating conjunctions
    marked IN (although/IN I/PRP..)
  • Except the preposition/complementizer to is
    just marked to.

13
POS Tagging
  • Words often have more than one POS back
  • The back door JJ
  • On my back NN
  • Win the voters back RB
  • Promised to back the bill VB
  • The POS tagging problem is to determine the POS
    tag for a particular instance of a word.

14
How hard is POS tagging? Measuring ambiguity
15
Algorithms for POS Tagging
  • Ambiguity In the Brown corpus, 11.5 of the
    word types are ambiguous (using 87 tags)

Worse, 40 of the tokens are ambiguous.
16
Algorithms for POS Tagging
  • Why cant we just look them up in a dictionary?
  • Words that arent in the dictionary

http//story.news.yahoo.com/news?tmplstorycid57
8ncid578e1u/nm/20030922/ts_nm/iraq_usa_dc
  • One idea P(ti wi) the probability that a
    random hapax legomenon in the corpus has tag ti.
  • Nouns are more likely than verbs, which are more
    likely than pronouns.
  • Another idea use morphology.

17
Algorithms for POS Tagging - Knowledge
  • Dictionary
  • Morphological rules, e.g.,
  • _____-tion
  • _____-ly
  • capitalization
  • N-gram frequencies
  • to _____
  • DET _____ N
  • But what about rare words, e.g, smelt (two verb
    forms, melt and past tense of smell, and one noun
    form, a small fish)
  • Combining these
  • V _____-ing I was gracking vs. Gracking
    is fun.

18
POS Tagging - Approaches
  • Approaches
  • Rule-based tagging
  • (ENGTWOL)
  • Stochastic (Probabilistic) tagging
  • HMM (Hidden Markov Model) tagging
  • Transformation-based tagging
  • Brill tagger
  • Do we return one best answer or several answers
    and let later steps decide?
  • How does the requisite knowledge get entered?

19
3 methods for POS tagging
  • 1. Rule-based tagging
  • Example Karlsson (1995) EngCG tagger based on
    the Constraint Grammar architecture and ENGTWOL
    lexicon
  • Basic Idea
  • Assign all possible tags to words (morphological
    analyzer used)
  • Remove wrong tags according to set of constraint
    rules (typically more than 1000 hand-written
    constraint rules, but may be machine-learned)

20
3 methods for POS tagging
  • 2. Transformation-based tagging
  • Example Brill (1995) tagger - combination of
    rule-based and stochastic (probabilistic) tagging
    methodologies
  • Basic Idea
  • Start with a tagged corpus dictionary (with
    most frequent tags)
  • Set the most probable tag for each word as a
    start value
  • Change tags according to rules of type if word-1
    is a determiner and word is a verb then change
    the tag to noun in a specific order (like
    rule-based taggers)
  • machine learning is usedthe rules are
    automatically induced from a previously tagged
    training corpus (like stochastic approach)

21
3 methods for POS tagging
  • 3. Stochastic (Probabilistic) tagging
  • Example HMM (Hidden Markov Model) tagging - a
    training corpus used to compute the probability
    (frequency) of a given word having a given POS
    tag in a given context

22
Hidden Markov Model (HMM) Tagging
  • Using an HMM to do POS tagging
  • HMM is a special case of Bayesian inference
  • It is also related to the noisy channel model
    in ASR (Automatic Speech Recognition)

23
Hidden Markov Model (HMM) Taggers
  • Goal maximize P(wordtag) x P(tagprevious n
    tags)
  • P(wordtag)
  • word/lexical likelihood
  • probability that given this tag, we have this
    word
  • NOT probability that this word has this tag
  • modeled through language model (word-tag matrix)
  • P(tagprevious n tags)
  • tag sequence likelihood
  • probability that this tag follows these previous
    tags
  • modeled through language model (tag-tag matrix)

Lexical information
Syntagmatic information
24
POS tagging as a sequence classification task
  • We are given a sentence (an observation or
    sequence of observations)
  • Secretariat is expected to race tomorrow
  • sequence of n words w1wn.
  • What is the best sequence of tags which
    corresponds to this sequence of observations?
  • Probabilistic/Bayesian view
  • Consider all possible sequences of tags
  • Out of this universe of sequences, choose the tag
    sequence which is most probable given the
    observation sequence of n words w1wn.

25
Getting to HMM
  • Let T t1,t2,,tn
  • Let W w1,w2,,wn
  • Goal Out of all sequences of tags t1tn, get the
    the most probable sequence of POS tags T
    underlying the observed sequence of words
    w1,w2,,wn
  • Hat means our estimate of the best the most
    probable tag sequence
  • Argmaxx f(x) means the x such that f(x) is
    maximized
  • it maximazes our estimate of the best tag
    sequence

26
Getting to HMM
  • This equation is guaranteed to give us the best
    tag sequence
  • But how do we make it operational? How do we
    compute this value?
  • Intuition of Bayesian classification
  • Use Bayes rule to transform it into a set of
    other probabilities that are easier to compute
  • Thomas Bayes British mathematician (1702-1761)

27
Bayes Rule
Breaks down any conditional probability P(xy)
into three other probabilities P(xy) The
conditional probability of an event x assuming
that y has occurred
28
Bayes Rule
We can drop the denominator it does not change
for each tag sequence we are looking for the
best tag sequence for the same observation, for
the same fixed set of words
29
Bayes Rule
30
Likelihood and prior
n
31
Likelihood and prior Further Simplifications
1. the probability of a word appearing depends
only on its own POS tag, i.e, independent of
other words around it
n
2. BIGRAM assumption the probability of a
tag appearing depends only on the previous tag
3. The most probable tag sequence estimated by
the bigram tagger
32
Likelihood and prior Further Simplifications
1. the probability of a word appearing depends
only on its own POS tag, i.e, independent of
other words around it
n
33
Likelihood and prior Further Simplifications
2. BIGRAM assumption the probability of a
tag appearing depends only on the previous tag
Bigrams are groups of two written letters, two
syllables, or two words they are a special case
of N-gram. Bigrams are used as the basis for
simple statistical analysis of text The bigram
assumption is related to the first-order Markov
assumption
34
Likelihood and prior Further Simplifications
3. The most probable tag sequence estimated by
the bigram tagger
--------------------------------------------------
--------------------------------------------------
-----------
n
biagram assumption
35
Two kinds of probabilities (1)
  • Tag transition probabilities p(titi-1)
  • Determiners likely to precede adjs and nouns
  • That/DT flight/NN
  • The/DT yellow/JJ hat/NN
  • So we expect P(NNDT) and P(JJDT) to be high
  • But P(DTJJ) to be?

36
Two kinds of probabilities (1)
  • Tag transition probabilities p(titi-1)
  • Compute P(NNDT) by counting in a labeled corpus

of times DT is followed by NN
37
Two kinds of probabilities (2)
  • Word likelihood probabilities p(witi)
  • P(isVBZ) probability of VBZ (3sg Pres verb)
    being is
  • Compute P(isVBZ) by counting in a labeled corpus

If we were expecting a third person singular
verb, how likely is it that this verb would be
is?
38
An Example the verb race
  • Secretariat/NNP is/VBZ expected/VBN to/TO race/VB
    tomorrow/NR
  • People/NNS continue/VB to/TO inquire/VB the/DT
    reason/NN for/IN the/DT race/NN for/IN outer/JJ
    space/NN
  • How do we pick the right tag?

39
Disambiguating race
40
Disambiguating race
  • P(NNTO) .00047
  • P(VBTO) .83
  • The tag transition probabilities P(NNTO) and
    P(VBTO) answer the question How likely are we
    to expect verb/noun given the previous tag TO?
  • P(raceNN) .00057
  • P(raceVB) .00012
  • Lexical likelihoods from the Brown corpus for
    race given a POS tag NN or VB.
  • P(NRVB) .0027
  • P(NRNN) .0012
  • tag sequence probability for the likelihood of an
    adverb occurring given the previous tag verb or
    noun
  • P(VBTO)P(NRVB)P(raceVB) .00000027
  • P(NNTO)P(NRNN)P(raceNN).00000000032
  • Multiply the lexical likelihoods with the tag
    sequence probabiliies the verb wins

41
Hidden Markov Models
  • What weve described with these two kinds of
    probabilities is a Hidden Markov Model (HMM)
  • Lets just spend a bit of time tying this into
    the model
  • In order to define HMM, we will first introduce
    the Markov Chain, or observable Markov Model.

42
Definitions
  • A weighted finite-state automaton adds
    probabilities to the arcs
  • The sum of the probabilities leaving any arc must
    sum to one
  • A Markov chain is a special case of a WFST in
    which the input sequence uniquely determines
    which states the automaton will go through
  • Markov chains cant represent inherently
    ambiguous problems
  • Useful for assigning probabilities to unambiguous
    sequences

43
Markov chain First-order observed Markov
Model
  • a set of states
  • Q q1, q2qN the state at time t is qt
  • a set of transition probabilities
  • a set of probabilities A a01a02an1ann.
  • Each aij represents the probability of
    transitioning from state i to state j
  • The set of these is the transition probability
    matrix A
  • Distinguished start and end states
  • Special initial probability vector ?
  • ?i the probability that the MM will start in
    state i, each ?i expresses the probability
    p(qiSTART)

44
Markov chain First-order observed Markov
Model
  • Markov Chain for weather Example 1
  • three types of weather sunny, rainy, foggy
  • we want to find the following conditional
    probabilities
  • P(qnqn-1, qn-2, , q1)
  • - I.e., the probability of the unknown weather
    on day n, depending on the (known) weather of
    the preceding days
  • - We could infer this probability from the
    relative frequency (the statistics) of past
    observations of weather sequences
  • Problem the larger n is, the more observations
    we must collect.
  • Suppose that n6, then we have to collect
    statistics for 3(6-1) 243 past histories

45
Markov chain First-order observed Markov
Model
  • Therefore, we make a simplifying assumption,
    called the (first-order) Markov assumption
  • for a sequence of observations q1, qn,
  • current state only depends on previous state
  • the joint probability of certain past and current
    observations

46
Markov chain First-order observable Markov
Model

47
Markov chain First-order observed Markov
Model
  • Given that today the weather is sunny, what's
    the probability that tomorrow is sunny and the
    day after is rainy?
  • Using the Markov assumption and the
    probabilities in table 1, this translates into

48
The weather figure specific example
  • Markov Chain for weather Example 2

49
Markov chain for weather
  • What is the probability of 4 consecutive rainy
    days?
  • Sequence is rainy-rainy-rainy-rainy
  • I.e., state sequence is 3-3-3-3
  • P(3,3,3,3)
  • ?1a11a11a11a11 0.2 x (0.6)3 0.0432

50
Hidden Markov Model
  • For Markov chains, the output symbols are the
    same as the states.
  • See sunny weather were in state sunny
  • But in part-of-speech tagging (and other things)
  • The output symbols are words
  • But the hidden states are part-of-speech tags
  • So we need an extension!
  • A Hidden Markov Model is an extension of a Markov
    chain in which the output symbols are not the
    same as the states.
  • This means we dont know which state we are in.

51
Markov chain for weather
52
Markov chain for words
Observed events words Hidden events tags
53
Hidden Markov Models
  • States Q q1, q2qN
  • Observations O o1, o2oN
  • Each observation is a symbol from a vocabulary V
    v1,v2,vV
  • Transition probabilities (prior)
  • Transition probability matrix A aij
  • Observation likelihoods (likelihood)
  • Output probability matrix Bbi(ot)
  • a set of observation likelihoods, each
    expressing the probability of an observation ot
    being generated from a state i, emission
    probabilities
  • Special initial probability vector ?
  • ?i the probability that the HMM will start in
    state i, each ?i expresses the probability
  • p(qiSTART)

54
Assumptions
  • Markov assumption the probability of a
    particular state depends only on the previous
    state
  • Output-independence assumption the probability
    of an output observation depends only on the
    state that produced that observation

55
HMM for Ice Cream
  • You are a climatologist in the year 2799
  • Studying global warming
  • You cant find any records of the weather in
    Boston, MA for summer of 2007
  • But you find Jason Eisners diary
  • Which lists how many ice-creams Jason ate every
    date that summer
  • Our job figure out how hot it was

56
Noam task
  • Given
  • Ice Cream Observation Sequence 1,2,3,2,2,2,3
  • (cp. with output symbols)
  • Produce
  • Weather Sequence C,C,H,C,C,C,H
  • (cp. with hidden states, causing states)

57
HMM for ice cream
58
Different types of HMM structure
Ergodic fully-connected
Bakis left-to-right
59
HMM Taggers
  • Two kinds of probabilities
  • A transition probabilities (PRIOR)
  • B observation likelihoods (LIKELIHOOD)
  • HMM Taggers choose the tag sequence which
    maximizes the product of word likelihood and tag
    sequence probability

60
Weighted FSM corresponding to hidden states of
HMM, showing A probs
61
B observation likelihoods for POS HMM
62
The A matrix for the POS HMM
63
The B matrix for the POS HMM
64
HMM Taggers
  • The probabilities are trained on hand-labeled
    training corpora (training set)
  • Combine different N-gram levels
  • Evaluated by comparing their output from a test
    set to human labels for that test set (Gold
    Standard)

65
The Viterbi Algorithm
  • best tag sequence for "John likes to fish in the
    sea"?
  • efficiently computes the most likely state
    sequence given a particular output sequence
  • based on dynamic programming

66
A smaller example
a
b
  • What is the best sequence of states for the input
    string bbba?
  • Computing all possible paths and finding the one
    with the max probability is exponential

67
A smaller example (cont)
  • For each state, store the most likely sequence
    that could lead to it (and its probability)
  • Path probability matrix
  • An array of states versus time (tags versus
    words)
  • That stores the prob. of being at each state at
    each time in terms of the prob. for being in each
    state at the preceding time.

Best sequence Best sequence Input sequence / time Input sequence / time Input sequence / time Input sequence / time
e --gt b b --gt b bb --gt b bbb --gt a
leading to q coming from q e --gt q 0.6 (1.0x0.6) q --gt q 0.108 (0.6x0.3x0.6) qq --gt q 0.01944 (0.108x0.3x0.6) qrq --gt q 0.018144 (0.1008x0.3x0.4)
leading to q coming from r r --gt q 0 (0x0.5x0.6) qr --gt q 0.1008 (0.336x0.5x 0.6) qrr --gt q 0.02688 (0.1344x0.5x0.4)
leading to r coming from q e --gt r 0 (0x0.8) q --gt r 0.336 (0.6x0.7x0.8) qq --gt r 0.0648 (0.108x0.7x0.8) qrq --gt r 0.014112 (0.1008x0.7x0.2)
leading to r coming from r r --gt r 0 (0x0.5x0.8) qr --gt r 0.1344 (0.336x0.5x0.8) qrr --gt r 0.01344 (0.1344x0.5x0.2)
68
Viterbi intuition we are looking for the best
path
S1
S2
S4
S3
S5
Slide from Dekang Lin
69
The Viterbi Algorithm
70
Intuition
  • The value in each cell is computed by taking the
    MAX over all paths that lead to this cell.
  • An extension of a path from state i at time t-1
    is computed by multiplying
  • Previous path probability from previous cell
    viterbit-1,i
  • Transition probability aij from previous state I
    to current state j
  • Observation likelihood bj(ot) that current state
    j matches observation symbol t

71
Viterbi example
72
Smoothing of probabilities
  • Data sparseness is a problem when estimating
    probabilities based on corpus data.
  • The add one smoothing technique

C- absolute frequency N no of training
instances B no of different types
  • Linear interpolation methods can compensate for
    data sparseness with higher order models. A
    common method is interpolating trigrams, bigrams
    and unigrams
  • The lambda values are automatically determined
    using a variant of the Expectation Maximization
    algorithm.

73
Viterbi for POS tagging
  • Let
  • n nb of words in sentence to tag (nb of input
    tokens)
  • T nb of tags in the tag set (nb of states)
  • vit path probability matrix (viterbi)
  • viti,j probability of being at state
    (tag) j at word i
  • state matrix to recover the nodes of the best
    path (best tag sequence)
  • statei1,j the state (tag) of the incoming
    arc that led to this most probable state j at
    word i1
  • // Initialization
  • vit1,PERIOD1.0 // pretend that there is
    a period before
  • // our
    sentence (start tag PERIOD)
  • vit1,t0.0 for t ? PERIOD

74
Viterbi for POS tagging (cont)
  • // Induction (build the path probability matrix)
  • for i1 to n step 1 do // for all words in the
    sentence
  • for all tags tj do // for all possible
    tags
  • // store the max prob of the path
  • viti1,tj max1kT(viti,tk x P(wi1tj) x
    P(tj tk))
  • // store the actual state
  • pathi1,tj argmax1kT ( viti,tk x
    P(wi1tj) x P(tj tk))
  • end
  • end
  • //Termination and path-readout
  • bestStaten1 argmax1jT vitn1,j
  • for jn to 1 step -1 do // for all the words in
    the sentence
  • bestStatej pathi1, bestStatej1
  • end
  • P(bestState1,, bestStaten ) max1jT
    vitn1,j

emission probability
state transition probability
probability of best path leading to state tk at
word i
75
Possible improvements
  • in bigram POS tagging, we condition a tag only on
    the preceding tag
  • why not...
  • use more context (ex. use trigram model)
  • more precise
  • is clearly marked --gt verb, past participle
  • he clearly marked --gt verb, past tense
  • combine trigram, bigram, unigram models
  • condition on words too
  • but with an n-gram approach, this is too costly
    (too many parameters to model)

76
Further issues with Markov Model tagging
  • Unknown words are a problem since we dont have
    the required probabilities. Possible solutions
  • Assign the word probabilities based on
    corpus-wide distribution of POS
  • Use morphological cues (capitalization, suffix)
    to assign a more calculated guess.
  • Using higher order Markov models
  • Using a trigram model captures more context
  • However, data sparseness is much more of a
    problem.

77
TnT
  • Efficient statistical POS tagger developed by
    Thorsten Brants, ANLP-2000
  • Underlying model
  • Trigram modelling
  • The probability of a POS only depends on its two
    preceding POS
  • The probability of a word appearing at a
    particular position given that its POS occurs at
    that position is independent of everything else.

78
Training
  • Maximum likelihood estimates

Smoothing context-independent variant of linear
interpolation.
79
Smoothing algorithm
  • Set ?i0
  • For each trigram t1 t2 t3 with f(t1,t2,t3 )gt0
  • Depending on the max of the following three
    values
  • Case (f(t1,t2,t3 )-1)/ f(t1,t2) incr ?3 by
    f(t1,t2,t3 )
  • Case (f(t2,t3 )-1)/ f(t2) incr ?2 by
    f(t1,t2,t3 )
  • Case (f(t3 )-1)/ N-1 incr ?1 by
    f(t1,t2,t3 )
  • Normalize ?i

80
Evaluation of POS taggers
  • compared with gold-standard of human performance
  • metric
  • accuracy of tags that are identical to gold
    standard
  • most taggers 96-97 accuracy
  • must compare accuracy to
  • ceiling (best possible results)
  • how do human annotators score compared to each
    other? (96-97)
  • so systems are not bad at all!
  • baseline (worst possible results)
  • what if we take the most-likely tag (unigram
    model) regardless of previous tags ? (90-91)
  • so anything less is really bad

81
More on tagger accuracy
  • is 95 good?
  • thats 5 mistakes every 100 words
  • if on average, a sentence is 20 words, thats 1
    mistake per sentence
  • when comparing tagger accuracy, beware of
  • size of training corpus
  • the bigger, the better the results
  • difference between training testing corpora
    (genre, domain)
  • the closer, the better the results
  • size of tag set
  • Prediction versus classification
  • unknown words
  • the more unknown words (not in dictionary), the
    worst the results

82
Error Analysis
  • Look at a confusion matrix (contingency table)
  • E.g. 4.4 of the total errors caused by
    mistagging VBD as VBN
  • See what errors are causing problems
  • Noun (NN) vs ProperNoun (NNP) vs Adj (JJ)
  • Adverb (RB) vs Particle (RP) vs Prep (IN)
  • Preterite (VBD) vs Participle (VBN) vs Adjective
    (JJ)
  • ERROR ANALYSIS IS ESSENTIAL!!!

83
Tag indeterminacy
84
Major difficulties in POS tagging
  • Unknown words (proper names)
  • because we do not know the set of tags it can
    take
  • and knowing this takes you a long way (cf.
    baseline POS tagger)
  • possible solutions
  • assign all possible tags with probabilities
    distribution identical to lexicon as a whole
  • use morphological cues to infer possible tags
  • ex. word ending in -ed are likely to be past
    tense verbs or past participles
  • Frequently confused tag pairs
  • preposition vs particle
  • ltrunninggt ltupgt a hill (prep) / ltrunning upgt a
    bill (particle)
  • verb, past tense vs. past participle vs.
    adjective

85
Unknown Words
  • Most-frequent-tag approach.
  • What about words that dont appear in the
    training set?
  • Suffix analysis
  • The probability distribution for a particular
    suffix is generated from all words in the
    training set that share the same suffix.
  • Suffix estimation Calculate the probability of
    a tag t given the last i letters of an n letter
    word.
  • Smoothing successive abstraction through
    sequences of increasingly more general contexts
    (i.e., omit more and more characters of the
    suffix)
  • Use a morphological analyzer to get the
    restriction on the possible tags.

86
Unknown words
87
Alternative graphical models for part of speech
tagging
88
Different Models for POS tagging
  • HMM
  • Maximum Entropy Markov Models
  • Conditional Random Fields

89
Hidden Markov Model (HMM) Generative Modeling
Source Model P(Y)
Noisy Channel P(XY)
y
x
90
Dependency (1st order)
91
Disadvantage of HMMs (1)
  • No Rich Feature Information
  • Rich information are required
  • When xk is complex
  • When data of xk is sparse
  • Example POS Tagging
  • How to evaluate P(wktk) for unknown words wk ?
  • Useful features
  • Suffix, e.g., -ed, -tion, -ing, etc.
  • Capitalization
  • Generative Model
  • Parameter estimation maximize the joint
    likelihood of training examples

92
Generative Models
  • Hidden Markov models (HMMs) and stochastic
    grammars
  • Assign a joint probability to paired observation
    and label sequences
  • The parameters typically trained to maximize the
    joint likelihood of train examples

93
Generative Models (contd)
  • Difficulties and disadvantages
  • Need to enumerate all possible observation
    sequences
  • Not practical to represent multiple interacting
    features or long-range dependencies of the
    observations
  • Very strict independence assumptions on the
    observations

94
  • Better Approach
  • Discriminative model which models P(yx) directly
  • Maximize the conditional likelihood of training
    examples

95
Maximum Entropy modeling
  • N-gram model probabilities depend on the
    previous few tokens.
  • We may identify a more heterogeneous set of
    features which contribute in some way to the
    choice of the current word. (whether it is the
    first word in a story, whether the next word is
    to, whether one of the last 5 words is a
    preposition, etc)
  • Maxent combines these features in a probabilistic
    model.
  • The given features provide a constraint on the
    model.
  • We would like to have a probability distribution
    which, outside of these constraints, is as
    uniform as possible has the maximum entropy
    among all models that satisfy these constraints.

96
Maximum Entropy Markov Model
  • Discriminative Sub Models
  • Unify two parameters in generative model into one
    conditional model
  • Two parameters in generative model,
  • parameter in source model
    and parameter in noisy channel
  • Unified conditional model
  • Employ maximum entropy principle
  • Maximum Entropy Markov Model

97
General Maximum Entropy Principle
  • Model
  • Model distribution P(Y X) with a set of features
    f1, f2, ?, fl defined on X and Y
  • Idea
  • Collect information of features from training
    data
  • Principle
  • Model what is known
  • Assume nothing else
  • ? Flattest distribution
  • ? Distribution with the maximum Entropy

98
Example
  • (Berger et al., 1996) example
  • Model translation of word in from English to
    French
  • Need to model P(wordFrench)
  • Constraints
  • 1 Possible translations dans, en, à, au course
    de, pendant
  • 2 dans or en used in 30 of the time
  • 3 dans or à in 50 of the time

99
Features
  • Features
  • 0-1 indicator functions
  • 1 if (x, y) satisfies a predefined condition
  • 0 if not
  • Example POS Tagging

100
Constraints
  • Empirical Information
  • Statistics from training data T
  • Expected Value
  • From the distribution P(Y X) we want to model
  • Constraints

101
Maximum Entropy Objective
  • Entropy
  • Maximization Problem

102
Dual Problem
  • Dual Problem
  • Conditional model
  • Maximum likelihood of conditional data
  • Solution
  • Improved iterative scaling (IIS) (Berger et al.
    1996)
  • Generalized iterative scaling (GIS) (McCallum et
    al. 2000)

103
Maximum Entropy Markov Model
  • Use Maximum Entropy Approach to Model
  • 1st order
  • Features
  • Basic features (like parameters in HMM)
  • Bigram (1st order) or trigram (2nd order) in
    source model
  • State-output pair feature (Xk xk, Yk yk)
  • Advantage incorporate other advanced features on
    (xk, yk)

104
HMM vs MEMM (1st order)
Maximum Entropy Markov Model (MEMM)
HMM
105
Performance in POS Tagging
  • POS Tagging
  • Data set WSJ
  • Features
  • HMM features, spelling features (like ed, -tion,
    -s, -ing, etc.)
  • Results (Lafferty et al. 2001)
  • 1st order HMM
  • 94.31 accuracy, 54.01 OOV accuracy
  • 1st order MEMM
  • 95.19 accuracy, 73.01 OOV accuracy

106
ME applications
  • Part of Speech (POS) Tagging (Ratnaparkhi, 1996)
  • P(POS tag context)
  • Information sources
  • Word window (4)
  • Word features (prefix, suffix, capitalization)
  • Previous POS tags

107
ME applications
  • Abbreviation expansion (Pakhomov, 2002)
  • Information sources
  • Word window (4)
  • Document title
  • Word Sense Disambiguation (WSD) (Chao Dyer,
    2002)
  • Information sources
  • Word window (4)
  • Structurally related words (4)
  • Sentence Boundary Detection (Reynar
    Ratnaparkhi, 1997)
  • Information sources
  • Token features (prefix, suffix, capitalization,
    abbreviation)
  • Word window (2)

108
Solution
  • Global Optimization
  • Optimize parameters in a global model
    simultaneously, not in sub models separately
  • Alternatives
  • Conditional random fields
  • Application of perceptron algorithm

109
Why ME?
  • Advantages
  • Combine multiple knowledge sources
  • Local
  • Word prefix, suffix, capitalization (POS -
    (Ratnaparkhi, 1996))
  • Word POS, POS class, suffix (WSD - (Chao Dyer,
    2002))
  • Token prefix, suffix, capitalization,
    abbreviation (Sentence Boundary - (Reynar
    Ratnaparkhi, 1997))
  • Global
  • N-grams (Rosenfeld, 1997)
  • Word window
  • Document title (Pakhomov, 2002)
  • Structurally related words (Chao Dyer, 2002)
  • Sentence length, conventional lexicon (Och Ney,
    2002)
  • Combine dependent knowledge sources

110
Why ME?
  • Advantages
  • Add additional knowledge sources
  • Implicit smoothing
  • Disadvantages
  • Computational
  • Expected value at each iteration
  • Normalizing constant
  • Overfitting
  • Feature selection
  • Cutoffs
  • Basic Feature Selection (Berger et al., 1996)

111
Conditional Models
  • Conditional probability P(label sequence y
    observation sequence x) rather than joint
    probability P(y, x)
  • Specify the probability of possible label
    sequences given an observation sequence
  • Allow arbitrary, non-independent features on the
    observation sequence X
  • The probability of a transition between labels
    may depend on past and future observations
  • Relax strong independence assumptions in
    generative models

112
Discriminative ModelsMaximum Entropy Markov
Models (MEMMs)
  • Exponential model
  • Given training set X with label sequence Y
  • Train a model ? that maximizes P(YX, ?)
  • For a new data sequence x, the predicted label y
    maximizes P(yx, ?)
  • Notice the per-state normalization

113
MEMMs (contd)
  • MEMMs have all the advantages of Conditional
    Models
  • Per-state normalization all the mass that
    arrives at a state must be distributed among the
    possible successor states (conservation of score
    mass)
  • Subject to Label Bias Problem
  • Bias toward states with fewer outgoing transitions

114
Label Bias Problem
  • Consider this MEMM
  • P(1 and 2 ro) P(2 1 and ro)P(1 ro)
    P(2 1 and o)P(1 r)
  • P(1 and 2 ri) P(2 1 and ri)P(1 ri)
    P(2 1 and i)P(1 r)
  • Since P(2 1 and x) 1 for all x, P(1 and 2
    ro) P(1 and 2 ri)
  • In the training data, label value 2 is the only
    label value observed after label value 1
  • Therefore P(2 1) 1, so P(2 1 and x) 1 for
    all x
  • However, we expect P(1 and 2 ri) to be
    greater than P(1 and 2 ro).
  • Per-state normalization does not allow the
    required expectation

115
Solve the Label Bias Problem
  • Change the state-transition structure of the
    model
  • Not always practical to change the set of states
  • Start with a fully-connected model and let the
    training procedure figure out a good structure
  • Prelude the use of prior, which is very valuable
    (e.g. in information extraction)

116
Random Field
117
Conditional Random Fields (CRFs)
  • CRFs have all the advantages of MEMMs without
    label bias problem
  • MEMM uses per-state exponential model for the
    conditional probabilities of next states given
    the current state
  • CRF has a single exponential model for the joint
    probability of the entire sequence of labels
    given the observation sequence
  • Undirected acyclic graph
  • Allow some transitions vote more strongly than
    others depending on the corresponding observations

118
Definition of CRFs
X is a random variable over data sequences to be
labeled Y is a random variable over corresponding
label sequences
119
Example of CRFs
120
Graphical comparison among HMMs, MEMMs and CRFs
HMM MEMM CRF
121
Conditional Distribution
122
Conditional Distribution (contd)
  • CRFs use the observation-dependent
    normalization Z(x) for the conditional
    distributions

Z(x) is a normalization over the data sequence x
123
Parameter Estimation for CRFs
  • The paper provided iterative scaling algorithms
  • It turns out to be very inefficient
  • Prof. Dietterichs group applied Gradient
    Descendent Algorithm, which is quite efficient

124
Training of CRFs (From Prof. Dietterich)
  • Then, take the derivative of the above equation
  • For training, the first 2 items are easy to get.
  • For example, for each lk, fk is a sequence of
    Boolean numbers, such as 00101110100111.
  • is just the total number of 1s in the
    sequence.
  • The hardest thing is how to calculate Z(x)

125
Training of CRFs (From Prof. Dietterich) (contd)
  • Maximal cliques

126
POS tagging Experiments
127
POS tagging Experiments (contd)
  • Compared HMMs, MEMMs, and CRFs on Penn treebank
    POS tagging
  • Each word in a given input sentence must be
    labeled with one of 45 syntactic tags
  • Add a small set of orthographic features whether
    a spelling begins with a number or upper case
    letter, whether it contains a hyphen, and if it
    contains one of the following suffixes -ing,
    -ogy, -ed, -s, -ly, -ion, -tion, -ity, -ies
  • oov out-of-vocabulary (not observed in the
    training set)

128
Summary
  • Discriminative models are prone to the label bias
    problem
  • CRFs provide the benefits of discriminative
    models
  • CRFs solve the label bias problem well, and
    demonstrate good performance
About PowerShow.com