Course Review Part 1

1
Course Review (Part 1)

• LING 572
• Fei Xia
• 1/19/06

2
Outline

• Recap
• Homework 1
• Project Part 1

3
Recap
4
Recap

• FSA and HMM
• DT, DL, TBL

5
A learning algorithm

• Modeling
• Representation
• Decomposition
• Parameters
• Properties
• Training
• Simple counting, hill-climbing, greedy algorithm,
  
• Pruning and filtering
• Smoothing issues

6
A learning algorithm (cont)

• Decoding
• Simply verify condition DT, DL, TBL
• Viterbi FSA and HMM
• Pruning during the search
• Relation with other algorithms
• Ex DNF, CNF, DT, DL and TBL
• Ex WFA and HMM, PFA and HMM

7
NLP task

• Choose a ML method e.g., DT, TBL
• Modeling
• Ex TBL What kinds of features?
• Ex HMM What are the states? What are the output
  symbols?
• Training e.g., DT
• Select a particular algorithm ID3, C4.5
• Choose pruning/filtering/smoothing strategies,
  thresholds, quality measures, etc.
• Decoding
• Pruning strategies

8
Homework 1
9
Hw1

• Problem 3 4 State-emission and arc-emission
  HMMs.
• Problem 5 Viterbi algorithm
• Problem 2 HMM
• Problem 1 FSA

10
Problem 3 State-emission HMM ? Arc-emission HMM
(a)
(b)
Given a path X1, X2, ..., Xn1 in HMM1 ? The path
in HMM2 is X1, X2, ..., Xn1.
11
Problem 3 (cont)
(c)
12
Problem 4 Arc-emission HMM ? state-emission HMM
(a)
13
Problem 4 (cont)
(b) Given a path X1, X2, ., Xn1 in HMM1,
the path in HMM2 is X1_X1, X1_X2, ., Xn_Xn1
(c)
14
Problem 5 Viterbi algorithm with e-emission
15
Problem 5 (cont)
Cost(i, j) is the max prob for a path from i to j
which produces nothing. To calculate
Cost(i, j), let

where N is the number of states in HMM.
16
Problems 1 2 Important tricks
Constants can be moved outside the sum signs
17
Tricks (cont)

• The order of sums can be changed

18
Tricks (cont)

• The order of sum and product

19
Problem 2 HMM

• Prove by induction
• When the length is 0
• When the length is n-1, we assume that

20
Problem 2 (cont)
21
Problem 1 FSA
22
Problem 1 (cont)
...
23
Project Part 1
24
Carmel a WFA package
WFA
Input/output symbols
Carmel
best path
25
Bigram tagging

• FST1
• Initial states BOS
• Final states EOS
• FST2

26
Trigram tagging
t2 P(t2 t1,t0)

• FST1
• Initial state BOS-BOS
• Final state EOS-EOS
• FST2

t0t1
t1t2
27
Minor details

• BOS and EOS
• No need for special treatment for BOS
• EOS
• Add two EOSs at the end of a sentence, or
• Replace input symbol EOS with e (a.k.a. e).

28
Results