Natural Language Processing 6 Hidden Markov Models HMMs

1
Natural Language Processing(6)Hidden Markov
Models(HMMs)

• Dr. Xuan Wang(? ?)
• Intelligence Computing Research Center
• Harbin Institute of Technology Shenzhen Graduate
  School
• Slides from Dr. Mary P. Harper
• ECE, Purdue University

2
Markov Assumptions
3
Definition of a Markov Chain
4
Markov Chain
5
Markov Process LanguageModels
6
MM

• Bigram MM(??MM)

7
MM

• Trigram MM(??MM)

8
Andrei A. Markov

• Andrei A. Markov
• Russian statistician
• 1856 1922

9
Hidden Markov Models (HMMs)
10
What is an HMM?
11
What is an HMM?
12
Elements of an HMM
13
HMM Formalism
14
HMM Formalism
15
Example of an Arc Emit HMM
16
HMMs
17
Example HMM
18
Example HMM
19
Simulating a Markov Process
20
Why Use Hidden Markov Models?
21
Fundamental Questions for HMMs
22
Probability of an Observation
23
Probability of an Observation
24
Probability of an Observation
25
Probability of an Observation
26
Probability of an Observation
27
Probability of an Observation(State Emit)
28
Probability of an Observation with an Arc Emit HMM
29
Making Computation Efficient
30
Trellis or Lattice(??)
31
Emit probability equals to 1
32
Emit probability NOT equals to 1
33
Forward Procedure
34
Forward Procedure
35
Forward Procedure
36
Forward Procedure
37
Forward Procedure
38
Forward Procedure
39
Forward Procedure
40
Forward Procedure
41
Forward Procedure
42
The Forward Procedure
43
A Example
44
The Backward Procedure
45
Backward Procedure
46
The Backward Procedure
47
Probability of an Observation
48
Finding the Best State Sequence
49
Best State Sequence
50
Finding the Best State Sequence The Viterbi
Algorithm
51
Viterbi Algorithm
52
Viterbi Algorithm
53
Finding the Best State SequenceThe Viterbi
Algorithm
54
Viterbi Algorithm
55
Andrew J. Viterbi

• Communications pioneer Andrew J.Viterbi who in
  1962 earned one of the first doctorates in
  electrical engineering granted at the University
  of Southern California has forever changed how
  people everywhere connect and communicate,
  whether from across a crowded city, between
  nations or from the infinite reaches of space.
• Born into an analog world, this visionary thinker
  opened the doors to the digital age with the
  Viterbi Algorithm, a groundbreaking mathematical
  formula for eliminating signal interference.
  Today, the Viterbi Algorithm is used in all four
  international standards for digital cellular
  telephones, as well as in data terminals, digital
  satellite broadcast receivers and deep space
  telemetry.
• Dr.Viterbis lifelong interest in communications
  began as a child, when his family fled Italy for
  America in 1939 to escape the persecution of
  Jews. Long absences from family members instilled
  a desire to find ways of communicating across
  political and geographical borders.

56
Example
57
Parameter Estimation
58
Parameter Estimation The Forward-Backward
Algorithm
59
Parameter Estimation
60
Definitions
61
The Probability of Traversing an Arc
62
Definitions
63
Reestimation Procedure
64
The Reestimation Formulas
65
Reestimation Formulas
66
Parameter Estimation
67
Parameter Estimation
68
Worlds Superstar2000 Turing Award Winners

• Andrew Chi-Chih Yao
• (???)
• In recognition of his fundamental contributions
  to the theory of computation, including the
  complexity-based theory of pseudorandom number
  generation, cryptography, and communication
  complexity.

69

• 1. Biographical and Personal Information
• Born December 24, 1946, Shanghai, China.
• Citizenship U.S.A.
• Mailing Address Computer Science Department,
  Princeton University, Princeton, NJ 08544, USA
• Email yao_at_cs.princeton.edu
• 2. Education
• National Taiwan University, B.S., Physics, 1967.
• Harvard University, A.M., Physics, 1969 Ph.D.,
  Physics, 1972.
• University of Illinois, Ph.D., Computer Science,
  1975.

70

• 3. Employment Record
• Assistant Professor, Mathematics Department,
  Massachusetts Institute of Technology,
  9/19758/1976.
• Assistant Professor, Computer Science Department,
  Stanford University, 9/19768/1981.
• Professor, Computer Science Division, University
  of California, Berkeley, 9/19819/1982.
• Professor, Computer Science Department, Stanford
  University, 10/19826/1986.
• William and Edna Macaleer Professor of
  Engineering and Applied Science, Computer Science
  Department,
• Princeton University, 7/19868/2004.

71
Andrew Chi-Chih Yao
72
(No Transcript)
73
(No Transcript)
74

• Thanks!