EM Algorithm

1
EM Algorithm

• ??????

2
Contents

• Introduction
• Example ? Missing Data
• Example ? Mixed Attributes
• Example ? Mixture
• Main Body
• Mixture Model
• EM-Algorithm on GMM

3
EM Algorithm

• Introduction

4
Introduction

• EM is typically used to compute maximum
  likelihood estimates given incomplete samples.
• The EM algorithm estimates the parameters of a
  model iteratively.
• Starting from some initial guess, each iteration
  consists of
• an E step (Expectation step)
• an M step (Maximization step)

5
Applications

• Filling in missing data in samples
• Discovering the value of latent variables
• Estimating the parameters of HMMs
• Estimating parameters of finite mixtures
• Unsupervised learning of clusters

6
EM Algorithm

• Example ?
• Missing Data

7
Univariate Normal Sample
Sampling
8
Maximum Likelihood
Sampling
We want to maximize it.
Given x, it is a function of ? and ?2
9
Log-Likelihood Function
Maximize this instead
By setting
and
10
Max. the Log-Likelihood Function
11
Max. the Log-Likelihood Function
12
Miss Data
Missing data
Sampling
13
E-Step
be the estimated parameters at the initial of the
tth iterations
Let
14
E-Step
be the estimated parameters at the initial of the
tth iterations
Let
15
M-Step
be the estimated parameters at the initial of the
tth iterations
Let
16
Exercise
17
EM Algorithm

• Example ?
• Mixed Attributes

18
Multinomial Population
Sampling
N samples
19
Maximum Likelihood
Sampling
N samples
20
Maximum Likelihood
Sampling
N samples
We want to maximize it.
21
Log-Likelihood
22
Mixed Attributes
Sampling
N samples
x3 is not available
23
E-Step
Sampling
N samples
x3 is not available
Given ?(t), what can you say about x3?
24
M-Step
25
Exercise
Estimate ? using different initial conditions?
26
EM Algorithm

• Example Mixture

27
Binomial/Poison Mixture
M married obasong
X Children
n0
Obasongs
28
Binomial/Poison Mixture
M married obasong
X Children
n0
Obasongs
Unobserved data
nA married Obs nB unmarried Obs
29
Binomial/Poison Mixture
M married obasong
X Children
n0
Obasongs
Complete data
30
Binomial/Poison Mixture
n0
Obasongs
Complete data
31
Complete Data Likelihood
n0
Obasongs
Complete data
32
Complete Data Likelihood
n0
Obasongs
Complete data
33
Log-Likelihood
34
Maximization
35
Maximization
36
E-Step
Given
37
M-Step
38
Example
39
EM Algorithm

• Main Body

40
Maximum Likelihood
41
Latent Variables
Incomplete Data
Complete Data
42
Complete Data Likelihood
43
Complete Data Likelihood
A function of latent variable Y and
parameter ?
A function of parameter ?
A function of random variable Y.
The result is in term of random variable Y.
Computable
If we are given ?,
44
Expectation Step
Let ?(i?1) be the parameter vector obtained at
the (i?1)th step.
Define
45
Maximization Step
Let ?(i?1) be the parameter vector obtained at
the (i?1)th step.
Define
46
EM Algorithm

• Mixture Model

47
Mixture Models

• If there is a reason to believe that a data set
  is comprised of several distinct populations, a
  mixture model can be used.
• It has the following form

with
48
Mixture Models
Let yi?1,, M represents the source that
generates the data.
49
Mixture Models
Let yi?1,, M represents the source that
generates the data.
50
Mixture Models
51
Mixture Models
52
Mixture Models
Given x and ?, the conditional density of y can
be computed.
53
Complete-Data Likelihood Function
54
Expectation
?g Guess
55
Expectation
?g Guess
56
Expectation
Zero when yi ? l
57
Expectation
58
Expectation
59
Expectation
1
60
Maximization
Given the initial guess ?g,
We want to find ?, to maximize the above
expectation.
In fact, iteratively.
61
The GMM (Guassian Mixture Model)
Guassian model of a d-dimensional source, say j
GMM with M sources
62
EM Algorithm

• EM-Algorithm on GMM

63
Goal
Mixture Model
subject to
To maximize
64
Goal
Mixture Model
Correlated with ?l only.
Correlated with ?l only.
subject to
To maximize
65
Finding ?l
Due to the constraint on ?ls, we introduce
Lagrange Multiplier ?, and solve the following
equation.
66
Finding ?l
1
N
1
67
Finding ?l
68
Finding ?l
Only need to maximize this term
Consider GMM
unrelated
69
Finding ?l
Only need to maximize this term
Therefore, we want to maximize
How?
knowledge on matrix algebra is needed.
unrelated
70
Finding ?l
Therefore, we want to maximize
71
Summary
EM algorithm for GMM
Given an initial guess ?g, find ?new as follows
Not converge
72
Demonstration
EM algorithm for Mixture models
73
Exercises

• Write a program to generate multidimensional
  Gaussian distribution.
• Draw the distribution for 2-dim data.
• Write a program to generate GMM.
• Write EM-algorithm to analyze GMM data.
• Study more EM-algorithm for mixture.
• Find applications for EM-algorithm.

74
References

• A Gentle Tutorial of the EM Algorithm and its
  Application to Parameter Estimation for Gaussian
  Mixture and Hidden Markov Models (1998), Jeff
  Bilmes
• The Expectation Maximization Algorithm A short
  tutorial, Sean Borman.