Expectation Maximization (EM)

1
Expectation Maximization (EM)

• Northwestern University
• EECS 395/495
• Special Topics in Machine Learning

2
Outline

• Objective
• Simple example
• Complex example

3
Objective

• Learning with missing/unobservable data

B
E
E B A J 1 1 1 1 1 0 1 1 0 0 0
0
A
Maximum likelihood
J
4
Objective

• Learning with missing/unobservable data

B
E
E B A J 1 1 ? 1 1 0 ? 1 0 0 ?
0
A
Optimize what?
J
5
Outline

• Objective
• Simple example
• Complex example

6
Simple example
7
Maximize likelihood
8
Same Problem with Hidden Information
Grade
Hidden
Score

• Observable

9
Same Problem with Hidden Information
10
Same Problem with Hidden Information
11
EM for our example
12
EM Convergence
13
Generalization

• X observable data (score h, c, d)
• z missing data (grade a, b, c, d)
• model parameters to estimate ( )
• E given , compute the expectation of z
• M use z obtained in E step, maximize the
  likelihood with respect to

14
Outline

• Objective
• Simple example
• Complex example

15
Gaussian Mixtures
16
Gaussian Mixtures

• Know
• Data
• -
• -
• Dont know
• Data label
• Objective
• -

17
The GMM assumption
18
The GMM assumption
19
The GMM assumption
20
The GMM assumption
21
The data generated
Label
Coordinates
22
Computing the likelihood
23
EM for GMMs
24
EM for GMMs
25
EM for GMMs
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
Generalization

• X observable data
• z unobservable data
• model parameters to estimate
• E given , compute the expectation of z
• M use z obtained in E step, maximize the
  likelihood with respect to

35
For distributions in exponential family

• Exponential family
• Yes normal, exponential, beta, Bernoulli,
  binomial, multinomial, Poisson
• No Cauchy and uniform
• EM using sufficient statistics
• S1 computing the expectation of the statistics
• S2 set the maximum likelihood

36
What EM really is

• X observable data
• z missing data

• Maximize expected log likelihood
• E-step Determine the expectation
• M-step Maximize the expectation above with
  respect to

37
Final comments

• Deal with missing data/latent variables
• Maximize expected log likelihood
• Local minima