Parameter Estimation

1
Parameter Estimation

• ??????

2
Contents

• Introduction
• Maximum-Likelihood Estimation
• Bayesian Estimation

3
Parameter Estimation

• Introduction

4
Bayesian Rule
We want to estimate the parameters of
class-conditional densities if its parametric
form is known, e.g.,
5
Methods

• The Method of Moments
• Not discussed in this course
• Maximum-Likelihood Estimation
• Assume parameters are fixed but unknown
• Bayesian Estimation
• Assume parameters are random variables
• Sufficient Statistics
• Not discussed in this course

6
Parameter Estimation

• Maximum-Likelihood Estimation

7
Samples
?1
?2
The samples in Dj are drawn independently
according to the probability law p(x?j).
Assume that p(x?j) has a known parametric form
with parameter vector ?j.
?3
e.g.,
?j
8
Goal
The estimated version will be denoted by
?1
?2
Use Dj to estimate the unknown parameter vector ?j
?3
9
Problem Formulation
Because each class is consider individually, the
subscript used before will be dropped.
Now the problem is
10
Criterion of ML
By the independence assumption, we have
Likelihood function
MLE
11
Criterion of ML
Often, we resort to maximize the log-likelihood
function
How?
MLE
12
Criterion of ML
Example
How?
13
Differential Approach if Possible
Find the extreme values using the method in
differential calculus.
Let f(?) be a continuous function, where ?(?1,
?2,, ?n)T.
Gradient Operator
Find the extreme values by solving
14
Preliminary
Let
15
Preliminary
Let
(?xf )T
0
16
The Gaussian Population

• Two cases
• Unknown ?
• Unknown ? and ?

17
The Gaussian Population Unknown ?
18
The Gaussian Population Unknown ?
Set
Sample Mean
19
The Gaussian Population Unknown ? and ?
Consider univariate normal case
20
The Gaussian Population Unknown ? and ?
Consider univariate normal case
unbiased
Set
biased
21
The Gaussian Population Unknown ? and ?
For multivariate normal case
The MLE of ? and ? are
unbiased
biased
22
Unbiasedness
Unbiased Estimator (Absolutely unbiase)
Consistent Estimator (asymptotically unbiased)
23
MLE for Normal Population
Sample Mean
Sample Covariance Matrix
24
Parameter Estimation

• Bayesian Estimation

25
Comparison

• MLE (Maximum-Likelihood Estimation)
• to find the fixed but unknown parameters of a
  population.
• Bayesian Estimation
• Consider the parameters of a population to be
  random variables.

26
Heart of Bayesian Classification
Ultimate Goal
Evaluate
What can we do if prior probabilities and
class-conditional densities are unknown?
27
Helpful Knowledge

• Functional form for unknown densities
• e.g., Normal, exponential,
• Ranges for the values of unknown parameters
• e.g., uniform distributed over a range
• Training Samples
• Sampling according to the states of nature.

28
Posterior Probabilities from Sample
29
Posterior Probabilities from Sample
Each class can be considered independently
30
Problem Formulation
Let D be a set of samples drawn independently
according to the fixed but known distribution
p(x).
We want to determine
This the central problem of Bayesian Learning.
31
Parameter Distribution
Assume p(x) is unknown but knowing it has a fixed
form with parameter vector ?.
is complete known
Assume ? is a random vector, and p(?) is a known
a priori.
32
Class-Conditional Density Estimation
33
Class-Conditional Density Estimation
The posterior density we want to estimate
The form of distribution is assumed known
34
Class-Conditional Density Estimation
35
Class-Conditional Density Estimation
36
The Univariate Gaussian Unknown ?
distribution form is known
assume ? is normal distributed
37
The Univariate Gaussian Unknown ?
38
The Univariate Gaussian Unknown ?
39
The Univariate Gaussian Unknown ?
40
The Univariate Gaussian Unknown ?
41
The Univariate Gaussian p(xD)
42
The Univariate Gaussian p(xD)
43
The Univariate Gaussian p(xD)
?
44
The Multivariate Gaussian Unknown ?
distribution form is known
assume ? is normal distributed
45
The Multivariate Gaussian Unknown ?
46
The Multivariate Gaussian Unknown ?
47
General Theory
1.
the form of class-conditional density is known.
2.
knowledge about the parameter distribution is
available.
samples are randomly drawn according to the
unknown probability density p(x).
3.
48
General Theory
1.
the form of class-conditional density is known.
2.
knowledge about the parameter distribution is
available.
samples are randomly drawn according to the
unknown probability density p(x).
3.
49
Incremental Learning
Recursive
50
Example
1.
2.
3.
51
Example
1.
2.
3.