# Bayesian Decision Theory (Sections 2.1-2.2) - PowerPoint PPT Presentation

PPT – Bayesian Decision Theory (Sections 2.1-2.2) PowerPoint presentation | free to download - id: 4b0515-YTkxN

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## Bayesian Decision Theory (Sections 2.1-2.2)

Description:

### Bayesian Decision Theory (Sections 2.1-2.2) Decision problem posed in probabilistic terms Bayesian Decision Theory Continuous Features All the relevant probability ... – PowerPoint PPT presentation

Number of Views:105
Avg rating:3.0/5.0
Slides: 89
Category:
Transcript and Presenter's Notes

Title: Bayesian Decision Theory (Sections 2.1-2.2)

1
Bayesian Decision Theory(Sections 2.1-2.2)
• Decision problem posed in probabilistic terms
• Bayesian Decision TheoryContinuous Features
• All the relevant probability values are known

2
(No Transcript)
3
(No Transcript)
4
(No Transcript)
5
Probability Density
6
Course Outline
MODEL INFORMATION
COMPLETE
INCOMPLETE
Supervised Learning
Unsupervised Learning
Bayes Decision Theory
Nonparametric Approach
Parametric Approach
Nonparametric Approach
Parametric Approach
Optimal Rules
Plug-in Rules
Density Estimation
Geometric Rules (K-NN, MLP)
Mixture Resolving
Cluster Analysis (Hard, Fuzzy)
7
Introduction
• From sea bass vs. salmon example to abstract
decision making problem
• State of nature a priori (prior) probability
• State of nature (which type of fish will be
observed next) is unpredictable, so it is a
random variable
• The catch of salmon and sea bass is equiprobable
• P(?1) P(?2) (uniform priors)
• P(?1) P( ?2) 1 (exclusivity and exhaustivity)
• Prior prob. reflects our prior knowledge about
how likely we are to observe a sea bass or
salmon these probabilities may depend on time of
the year or the fishing area!

8
• Bayes decision rule with only the prior
information
• Decide ?1 if P(?1) gt P(?2), otherwise decide ?2
• Error rate Min P(?1) , P(?2)
• Suppose now we have a measurement or feature on
the state of nature - say the fish lightness
value
• Use of the class-conditional probability density
• P(x ?1) and P(x ?2) describe the difference
in lightness feature between populations of sea
bass and salmon

9
Amount of overlap between the densities
determines the goodness of feature
10
• Maximum likelihood decision rule
• P(x ?1) gt P(x ?2), otherwise ?2
• How does the feature x influence our attitude
(prior) concerning the true state of nature?
• Bayes decision rule

11
• Posteriori probability, likelihood, evidence
• P(?j , x) P(?j x)p (x) p(x ?j) P (?j)
• Bayes formula
• P(?j x) p(x ?j) . P (?j) / p(x)
• where
• Posterior (Likelihood. Prior) / Evidence
• Evidence P(x) can be viewed as a scale factor
that guarantees that the posterior probabilities
sum to 1
• P(x ?j) is called the likelihood of ?j with
respect to x the category ?j for which P(x
?j) is large is more likely to be the true
category

12
(No Transcript)
13
• P(?1 x) is the probability of the state of
nature being ?1 given that feature value x has
been observed
• Decision based on the posterior probabilities is
called the Optimal Bayes Decision rule
• For a given observation (feature value) X
• if P(?1 x) gt P(?2 x) decide ?1
• if P(?1 x) lt P(?2 x) decide ?2
• To justify the above rule, calculate the
probability of error
• P(error x) P(?1 x) if we decide ?2
• P(error x) P(?2 x) if we decide ?1

14
• So, for a given x, we can minimize te rob. Of
error, decide ?1 if
• P(?1 x) gt P(?2 x) otherwise decide ?2
• Therefore
• P(error x) min P(?1 x), P(?2 x)
• Thus, for each observation x, Bayes decision rule
minimizes the probability of error
• Unconditional error P(error) obtained by
integration over all x w.r.t. p(x)

15
• Optimal Bayes decision rule
• Decide ?1 if P(?1 x) gt P(?2 x) otherwise
decide ?2
• Special cases
• (i) P(?1) P(?2) Decide ?1 if
• p(x ?1) gt p(x ?2), otherwise ?2
• (ii) p(x ?1) p(x ?2) Decide ?1 if
• P(?1) gt P(?2), otherwise ?2

16
Bayesian Decision Theory Continuous Features
• Generalization of the preceding formulation
• Use of more than one feature (d features)
• Use of more than two states of nature (c classes)
• Allowing other actions besides deciding on the
state of nature
• Introduce a loss function which is more general
than the probability of error

17
• Allowing actions other than classification
primarily allows the possibility of rejection
• Refusing to make a decision when it is difficult
to decide between two classes or in noisy cases!
• The loss function specifies the cost of each
action

18
• Let ?1, ?2,, ?c be the set of c states of
nature
• (or categories)
• Let ?1, ?2,, ?a be the set of a possible
actions
• Let ?(?i ?j) be the loss incurred for taking
action ?i when the true state of nature is ?j
• General decision rule ?(x) specifies which action
to take for every possible observation x

19
• Conditional Risk
• Overall risk
• R Expected value of R(?i x) w.r.t. p(x)
• Minimizing R Minimize R(?i x) for i
1,, a

For a given x, suppose we take the action ?i
if the true state is ?j , we will incur the loss
?(?i ?j). P(?j x) is the prob. that the true
state is ?j But, any one of the C states is
possible for given x.
Conditional risk
20
• Select the action ?i for which R(?i x) is
minimum
• The overall risk R is minimized
and the resulting risk is called the Bayes risk
it is the best performance that can be achieved!

21
• Two-category classification
• ?1 deciding ?1
• ?2 deciding ?2
• ?ij ?(?i ?j)
• loss incurred for deciding ?i when the true state
of nature is ?j
• Conditional risk
• R(?1 x) ??11P(?1 x) ?12P(?2 x)
• R(?2 x) ??21P(?1 x) ?22P(?2 x)

22
• Bayes decision rule is stated as
• if R(?1 x) lt R(?2 x)
• Take action ?1 decide ?1
• This results in the equivalent rule
• decide ?1 if
• (?21- ?11) P(x ?1) P(?1) gt
• (?12- ?22) P(x ?2) P(?2)
• and decide ?2 otherwise

23
• Likelihood ratio
• The preceding rule is equivalent to the following
rule
• then take action ?1 (decide ?1) otherwise take
action ?2 (decide ?2)
• Note that the posteriori porbabilities are scaled
by the loss differences.

24
• Interpretation of the Bayes decision rule
• If the likelihood ratio of class ?1 and class ?2
exceeds a threshold value (that is independent of
the input pattern x), the optimal action is to
decide ?1
• Maximum likelihood decision rule the threshold
value is 1 0-1 loss function and equal class
prior probability

25
Bayesian Decision Theory(Sections 2.3-2.5)
• Minimum Error Rate Classification
• Classifiers, Discriminant Functions and Decision
Surfaces
• The Normal Density

26
Minimum Error Rate Classification
• Actions are decisions on classes
• If action ?i is taken and the true state of
nature is ?j then
• the decision is correct if i j and in error if
i ? j
• Seek a decision rule that minimizes the
probability of error or the error rate

27
• Zero-one (0-1) loss function no loss for correct
decision and a unit loss for any error
• The conditional risk can now be simplified as
• The risk corresponding to the 0-1 loss function
is the average probability of error
• ?

28
• Minimizing the risk requires maximizing the
posterior probability P(?i x) since
• R(?i x) 1 P(?i x))
• For Minimum error rate
• Decide ?i if P (?i x) gt P(?j x) ?j ? i

29
• Decision boundaries and decision regions
• If ? is the 0-1 loss function then the threshold
involves only the priors

30
(No Transcript)
31
Classifiers, Discriminant Functionsand Decision
Surfaces
• Many different ways to represent pattern
classifiers one of the most useful is in terms
of discriminant functions
• The multi-category case
• Set of discriminant functions gi(x), i 1,,c
• Classifier assigns a feature vector x to class ?i
if
• gi(x) gt gj(x) ?j ? i

32
Network Representation of a Classifier
33
• Bayes classifier can be represented in this way,
but the choice of discriminant function is not
unique
• gi(x) - R(?i x)
• (max. discriminant corresponds to min. risk!)
• For the minimum error rate, we take
• gi(x) P(?i x)
• (max. discrimination corresponds to max.
posterior!)
• gi(x) ? P(x ?i) P(?i)
• gi(x) ln P(x ?i) ln P(?i)
• (ln natural logarithm!)

34
• Effect of any decision rule is to divide the
feature space into c decision regions
• if gi(x) gt gj(x) ?j ? i then x is in Ri
• (Region Ri means assign x to ?i)
• The two-category case
• Here a classifier is a dichotomizer that has
two discriminant functions g1 and g2
• Let g(x) ? g1(x) g2(x)
• Decide ?1 if g(x) gt 0 Otherwise decide ?2

35
• So, a dichotomizer computes a single
discriminant function g(x) and classifies x
according to whether g(x) is positive or not.
• Computation of g(x) g1(x) g2(x)

36
(No Transcript)
37
The Normal Density
• Univariate density N(? , ?2)
• Normal density is analytically tractable
• Continuous density
• A number of processes are asymptotically Gaussian
• Patterns (e.g., handwritten characters, speech
signals ) can be viewed as randomly corrupted
versions of a single typical or prototype
(Central Limit theorem)
• where
• ? mean (or expected value) of x
• ?2 variance (or expected squared
deviation) of x

38
(No Transcript)
39
• Multivariate density N(? , ?)
• Multivariate normal density in d dimensions
• where
• x (x1, x2, , xd)t (t stands for
the transpose of a vector)
• ? (?1, ?2, , ?d)t mean vector
• ? dd covariance matrix
• ? and ?-1 are determinant and
inverse of ?, respectively
• The covariance matrix is always symmetric and
positive semidefinite we assume ? is positive
definite so the determinant of ? is strictly
positive
• Multivariate normal density is completely
specified by d d(d1)/2 parameters
• If variables x1 and x2 are statistically
independent then the covariance of x1 and x2
is zero.

40
Multivariate Normal density
Samples drawn from a normal population tend to
fall in a single cloud or cluster cluster center
is determined by the mean vector and shape by the
covariance matrix The loci of points of constant
density are hyperellipsoids whose principal axes
are the eigenvectors of ?
41
Transformation of Normal Variables
Linear combinations of jointly normally
distributed random variables are normally
distributed Coordinate transformation can
convert an arbitrary multivariate normal
distribution into a spherical one
42
Bayesian Decision Theory (Sections 2-6 to 2-9)
• Discriminant Functions for the Normal Density
• Bayes Decision Theory Discrete Features

43
Discriminant Functions for the Normal Density
• The minimum error-rate classification can be
achieved by the discriminant function
• gi(x) ln P(x ?i) ln P(?i)
• In case of multivariate normal densities

44
• Case ?i ?2.I (I is the identity matrix)
• Features are statistically independent and each
feature has the same variance

45
• A classifier that uses linear discriminant
functions is called a linear machine
• The decision surfaces for a linear machine are
pieces of hyperplanes defined by the linear
equations
• gi(x) gj(x)

46
• The hyperplane separating Ri and Rj
• is orthogonal to the line linking the means!

47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
• Case 2 ?i ? (covariance matrices of all
classes are identical but otherwise arbitrary!)
• Hyperplane separating Ri and Rj
• The hyperplane separating Ri and Rj is generally
not orthogonal to the line between the means!
• To classify a feature vector x, measure the
squared Mahalanobis distance from x to each of
the c means assign x to the category of the
nearest mean

51
(No Transcript)
52
(No Transcript)
53
Discriminant Functions for 1D Gaussian
54
• Case 3 ?i arbitrary
• The covariance matrices are different for each
category
• In the 2-category case, the decision surfaces
are hyperquadrics that can assume any of the
general forms hyperplanes, pairs of hyperplanes,
hyperspheres, hyperellipsoids, hyperparaboloids,
hyperhyperboloids)

55
Discriminant Functions for the Normal Density
56
(No Transcript)
57
Discriminant Functions for the Normal Density
58
Discriminant Functions for the Normal Density
59
Decision Regions for Two-Dimensional Gaussian Data
60
Error Probabilities and Integrals
• 2-class problem
• There are two types of errors
• Multi-class problem
• Simpler to computer the prob. of being correct
(more ways to be wrong than to be right)

61
Error Probabilities and Integrals
Bayes optimal decision boundary in 1-D case
62
Error Bounds for Normal Densities
• The exact calculation of the error for the
general Guassian case (case 3) is extremely
difficult
• However, in the 2-category case the general error
can be approximated analytically to give us an
upper bound on the error

63
Error Rate of Linear Discriminant Function (LDF)
• Assume a 2-class problem
• Due to the symmetry of the problem (identical ?),
the two types of errors are identical
• Decide if or
• or

64
Error Rate of LDF
• Let
• Compute expected values variances of
when
• where
• squared Mahalanobis distance between

65
Error Rate of LDF
• Similarly

66
Error Rate of LDF
67
Error Rate of LDF
68
Error Rate of LDF
69
Chernoff Bound
• To derive a bound for the error, we need the
following inequality

Assume conditional prob. are normal
where
70
Chernoff Bound
Chernoff bound for P(error) is found by
determining the value of ? that minimizes
exp(-k(?))
71
Error Bounds for Normal Densities
• Bhattacharyya Bound
• Assume ? 1/2
• computationally simpler
• slightly less tight bound
• Now, Eq. (73) has the form

When the two covariance matrices are equal,
k(1/2) is te same as the Mahalanobis distance
between the two means
72
Error Bounds for Gaussian Distributions
Chernoff Bound
Best Chernoff error bound is 0.008190
Bhattacharya Bound (ß1/2)
2category, 2D data
Bhattacharya error bound is 0.008191
True error using numerical integration 0.0021
73
Neyman-Pearson Rule
Classification, Estimation and Pattern
recognition by Young and Calvert
74
Neyman-Pearson Rule
75
Neyman-Pearson Rule
76
Neyman-Pearson Rule
77
Neyman-Pearson Rule
78
Neyman-Pearson Rule
79
Signal Detection Theory
• We are interested in detecting a single weak
pulse, e.g. radar reflection the internal signal
(x) in detector has mean m1 (m2) when pulse is
absent (present)

The detector uses a threshold x to determine the
presence of pulse
Discriminability ease of determining whether the
pulse is present or not
For given threshold, define hit, false alarm,
miss and correct rejection
80
• Experimentally compute hit and false alarm rates
for fixed x
• Changing x will change the hit and false alarm
rates
• A plot of hit and false alarm rates is called the
ROC curve

Performance shown at different operating points
81
Operating Characteristic
• In practice, distributions may not be Gaussian
and will be multidimensional ROC curve can
still be plotted
• Vary a single control parameter for the decision
rule and plot the resulting hit and false alarm
rates

82
Bayes Decision Theory Discrete Features
• Components of x are binary or integer valued x
can take only one of m discrete values
• v1, v2, ,vm
• Case of independent binary features for
2-category problem
• Let x x1, x2, , xd t where each xi is
either 0 or 1, with probabilities
• pi P(xi 1 ?1)
• qi P(xi 1 ?2)

83
• The discriminant function in this case is

84
Bayesian Decision for Three-dimensional Binary
Data
• Consider a 2-class problem with three
independent binary features class priors are
equal and pi 0.8 and qi 0.5, i 1,2,3
• wi 1.3863
• w0 1.2
• Decision surface g(x) 0 is shown below

Decision boundary for 3D binary features. Left
figure shows the case when pi.8 and qi.5. Right
figure shows case when p3q3 (Feature 3 is not
providing any discriminatory information) so
decision surface is parallel to x3 axis
85
Handling Missing Features
• Suppose it is not possible to measure a certain
feature for a given pattern
• Possible solutions
• Reject the pattern
• Approximate the missing feature
• Mean of all the available values for the missing
feature
• Marginalize over the distribution of the missing
feature

86
Handling Missing Features

87
Other Topics
• Compound Bayes Decision Theory Context
• Consecutive states of nature might not be
statistically independent in sorting two types
of fish, arrival of next fish may not be
independent of the previous fish
• Can we exploit such statistical dependence to
gain improved performance (use of context)
• Compound decision vs. sequential compound
decision problems
• Markov dependence
• Sequential Decision Making
• Feature measurement process is sequential (as in
medical diagnosis)
• Feature measurement cost
• Minimize the no. of features to be measured while
achieving a sufficient accuracy minimize a
combination of feature measurement cost
classification accuracy

88
Context in Text Recognition