Introduction To Linear Discriminant Analysis

1
Introduction To Linear Discriminant Analysis
??? ????????????? ?????????????
2
Linear Discriminant Analysis

For a given training sample set,
determine a set of optimal projection axes such
that the set of projective feature vectors of the
training samples has the maximum between-class
scatter and minimum within-class scatter
simultaneously.
3
Linear Discriminant Analysis

Linear Discriminant Analysis seeks a projection
that best separate the data .
Sb between-class scatter matrix Sw
within-class scatter matrix
4
LDA

Fisher discriminant analysis
Sol
5
LDA

Fisher discriminant analysis
where ,
k1k2 and let
6
LDA

Fisher discriminant analysis
7
LDA

Generalized eigenvalue problem.....Theorem 2
Let M be a real symmetric matrix with largest
eigenvalue then and the maximum occurs when ,
i.e. the unit eigenvector
associated with .
Proof
8
LDA

Generalized eigenvalue problem.....proof of
Theorem 2
9
LDA

Generalized eigenvalue problem.....proof of
Theorem 2
Cor
If M is a real symmetric matrix with largest
eigenvalue . And the maximum is achieved
whenever ,where is the unit
eigenvector associated with .
10
LDA

Generalized eigenvalue problem.. Theorem 1
Let Sw and Sb be nn real symmetric matrices .
If Sw is positive definite, then there exists an
nn matrix V which achieves
The real numbers ?1.?n satisfy the generalized
eiegenvalue equation
generalized
eigenvector
generalized eigenvalue
11
LDA

Generalized eigenvalue problem.....proof of
Theorem 1
Let and be the unit
eigenvectors and eigenvalues of Sw, i.e
Now define then
where
Since ri ?0 (Sw is positive definite) , exist
12
LDA

Generalized eigenvalue problem.....proof of
Theorem 1
13
LDA

Generalized eigenvalue problem.....proof of
Theorem 1
We need to claim
(applying a unitary matrix to a whitening process
doesnt affect it!)
(VT)-1 exists since det(VTSwV) det (I ) ?
det(VT) det(Sw) det(V) det(I) Because det(VT)
det(V) ? det(VT)2 det(Sw) 1 gt 0 ? det(VT)
0
14
LDA

Generalized eigenvalue problem.....proof of
Theorem 1
Procedure for diagonalizing Sw (real symmetric
and positive definite) and Sb (real symmetric)
simultaneously is as follows
1. Find ?i by solving And then find
normalized ,
i1,2..,n 2. normalized