Loading...

PPT – Multivariate Statistics PowerPoint presentation | free to download - id: 688427-MmIwO

The Adobe Flash plugin is needed to view this content

Multivariate Statistics

- Matrix Algebra II
- W. M. van der Veld
- University of Amsterdam

Overview

- The determinant of a matrix
- The matrix inverse
- System of equations

The determinant of a matrix

- The determinant of a matrix is a scalar and is

denoted as A or det(A). Det(A) only exists

when A is a square matrix. - It has very important mathematical properties,

but it is very difficult to provide a substantive

definition. - The determinant is necessary to compute the

inverse of a matrix (A-1). - The inverse of a matrix is needed for solving

systems of linear equations multivariate

statistics often comes down to this. - When the determinant is zero, there exists no

solution to a system of linear equations. - Lets see how the value of the determinant is

found.

The determinant of a matrix

- How to do it? The most simple case, a 2 by 2

matrix . - Det(A)A?

The determinant of a matrix

- One step further, a 3 by 3 matrix.
- Det(A)A?

Cofactor

The determinant of a matrix

- You should have noted that for matrices larger

than first order, computation of the determinant

is a recursive process. This process stops each

time a 1 by 1 determinant is encountered, and

involves multiplication by the cofactors.

The determinant of a matrix

- Let A be a matrix of order n x n. If we omit one

or more rows or columns from A, we obtain a

matrix of smaller order, called a minor of the

matrix. - Similarly, we have minors of a determinant, and

in particular, if we omit from the determinant

the ith row and the jth column, the resulting

minor will be square and its determinant will be

symbolized Mij. This determinant is called a

cofactor (cij) if we give it a sign equal to

(-1)ij, so that cij (-1)ij Mij. Using

this notation we can write a formula for the

expansion of a determinant of order n

In this version the determinant is expanded

according to its ith row.

The determinant of a matrix

- The following rules are important for

determinants, and can help you sometimes to

simplify calculations - The determinant of A has the same value as the

determinant of A. - The value of the determinant changes sign if one

row (column) is interchanged with another row

(column). - If a determinant has two equal rows (columns),

its value is zero. - If a determinant has two rows (columns) with

proportional elements, its value is zero. - If all elements in a row (column) are multiplied

by a constant, the value of the determinant is

multiplied by that constant. - If a determinant has a row (column) in which all

elements are zero, the value of the determinant

is zero. - The value of the determinant remains unchanged if

one row (column) is added to or subtracted form

another row (column). Moreover, if a row (column)

is multiplied by a constant and then added to or

subtracted from another row (column) the value

remains unchanged.

The determinant of a matrix

- What is the determinant of

The matrix inverse

- Let A be a square matrix. If we can find a matrix

B of the same order as A such that ABBAI, then

B is said to be the inverse of A and is

symbolized A-1. A-1, if exists, can be found as

follows. - Let C be the matrix of cofactors of A (i.e., cij

is the cofactor obtained from the minor Mij)

then

- Where C is the transpose of C (or if one

prefers, C is the matrix of cofactors of A). It

is immediately seen that the inverse is undefined

if A is not square (since then there is no

determinant A), and also if A is equal to

zero.

The matrix inverse

- Illustration that AA-1 A-1A I.

The matrix inverse

- How did I get A-1?

Now Compute C

C transpose gt C

Calculate A-1

The matrix inverse

- Another way to calculate A-1. This way introduces

you to solving systems of equations.

The matrix inverse

- Rules for algebra with inverse matrices
- AA-1 A-1A I
- (AB)-1 B-1A-1
- (ABC)-1 C-1B-1A-1
- Proof that (AB)-1 B-1A-1.

System of equations

- In the introduction I already mentioned that the

basic linear equation ybx will be very important

for multivariate methods. - Here we will discuss how to solve systems of such

linear equations.

System of equations

- Illustration. Suppose we have the following set

of equations -31x14x2 13x12x2 - The basic way to think about this problem set is

finding the intersection, i.e. for which unknowns

are the equations satisfied. - This can be solved in a simple way (old style).

- The solution is basically the intersection of the

lines represented by the equation. - You wont be surprised that there is a more

general way to solve systems of linear equations,

using matrix algebra.

System of equations

- We shall distinguish between homogeneous and

nonhomogeneous equations. - A homogeneous equation is of type ax0,
- With a and x being vector of order nx1 a is a

vector of coefficients, and x a vector of

unknowns. - Eg., if a(2 -3) and x(x1 x2), then ax0

gives a homogeneous equation with two unknowns

2x1-3x20. - A nonhomogeneous equation is of type axk, with

k some constant (?0). - Eg., 2x1-3x22.

System of equations

- Geometrically equations with two unknowns

represent lines in a plane defined by the axes x1

and x2.

- Similarly, an equation with three unknowns,

3x13x25x34, represents a plane in an S3 - The space S3 is defined by the three coordinates.

Each triplet of values that satisfies the

equation refers to a point located on the plane

within S3.

System of equations

- This can be generalized, so that an equation with

n unknowns will stand for an (n-1) dimensional

subspace Sn. - Such a subspace is called a hyperplane,

symbolized by the letter V. So that Vn-1 is a

hyperplane in subspace Sn., but Sn will also of

course contain the hyperplanes of a lower order. - Recognize that principal component analyses is

based upon this idea, except that the axis in the

hyperplane are called factors or better principal

components.

System of equations

- Rules for the intersection of hyperplanes
- In general, two lines (V2) will intersect in a

point (V1), except for parallel or coinciding

lines. - Also two planes (V3) will intersect in a line

(V2), except for similar exceptions. - The general rule is in Sn two hyperplanes Vn-p

and Vn-q intersect in a Vn-(pq). In addition,

when pqn, then the intersection is a point,

when pqgtn then there is no intersection. - What does the rule mean for equations?
- That, in general there will be a unique solution

only for n equations in n unknowns.

System of equations

- Solution for n nonhomogeneous equations with n

unknowns

- What to do? Normally you divide by A so that you

obtain a solution for x (give example 153x). - Matrix division is defined as multiplication by

the inverse, so

System of equations

- Solution for m equations with n unknowns mn.

- What to do? Normally you divide by A so that you

obtain a solution for x (give example 153x). - Matrix division is defined as multiplication by

the inverse, so

System of equations

- Example. Suppose we have the following set of

equations -31x14x2 13x12x2 - We already solved this one, resulting in x11 and

x2-1. - The set of equations can be written as a matrix

operation.

System of equations

- Thus, we have to find the inverse of A gt A-1

C/A

- We have to take the transpose of C

System of equations

- We have to divide by A.

- Thus the inverse matrix is.

System of equations

- Thus a solution for -31x14x2 13x12x2 is

found via

System of equations

- Solution for n-1 homogeneous equations with n

unknowns

- There are many solutions. Why?
- Because unknownsgtknowns or more xs than ns.
- The intersection (solution) therefore will

represent a hyperplane, and all points in the

hyperplane satisfy the equation. - How to find this hyperplane?

System of equations

- How to find this hyperplane?
- One solution can be found when one of the

unknowns is fixed to an arbitrary value k, with

k?0. - The resulting set of equations are then solvable

using A-1kx. - The solution then is x (x1 x2 .. xnk).
- However, all vectors proportional to this vector

will also satisfy the equation. - Hence the general solution is cx, with c an

arbitrary constant, and x is column vector

including the coordinates of the intersection. - The solution includes c0, because 0 is also a

solution. - This is very abstract, so.

System of equations

- An illustration.
- The problems set is n-1 homogeneous equations in

n unknowns. - Two equations and three unknowns.

2x1-3x21x30 4x11x2-2x30 - Now fix x3 to an arbitrary value k1.
- Therefore 2x1-3x210 gt 2x1-3x2-1

4x11x2-20 gt 4x11x22 - Now there are two nonhomogeneous equations with

two unknowns, which is solvable using A-1kx

System of equations

- Thus a solution will be

- Note that x31

- Now any vector proportional with this vector will

satisfy the set of equations e.g. x(5 8 14) - All these vectors are considered one solution,

and they form in this case a hyperplane V2.

System of equations

Exercise, solve x1 2x2 0 3x1 7x2 1

A-1Ax Ix x A-1k

- So if Ax k solve via x A-1k.
- .... But it is not always so simple

System of equations

- Sometimes, the requirement that mn seems to be

fulfilled, so that there should exist a solution. - But consider the following cases.

(Row 2 2 x Row 1)

(Row 3 Row 1 Row 2)

(Column 3 Column 1 Column 2), etc.

System of equations

- These situations are called linear dependence
- Given vectors x1, x2,, xn-1
- Another vector xn is linearly dependent if there

exists constants a1, a2,, an-1 such that xn

a1x1a2x2 an-1xn-1 - Otherwise the vector xn is linearly independent.
- In case of linear dependence A 0.
- And then the inverse is not defined A-1C/A.
- And when the inverse is not defined we cannot

find a solution via A-1kx.

System of equations

- Generally a unique solution exists only if mn,

and A?0 - When are there problems?
- If mltn there are many solutions, the problem is

underdetermined. 8x110x214x39 4x112x216x310

- if mgtn there are no solutions, the problem is

overdetermined. 8x110x29 4x112x210 4x110x22

System of equations

- Using the idea of linear dependency, the rank of

a matrix can be introduced. - rank(A) number of linearly independent rows or

columns. - Given an mxn matrix, with m n, then if
- A ? 0 ? rank(A) n ? full rank, solvable
- A 0 ? rank(A) lt n ? rank deficient
- We will get back to the issue of rank.

Overdetermined Systems

- Find Ax closest to k
- Least-squares distance measure

- Minimization problem
- Normal equations (AA)x Ak
- Solution x (AA)-1Ak
- AA must be nonsingular i.e. AA?0
- (AA)-1A is called the left inverse matrix

Underdetermined Systems

- Find smallest x that satisfies equations
- Minimum norm objective

- Solution x A(AA)-1k
- AA must be nonsingular
- A(AA)-1 is called the right inverse