Multivariate Statistics - PowerPoint PPT Presentation


PPT – Multivariate Statistics PowerPoint presentation | free to download - id: 688427-MmIwO


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Multivariate Statistics


The determinant of a matrix You should have noted that for matrices larger ... rules are important for determinants, ... is needed for solving systems of ... – PowerPoint PPT presentation

Number of Views:4
Avg rating:3.0/5.0
Date added: 1 April 2020
Slides: 29
Provided by: Wvande5


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Multivariate Statistics

Multivariate Statistics
  • Matrix Algebra II
  • W. M. van der Veld
  • University of Amsterdam

  • The determinant of a matrix
  • The matrix inverse
  • System of equations

The determinant of a matrix
  • The determinant of a matrix is a scalar and is
    denoted as A or det(A). Det(A) only exists
    when A is a square matrix.
  • It has very important mathematical properties,
    but it is very difficult to provide a substantive
  • The determinant is necessary to compute the
    inverse of a matrix (A-1).
  • The inverse of a matrix is needed for solving
    systems of linear equations multivariate
    statistics often comes down to this.
  • When the determinant is zero, there exists no
    solution to a system of linear equations.
  • Lets see how the value of the determinant is

The determinant of a matrix
  • How to do it? The most simple case, a 2 by 2
    matrix .
  • Det(A)A?

The determinant of a matrix
  • One step further, a 3 by 3 matrix.
  • Det(A)A?

The determinant of a matrix
  • You should have noted that for matrices larger
    than first order, computation of the determinant
    is a recursive process. This process stops each
    time a 1 by 1 determinant is encountered, and
    involves multiplication by the cofactors.

The determinant of a matrix
  • Let A be a matrix of order n x n. If we omit one
    or more rows or columns from A, we obtain a
    matrix of smaller order, called a minor of the
  • Similarly, we have minors of a determinant, and
    in particular, if we omit from the determinant
    the ith row and the jth column, the resulting
    minor will be square and its determinant will be
    symbolized Mij. This determinant is called a
    cofactor (cij) if we give it a sign equal to
    (-1)ij, so that cij (-1)ij Mij. Using
    this notation we can write a formula for the
    expansion of a determinant of order n

In this version the determinant is expanded
according to its ith row.
The determinant of a matrix
  • The following rules are important for
    determinants, and can help you sometimes to
    simplify calculations
  • The determinant of A has the same value as the
    determinant of A.
  • The value of the determinant changes sign if one
    row (column) is interchanged with another row
  • If a determinant has two equal rows (columns),
    its value is zero.
  • If a determinant has two rows (columns) with
    proportional elements, its value is zero.
  • If all elements in a row (column) are multiplied
    by a constant, the value of the determinant is
    multiplied by that constant.
  • If a determinant has a row (column) in which all
    elements are zero, the value of the determinant
    is zero.
  • The value of the determinant remains unchanged if
    one row (column) is added to or subtracted form
    another row (column). Moreover, if a row (column)
    is multiplied by a constant and then added to or
    subtracted from another row (column) the value
    remains unchanged.

The determinant of a matrix
  • What is the determinant of

The matrix inverse
  • Let A be a square matrix. If we can find a matrix
    B of the same order as A such that ABBAI, then
    B is said to be the inverse of A and is
    symbolized A-1. A-1, if exists, can be found as
  • Let C be the matrix of cofactors of A (i.e., cij
    is the cofactor obtained from the minor Mij)
  • Where C is the transpose of C (or if one
    prefers, C is the matrix of cofactors of A). It
    is immediately seen that the inverse is undefined
    if A is not square (since then there is no
    determinant A), and also if A is equal to

The matrix inverse
  • Illustration that AA-1 A-1A I.

The matrix inverse
  • How did I get A-1?

Now Compute C
C transpose gt C
Calculate A-1
The matrix inverse
  • Another way to calculate A-1. This way introduces
    you to solving systems of equations.

The matrix inverse
  • Rules for algebra with inverse matrices
  • AA-1 A-1A I
  • (AB)-1 B-1A-1
  • (ABC)-1 C-1B-1A-1
  • Proof that (AB)-1 B-1A-1.

System of equations
  • In the introduction I already mentioned that the
    basic linear equation ybx will be very important
    for multivariate methods.
  • Here we will discuss how to solve systems of such
    linear equations.

System of equations
  • Illustration. Suppose we have the following set
    of equations -31x14x2 13x12x2
  • The basic way to think about this problem set is
    finding the intersection, i.e. for which unknowns
    are the equations satisfied.
  • This can be solved in a simple way (old style).
  • The solution is basically the intersection of the
    lines represented by the equation.
  • You wont be surprised that there is a more
    general way to solve systems of linear equations,
    using matrix algebra.

System of equations
  • We shall distinguish between homogeneous and
    nonhomogeneous equations.
  • A homogeneous equation is of type ax0,
  • With a and x being vector of order nx1 a is a
    vector of coefficients, and x a vector of
  • Eg., if a(2 -3) and x(x1 x2), then ax0
    gives a homogeneous equation with two unknowns
  • A nonhomogeneous equation is of type axk, with
    k some constant (?0).
  • Eg., 2x1-3x22.

System of equations
  • Geometrically equations with two unknowns
    represent lines in a plane defined by the axes x1
    and x2.
  • Similarly, an equation with three unknowns,
    3x13x25x34, represents a plane in an S3
  • The space S3 is defined by the three coordinates.
    Each triplet of values that satisfies the
    equation refers to a point located on the plane
    within S3.

System of equations
  • This can be generalized, so that an equation with
    n unknowns will stand for an (n-1) dimensional
    subspace Sn.
  • Such a subspace is called a hyperplane,
    symbolized by the letter V. So that Vn-1 is a
    hyperplane in subspace Sn., but Sn will also of
    course contain the hyperplanes of a lower order.
  • Recognize that principal component analyses is
    based upon this idea, except that the axis in the
    hyperplane are called factors or better principal

System of equations
  • Rules for the intersection of hyperplanes
  • In general, two lines (V2) will intersect in a
    point (V1), except for parallel or coinciding
  • Also two planes (V3) will intersect in a line
    (V2), except for similar exceptions.
  • The general rule is in Sn two hyperplanes Vn-p
    and Vn-q intersect in a Vn-(pq). In addition,
    when pqn, then the intersection is a point,
    when pqgtn then there is no intersection.
  • What does the rule mean for equations?
  • That, in general there will be a unique solution
    only for n equations in n unknowns.

System of equations
  • Solution for n nonhomogeneous equations with n
  • What to do? Normally you divide by A so that you
    obtain a solution for x (give example 153x).
  • Matrix division is defined as multiplication by
    the inverse, so

System of equations
  • Solution for m equations with n unknowns mn.
  • What to do? Normally you divide by A so that you
    obtain a solution for x (give example 153x).
  • Matrix division is defined as multiplication by
    the inverse, so

System of equations
  • Example. Suppose we have the following set of
    equations -31x14x2 13x12x2
  • We already solved this one, resulting in x11 and
  • The set of equations can be written as a matrix

System of equations
  • Thus, we have to find the inverse of A gt A-1
  • We have to take the transpose of C

System of equations
  • We have to divide by A.
  • Thus the inverse matrix is.

System of equations
  • Thus a solution for -31x14x2 13x12x2 is
    found via

System of equations
  • Solution for n-1 homogeneous equations with n
  • There are many solutions. Why?
  • Because unknownsgtknowns or more xs than ns.
  • The intersection (solution) therefore will
    represent a hyperplane, and all points in the
    hyperplane satisfy the equation.
  • How to find this hyperplane?

System of equations
  • How to find this hyperplane?
  • One solution can be found when one of the
    unknowns is fixed to an arbitrary value k, with
  • The resulting set of equations are then solvable
    using A-1kx.
  • The solution then is x (x1 x2 .. xnk).
  • However, all vectors proportional to this vector
    will also satisfy the equation.
  • Hence the general solution is cx, with c an
    arbitrary constant, and x is column vector
    including the coordinates of the intersection.
  • The solution includes c0, because 0 is also a
  • This is very abstract, so.

System of equations
  • An illustration.
  • The problems set is n-1 homogeneous equations in
    n unknowns.
  • Two equations and three unknowns.
    2x1-3x21x30 4x11x2-2x30
  • Now fix x3 to an arbitrary value k1.
  • Therefore 2x1-3x210 gt 2x1-3x2-1
    4x11x2-20 gt 4x11x22
  • Now there are two nonhomogeneous equations with
    two unknowns, which is solvable using A-1kx

System of equations
  • Thus a solution will be
  • Note that x31
  • Now any vector proportional with this vector will
    satisfy the set of equations e.g. x(5 8 14)
  • All these vectors are considered one solution,
    and they form in this case a hyperplane V2.

System of equations
Exercise, solve x1 2x2 0 3x1 7x2 1
A-1Ax Ix x A-1k
  • So if Ax k solve via x A-1k.
  • .... But it is not always so simple

System of equations
  • Sometimes, the requirement that mn seems to be
    fulfilled, so that there should exist a solution.
  • But consider the following cases.

(Row 2 2 x Row 1)
(Row 3 Row 1 Row 2)
(Column 3 Column 1 Column 2), etc.
System of equations
  • These situations are called linear dependence
  • Given vectors x1, x2,, xn-1
  • Another vector xn is linearly dependent if there
    exists constants a1, a2,, an-1 such that xn
    a1x1a2x2 an-1xn-1
  • Otherwise the vector xn is linearly independent.
  • In case of linear dependence A 0.
  • And then the inverse is not defined A-1C/A.
  • And when the inverse is not defined we cannot
    find a solution via A-1kx.

System of equations
  • Generally a unique solution exists only if mn,
    and A?0
  • When are there problems?
  • If mltn there are many solutions, the problem is
    underdetermined. 8x110x214x39 4x112x216x310
  • if mgtn there are no solutions, the problem is
    overdetermined. 8x110x29 4x112x210 4x110x22

System of equations
  • Using the idea of linear dependency, the rank of
    a matrix can be introduced.
  • rank(A) number of linearly independent rows or
  • Given an mxn matrix, with m n, then if
  • A ? 0 ? rank(A) n ? full rank, solvable
  • A 0 ? rank(A) lt n ? rank deficient
  • We will get back to the issue of rank.

Overdetermined Systems
  • Find Ax closest to k
  • Least-squares distance measure
  • Minimization problem
  • Normal equations (AA)x Ak
  • Solution x (AA)-1Ak
  • AA must be nonsingular i.e. AA?0
  • (AA)-1A is called the left inverse matrix

Underdetermined Systems
  • Find smallest x that satisfies equations
  • Minimum norm objective
  • Solution x A(AA)-1k
  • AA must be nonsingular
  • A(AA)-1 is called the right inverse