Elementary Linear Algebra

1
Elementary Linear Algebra

• Additional Topics

2
Content

• LU-Decompositions
• Least Squares Fitting to Data
• Quadratic Forms
• Diagonalizing Quadratic Forms Conic Sections
• Quadratic Surfaces

3
Solving Linear Systems by Factoring

• If an n?n matrix A can be factored into a product
  of n?n matrices as
• A LU
• where L is lower triangular and U is upper
  triangular, the linear system Ax b can be
  solved as follows
• Rewrite the system Ax b as LUx b
• Define a new n?1 matrix y by Ux y
• Solve the system Ly b for y
• Solve the system Ux y for x

4
Solving Linear Systems by Factoring

• This procedure replaces the problem of solving
  the single system
• Ax b
• by the problem of solving two systems
• Ly b and Ux y.
• However, the latter systems are easy to solve
  since the coefficient matrices are triangular.

5
Example

• Solve
• Solution

6
LU-Decompositions

• Suppose that an n?n matrix A has been reduced to
  a row-echelon form U by a sequence of elementary
  row operations, then each of these operations can
  be accomplished by multiplying on the left by an
  appropriate elementary matrix.
• Thus, we can find elementary matrices E1, E2, ,
  Ek such that
• Ek E2 E1 A U

7
LU-Decompositions

• Since Eis are invertible, we have A E1-1 E2-1
  Ek-1 U
• Since a product of lower triangular matrices is
  also lower triangular, the matrix L defined by
• L E1-1 E2-1 Ek-1
• is lower triangular provided that no row
  interchanges are used in reducing A to U.
• Thus, we have A LU, which is a factorization of
  A into a product of a lower triangular matrix and
  a upper triangular matrix.

8
LU-Decompositions

• Theorem 9.9.1
• If A is a square matrix that can be reduced to a
  row-echelon form U by Gaussian elimination
  without row interchanges, then A can be factored
  as A LU, where L is a lower triangular matrix.
• Definition
• A factorization of a square matrix A as A LU,
  where L is lower triangular and U is upper
  triangular, is called an LU-decomposition or
  triangular decomposition of the matrix A.

9
Example

• Find an LU-decomposition of
• Solution

10
Procedure for Find LU-Decompositions

• Observing from the previous example on the
  inverses, the following procedure for
  constructing an LU-decomposition of a square
  matrix A provide that A can be reduced to
  row-echelon form without row interchanges.
• Reduce A to a row-echelon form U by Gaussian
  elimination without row interchanges, keeping
  track of the multipliers used to introduce the
  leading 1s and the multipliers used to introduce
  the zeros below the leading 1s.

11
Procedure for Find LU-Decompositions

• In each position along the main diagonal of L,
  replace the reciprocal of the multiplier that
  introduced the leading 1 in that position in U.
• In each position below the main diagonal of L,
  place the negative of the multiplier used to
  introduce the zero in that position in U.
• Form the decomposition A LU.

12
Example

• Find an LU-decomposition of
• Solution

13
Remarks

• Not every square matrix has an LU-decomposition.
• If a square matrix A can be reduced to
  row-echelon form by Gaussian elimination with row
  interchanges, then A has an LU-decomposition.
• In general, if row interchanges are required to
  reduce matrix A to row-echelon form, then there
  is no LU-decomposition of A.
• However, in such case it is possible to factor A
  in the form A PLU, where L is low triangular, U
  is upper triangular, and P is a matrix obtained
  by interchanging the rows of In appropriately.

14
Remarks

• LU-decomposition of a square matrix is not
  unique!
• The LU-decomposition can be written as A LU
  (L?D)U L?(DU) L?U?, where L? and U? are lower
  and upper triangular, respectively.

15
Fitting a Curve to Data

• Given a set of data points, say (x1,y1), ,
  (xn,yn), find a curve y f(x) to fit all the
  data points.
• Some possible fitting curves are
• Straight line y a bx
• Quadratic polynomial y a bx cx2
• Cubic polynomial y a bx cx2 dx3

16
Fitting a Curve to Data

• It is not necessary for the fitting curve y
  f(x) to pass the data points.
• The goal is to minimize the overall error between
  the data points and the fitting curve

17
Least Squares Fit of a Straight Line

• Suppose we want to fit a straight line y a bx
  to the points (x1,y1), , (xn,yn).
• If the points are collinear, we have
• y1 a bx1
• ?
• yn a bxn
• In matrix form
• Mv y
• where

18
Least Squares Fit of a Straight Line

• If the data points are not collinear, we want to
  find a line y a bx such that the error is
  minimized.
• The error can be think as the distance to the
  line along the vertical projection, i.e., ei
  (yi (a bxi))2
• Thus, we want to minimize ?ei , which corresponds
  to
• min y Mv2

19
Least Squares Fit of a Straight Line

• From Theorem 6.4.2, the least squares solutions
  of Mv y can be obtained by solving the
  associate normal system
• MTMv MTy
• Thus, the solution is given by
• v (MTM)-1MTy

20
Theorem (Least Squares Solution)

• Let (x1,y1), , (xn,yn) be a set of two or more
  data points, not all lying on a vertical line,
  and let
• Then there is a unique least squares straight
  line fit
• y a bx
• to the data points.

21
Theorem (Least Squares Solution)

• Moreover, v a bT is given by the formula
• which expresses the fact that v v is the
  unique solution of the normal equations MTMv MTy

22
Example

• Find the least squares straight line fit to the
  four points (0,1), (1,3), (2,4), and (3,4).
• Solution

23
9.4Approximation problems Fourier series

• In this section we shall use results about
  orthogonal projections in inner product spaces to
  solve problems that involve approximating a given
  function by simpler function.
• Such problems arise in a variety of engineering
  and scientific applications.

24
Best Approximations (1/2)

• All of the problems that we will study in this
  section will be special cases of the following
  general problem.
• Approximation problem
• Given a function f that is continuous on an
  interval a,b, find the best possible
  approximation to f using only functions from a
  specified subspace W of Ca,b.

25
Best Approximations (2/2)
26
Measurements of error (1/6)

• We must make the phrase best approximation over
  a,b mathematically precise to do this we need
  a precise way of measuring the error that results
  when one continuous function is approximated by
  another over a,b.
• if we were concerned only with approximating f(x)
  at a single point x0, then the error at x0 x by
  an approximation g(x) would be simply
• Sometimes called the deviation between f and g at
  x0 (Figure 9.4.1).

27
Figure 9.4.1
Go back
28
Measurements of error (2/6)

• Consequently, in one part of the interval an
  approximation g1 to f may have smaller deviations
  from f than an approximation g2 to f, and in
  another part of the interval it might be the
  other way around.
• How do we decide which is the better overall
  approximation?
• What we need is some way of measuring the overall
  error in an approximation g(x).
• One possible measure of overall error is obtained
  by integrating the deviation f(x)-g(x) over the
  entire interval a,b that is,

29
Measurements of error (3/6)

• Geometrically (1) is the area between the graphs
  of f(x) and g(x) over the interval a,b (Figure
  9.4.2) the greater the area, the greater the
  overall error.
• While (1) is natural and appealing geometrically,
  most mathematicians and scientists generally
  favor the following alternative measure of error,
  called the mean square error.

30
Figure 9.4.2
Go back
31
Measurements of error (4/6)

• Mean square error emphasizes the effect of larger
  errors because of the squaring and has the added
  advantage that it allows us to bring to bear the
  theory of inner product spaces.
• To see how, suppose that f is a continuous
  function on a,b that we want to approximate by
  a function g from a subspace W of Ca,b, and
  suppose that Ca,b is given the inner product

32
Measurements of error (5/6)

• It follows that
• so that minimizing the mean square error is the
  same as minimizing f-g.
• Thus, the approximation problem posed informally
  at the beginning of this section can be restated
  more precisely as follows

33
Least Square Approximation
34
Measurements of error (6/6)

• Since f-g2 and f-g are minimized by the
  same function g, the preceding problem is
  equivalent to looking for a function g in W that
  is closest to f.
• But we know from Theorem 6.4.1 that gprojwf is
  such a function (Figure 9.4.3). Thus, we have the
  following result.

35
Figure 9.4.3
Go back
36
Example 1Least squares approximations

• Find the least squares approximation of f(x)x on
  0,2p by
• A) a trigonometric polynomial of order 2 or less
• B) a trigonometric polynomial of order n or less.

37
Solution (a)
38
Solution (b) (1/2)
Figure 9.4.4
39
Figure 9.4.4
Go back
40
Solution (b) (2/2)
41
9.5 Quadratic forms

• In this section we shall study functions in which
  the terms are squares of variables or products of
  two variables.
• Such functions arise in a variety of
  applications, including geometry, vibrations of
  mechanical systems, statistics, and electrical
  engineering.

42
Quadratic forms (1/4)
43
Quadratic forms (2/4)
44
Quadratic forms (3/4)
45
Quadratic forms (4/4)
46
Example 1Matrix Representation of Quadratic Forms
47

• Symmetric matrices are useful, but not essential,
  for representing quadratic forms.
• For example, the quadratic form 2x26xy-7y2,
  which we represented in Example 1 as xTAx with a
  symmetric matrix A, can be written as
• where the coefficient 6 of the cross-product
  term has been split as 51 rather than 33, as in
  the symmetric representation.

48

• However, symmetric matrices are usually more
  convenient to work with, so it will always be
  understood that A is symmetric when we write a
  quadratic form as xTAx, even if not stated
  explicitly.
• When convenient, we can use Formula (7) of
  Section 4.1 to express a quadratic form xTAx in
  terms of the Euclidean inner product as
• If preferred, we can use the notation uvltu,vgt
  for the dot product and write these expression as

xTAxAxx or by symmetry of the dot product
xTAxxAx
xTAxxT(Ax)ltAx,xgtltx,Axgt (6)
49
Theorem 9.5.1
50
Example 2Consequences of Theorem 9.5.1
51
Solution
52
Definition
53
Theorem 9.5.2
54
Example 3Showing that a matrix is positive
definite
55
Example 4Working with principle submatrices
56
Theorem 9.5.3