Total Least Squares and ErrorsinVariables Modeling : Problem formulation, Algorithms, and Applicatio

1
Total Least Squares and Errors-in-Variables
Modeling Problem formulation, Algorithms, and
Applications

• By Sabine Van Huffel
• Katholieke Universiteit Leuven
• Dept Electrical Engineering, Division
  ESAT/SCD(SISTA)
• Leuven, Belgium

2
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

3
Introduction

Linear parameter estimation problem
?1?1 ? 2?2 ? p?p ?
?j parameters, ?j,? variables
Take n measurements xij, yj , j1,,n(ngtp)
solve for ? ?1,, ?pT
Xnxp ? ? ynx1
Classical approach
Linear Least Squares
(X exact)
Unrealistic if errors occur in both X and ye.g.
in automatic control, signal processing, system
theory, statistics, biology, chemistry,
economics, physics, ... Requires a more general
approach Total Least Squares (TLS)
4
The geometry of least squares inverse least
squares total least squares
5
Introduction (contd.)
Consider the simplest linear model between 2
variables ? and ? with underlying
relationship
The real measurements x and y may have errors ?
and ?, i.e., x ? ? and y ? ?
(errors-in-variables model)
A wide variety of situations exist, depending on
the error structures in x and y. We will examine
a number of these.
6
Introduction (contd.)
Least Squares vs.
Total Least Squares
Orthogonal Projection Sensitive to Scale
Change Invariant to Exchange of Axes ML solution
whensx sy const.
Vertical Projection Invariant to Scale
Change Sensitive to Exchange of Axes ML solution
whensx 0, sy const.
7
Problem Formulation
8
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

9
Univariate EIV regression a statistical approach

• Model formulation fit n measurements (xi,yi),
  i1,,n to a straight line
• Standard regression xi error-free, yi with error
• no error
• Estimate ?0, ?1 using Least Squares (LS)
• EIV regression xi, yi with error

10
Univariate EIV regression a statistical approach
(contd.)

• Three EIV models functional
  fixed values
• structural
  random E( )?, var( )?2
• ultrastructural
  random E( )?i, var( )?2
• Rewrite EIV regression as
• EIV regression ? standard regression xi random
  and cov(xi,?i)-?1??2?0
• Standard regression estimator not consistent
• EIV model not identifiable unless side conditions
  are imposed
• ratio of error variances ???2 /??2 known
• ??2 known
• ??2 known
• both ??2 and ??2 known
• Assumption 1 most popular (Adcock, 1877)
  Orthogonal Regression (OR) if ?1

11
Univariate EIV regression a statistical approach
(contd.)

• EIV model parameter estimation
• Orthogonal regression (Adcock, 1877) maximizing
  likelihood function (?1) reduces to minimizing
  the sum of squared orthogonal distances from data
  points to regression line
• Weighted LS (Lindley, 1947) or generalized LS
  (Sprent, 1966) weight residuals with reciprocal
  of their variance and minimize sum
• only holds for no-equation-error model
• Modified LS (Cheng, 1999) ?most general approach
  minimize unbiased and consistent estimator of the
  appropriate unknown error variance, e.g. estimate
  ??2 (? known)

12
Univariate EIV regression a statistical approach
(contd.)

• Minimizing with respect to ?0, ?1 yields

13
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

14
TLS and EIV regression a computational approach

• Statistical approach estimate EIV parameters ?0,
  ?1 by optimizing statistical properties (remove
  bias, max. likelihood, consistency)
• Applied mathematics approach (TLS) estimate ?0,
  ?1 by minimally correcting measured data xi,yi
  such that the corrected data
• exactly satisfy the imposed model, i.e.
• TLS for univariate EIV model OR
• For more than one explanatory variable (X
  matrix) no analytical solution, use SVD or EVD
  approach

15
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

16
Historical remarks TLS (re) discovered many times

• Statistics, regression
• errors-in-variables regression (Gleser, 1981)
• measurement error models (Fuller, 1987)
• orthogonal (distance) regression (Pearson ,1902
  Adcock,1877) York 1966)
• generalized (weighted, modified) least squares
  (Madansky, 1959 Lindley 1947 Sprent 1966, Cheng
  van Ness 1999 )
• Numerical analysis Total Least Squares (Golub
  1973 Golub Van Loan, 1980)
• System Identification
• global linear least squares (Staar, 1981)
• eigenvector method (Levin, 1964)
• Koopmans-Levin method (Fernando Nicholson,
  1985)
• Compensated least squares (Stoica Söderström,
  1982)
• Experimental modal analysis Hv technique
  (Leuridan et al., 1986)
• Signal Processing minimum norm method (Kumaresan
  Tufts, 1983)
• Chemometrics Maximum Likelihood PCA (Wentzell et
  al., 1997)

17
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

18
The SVD revisited

• Given nxp complex matrix X
• with
• r rank number of independent columns/rows of
  X
• U1 basis of column space of X, U2?U1
• V1 basis of row space of X, V2?V1

19
The SVD revisited (contd.) SVD example
forward map
Easy to compute Matlab, calculator
inverse map
20
The SVD revisited (contd.) SVD properties
21
The SVD revisited (contd.) Oriented energy
Consider a matrix A representing a set of p
vector samples from an n-dimensional signal space

Axes of symmetry singular vectors Length axes
squared singular values
22
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

23
Recall Problem Formulation
24
Basic TLS algorithm and computational issues
(contd.)
25
Basic TLS algorithm and computational issues
(contd.)Extensions TLS algorithm
1. not unique
minimum norm solution 2.
non-generic case additional
constraint TLS solution
26
Basic TLS algorithm and computational issues
(contd.) Extensions TLS algorithm
3. multivariate case
p d
software available (SLICOT, NETLIB)or
equivalently,
27
Basic TLS algorithm and computational issues
(contd.)More efficient TLS algorithms
TLS solution entirely determined by - lowest
right singular vector of Xy (d1)- more
generally basis of lowest right singular
subspace of XY corresponding to
smallest singular values. ? compute ONLY those
vectors instead of whole SVD a. directly
compute SVD of XY partially ? partial TLS
(halves CPU time), rank-revealing URV/ULV
(Van Huffel, Stewart, Zha, Hansen,
Fierro,...) b. iteratively recommended for
slowly varying TLS problems. ? inverse iteration,
Rayleigh quotient iteration, conjugate gradient,
(inverse) Chebyshev iteration, (implicitly
restarted) Lanczos methods, neural computing, ...
(Van Huffel, Fierro, Hansen, Barlow,
Cirrincione, Björck, Kamm, Nagy,...)
28
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

29
TLS Properties

• Compare TLS directly with LS (solutions,
  residuals, approximation efforts, subspaces)
  
• LS
• TLS
• TLS minimizes a sum of weighted squared residuals
  

30
TLS Properties (contd.) Algebraic properties
and Sensitivity

• Differences increase when ?p (Xy)/ ?p (X)
  grows
• X? ? y less compatible
• vector y growing
• X (nearly) rank-deficient
• y oriented along the lowest singular vectors
  of X
• Accuracy TLS maximal when y parallel with
  lowest singular vector up of X

31
TLS Properties (contd.) Sensitivity
32
TLS Properties (contd.) Consistency TLS
solution
Errors-in-variables model ? ? ? X ? ? y ?
? error matrix ?? Theorem (Gleser, 1981
Fuller, 1987) if ?i?i i.i.d. with
zero mean and E(?i?i ?i?i T) ?I,
(i.e. all errors equally sized
and uncorrelated) and
exists, positive definite then TLS
strongly consistent (LS inconsistent)
33
TLS Properties (contd.) Consistency TLS
solution
34
TLS Properties (contd.) extend consistency
TLS

• ? ? ?
• X ? ?
• y ? ?
• Assume errors ?i ?i are i.i.d. with zero
  mean but correlated
• Error covariance matrix E(?i ?i ?i ?i
  T)?CCTC known, positive definite
• Consistent solution ?
• 1. transform data XyXyC-1
• 2. compute TLS solution
  3. transform back
• implicit transformations via Generalized SVD /
  QSVD of (Xy,C)
• Solve GENERALIZED TLS problem ? strongly
  consistent

(Van Huffel and Vandewalle, 1989 Gallo, 1981
Gleser, 1981 Fuller, 1987)
35
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• Case Study TLS in near-infrared spectroscopy
• Case Study TLS in magnetic resonance
  spectroscopy
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

36
Total Least Squares Applications

• CASE STUDIES IN SIGNAL PROCESSING SYSTEM
  IDENTIFICATION
• Subset selection in EIV models
• Blackbox modeling in signal processing and system
  identification
• Linear prediction problem
• Impulse response estimation by discrete
  deconvolution
• Parameter estimation in transfer function models
• TLS based estimation of frequency response
  functions in Modal Analysis
• STLS based linear prediction modeling in Speech
  Compression
• CASE STUDIES IN BIOMEDICAL SIGNAL PROCESSING
• Renography
• Magnetic Resonance Spectroscopy
• Near-Infrared Spectroscopy
• Polysomnography
• Fetal Electrocardiography
• OTHER TLS APPLICATIONS

37
Some Biomedical Examples

• HOW? preprocessing --gt modeling --gt parameter
  estimation --gt validation --gt GUI
• TOPICS in collaboration with UZ Leuven
• Renogram deconvolution structured TLS,
  displacement rank
• Fetal ECG extraction generalized SVD,
  Independent Component Analysis
• MR Spectroscopic quantitation TLS, FIR filter,
  nonlinear LS fitting
• Near-Infrared Spectroscopy weighted TLS, PCA,
  CCA, constrained LS
• Polysomnography multichannel SVD based rank
  reduction TLS LP modeling
• Preoperative cancer diagnosis neural Bayesian
  networks (MLP)
• Osseoperception, brain tumor recognition, .

38
NEAR-INFRARED SPECTROSCOPY quantitation of
neonatal oxygenation

• In collaboration with the Department of
  Paediatrics and Neonatal Medicine, Univ.
  Hospitals Leuven, Belgium
• Co-workers G. Morren, G. Naulaers, P. Casaer,
  H. Devlieger

39
Near Infrared Spectroscopy (NIRS)

• Non-invasive measurement of cerebral oxygenation
  of infants to study
• the effects of apneas, drugs, temperature
  changes, sucking,
• the correlation with other physiological
  parameters (heart rate, respiration, EMG )

40
Beer-Lambert Law Model
d
41
NIRS algorithm
42
Algorithm refinements
43
Total least squares
44
MAGNETIC RESONANCE SPECTROSCOPY in-vivo
quantitation of metabolite concentrations and
images

• In collaboration with the biomedical NMR unit,
  Department of radiology, Univ. Hospitals Leuven,
  Belgium and the Spin Imaging Group, Fac. Of Appl.
  Physics, Delft Univ. of Technology, The
  Netherlands
• Co-workers L. Vanhamme, R. in t Zandt, P. Van
  Hecke, Y. Wang, D. van Ormondt, N. Mastronardi,
  T. Sundin, H. Chen, D. Graveron, I. Dologlou,
  E. Heyvaert, C. Decanniere

45
Nuclear magnetic resonance
static magnetic field
46
Different applications of NMR
47
MR Spectroscopic Imaging
Survey Image and 1H Spectroscopic Images of a
Patient Suffering from a Glioblastoma
48
Model function
Theoretical Model
49
The real world ...

• accuracy
• automation
• efficiency

50
Proton MRS
51
Preprocessing Data Quantification
HSVD/HTLS
subspace- based approach
? minimal interaction expertise, limited
possibilities, suboptimal
52
Results using Simulated MRS data
53
Results using Simulated MRS data
54
Results using Simulated MRS data
55
Results using in-vivo MRS data
56
Preprocessing
HTLS filtering
HLR replace SVD by low rank revealing ULV ? 2
to 100 times faster
57
Extension to time series
58
Time Series Quantitation
extension of HTLS
HTLSstack

• Select for every signal the corresponding
  frequencies dampings

• Determine for every signal the corresponding
  amplitudes phases

59
HTLSstack versus HTLS
60
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

61
TLS extensions

• Mixed LS-TLS and extensions submatrix of XY
  error-free (Demmel, 87 88)
• Generalized TLS XX1X2, X1 error-free,
  errors on X2 correlated
  (Gallo 81, Fuller 87,
  Golub, Hoffman, Stewart  87,
• 
  Van
  Huffel Vandewalle, 91)
• Weighted (scaled) TLS DX Y T, D and T
  diagonal, unequal error variances in X and Y
  (Golub, Van Loan  81, Rao  97, Paige and
  Strakos, 01-02)
• Restricted TLS error matrix of the form EDEC
  (includes equality constraints, LS, TLS, mixed
  LS-TLS, ) (Van Huffel, Zha  91)
• Total (Lp) approximation uses other norms
  (Watson, Späth, Osborne, 82)
• Nonlinear measurement error model (Caroll et al.
   95) XBG?Y bilinear TLS
  approach min
  
  inconsistent (Fuller 87) adjusted LS
  estimator
  consistentcorrection for small
  samples (A. Kukush, I. Markovsky, Van Huffel,
  01) Other nonlinear models semi-linear,
  quadratic, polynomial (Caroll et al, S. Zwanzig,
  A. Kukush, I. Markovsky, Amari 2002)

62
TLS extensions (contd.)

• Elementwise-Weighted TLS (differently sized
  errors) (M.L.
  Rastello, A. Premoli, Markovsky, Kukush, Van
  Huffel 2002)
• Rowwise-Weighted TLS (rowwise different error
  covariance Matrix)
  (Markovsky, Kukush, Van Huffel 2004)
• Bounded Uncertainties (El Ghaoui 1997, Sayed,
  Chandrasekaran, Golub 1997)
• Structured TLS (e.g. Toeplitz/Hankel,
  displacement rank, regularisation)
  
• (Rosen, Park, Glick, 96, Lemmerling, De
  Moor, N. Mastronardi, Van Huffel, M. Schuermans)
• Regularized TLS (truncated TLS, quadratic
  eigenvalue problems,)
  (Guo, Renaut 2002, P.C. Hansen, D. OLeary, G.
  Golub, R. Fierro 1997, D. Sima et al, 2004)
• Latency error (equation error) (De Moor,
  Lemmerling, A. Yeredor 2002)
• Cox proportional Hazards model with EIV (H.
  Kuchenhoff 2002)
• TLS for large scale problems
  
  (using a preconditioned conjugate gradient
  method proposed by A. Björck ,1997)
• TLS for large scale Toeplitz systems of equations
  (estimate no longer ML)
  (J. Kamm and J. Nagy, 1998, applied Newton
  iterations combined with a bisection scheme and
  circulant factorization preconditioners)
• (S. Van Huffel and P. Lemmerling, eds, TLS and
  EIV modeling, Kluwer 2002)

63
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

64
Structured TLS
Park, Rosen, Glick  94, Lemmerling 99,
Mastronardi 01)
(Abatzoglou,Mendel  87, De Moor  92,
Structured TLS

• Why structured TLS ?
• is structured and noise on different entries of S
  is i.i.d. Gaussian white noise
• Example Toeplitz matrix
• Computation constrained nonlinear optimization
  (Newton) exploit matrix structure ? displacement
  rank
• Note STLS solution consistent for affine
  structures (Kukush 02)

65
Structured TLS Is STLS a simple extension
of TLS ?

• fTLS(z) is the objective function we have to
  minimize for solving the TLS problem
• fSTLS(z) is the objective function we have to
  minimize for solving the structured TLS problem

66
Structured TLS (contd.) for structured X, y
unstructured
Assume that q lt np different elements of X are
subject to error, e.g. X Toeplitz q lt np-1, X
sparse qltltnp represents the
corrections applied to these elements Vector ?
and correction matrix E are equivalent ry-(XE)?
gives rr(?, ?) STLS problem
Dqxq diagonal matrix of positive
weights Equivalent to TLS when qnp and Lp2 In
order to solve STLS, an nxq matrix X is needed
such that E ? B? B has the following
characteristics - Elements of B are the ? is
with suitable repetition- Number of non-zeros in
B equals number of non-zeros in E- B and E have
similar structure
67
Structured TLS (contd.) Construction of E
and B
If ?k is (i,j)th element of E then ? j is (i,k)th
element of B Example
68
Structured TLS (contd.) linearize
r(?,?)

• Let ??, ?E and ?? represent small changes
• Use (?E) ? B?? and neglect 2nd order terms in
  ??, ??.
• Linearization gives
• r(???, ???) y-(XE?E)(???)
• ?r(?, ?)-B??-(XE)??
• At each iteration, solve the linearized
  minimization
• with ,
  rank (M)pq if (XE) is of full rank

69
Structured TLS (contd.) for structured X
unstructured y

• Input X, D,y, structure on X, tolerance ?
• Output correction matrix E, solution ?
• 1. Set E0, ?0, compute ? from B from ?, set
  ry-X ?
• 2. repeat
• (a)
• (b) set (c) construct E from ? and B
  from ? compute ry-(XE) ? until
• p2?2(a)LS problem

70
Structured TLS (contd.) exploit structure
of

• assume Lp2 kernel step of basic algorithm LS
  problem, X Toeplitz, DI ? exploit low
  displacement rank of involved matrices
  sparsity of generators ? O(npp2) flops
• comparison in efficiency, using simulation
  example- alg1 see above (exploiting
  displacement structure sparsity)- alg2
  basic algorithm without exploitation of structure
  (O(np)3)

71
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Case Study STLS in renography
• Other TLS applications
• Conclusions

72
Case Study STLS in RENOGRAPHY renogram
deconvolution in kidneys

• In collaboration with the division of nuclear
  medicine, Univ. Hospital Leuven, Belgium
• co-workers P. Lemmerling, N. Mastronardi, J.
  Baetens

73
Measurement setup
74
Measurement setup
75
Overview of the renal scintigraphy
76
Used renal regions of interest
Heart
Right kidney
Left kidney
Background region
77
TAC (Time-Activity) curves
Kidneys y(t)
Heart or renal artery u(t) y(t)u(t)h(t) where
y(t)renal TAC (OUT) u(t)heart TAC
(IN) h(t)impulse response (unknown) convolution
operator
78
Impulse response estimation by discrete
deconvolution

• assume system - linear - time invariant -
  causal - zero initial state - finite state
  dimension

d(t)
1
t
impulse response
impulse
u(t)
79
Convolution illustrated
80
Example Impulse response estimation
by discrete deconvolution

Measure u(t) and y(t), find h(t)discrete
deconvolution
0
Ymx1
H
Umxn
u(t) and y(t) noisy ? TLS recommended exploit
matrix structure of U ? STLS (max. likelihood)
81
Simulation setup
82
Comparison in accuracy
MA versus TLS (MAMatrix Algorithm Solves a
square system YUH Via Gaussian Elimination
with Partial pivoting)
- TLS more accurate than MA, even if curves are
smoothed- accuracy of MA depends heavily on the
number of smoothings- TLS needs no smoothing-
overdetermination not possible with MA- MA fails
to solve rank - deficient problems TLS more
reliable
83
Comparison in accuracy
Average relative error of MA and TLS in function
of ?v for 4 different degrees of smoothing

84
Renogram deconvolution via STLS

• relations between in- and output at time t
  y(t)u(t)h(0)u(t-1)h(1)...u(1)h(t-1)u(0)h(t
  )

model selection criterion statistically optimal
relation between in- and outputs
structure to preserve
85
STLS solution H can be computed via STLS
algorithm provided Lp2, nM, pN, qM, DI
weighted matrix, ? set to 10-6, XEU?U,
yrY?Y, ?H
0
0
86
TLS versus STLS in renal deconvolution
TLS more reliable robust than currently used
algorithm (Gaussian elim. with partial pivoting)
Add regularisation as noise st. dev. increases
(Mastronardi et al, 2004)
87
Other STLS Applications

• Medical diagnosis (renography, brain tumor
  recognition)
• System identification (EIV Kalman filtering,
  Hidden Markov)
• Modal analysis
• Signal Processing (audio, NMR, speech)
• Astronomy
• Information Retrieval
• Image (Reconstruction, Deblurring, )
• Computer Vision
• Metrology
• Multivariate calibration
• (S. Van Huffel and P. Lemmerling,
  eds, TLS and EIV modeling, Kluwer 2002)

88
Overview

• Introduction and Problem Formulation
• Univariate EIV regression a statistical approach
• TLS and EIV regression a computational approach
• Historical remarks
• The SVD revisited
• Basic TLS algorithm and computational issues
• TLS properties
• TLS applications
• TLS extensions
• Structured TLS
• STLS applications
• Conclusions

89
Conclusions Collaboration must continue ...
between STATISTICS, COMPUTATIONAL
MATHEMATICS and ENGINEERING