References: Nonlinear Time Series Analysis

1
References Nonlinear Time Series Analysis
Books
Review Articles

• H. Kantz, T. Schreiber (1997) Nonlinear Time
  Series Analysis. Cambridge University Press.
• H.D.I. Abarbanel (1996) Analysis of Observed
  Chaotic Data. Springer-Verlag.
• R.M.A. Urbauch (2000) Footprints of Chaos in the
  Markets. Prentice-Hall.
• S. Strogatz (1994) Nonlinear Dynamics and Chaos.
  Addison-Wesley, Reading, MA.
• E. Ott, T. Sauer, J.A. Yorke (1994) Coping with
  Chaos Analysis of Chaotic Data and the
  Exploitation of Chaotic Systems. Wiley, New York.
  

• P. Eckmann, D. Ruelle (1985) Ergodic theory of
  chaos and strange attractors. Reviews of Modern
  Physics 57 617-656.
• T. Parker, L. Chua (1987) Chaos a tutorial for
  engineers. Proceedings of IEEE 75 982-1008.
• P. Grassberger, T. Schreiber, C. Schaffrath
  (1991) Nonlinear time sequence analysis.
  International Journal for Bifurcation Chaos 1
  521-547.
• H.D.I. Abarbanel et al. (1993) The analysis of
  observed chaotic data in physical systems.
  Reviews of Modern Physics 65 1331-1392.
• Schreiber T (1999) Interdisciplinary application
  of nonlinear time series methods. Phys. Rep.
  308 1-40.

2
Some Key Papers
Mathematical Concepts of Embedding Packard NH et
al. (1980) Geometry from a time series. Phys.
Rev. Lett. 45 712-716. Sauer T, Yorke JA ,
Casdagli M. (1991) Embedology. J. Stat. Phys. 65
579-616. Takens F (1981) Detecting strange
attractors in fluid turbulence. In D Rand and LS
Young, Eds. Dynamical Systems and Turbulence,
pp. 366-381, Springer, Berlin.
Embedding in the Presence of Noise Casdagli M et
al. (1991) State space reconstruction in the
presence of noise. Physica D 51
52-98. Gibson JF et al. (1992) An analytical
approach to practical state space reconstruction.
Physica D 57 1-30.
Principal Component Analysis Albano AM,
Passamante A, Farrell ME (1991) Using higher
order correlations to define an embedding
window. Physica D 54 85-97. Broomhead DS, King
GP (1986) Extracting qualitative dynamics from
experimental data. Physica D 20
217-236. Palus M, Dvorak I (1992) Singular value
decomposition in attractor reconstruction
pitfalls and precautions. Physica D 55
221-234.
3
Choice of Time Delay ( t ) Buzug T, Pfister G
(1992) Optimal delay time and embedding dimension
for delay-time coordinates by analysis of
the global and local dynamical behavior of
strange attractors. Phys. Rev. A 45
7073-7084. Fraser AM, Swinney HL (1986)
Independent coordinates for strange attractor
from mutual information. Phys. Rev. A 33
1134-1140.
Choice of Embedding Dimension ( d ) Kennel MB,
Brown R, Abarbanel HDI (1992) Determining
embedding dimension for phase space
reconstruction using a geometrical construction.
Phys. Rev. A 45 3403-3411.
Correlation Dimension d2 Grassberger P, Procaccia
I (1983) Measuring the strangeness of strange
attractors. Physica D 9 189-208.
Estimation of d2 Theiler J (1986) Spurious
dimensions from correlation algorithms applied to
limited time series data. Phys. Rev. A. 34
2427-2432. Theiler J (1990) Estimating fractal
dimension. J. Opt. Soc. Am. A. 7 1055-1073.
4
Fractal Noise Provenzale A et al. (1992)
Distinguishing between low-dimensional dynamics
and randomness in measured time series. Physica
D 58 31-49.
Lyapunov Exponents Wolf A et al. (1985)
Determining Lyapunov exponents from a time
series. Physica D 16 285-317. Eckmann JP et al.
(1986) Lyapunov exponents from time series.
Physical Review A 34 4971-4879. Rosenstein
MT, Collins JJ, De Luca CJ (1993) A practical
method for calculating largest Lyapunov
exponents from small data sets. Physica D 65
117-134.
5
Sources of Errors in the Estimation of d2
Any physical quantity can never be known exactly
so it is appropriate to provide a confidence
interval of the estimated quantity. Or in other
words to identify the errors in estimation.
Three primary sources of errors

• Geometrical ErrorsCaused by geometrical effects
  in the state space (either reconstructed or true
  one) ? edge effects, singularities, lacunarity,
  measurement noise, finite precision etc
• Dynamical ErrorsCaused by certain properties of
  the trajectories that are related with dynamics
  ? autocorrelation effects, finite d2 by certain
  types of filtered noise
• Statistical ErrorsRelated with the method of
  estimation itself ? limited number of data
  points, high dimensional process

6
Dynamical Errors
Introduced by the spurious correlations in the
data induced by oversampling. ? Produces
knee in the correlation sum C(r) ?
Underestimation of slope of C(r) vs r ?
Tendency towards convergent d2
Remedy Apply Theilers correction (window) in the
calculation of C(r)
Recipe to Choose Theilers Window
The length of window should be proportional to
correlation time of the series. Or Choose a
reasonably large value because loss of
vector-pairs will be negligible.Or Apply
space-time separation plot.
7
Space Time Separation Plot
Provenzale et al. (1992) Physica D
In the presence of temporal correlations, the
probability that a given pair of pointshas a
distance smaller than r does not only depend on r
but also on the time thathas elapsed between the
two measurements.
This dependence can be estimated by plotting the
number of pairs as a function of(i) the time
separation Dt, and (ii) the spatial distance r.
Scatter Plot of Dt vs r for Lorenz Flow
Spatial separation
Temporal separation
For small Dt, points are always close in space.
Investigate the contour maps of the fraction of
points closer than a distance r at a given time
separation Dt as a function of Dt. ?
P(x(tDt) x(t) lt r)
8
NMR Laser Data
The temporal correlation would not induce
artifacts when the contour linesare flat.
r
Dt
Taylor-Couette Flow Data
The oscillations would not cause anyartifact as
long as the observation period is much larger
than a cycle length. tmin should be greater than
the rising time of the first segment.
r
Dt
In stead of choosing a time window as Theilers
correction, in practice, you also can exclude a
fixed number of points from the calculation.
Temporal correlations are present as long as the
contour curves do not saturate.
9
If a system possesses a chaotic attractor, then
d2 is computationally very efficientmeasure to
estimate the underlying fractal dimension,
but a small non-integer d2 does not
necessarily imply low dimensional chaotic
dynamics.
Why?
Because the correlation sum only reflects spatial
distribution of the experimentaltrajectory
points, but not their temporal relationships.So,
d2 does not represent any dynamical property of
the system!
Simple stochastic processes, characterized by a
power-law power spectrum with random,
independent, uniformly distributed Fourier
phases, generate time serieswith finite d2.
10
Consider the stochastic signals
In short, the process has power-law Fourier power
spectra (phase fk can be random)
Although such system is intrinsically infinite
dimensional, they produce finite d2
11
There are also other types of stochastic signals
which yields a finite estimate of correlation
dimension.
q -0.9, a b 1, ?(t) is a standard Gaussian
white noise process.
Linear FN
Nonlinear FN
12
Both these time series have very similar power
spectra P(f) ? f-2 The linear signal is
statistically self-similar, an homogeneous
fractal signal.The nonlinear signal is
multifractal and intermittent. The Fourier
phases of x(t) are uniformly distributed with no
correlation. But some of the Fourier phases of
y(t) can be correlated.
13
Linear Noise
Nonlinear Noise
C(r)
C(r)
d2
K2
d
d
Finite correlation dimension correlation
entropy!
14
Signal Differentiation
Differentiate the signal and estimate d2
If a system is governed by a low-dimensional
strange attractor, d2 will remain same.
But for a stochastic signal, d2 of the first
derivative will be higher.
15
Lorenz Flow
Fractal Noise
d2
d2
diff.
orig.
Embedding Dimension
Embedding Dimension
16
Structure Function
For a fractal signal, S(n) ? n2s ? s is the
scaling exponent
For a fractal process with 1/fa power spectra a
2s 1
Then,
17
Lorenz Flow
Fractal Noise
Original
Structure Function
First Difference
S(n) for a stochastic process would display a
long scaling region, but S(n) for strange
attractor shows oscillatory behavior.
18
Space-Time Plot for 1/fa Noise
The only points with small separationare
dynamically near neighbors ? No recurrence in
phase space
There exists no time scales on which the
distribution is stable.
Spatial separation
In short, such time series is inherently non-stati
onary!
Temporal separation
19
Generalized Dimensions
Theoretically, attractor of a chaotic process is
self-similar, possessing structure on all
possible lengths scales, and will be
statistically self-similar for practical data.
Self similarity is best characterized by
Hausdorff dimension or by box-counting
dimension.
Recapitulating box-counting a point set located
in ?d, covered with a regular grid of boxes of
length e of N(e), then ?
But, natural measures of dynamical system are not
homogenous, i.e. parts of the trajectory will
be more visited thus contain larger fraction of
the measure. Standard dimension of the support of
the trajectory may not be the most useful.
Assume a phase space has fractal measure m.
? the probability of finding a typical trajectory
in a ball of radius e around x
20
The generalized correlation integral
which will also be equal to
If the set of points is self-similar,
Generalized Dimension Dq can be defined as
? Information Dimension
For q 1
ltlnpegt ? average information needed to specify a
point x with precision e
D1 indicates how this information scales with
resolution e.
21
If Dq depends on q, the set is multifractal.
Dq is a nonincreasing function of q, d2 is
approximately the lower bound of D1.
Since d2 can be easily obtained from time series
data, d2 is the best practical approximation to
D1. If the data are multifractal, i.e. Dq
strongly depends on q, d2 is not a
goodapproximation to D1.
The generalized correlation integral can be
written as
For a discrete time series
22
In practice, to remove the bias due to temporal
correlation, the generalizedcorrelation integral
is computed as
Finally, we have to look for scaling range at
finite length scales and look for a plateau of
dq(e) d(logCq(e))/d(log(e))
23
More Invariant Measures
Other than generalized dimensions, there are more
geometric quantities whichremain invariant under
embedding ? Lyapunov exponents
Generalised entropies
Lyapunov Exponents
Motion
The initial infinitesimal sphere is distorted
into an infinitesimal ellipsoid whoseprincipal
axes are oriented in the expanding and
contracting directions as well as the direction
of no distortion.
24
But, there is no distortion in the direction of
the flow ? One of the axes of the ellipsoid
will coincide with the tangent of the trajectory
of the center point. This axis
changes direction with time, but the magnitude
remains preserved. Other axes change in
both magnitude and direction.
The Lyapunov exponents are determined by the long
term rate of change in themagnitude of the
principal axes of the evolving ellipsoid. They
are termed as exponents because the rate is
measured in exponential forms.
Let, pi(t) the magnitude of the i-th axis at
time t Then, its exponential rate if change
li(t) is given by
The limiting value (t??) is the Lyapunov exponent
corresponding to the i-th axis.
25
Usually, this exponent is denoted by li and
expressed in units of binary digits (or nats)
per unit time by choosing b 2 (or b e) . The
Lyapunov exponents are arranged in decreasing
order, l1 l2 l3 ? Lyapunov Spectrum
The li are scalars measuring the time average
growth of the magnitude of the principal axes.
No information of direction, but only measure
the amount of stretching and folding
experienced by a small rotating volume element
moving under the flow.
An initial magnitude q1(0) in the direction of
first principal axis grows to ? The linear
segment grows like
The area in the plane of the first two principal
axes will grow like
The volume defined by the first three axes will
grow like
26
?li - the time average divergence of the flow
or the long term exponential rate of
change in the phase space volume
Now, a system with an attractor is dissipative,
i.e, net volume contracts. This implies, ?li lt
0, means at least one li is negative.
But if the system is chaotic, it is producing
information (sensitive dependence oninitial
condition produces exponential divergence of
nearby trajectories) ? at least one li is
positive.
27
Some Critical Remarks
Since the attractor is bounded, there cannot be
exponential divergence all along, So the folding
makes the attractor not diverge to infinity. But
Lyapunov exponents do not give information on
this folding process.
An n-dimensional attractor has n true Lyapunov
exponents, but the number ofdegrees of freedom
in the phase space defines the number of
principal axes of theexpanding ellipsoid that is
used to define the Lyapunov exponents (LE). If
an n-dimensional attractor is embedded in a
d-dimensional (d gt n) space, any numerical
procedure will produce d Lyapunov exponents. But
only n of these are true exponents, other (d-n)
are called phantom exponents.
Simple Recipe Under time inversion, all true
Lyapunov exponents change their signs. Compute
Lyapunov spectrum for both the original data and
for the time reversed dataand find which ones
change their sign. They are the true exponents.
28
Wolf et al. (1986) Physica D
If the state flow is known, the whole Lyapunov
spectrum can be analyticallydetermined. In
short, with a system of d ODE, one numerically
solves for d1initial conditions. The growth of
a corresponding set of vectors is measured, and
as the system evolves, the vectors are
repeatedly orthonormalized by Gram-Schmidt
procedure. This procedure guarantees that only
one vector has a component in the direction of
most rapid expansion.
Thus, one cannot calculate the entire Lyapunov
spectrum by choosing arbitrary directions for
measuring the separation of nearby initial
conditions. This requires that one should
measure the separation along the Lyapunov
directions that corresponds to the principal
axes of the ellipsoids. These Lyapunov
directions depend on the system flow and are
defined by the Jacobian matrix, i.e. the
tangent map, at each point of interest along the
flow. Thus, one should preserve the proper
phase space orientation by using a suitable
approximation of the tangent map.
29
If there exists an ergodic measure of the system,
then Oseledecs multiplicative ergodic theorem
establishes the usages of arbitrary phase space
directions while calculating the largest LE.
In other words, two randomly chosen initial
conditions will diverge exponentially at a rate
decided by the largest LE. ? a random vector
of initial conditions will likely to converge to
the most unstable manifold since growth
along this direction dominates growth along
other Lyapunov directions.
Thus, the largest LE can be defined as
d(t) average divergence at time tC a
constant
Estimation of the largest LE is preferable over
the estimation of Lyapunov spectrum (i) it
requires less assumptions (ii) it requires less
data points (iii) it is easier to compute
30
Estimating Maximal LE
Rosenstein et al. (1993) Physica D

• Reconstruct the state space with a suitable
  embedding dimension d and time delay t
• Locate the nearest neighbor of each vector on the
  trajectory. The nearest neighbor, x(j), is found
  by searching for the point minimizes the
  distance to the particular reference vector,
  x(r).

dr(0) the initial distance from the r-th point
to its nearest neighbor. A further constraint is
imposed such that nearest neighbors have a
temporalseparation greater than the mean period
of the time series
mean period - reciprocal of the mean frequency
of the power spectrum
31
3. We assume the r-th pair of nearest neighbors
diverge approximately at a rate given by
the largest LE.
Cr initial separation
4. The above equation represents a set of
approximately parallel lines (r 1,2, , Nr)
each with a slope proportional to l1.
5. The largest LE is calculated by using a
least-squares fit to the average line
defined by
32
Lorenz Flow
Scaling region
After a short transition, a long scaling region
is found. This scaling region is used to extract
the largest LE. The profile saturates at longer
times because (i) the attractor is bounded, and
(ii) the average divergence cannot be more than
the length of the attractor.
33
Effect of Embedding Dimension
Henon map
Logistic map
Lorenz flow
Rössler flow
The algorithm is fairly robust against the
choices of embedding dimension.
34
Lyapunov Dimension
Relationship between dimensions and Lyapunov
exponents If li lt 0 for all i, the attractor
will be a fixed point, the dimension will be
zero. If li gt 0 for all i, the attractor will
diverge to infinity, so does the dimension.
Kaplan York (1983) J. Diff. Eq.
The Lyapunov or Kaplan-York dimension
where k is determined by the relation
It is conjectured that DKY is equal to the
information dimension D1.