Analyzing possible causes of bias of hydrological models with stochastic, timedependent parameters

1
Analyzing possible causes of bias of hydrological
models with stochastic, time-dependent parameters
Peter Reichert Eawag Dübendorf, ETH Zürich, and
SAMSI
2
Contents

• Motivation
• Approach
• Implementation
• Application
• Discussion

Motivation Approach Implementation Application Dis
cussion
3
Motivation
Motivation Approach Implementation Application Dis
cussion
Motivation
4
Motivation
Typical results of a hydrological model
Motivation Approach Implementation Application Dis
cussion

• Overall quality of fit demonstrates that the
  model describes the most relevant mechanisms in
  the system adequately.
• However, remaining systematic deviations of model
  results from data make uncertainty analysis
  difficult.

5
Motivation
Residuals of Box-Cox transformed results
Motivation Approach Implementation Application Dis
cussion

• Problems
• Heteroscedasticity of residuals (even after
  Box-Cox transformation).
• Autocorrelation of residuals.

6
Motivation
These problems are typical for any kind of
deterministic dynamic environmental
modelling. They make uncertainty analysis
difficult as this can only be done if the
statistical model assumptions are not seriously
violated.
Motivation Approach Implementation Application Dis
cussion
7
Motivation
Suggested solution (Kennedy and OHagan,
etc.) Extend the model by a discrepancy or bias
term.Replacebywhere yD deterministic
model, x model inputs, q model parameters, Ey
observation error, B bias or model
discrepancy, YM random variable representing
model results.
Motivation Approach Implementation Application Dis
cussion
The bias term is usually formulated as a
non-parametric statistical description of the
model deficits (often as a Gaussian Stoachastic
Process).
8
Motivation

• Advantage of this approach
• The statistical description of the model
  discrepancy allows for improved uncertainty
  analysis.
• Disadvantage
• Lack of understanding of the cause of the
  discrepancy makes it difficult to extrapolate.

Motivation Approach Implementation Application Dis
cussion
We are interested in a technique that supports
identification of the causes of model
discrepancies. This can lead to an improved model
formulation that reduces the discrepancies. This
cannot be done by a purely statistical approach,
but statistics can be supportive.
9
Motivation

• Causes of deficits of deterministic models
• Errors in parameter values.
• Errors in model structure.
• Errors in model input.
• Inadequateness of a deterministic description of
  systems that contain intrinsic non-deterministic
  behaviour due to
• influence factors not considered in the model,
• model simplifications (e.g. aggregation,
  adaptation, etc.),
• chaotic behaviour.

Motivation Approach Implementation Application Dis
cussion
10
Motivation
Because of these deficits we cannot expect a
deterministic model to describe nature
appropriately.
Motivation Approach Implementation Application Dis
cussion

• Pathway for improving models
• Reduce errors in deterministic model structure to
  improve average behaviour.
• Add adequate stochasticity to the model structure
  to account for random influences.

This requires the combination of statistical
analyses with scientific judgment. This talk is
about support of this process by statistical
techniques.
11
Approach
Motivation Approach Implementation Application Dis
cussion
Approach
12
Approach

• Questions
• How to make a deterministic, continuous-time
  model stochastic?
• How to distinguish between deterministic and
  stochastic model deficits?

Motivation Approach Implementation Application Dis
cussion

• Replacement of differential equations
  (representing conservation laws) by stochastic
  differential equations can violate conservation
  laws and does not address the cause of
  stochasticity directly.
• It seems to be conceptually more satisfying to
  replace model parameters (such as rate
  coefficients, etc.) by stochastic processes, as
  stochastic external influence factors usually
  affect rates and fluxes rather than states
  directly.

13
Approach
Motivation Approach Implementation Application Dis
cussion
14
Approach
Note that the basic idea of this approach is very
old. The original formulation was, however,
limited to discrete-time systems with slowly
varying driving forces (e.g. Beck 1987).
Motivation Approach Implementation Application Dis
cussion

• Our suggestion is to
• extend this original approach to continuous-time
  systems
• allow for rapidly varying external forces
• embed the procedure into statistical
  bias-modelling techniques.

This requires more complicated numerical
techniques and more extensive analyses of the
results.
15
Implementation
Motivation Approach Implementation Application Dis
cussion
Implementation
16
Model
Deterministc model
Motivation Approach Implementation Application Dis
cussion
Consideration of observation error
17
Model
Model with parameter i time-dependent
Motivation Approach Implementation Application Dis
cussion
18
Time Dependent Parameter
The time dependent parameter is modelled by a
mean-reverting Ornstein Uhlenbeck process
Motivation Approach Implementation Application Dis
cussion
This has the advantage that we can use the
analytical solution
or, after reparameterization
19
Inference

• We combine the estimation of
• constant model parameters, , with
• state estimation of the time-dependent
  parameter(s), , and with
• the estimation of (constant) parameters of the
  Ornstein-Uhlenbeck process(es) of the time
  dependent parameter(s), .

Motivation Approach Implementation Application Dis
cussion
20
Inference
Gibbs sampling for the three different types of
parameters. Conditional distributions
Motivation Approach Implementation Application Dis
cussion
simulation model (expensive)
Ornstein-Uhlenbeck process (cheap)
Ornstein-Uhlenbeck process (cheap)
simulation model (expensive)
Tomassini et al. 2007
21
Inference
Metropolis-Hastings sampling for each type of
parameter
Motivation Approach Implementation Application Dis
cussion
Multivariate normal jump distributions for the
parameters qM and qP. This requires one
simulation to be performed per suggested new
value of qM.
The discretized Ornstein-Uhlenbeck parameter,
, is split into subintervals for which
OU-process realizations conditional on initial
and end points are sampled. This requires the
number of subintervals simulations per complete
new time series of .
Tomassini et al. 2007
22
Estimation of Hyperparametersby Cross -
Validation
Motivation Approach Implementation Application Dis
cussion
Due to identifiability problems we select the two
hyperparameters (s,t) by cross-validation
Tomassini et al. 2007
23
Estimation of Hyperparametersby Cross -
Validation
For a state-space model of the form
Motivation Approach Implementation Application Dis
cussion
we can estimate the pseudo-likelihood from the
sample
Tomassini et al. 2007
24
Application
Motivation Approach Implementation Application Dis
cussion
Application
25
Hydrological Model
Simple Hydrological Watershed Model (1)
Motivation Approach Implementation Application Dis
cussion
Kuczera et al. 2006
26
Hydrological Model
Simple Hydrological Watershed Model (2)
Motivation Approach Implementation Application Dis
cussion
8 model parameters 3 initial conditions 1
standard dev. of obs. err. 3 modification
parameters
Kuczera et al. 2006
27
Hydrological Model
Simple Hydrological Watershed Model (3)
Motivation Approach Implementation Application Dis
cussion
28
Model Application

• Data set of Abercrombie watershed, New South
  Wales, Australia (2770 km2), kindly provided by
  George Kuczera (Kuczera et al. 2006).
• Box-Cox transformation applied to model and data
  to decrease heteroscedasticity of residuals.
• Step function input to account for input data in
  the form of daily sums of precipitation and
  potential evapotranspiration.
• Daily averaged output to account for output data
  in the form of daily averaged discharge.

Motivation Approach Implementation Application Dis
cussion
29
Analyses and Prior Distributions
A) Estimation of constant parameters Independent
lognormal distributions for all parameters
(83111) with the exception of the measurement
standard deviation (1/s), keeping correction
factors (frain, fpet, fQ) equal to unity. B)
Estimation of time-dependent parameters Ornstein-
Uhlenbeck process applied to the log of the
parameter. Hyperparameters t 1d, s 0.2 (22)
fixed, only estimation of initial value and mean
(0 for log frain, fpet, fQ). Constant parameters
as above.
Motivation Approach Implementation Application Dis
cussion
30
Estimation of Constant Parameters
A) Estimation of Constant Parameters Try to
find a reasonably good fit in which the
deterministic model with constant parameters
reproduces the major features of the data. The
goal of the second analysis with time-dependent
parameters will then be to support finding causes
of remaining model deficiencies.
Motivation Approach Implementation Application Dis
cussion
31
Estimation of Constant Parameters
Prior and Posterior Marginals
Motivation Approach Implementation Application Dis
cussion
32
Estimation of Constant Parameters
Max. post. simulation with constant parameters
Motivation Approach Implementation Application Dis
cussion
33
Estimation of Constant Parameters

• Results of Constant Parameter Fit
• The hydrological model with constant parameters
  leads to
• a fit that reasonably well reproduces the
  features shown by the data
• a simulation with physically meaningful
  be-haviour of state variables with respect to
  their values and to their time scales of
  variation
• identifiable model parameters (with the exception
  of the initial condition of hr).
• Despite this basic agreement, the remaining
  systematic deviations violate simple statistical
  assumptions and make uncertainty analysis
  difficult.

Motivation Approach Implementation Application Dis
cussion
34
Estimation of Time-Dependent Parameters
B) Estimation of time-dependent parameters
Sequentially replace constant parameters by
time-dependent parameters. Try to learn from the
results about deficits of the deterministic model
structure as well as about the need for
stochastic model extensions.
Motivation Approach Implementation Application Dis
cussion

• How to learn from the results of the analysis?
• Analysis of temporal behaviour of parameters.
• Analysis of posterior distributions of const.
  parameters.
• Analysis of behaviour of model results.
• Analysis of indicators of the quality of the fit.
• Explorative analysis of the relationships between
  time-dependent parameters and system variables.

35
1. Temporal Behaviour of Parameters
Time dependent parameter k_s
Motivation Approach Implementation Application Dis
cussion
36
1. Temporal Behaviour of Parameters
Time dependent parameter f_rain
Motivation Approach Implementation Application Dis
cussion
37
1. Temporal Behaviour of Parameters
Time dependent parameter f_Q
Motivation Approach Implementation Application Dis
cussion
38
1. Temporal Behaviour of Parameters
Time dependent parameter s_F
Motivation Approach Implementation Application Dis
cussion
39
1. Temporal Behaviour of Parameters

• Assessment
• In cases with highly dynamic external forcing,
  identified parameter time series are difficult to
  interpret directly.
• The variation of width measures of the posterior
  time-dependent parameter allows us to distinguish
  time periods during which we can gain information
  about variations in the parameter from periods
  during which we cannot.
• In our example, this varies somewhat from one
  parameter to the other, with a general tendency
  that we can learn more during periods with rain
  events than during dry weather periods.

Motivation Approach Implementation Application Dis
cussion
40
2. Posterior of Constant Parameters
Results for time dependent parameter k_s
Motivation Approach Implementation Application Dis
cussion
41
2. Posterior of Constant Parameters
Results for time dependent parameter f_rain
Motivation Approach Implementation Application Dis
cussion
42
2. Posterior of Constant Parameters
Results for time dependent parameter f_Q
Motivation Approach Implementation Application Dis
cussion
43
2. Posterior of Constant Parameters
Results for time dependent parameter s_F
Motivation Approach Implementation Application Dis
cussion
44
2. Posterior of Constant Parameters

• Assessment
• The marginal posterior distributions of some
  parameters depend significantly on which of the
  parameters was made time-dependent.
• In particular, making the modification factor for
  rain intensity time dependent, changes the
  posterior distributions of the other parameters
  strongly.
• This demonstrates the importance of addressing
  input (rainfall) intensity carefully.

Motivation Approach Implementation Application Dis
cussion
45
3. Behaviour of Model Results
Results for time dependent parameter k_s
Motivation Approach Implementation Application Dis
cussion
46
3. Behaviour of Model Results
Results for time dependent parameter f_rain
Motivation Approach Implementation Application Dis
cussion
47
3. Behaviour of Model Results
Results for time dependent parameter f_Q
Motivation Approach Implementation Application Dis
cussion
48
3. Behaviour of Model Results
Results for time dependent parameter s_F
Motivation Approach Implementation Application Dis
cussion
49
3. Behaviour of Model Results

• Assessment
• The basic features of the solutions are not
  changed by introducing a time-dependent
  parameter.
• For some of the parameters, making them
  time-dependent significantly reduces the bias in
  model output, for others this is not the case.

Motivation Approach Implementation Application Dis
cussion
50
4. Quality of Fit
Improvement with time-dependent parameters
Motivation Approach Implementation Application Dis
cussion
Nash-Sutcliffe indices ks 0.83 frain 0.78 fQ 0
.68 sF 0.64 kr 0.59 fpet 0.57 qgw,max 0.54 ql
at,max 0.53 kdp 0.53 kbf 0.53 base 0.53

• Assessment
• Input (frain) and out-put (fQ) corrections.
• Potential for soil / runoff model (ks, SF)
  improvements.
• Some potential for river and evaporation
  improvements.

Random or deterministic?
51
5. Relationsship with Model Variables
Scatter plot of k_s vs. model variables
Motivation Approach Implementation Application Dis
cussion
52
5. Relationsship with Model Variables
Scatter plot of f_rain vs. model variables
Motivation Approach Implementation Application Dis
cussion
53
5. Relationsship with Model Variables
Scatter plot of f_Q vs. model variables
Motivation Approach Implementation Application Dis
cussion
54
5. Relationsship with Model Variables
Scatter plot of s_F vs. model variables
Motivation Approach Implementation Application Dis
cussion
55
5. Relationsship with Model Variables

• Assessment
• Most of the time dependent parameters do not show
  deterministic variation with any of the system
  variables.
• The only exception is the parameter ks of the
  soil submodel that varies significantly with the
  saturated area (which it parameterizes).

Motivation Approach Implementation Application Dis
cussion
56
Conclusions

• Assessment
• Stochasticity seems to be the dominating cause of
  deviations of model results from measurements.
• This is likeli to be dominated by input
  (rainfall) uncertainty.
• The highest chance to find an improvement of the
  deterministic model is for the soil/runoff
  submodel of the hydrological model.
• It seems difficult to significantly improve the
  model by changes to the groundwater and river
  sub-models.

Motivation Approach Implementation Application Dis
cussion
57
Hydrological Model
Model extensions
Motivation Approach Implementation Application Dis
cussion
Extension 1 Modification of runoff flux
Extension 2 Modification of sat. area funct.
Both extensions lead to three more model
parameters.
58
Hydrological Model
Previous results
Extended models
Motivation Approach Implementation Application Dis
cussion
Nash-Sutcliffe indices ks 0.83 frain 0.78 fQ 0
.68 sF 0.64 kr 0.59 fpet 0.57 qgw,max 0.54 ql
at,max 0.53 kdp 0.53 kbf 0.53 base 0.53
Nash-Sutcliffe indices ext. 1 0.72 ext.
2 0.54
Assessment Model extension 1 significantly
improves the description of the system.
59
Approach
Motivation Approach Implementation Application Dis
cussion
60
Hydrological Model

• Next Steps
• Redo analysis with model extension 1.
• Compare remaining stochastic uncertainty with
  knowledge on input uncertainty.
• Do uncertainty analysis for model with extensions
  1 and input uncertainty.
• Investigate alternative ways of describing
  rainfall input uncertainty

Motivation Approach Implementation Application Dis
cussion
Conclusions The application of the technique led
to the dis-covery of improvements of the
deterministic model structure as well as to the
inclusion of stochasticity.
61
Discussion
Motivation Approach Implementation Application Dis
cussion
Discussion
62
Discussion

• The suggested procedure seems to fulfill the
  expectations of supporting the identification of
  model deficits and of introducing stochasticity
  into a deterministic model.
• There is need for future research in the
  following areas
• Explore alternative ways of learning from the
  identified parameter time series.
• Different formulation of time-dependent parameter
  (for some applications smoother behaviour).
• Improve efficiency (linearization, emulation).
• Learn from more applications.

Motivation Approach Implementation Application Dis
cussion
63
Transition

• On-going projects in various fields
• Applications for gaining more experience
• Reichert et al. hydrological model
• Cintron et al. epidemiological model
• Emulation of dynamic models
• Reichert et al. simple physical-based prior
• White et al. extended physical-based prior
• Liu et al. statistical prior
• Gosling et al. emulation of time step
• Linearization for improving efficiency
• Paulo et al. emulation of linearized model
• Liu et al. direct use of linearized model
• Other persons Bayarri, Santner, Pitman,
  OHagan, Wolpert
• More ideas on the way. Post-program workshop
  next year?

Motivation Approach Implementation Application Dis
cussion
64
Acknowledgements

• Development of the techniqueHans-Rudolf Künsch,
  Roland Brun, Lorenzo Tomassini, Mark Borsuk,
  Christoph Buser.
• Hydrological exampleJohanna Mieleitner, George
  Kuczera.
• Interactions at SAMSISusie Bayarri, Tom
  Santner, Gentry White, Ariel Cintron, Fei Liu,
  Rui Paulo, Robert Wolpert, John Paul Gosling,
  Tony OHagan, Bruce Pitman, Jim Berger, and many
  more.

Motivation Approach Implementation Application Dis
cussion
I would like to thank in particular to Jim Berger
and Susie Bayarri for setting up this program
that lead to a very stimulating and fruitful stay
for me at SAMSI.