ECMWF Operational 4DVar and Future Developments

1
ECMWF Operational 4D-Varand Future Developments

• Yannick Trémolet
• August 2004
• Mike Fisher, Erik Andersson, Lars Isaksen,
  Dick Dee, Philippe Lopez, and many others

2
Producing a Weather Forecast
3
Outline

• Operational 4D-Var
• Observations,
• Forecast Model,
• Incremental 4D-Var,
• Operational Constraints.
• Future improvements
• Rain and Clouds Assimilation,
• Variational Bias Correction.
• Model Error
• Long Window 4D-Var.
• Summary

4
Observations
Data sources for the ECMWF Meteorological
Operational System. The numbers refer to all data
items received over a 24 hour period
5
Conventional observations used
SYNOP/SHIP
DRIBU MSL Pressure, Wind-10m
MSL Pressure, 10m-wind, 2m-Rel.Hum.
TEMP
Wind, Temperature, Spec. Humidity
PILOT/Profilers Wind
PAOB MSL Pressure
Aircraft Wind, Temperature
6
27 satellite data sources used in 4D-Var
DMSP SSM/I
NOAA AMSUA/B HIRS, AQUA AIRS
SCATTEROMETERS
GEOS
TERRA / AQUA MODIS
OZONE
7
Observation data count (10/03/04 00 UTC)
Only 3.5 of screened data is assimilated.
Lars Isaksen
8
Number of Used Data per Analysis Cycle
2,500,000
Millions of Observations
Erik Andersson
9
The ECMWF forecast model

• The model has more than 20 million grid points
• Spacing of grid points 40 km
• 60 levels from surface to 65 km
• Temperature, wind, humidity and ozone are
  specified at each point.
• This is done in 15 minute time-steps.
• Number of computations required to make a ten-day
  forecast 188,000,000,000,000

10
Physical processes in the ECMWF model
11
4D-Var Characteristics
All data within a 12-hour period are used
simultaneously, in one global (iterative)
estimation problem.

• 4D-Var finds the 12-hour forecast evolution that
  best fits the available observations.
• It does so by adjusting 1) surface pressure, and
  the upper-air fields of 2) temperature, 3) wind,
  4) specific humidity and 5) ozone.

The cost function measuring the discrepancy
between Observations and Forecast is minimised.
12
Incremental 4D-Var Formulation
In the incremental formulation (Courtier et al.
1994) the cost function is expressed in terms of
increments with respect to the background state,
with and linearized
around .

• The i-summation is over 1h or ½h-long
  sub-divisions (or time slots) of the 12-hour
  assimilation period.

• The innovations are calculated using the
  non-linear operators, and

This ensures the highest possible accuracy for
the calculation of the innovations, which are the
primary input to the assimilation!
13
Incremental 4D-Var
14
Approximations at inner iterations

• 1) The tangent-linear approximation
• 
  and
• 2) Approximations to reduce the cost this
  involves degrading the tangent-linear (and its
  adjoint) with respect to the full model.
• Lower resolution (T159 instead of T511),
• Simplified physics (some processes ignored),
• Simpler dynamics (e.g. spectral instead of
  grid-point humidity).

This results in a shorter control vector, and
cheaper TL and AD model during the minimisation -
i.e. the inner iterations.
The inner loop minimization problem is solved
using a Conjugate Gradient algorithm.
15
Humidity Analysis

• Humidity short-range FC-errors show large
  variability over short distances and can vary
  with several orders of magnitude in the vertical.
• The analysis needs to respect the physical limits
  due to condensation effects near saturation and
  the strict limit at zero humidity.
• Following the approach by Dee and DaSilva (2003),
  a normalized relative humidity variable with a
  more nearly Gaussian distribution was chosen
  as the new analysis variable for the humidity
  analysis.

Elias Holm
16
Normalized humidity control variable

• A control variable with normal distribution
  N(0,1) is obtained by normalization
• Implementation requires linear inner loops
• Inner loops use
• Outer loops get from using the
  nonlinear definition of .
• The background error cost function is now a
  nonlinear function of relative humidity

17
Humidity Background Errors

• The new statistical model for humidity background
  errors gives
• Lower background error in near-saturated regions
  and in cold-air out-break.
• Higher background error in the depression and
  along cold front

Elias Holm
18
Analysis Increments
2nd minimization Additional T159 T increment
1st minimization T95 T increments
0.2K temperature intervals (blue is negative red
is positive)
Most of the increment is formed at the lower
resolution with smaller additions and corrections
obtained at the higher resolution.
Lars Isaksen
19
Incremental 4D-Var Algorithm

• Quadratic inner iterations. Variational quality
  control and SCAT ambiguity removal moved to outer
  level.
• Conjugate Gradient minimisation. With objective
  stopping-criterion based on the gradient-norm
  reduction.
• Hessian eigenvector pre-conditioning. Updated
  after each inner minimisation.
• Multi-Incremental, T95/T159 (some tests at T255).
• Interpolation of the trajectory from T511 to
  T95/T159.
• More TL physics during second minimisation.
• Normalised humidity control variable.
• Implemented operationally in January 2003
  (T95/T159).

20
Outline

• Operational 4D-Var
• Observations,
• Forecast Model,
• Incremental 4D-Var,
• Operational Constraints.
• Future improvements
• Rain and Clouds Assimilation,
• Variational Bias Correction.
• Model Error
• Long Window 4D-Var.
• Summary

21
The operational schedule for the 00Z cycle
22
Computer configuration March 2004
23
Supercomputer Configuration
24
Performance of Operational Runs
The total cost of the 51-members EPS is
roughly twice the cost of the 10 days T511
forecast.
25
Outline

• Operational 4D-Var
• Observations,
• Forecast Model,
• Incremental 4D-Var,
• Operational Constraints.
• Future improvements
• Rain and Clouds Assimilation,
• Variational Bias Correction.
• Model Error
• Long Window 4D-Var.
• Summary

26
The Future of 4D-Var at ECMWF
The main challenge new types and higher
resolution of observations.

• Observation equivalents have to be computed
  accurately in the minimisation (model and
  observation operators) in terms of
• Resolution,
• Physical processes.
• Error statistics are needed at smaller scales.
• Observation error correlations should be taken
  into account.

27
The Future of 4D-Var at ECMWF

• More Accurate Minimisation
• Higher resolution (T255 or higher),
• More physics in model and observation operators,
• Grid point humidity in inner loop,
• Mass preserving grid point interpolations,
• Review use of topography in observation
  operators.
• Maintaining a long assimilation window while
  increasing the resolution emphasises the need
  for
• Deal with nonlinearities,
• Account for Model Errors,
• Investigate nonlinear minimisation algorithms.

28
The Future of 4D-Var at ECMWF

• Removes Biases Observations and Model.
• Background Error Statistics
• Wavelet Jb provides regional variations,
• Real time EnDA at reduced resolution could
• Improve flow dependence,
• Provide initial conditions for EPS,
• Improve quality control of observations.
• Accounting for Observation Error Correlations
• Use randomisation method.
• All this will require even more computer power !

29
Linearized Physics
Marta Janiskova
30
Linear Model Improvements
Temperature
Marta Janiskova
31
Linear Model Improvements
Marta Janiskova
32
Rain and Clouds Assimilation 1D-Var4D-Var
Philippe Lopez
33
1D-Var4D-Var SSM/I 22 GHz TB
Philippe Lopez
34
Assimilation of SSM/I rainy 22Ghz TB
Tropics (20S/20N) Forecast Root Mean Square
Error of Z in meters
Philippe Lopez
35
Variational Bias Correction The Idea
The bias in a given instrument/channel tends to
be modelled in terms of a relatively small number
of parameters It is possible to estimate these
parameters and correct the observations during
the analysis (Derber and Wu, 1998)
Dick Dee
36
Variational Bias Correction Implementation
Dick Dee
37
Evolution of Bias Parameters
NOAA-15 AMSUA Ch5
p(0) global constant p(1) 1000-300hPa
thickness p(2) 200-50hPa thickness p(3) surface
temperature p(4) total column water
Dick Dee
38
Departure Statistics
20040323 - 20040414 (21 days)brightness
temperatures
Dick Dee
39
Departure Statistics
20040323 - 20040414 (21 days)conventional
temperatures
Dick Dee
40
Cautionary note What to do about model bias?
Dick Dee
41
Model Error in 4D-Var
42
Weak constraint 4D-Var
43
Size of the problem
44
Sources of information
45
Characteristics of model error
46
Choice of control variable
47
Model Error Covariance Matrix
48
Average Vorticity Vertical Correlations
49
Temperature Horizontal Correlations
50
Divergence Standard Deviation/1.0E-6
51
Average Standard Deviation Profiles
52
4D-Var Kalman Smoother

• The equivalence 4dVar Kalman smoother is well
  known (and easy to prove) for a perfect, linear
  model.
• The equivalence is less well known for the case
  of an imperfect, linear model. A proof is given
  be Ménard and Daley (1996, Tellus)
• Weak-constraint 4dVar with an imperfect, linear
  model and background covariance matrix B is
  equivalent to (i.e. gives the same state
  estimates as) a fixed-interval Kalman smoother
  that uses the same model, observations,
  observation operators, and initial covariance
  matrix B.
• In fact, 4dVar can handle more general pdfs
  (time-correlated model errors, non-Gaussian
  observation errors, nonlinear model, etc.) than
  the Kalman smoother.
• In this sense, 4dVar is a more fundamental method
  than the Kalman smoother!

Mike Fisher
53
Filtering v. Smoothing

• The Kalman filter codifies all information from
  past observations in the current background state
  and covariance matrix.
• This allows it to remove the time dimension from
  the problem.
• This doesnt seem such an advantage if it
  requires us to handle explicitly a covariance
  matrix of dimension 106106.
• Any reduced-rank approximation to the Kalman
  filter is necessarily sub-optimal.
• We might be better off retaining the time
  dimension, and minimizing the 4dVar cost function
  directly.
• Crucial to this approach is the fact that the
  Kalman filter has limited memory.

Mike Fisher
54
Mean RMS Analysis Error
Lorenz 95 Model
RMS error for 4dVar
RMS error for OI
RMS error for EKF
Mike Fisher
55
Limited Memory
Analysis experiments started with/without
satellite data on 1st August 2002
Mike Fisher
56
Summary

• Long-window, weak-constraint 4dVar is an
  efficient algorithm for solving the Kalman
  smoothing problem for large-dimensional systems.
• No rank-reduction required.
• If the window is long enough, the analysis (and
  its covariance matrix) is independent of the
  background (and its covariance matrix).
• Long enough is probably somewhere between 3 and
  10 days.
• Weak-constraint is a less stiff problem than
  strong-constraint Minimization should be better
  conditioned (i.e. faster).
• Long-window, weak-constraint 4dVar isnt cheap.
  But, you get a full-rank Kalman filter that
  should at least be a useful tool to evaluate
  other, sub-optimal methods (e.g. EnKF).

Mike Fisher
57
Outline

• Operational 4D-Var
• Observations,
• Forecast Model,
• Incremental 4D-Var,
• Operational Constraints.
• Future improvements
• Rain and Clouds Assimilation,
• Variational Bias Correction.
• Model Error
• Long Window 4D-Var.
• Summary

58
Forecasts Scores
Adrian Simmons
59
Forecasts Scores
60
Conclusions

• 4D-Var has performed well since its operational
  implementation in 1997.
• It should be improved further by new developments
  to allow for
• Use of higher resolution observations,
• Use of new types of observations (clouds, rain),
• Better modelling of errors (background and
  observations),
• Correction of biases (observations and model).