Assessing forecast uncertainty from synoptic to sub-seasonal scales.

1
Assessing forecast uncertainty from synoptic to
sub-seasonal scales.

• Celeste Saulo and Juan Ruiz
• CIMA (CONICET/UBA) DCAO (FCEN UBA)

2
Motivation and general context

• Many meteorological services run operational
  ensemble prediction systems (EPS), which provide
  estimates of the uncertainty of the forecast.
• Many of these outputs are readily available to
  the scientific community through, e.g. TIGGE
  (THORPEX Interactive Grand Global Ensemble).
• Obtaining useful (valuable) information from EPS
  requires statistical post-processing and specific
  research depending on the variable/problem/region.
  
• There is growing interest in obtaining useful
  information from EPS on time scales between 2
  weeks and 2 months.

3
Motivation and general context

• Active research is being pursued in numerous
  places on the definition of initial ensembles,
  multimodel (or stochastic physics) as well as on
  the evaluation of ensemble predictions.
• During the first half of THORPEX it was realized
  that model error diagnosis is one area where
  universities and research institutions can make
  substantial contributions to the further
  development of models (and hence forecast skill),
  thereby supporting the relatively small community
  of model developers.

THORPEX The Observing System Research and
Predictability Experiment
4
Potential areas of research under UMI-IFAECI

• Predictability studies
• Ensemble generation (including data assimilation)
• Probabilistic forecasts
• Verification strategies

5
Related ongoing studies

• How sensitive are probabilistic precipitation
  forecasts to the choice of calibration algorithms
  and the ensemble generation method?
• Part I Sensitivity to calibration methods (Ruiz
  and Saulo, Meteorol. Appl., 2011)
• Part II sensitivity to ensemble generation
  method (Ruiz, Saulo and Kalnay, Meteorol. Appl.
  2011)

6

• Three different ensemble generation strategies,
  using WRF regional model as the basis
• Breeding (11 members)
• Multi-model (11 members)
• Pragmatic spatially shifted ((2m 1)2 members,
  e.g., 121)

7

• In order to correct the effect of the ensemble
  systematic errors, several techniques have been
  developed, all of them based in the study of the
  relationship between error and forecasted value
  and in the development of statistical models to
  compute a calibrated probability given the
  forecasts of the ensemble members

• A logistic regression is used to represent
  h(ygt0f) and a GAMMA function is used to
  represent h(ygttrf,ygt0)
• BMA ? weighted calibrated probability for each
  member
• GAMMA-ENS ?all weights are equal calibrated
  probability for each member
• GAMMA? no weights calibration applied to the
  ensemble mean
• WMEAN ?weighted ensemble mean and then
  calibration is applied

8
Weights associated to each member of the
spatially shifted ensemble as a function of the
corresponding shift in the southnorth (y axis)
and the westeast (x axis) directions. Negative
shift values indicate southward and westward
shifts respectively.
9
Continuous ranked probability score (CRPS)
GAMMA calibration has been adopted
The computation of a weighted ensemble mean can
lead to moderate better results however the best
choice for a weight computation algorithm is
still an open question. The PQPF derived from the
un-weighted ensemble mean produces, if not the
best results, almost as good results as any other
approach.
10
Shifted
MM
Breeding
48 hours forecast
24 hours forecast
11
Shifted-MM
Shifted-Breeding
Shifted combined
shifted multimodel 1331 members shifted breeding
1331 members shifted combined 2541 members
12

• The spatially shifted ensemble proves to be quite
  competitive at short forecast ranges,

Precipitation uncertainty at these ranges is
mostly related with the location of rain areas

• yet its skill drops rapidly with increasing lead
  times

uncertainties associated with the existence, or
intensity of pp, tend to become more important
with increasing lead times.

• multimodel ensemble (physics) outperformed the
  breeding ensemble (IBC). Still, the improvement
  combining both is modest

most of the PQPF limitations during summer arise
from errors in model physics rather than problems
in the initial and/or boundary conditions
13

• Among the alternatives that have been evaluated,
  the most important improvement has been obtained
  with the combination of the multimodel ensemble
  approach (and/or the combined approach) and the
  spatial shift technique even at 48-hours lead
  time. This approach is particularly interesting
  and promising for implementing high resolution
  ensembles in small operational or research
  centers for which computational costs largely
  restrict ensemble size.

14
Ensemble Forecast Object Oriented Verification
Method

• Work in progress Juan Ruiz (postdoc at LMD) and
  Olivier Talagrand
• The method has been designed to be applied to the
  500 hPa field, however it can be easily extended
  to other fields as well (and probably other
  objects i.e. jet streak position, low level jet
  maximum possition, etc).
• It is based in the identification of local minima
  and the system associated with each local minima.
• As in 500 hPa, usually low pressure systems
  appear in the form of troughs rather than in the
  form of closed systems, the geopotential height
  anomaly is used instead of the full 500 hPa
  field.

15
Cyclone trajectories at 500 hPa, for a particular
day derived from the NCEP ensemble system
16
Questions for future research

• How much information can be obtained from the
  ensemble spread about the forecast skill? Are
  there specific scores to quantify this
  relationship in terms that it becomes useful for
  particular applications?
• Which is the most convenient way to combine
  different ensemble members? Is it necessary to
  take into account the different skill of each
  member? (i.e. Bayesian model averaging trying
  different weights against simpler techniques like
  logistic regression for precip)
• Which kind of information/type of scores could be
  used to provide valuable information about
  weather states with more than two weeks in
  advance?
• How can we use model error statistics to
  understand which processes are strongly affecting
  forecast quality so that key problems can be
  isolated and models improved?
• Which methodologies should we apply to forecast
  probability of extreme events?