BMA at KNMI SWITCHfc a system to optimally combine model info, using the Bayesian Model Averaging te

1

BMA at KNMI (SWITCHfc) a system to optimally
combine model info, using the Bayesian Model
Averaging technique
Frans Alkemade (credits to Ben Wichers Schreur,
Kees Kok) Presentation LAMEPS Vienna, November
13, 2006
2
Overview

• Introduction What is SWITCHfc?
• What motivates the development of SWITCHfc?
• How does BMA work?
• Possible interface and some examples from
  University of Washington
• BMA applied to HIRLAM lagged forecasts
• BMA applied to SRNWP-PEPS ensemble
• Plans for the near future
• Some issues limitations
• Project overview

3
What is SWITCHfc ? Statistical Weighting of
Information Targeted at Combining High-resolution
forecasts

• SWITCHfc is a Bayesian framework for the optimal
  combination of available deterministic forecasts.
  
• These forecasts may come from a single model at
  different lead times, or from single model
  ensemble forecasts, or from multi-model ensemble
  forecasts or from any collection of model
  forecasts.
• The framework should be flexible enough to even
  include subjective forecasts.

4
What motivates the development of SWITCHfc?

• The introduction of SWITCHfc to combine available
  deterministic forecasts in an optimal way
  provides a framework for solving some outstanding
  problems in the use of deterministic forecasts
• Some problems that could be solved
• The problem of finding the best deterministic
  forecast
• The problem of jumpiness
• Comparative verification
• Introduction of new models (verification and
  training)
• Some other possible advantages
• The subjectivity of a model switch can be
  avoided.
• The use of deterministic forecasts is supported
  by providing a calibrated model spread as a
  measure of predictability
• Calibrated reliable probabilistic forecasts can
  be derived by determining complete PDFs of
  atmospheric variables

5
How does BMA work? (Raftery et al.)

• At observation points a continuous PDF is
  calculated as a weighted combination of PDFs of
  individual forecasts.
• Weights correspond to the quality of information
  of individual models (as determined during a
  flexible training period).
• Point information is then used to construct
  forecast fields.

6
Basic principle
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
BMA on Lagged HIRLAM T2m forecasts

• Consider the lagged forecasts (6h, 12h, , 48h)
  to be individual members of an ensemble
• Training period was set at 25 days or 40 days
• BMA was performed for
• 1 1 yr at 6h time steps (a series of
  1500)
• 2 3 yr at 6h time steps (a series of
  4500)
• BMA was tested for 6h predictions, for 12 h and
  for 24 h predictions.

17
Mean resulting BMA-weights for 1 yr of 6 h
forecasts
18
Corresponding RMSEs
19
Results for other lagged-HIRLAM cases

• Results for other lagged-HIRLAM cases are
  similar. e.g.

20
PEPS calibration by BMA

• BMA was applied to archived temperature forecasts
  of 14 members of the SRNWP-PEPS Data were
  available for gt 100 German stations
• Some results are shown for the period Dec. 2005
  Jan. 2006
• Effect of different methods of BIAS correction is
  being studied.

21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
Some plans for the near future

• Get more PEPS data
• Implement BMA for parameters with non-normally
  distributed PDFs (precipitation, wind)
• Find optimal training periods for various
  seasons/areas
• Quantify effect of different methods of bias
  correction
• Attractive interface ( other marketing tools)

31
Some issues and limitations of BMA

• A separate PDF is calculated for each
  meteorological parameter. Therefore consistency
  is not guaranteed.
• BMA was originally developed for quantities whose
  PDFs can be approximated by normal distributions,
  such as temperature and sea-level pressure. This
  approximation may however have an effect on the
  tail of the probabilistic forecast.
• For the same reason forecasting wind and
  precipitation (that have non-normal
  distributions) is not straightforward.
• Resolution may be an issue.

32
Project

• R-based prototype validation, lagged-average
  demonstration
• Operational framework
• Implementations. (MUM, PEPS, SREF?)
• Science extremes, verification, persistence

33
Throw the SWITCH

• Dont choose, USE!

34
Thank you
35
Example Forecast of extreme precipitation

• At ECMWF the Extreme Forecast Index (EFI) was
  developed for the ECMWF EPS, to help the
  forecasters to synthesize the deviation from
  climatology of the EPS forecasts. This EFI is an
  integral measure of the departure between the EPS
  forecasts and the reference climate
  distributions.
• The next slide shows an example of an EFI map
  that shows areas of potential extreme
  precipitation, 10m winds and 2m temperature on
  the same plot.

36
WMO 2006

• The resolution of global EPS models is not
  sufficient to directly predict the intensity of
  severe events. Appropriate post-processing and
  calibration of EPS fields is required. While for
  specific events relevant diagnostics may be
  devised, a more general approach may also be
  useful.
• Although it may be difficult to predict absolute
  values, comparing an EPS forecast to a model
  climate distribution can indicate times when the
  EPS predicts an increased probability of an
  extreme event.

37
(No Transcript)
38
Non normal distributions Precipitation

• As said before BMA was developed initially for
  quantities whose PDFs can be approximated by
  normal distributions, such as temperature and
  sea-level pressure. BMA does for instance not
  apply in its original form to precipitation,
  because the predictive PDF of precipitation is
  nonnormal in two major ways it has a positive
  probability of being equal to zero, and it is
  skewed.
• However BMA can be extended to probabilistic
  quantitative precipitation forecasting. The
  predictive PDF corresponding to one ensemble
  member is than assumed to be a mixture of a
  discrete component at zero and a gamma
  distribution.
• This yields predictive distributions that are
  calibrated and sharp. It also gives probability
  of precipitation (PoP) forecasts that are much
  better calibrated than those based on consensus
  voting of the ensemble members.

39

Klimaatbeheersing
Frans Alkemade Westerbouwing, 20-03-2006
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
Overcoming best member problems by an BMA-like
approach

• The best member method individual members of
  an ensemble are dressed with error patterns
  drawn from a database of past errors made by the
  best member of the ensemble at each time step.
• It has been shown that the best-member method can
  lead to both underdispersive and overdispersive
  ensembles.
• The error patterns can be rescaled so as to
  obtain ensembles which display the desired
  variance. However, this approach can lead to an
  overestimation of the probability of extreme
  events.

46
Another approach

• VINCENT FORTIN et al. (Q. J. R. Meteorol. Soc.
  (2006), 132, pp. 13491369 doi
  10.1256/qj.05.167)
• Probabilistic forecasting from ensemble
  prediction systems Improving upon the
  best-member method by using a different weight
  and dressing kernel for each member

47

• Fortin et al. propose to overcome both
  difficulties by dressing and weighting each
  member differently, using a different error
  distribution for each order statistic of the
  ensemble.
• This method is closely related to the BMA. It can
  be seen as a mixture model where each order
  statistic of the ensemble is viewed as a
  different model Mk,