Anders Persson SMHI

1

• Some ideas related to the use of EPS
• Anders Persson, SMHI, Norrkoping
• D6 forecast worse today than 1954?
• EPS charts a la the Bergen School
• Systematic errors are not only biases
• Why might a better model yield higher RMSE?
• Appendices

2

• Some ideas related to the use of EPS
• Anders Persson, SMHI, Norrkoping
• D6 forecast worse today than 1954?
• EPS charts a la the Bergen School
• Systematic errors are not only biases
• Why might a better model yield higher RMSE?
• Appendices

3
The pre-NWP forecast accuracy

• A schematic illustration of the increase of
  RMSE with forecast time. The pre-NWP forecaster
  started from a persistence forecast which he
  skillfully extrapolated into the future,
  converging towards climate for longer ranges

persistence
A?2
meteorologist
A

• The time unit can be anything from hours to
  days depending on the parameter (hours for
  clouds, days for temperature)

4
NWP more accurate - but also less

• A good NWP model is able to simulate all
  atmospheric scales throughout the forecast. It
  has the same variance as the observations and the
  persistence forecasts, which yields an error
  saturation level 41 above the climate

persistence
A?2
worlds best NWP
meteorologist
A
5
The art of good forecasting
persistence

• The way out of the dilemma
• Combine the high accuracy of NWP in the
  short range with a filtering of the
  non-predictable scales for longer ranges
• This can be done both with and without the EPS

A?2
worlds best NWP
meteorologist
A
modified forecast
6
Observed 2 m T error growth (Karlstad)

• The accuracy of different forecast methods
  for a city in central Sweden winter 2001-2002
  persistence, climate, T511, an average of the
  last three days T511 (poor mans ensemble mean)
  and the T255 EPS ensemble mean

error saturation level
persistence
T511
error level for climatological statements
mean of last T511s
EPS ensemble mean

• Forecast days

7
But almost as important as forecast skill is
forecast consistency
8
Jumpiness in 2 m T forecasts (Karlstad)

• The ensemble mean is not only the most
  accurate, it is by far also the least jumpy
• While the T511 makes an average 24 hour
  jump of 2 - 4K, the T255 ensemble mean is
  stable at around 1 K

upper level for jumpiness
error level for climatological statements
T511
EPS ensemble mean

• Forecast days

9

• The 24 h jumpiness for T511 and the T255 ensemble
  mean

• Southern Sweden

• Central Sweden

10

• Some ideas related to the use of EPS
• Anders Persson, SMHI, Norrkoping
• D6 forecast worse today than 1954?
• EPS charts a la the Bergen School
• Systematic errors are not only biases
• Why might a better model yield higher RMSE?
• Appendices

11
The EPS contains a lot of information, spread out
in several locations
One solution is to create synoptic EPS maps
in the old Bergen School tradition
combine several parameters in the same map!
12
Probability of strong winds, precipitation and
anomalous (850 hPa) temperatures 6 February 2001
D5
13
Probability of strong winds, precipitation and
anomalous (850 hPa) temperatures 6 February 2001
D5
gt15 m/s
gt10 m/s
gt5 mm/24t
gt10 mm/24t
gt1 mm/24t
gt20 mm/24t
gt 8K
lt -8 K
lt -4 K
gt 4 K
14
Probabilities that the 850 hPa temperature
anomalies are gt- 4 K or gt 4 K

and combine them with the ensemble mean map...
15
500 hPa 6 February 2001 120 h
16
850 hPa temperature anomalies overlaid the
ensemble mean 500 hPa 6 February 2001 120 h
17
The same for gale force winds and precipitation
according to EPS from 6 February 2001 120 h
Vind gt 15m/s
Nbd gt5 mm/24 t
18
in combination with the ensemble mean forecast
19
6 February 2001 12 UTC 120 h
Prob gt 15 m/s
Prob gt 5mm/24h
20
The Synoptic EPS Map combines the mean and the
spread
21
A situation from 11 November 2001, which
contained disturbances both in the north and
south
22
EPS forecast 4 Nov 2001 D7 verifying 11 November
Prob gt 15 m/s
Prob gt 5mm/24h
1000 hPa geop.
23
EPS forecast 5 Nov 2001 D6 verifying 11 November
24
EPS forecast 6 Nov 2001 D5 verifying 11 November
25
EPS forecast 7 Nov 2001 D4 verifying 11 November
26
EPS forecast 8 Nov 2001 D3 verifying 11 November
27
The increase in anomaly correlation coefficient
of the EM of 2 m-temperature D7 forecasts
compared with different other deterministic
methods (9 European stations)
T511
T255
2
7
8
Ensemble mean T255
3
4
3
3
Poor Mans T511
Poor Mans T255
2
28
T511 forecasts compared with T255 ensemble mean
valuesAdvantages with T511 The same for T255
ensemble mean values

• Occasionally a bit unrealistic flow patterns
• The best categorical forecast values beyond 2
  days
• Drastically reduced jumpiness
• The ensemble mean can be supplemented with
  probabilities, which is not possible with the T511

• The best simulation of meteorological features
• The best forecasts 1-2 days ahead, but then worse
• Jumpy forecasts after 3-4 days
• Good geographical resolution

29

• Some ideas related to the use of EPS
• Anders Persson, SMHI, Norrkoping
• D6 forecast worse today than 1954?
• EPS charts a la the Bergen School
• Systematic errors are not only biases
• Why might a better model yield higher RMSE?
• Appendices

30
Problem in the ECMEF EPS system T255 Control
31
Common problem in EPS systemsunder-spread
error
spread
Too much spread
Too little spread
72h
32
The T255 (and T511) have problems with
temperatures below -25 C
Range 20 K
Too warm forecasts
33
27 Dec 2002
34
The mean error (ME) decreased by 1-2 K but the
RMSE only by 0.5 K The solution lies in the
ensemble approach...
27 Dec 2002
35
Range 30 K
Non-biased forecasts
36
Principle of a two-dimensionel error equation
Based on its experience of forecasts gt -8 C
the filtersees a certain linear relation
between forecast and error
Experience 21-29 October 2002
37
Principles of a two-dimensional error equation
Based on experiences in the range gt -8 C
the filter i capable of producing corrections
for much lower temperatures for which it has no
direct experience
Experience 21-29 Oct 2002
Extrapolation of regression line
38
Principen för en två-dimensionell felekvation
När verifikationen för temperatur-prognoserna
under -8 C blir tillgängliga visar det sig att
korektionerna var realistiska
Erfarenhet 30 oktober - 6 november 2002
39
Ölands södra udde januari - mars 2004
40

• Some ideas related to the use of EPS
• Anders Persson, SMHI, Norrkoping
• D6 forecast worse today than 1954?
• EPS charts a la the Bergen School
• Systematic errors are not only biases
• Why might a better model yield higher RMSE?
• Appendices

41
Understanding statistical interpretation and why
better forecasts can look worse...
42
The verification yielded RMSE5.0 Emean2.6
HIRLAM-44 24 hour 2 m temperature forecast for
Kiruna in Lapland winter 2001-2002
43
The corrections yielded a reduction of the RMSE
from 5.0 to 4.2 and of the EMEAN from 2.6 to 0.2
A 1-dimensional Kalman filter can reduce an
overall bias
But cooling the warm forecasts made the results
worse
Cooling the cold forecasts improved the results
44
A 2-dimensional Kalman filter can provide
different corrections to different regimes.
The cooling the cold forecasts did not affect the
warm forecast
Cooling the cold forecasts still improved the
results
45
The Kalman filtering has reduced two systematic
errors a positive mean error and an
underestimation of the variability
The mean error is reduced from 2.6 to 0.3!
HIRLAM variance 25 K
Observed variance 40 K
but the RMSE is only reduced from 5.0 to 4.6
46
To show that what looks like equal improvements
are not quite equal, we will make a simple
manipulation of the data the observations and
the numerical forecasts are swopped
47
24 hour 2 m temperature forecast for Kiruna
winter 2001-2002 - with observations and
forecasts swopped
obs
model
48
After Kalman filtering the EMEAN is reduced to
zero and the RMSE is reduced from 5.0 to 2.9
Extreme cold forecasts warmed
49
(No Transcript)
50

• Some ideas related to the use of EPS
• Anders Persson, SMHI, Norrkoping
• D6 forecast worse today than 1954?
• EPS charts a la the Bergen School
• Systematic errors are not only biases
• Why might a better model yield higher RMSE?
• Appendices

51
The complete formula for RMSE
52
From the RMSE to the MSE
53
Simplifying the notations
54
The power of simple mathematics
55
Decomposing the MSE around climate
56
What looks bad might be good
57
The accuracy of a climate statement
58
The RMS error saturation level
59
Validating the EPS system
To what degree is the EPS system able to cover
all possible scenarios?
To what degree does the EPS system manage to
include the verification?
Is there a consistent relation between EPS
spread an the error of the EPS mean?
These questions are not related to the
verification
60
A simple vector-geometrical alternative
61
The RMS error saturation level
f-a a-c?2
90º
With decreasing forecast accuracy the angle ?
will increase. The maximum RMSE equals the
variability times ?2
62
The spurious consistency-skill correlation

• Two forecasts systems (f) and (g) lack
  predictive skill and are mutually uncorrelated

This implies that all three vectors are
perpendicular (90)
Let us now watch this 3D figure from upper
right...
63
The spurious consistency-skill correlation

• Whereas the analysis vector (a) and the
  forecast vectors (f and g) are perpendicular,
  their differences are not! Their mutual angles
  are 60 which implies correlations of 50.
• It is when the forecasts start to display
  skill and mutual correlation that the 50
  correlation starts to decrease to the 30 level
  sometimes reported at a D5 or D6 range