UPDATED STATISTICAL POSTPROCESSING RESULTS WITH THE NCEP AND MSC GLOBAL ENSEMBLE FORECASTS

1
UPDATED STATISTICAL POST-PROCESSING RESULTS WITH
THE NCEP AND MSC GLOBAL ENSEMBLE FORECASTS

• Bo Cui1, Zoltan Toth2, Yuejian Zhu2, Dingchen
  Hou1
• David Unger3, Stéphane Beauregard4
• 1SAIC at Environmental Modeling Center, NCEP/NWS
• 2Environmental Modeling Center, NCEP/NWS
• 3Climate Prediction Center, NCEP/NWS
• 4Canadian Meteorological Centre, Meteorological
  Service of Canada
• Acknowledgements
• Jeff Whitaker, Tom Hamill, Richard Verret,
  Richard Wobus

2
Outline

• Statistical post-processing results
• Methods
• decaying averaging bias estimate
• climatological mean bias estimate, CDC GFS
  refcst. data set (Hamill Whitaker)
• bias estimate using 31-day centered running mean
  fcst. error
• (optimal benchmark )
• Variables bias corrected, 2004
• 500 mb height, 850 mb temperature, 2m
  temperature, 10m U and V
• components
• Comparison of NCEP CMC ensemble performance
  before and after CMC January upgrade
• Canadian ens. fcst on Jan.12, 2005, the Optimal
  Interpolation
• Technique for the analysis cycle was replaced
  with the Ensemble Kalman
• Filter Technique
• Current work and future plan
• Correlation between observed anomalies and ens.
  mean fcst. errors
• The 2nd moment bias correction

3
Bullet 1
RPSS 500 mb Height, Northern Hemisphere March,
2004 February, 2005

OPR_DAV2 RPSS improved for all
lead time RFC_COR significant
improvement for all lead time vs.
RFC_RAW
4
Bullet 1
RMS 500 mb Height, 2004 Summer Northern
Hemisphere
OPR_DAV2 RMS error reduced for first
week RFC_COR improvement for all lead
times wrt RFC_RAW

5
Bullet 1
PAC 500 mb Height, 2004 Summer Northern
Hemisphere
OPR_DAV2 PAC scores slightly
improved for first few days RFC_COR
very limited improvement over
RFC_RAW

6
Bullet 1
Excessive Outliers 500 mb Height, 2004 Summer
Northern Hemisphere
OPR_DAV2 improved performance
for up to 5-7 days RFC_COR improvement
for all lead time vs. RFC_RAW

7
Bullet 1
RPSS 850 mb Temperature, 2004 Summer Northern
Hemisphere
8
Bullet 1
ROC 850 mb Temperature, 2004 Summer Northern
Hemisphere
9
Bullet 1
RPSS 10m V component, Northern Hemisphere
March, 2004 February, 2005


10
Bullet 1
RPSS 2m Temperature, Northern Hemisphere
March, 2004 February, 2005

11
Bullet 1
Comparison of the Decaying Average Equal Weight
Approaches
RPSS 500 mb Height of Northern Hemisphere,
2004 Autumn

• Bias is calculated over some previous days with
  equal weight for each day
• Difficult to distinguish the difference between
  OPR_DAV2 and OPR_EQU curves by eye
• The decaying average approach and the equal
  weight approach are equivalent.
• Decaying average approach has the advantage
  the bias updated every day by only considering
  last day data

12
Second Moment Bias-Correction Algorithm RPSS
500 mb Height, 2004 Summer, Northern Hemisphere


OPR_DAV2S no improvement vs. OPR_DAC2
OPR_RFC_DAV2 no improvement for all
lead time vs. OPR_DAC2
13
PRELIMINARY RESULTS

• 1. Decaying averaging ( 2 weight, 30-day oper.
  training data)
• Short range Works very well, all measures
  improved (Day 5)
• Week 2 Limited success
• Degrades ensemble mean (rms, PAC)
• Improves probabilistic performance (ie, outlier
  stats, RPSS)
• 2. Climatological mean error removed (25-yr CDC
  training data)
• RMS and PAC Very limited improvement
• Probabilistic measures (RPSS, etc) significant
  gain
• 3. Operational (raw or bias-corrected) vs. CDC
  bias-corrected ens
• Ensemble mean Operational much better than CDC
  hindcast
• CDC has 50 larger initial error
• Probabilistic scores Operational much better
  for out to day 10
• For some measures, CDC hindcasts better beyond
  day 10

14
TENTATIVE CONCLUSIONS
Bullet 1

• 1. Adaptive, regime dependent bias correction
  works well for first few days (almost as good as
  optimal)
• Frequent updates of DA/NWP modeling system
  possible
• 2. Climate mean bias correction can add value,
  especially for wk2 prob. fcsts
• Generation of large hind-cast ensemble is
  expensive but can be helpful

15
Bullet 2
Before CMC January Upgrade RPSS 500 mb Height,
Northern Hemisphere Dec.1, 2004 Jan. 10, 2005


The two systems have big different performances
16
Bullet 2
After CMC January Upgrade RPSS 500 mb Height,
Northern Hemisphere Jan. 15, 2005 Feb. 28,
2005


The two systems now appear to have a similar
level of skill over the NH in terms of
probabilistic scores.
17
Bullet 3

• Correlation between the Obs. Anomalies Fcst.
  Errors
• Top 500 mb Height Fcst. Error Observed Anomaly
  at (40 N, 95 W), 2004 ( the anomaly added as
  day 17)
• Bottom 500 mb Height Residual Error Observed
  Anomaly
• the fcst. errors increase with fcst. lead time.
  The growth is nearly linear. The fcst. errors are
  a function of lead time, and longer lead times
  are strongly correlated with observed events.
• the fcst. errors have higher frequency details
  compared the residual errors.

18
Bullet 3

Current Work Future Plans

• Utilizes the correlation between the obs.
  anomalies ens. perturbation and fcst. errors
• Preliminary Results
• The ens. mean fcst. error is a function of lead
  time. The correlation between the
• observed anomalies and the ens. mean fcst error
  is very high. This suggests that
• the ens. mean fcst error is dominated by the
  observed verifying anomalies. The
• time mean errors may not be closely related to
  systematic errors.
• Purpose
• remove the observed anomaly from the error fields
  before they are used as
• estimates of the bias.
• Method
• decompose the total error into (a) component
  parallel to obs. anomaly
• (b) residual error, orthogonal to obs.
  Anomaly (M. Wei).
• remove error component along obs. anomaly from
  total error and work with residual component for
  bias estimation.
• The 2nd moment bias correction
• Current method, no significant improvement, the
  calculation of the 2nd
• moment ratio needs more consideration (David
  Unger).

19
Questions and Comments?More plots on
http//www.emc.ncep.noaa.gov/gmb/ens/
20
Bullet 1
METHOD / APPLICATION 1

• Adaptive (Kalman Filter type) Bias-Correction
  Algorithm

Implementation of decaying averaging for 1st
moment bias
decaying averaging mean error (1-w) prior
t.m.e w (f a)
For each lead time separately, tme time mean
error

• Application to NCEP Operational Ensemble

• OPR_RAW NCEP T00Z 10 ensemble forecasts
• OPR_DAV2 w 2 (most recent 30 days used)
• OPR_OPT 31-day centered running mean forecast
  error is removed, operationally not feasible,
  used as optimal benchmark

21
METHOD / APPLICATION 2
Bullet 1

• CDC GFS Reforecast Data Set (Hamill Whitaker)

• Model T62L28 MRF, circa 1998
• Initial states NCEP Reanalysis
• Duration 15 days runs at 00Z from 19781101 to
  now
• Ensemble Breeding, 10 members used from 15

• Bias correction

Climatological (out of sample) mean forecast
error (25 yrs) removed (1979-2003, 1st moment)

• Experiments

• RFC_RAW
• CDC reforecast ensemble forecasts (no bias
  correction)
• RFC_COR
• Calibrated CDC reforecast
• RFC_OPT
• 31-day centered running mean forecast error
  is removed, operationally not feasible, used as
  optimal benchmark

22
METHOD / APPLICATION 5

• Use large hindcast data set for correcting
  operational fcst. by using decaying average
  difference between operational and reforecast

FCSTclibriated FCSTOPR BIAS25yr_clim
BIASOPR-RFC

• Application to NCEP Operational Ensemble

• OPR_RFC_DAV2 25-year climatological mean
  fcst. errors and decaying averaging mean error
  (w2) between NCEP operational and CDC refcst.
  removed

23
METHOD / APPLICATION 6

• Second Moment Bias-Correction Algorithm

ratio r.m.s of ensemble mean / standard
deviation
decaying averaging mean ratio (1-w) prior
time mean ratio w ratio
FCSTclibriated FCSTmean Ratio ( FCSTm-
FCSTmean )
For each lead time separately

• Application to NCEP Operational Ensemble
• OPR_DAV2S w 2 (most recent 30 days
  used)

24
Bullet 2
Before CMC January Upgrade RPSS 500 mb Height,
Northern Hemisphere Dec.1, 2004 Jan. 10, 2005


The two systems have big different performances
25
Bullet 2
After CMC January Upgrade RPSS 500 mb Height,
Northern Hemisphere Jan. 15, 2005 Feb. 28,
2005


The two systems now appear to have a similar
level of skill over the NH in terms of
probabilistic scores.