Using reforecasts for probabilistic forecast calibration

1
Using reforecasts for probabilistic forecast
calibration
NOAA Earth System Research Laboratory

• Tom Hamill Jeff Whitaker
• NOAA Earth System Research Lab, Boulder, CO
• tom.hamill_at_noaa.gov

2
Reforecasts?

• A hindcast, a numerical prediction for a date in
  the past using the model and data assimilation
  system that is currently operational.
• Uses
• (1) Post-processing of ensemble weather and
  climate forecasts correcting for systematic
  bias, spread deficiencies, downscaling. Crucial
  for implementing reliable uncertainty forecasts
  in NWS.
• (2) Data assimilation first-guess forecasts
  corrected by observations forecasts assumed
  unbiased reforecasts can help adjust to make
  sure they are.
• (3) Diagnosing model errors sometimes model
  deficiencies arent obvious without large sample.

3
NOAAs 1st-generation reforecast data set

• Model T62L28 NCEP GFS, circa 1998
• Initial States NCEP-NCAR Reanalysis II plus 7
  /- bred modes.
• Duration 15 days runs every day at 00Z from
  19781101 to now. (http//www.cdc.noaa.gov/people/j
  effrey.s.whitaker/refcst/week2).
• Data Selected fields (winds, hgt, temp on 5
  press levels, precip, t2m, u10m, v10m, pwat,
  prmsl, rh700, heating). NCEP/NCAR reanalysis
  verifying fields included (Web form to download
  at http//www.cdc.noaa.gov/reforecast). Data
  saved on 2.5-degree grid.
• Experimental precipitation forecast products
  http//www.cdc.noaa.gov/reforecast/narr .

4
Reforecastingadvantages disadvantages

• Advantages
• Very large gains in skill and reliability from
  removing systematic errors (demonstrated later).
• A real pathway to the NWS providing objective,
  reliable, skillful probabilistic forecasts
  without a lot of human intervention (no fleet of
  forecasters working on probabilistic IFPS).
• Disadvantages
• Computationally expensive, especially if model is
  changing frequently.
• Not getting forecast improvement directly through
  improving the model.

5
Application NCEP/CPCs 6-10 day outlook
Map of probabilities of above / below / near
normal. 33 percent probability assumed in near
normal unless above or below gt 67 percent.
6
Comparison against NCEP / CPC forecasts at 155
stations, 100 days in winter 2001-2002
temperature forecasts

Reforecast calibrated Week-2 forecasts more
skillful than operational NCEP/CPC 6-10 day,
which was based on human blending of NCEP, ECMWF,
other tools.
precipitation forecasts
7
Reforecast-based example floods causing La
Conchita, California landslide, 12 Jan 2005
week-2 from reforecast
6-10 day from reforecast
8
Reforecast model brought back into production
and used operationally at NCEP/CPC because of
the usefulness of reforecast products. Also
working on transition of products to NCEP/HPC
9
Application downscaled precipitation forecasts
using analog technique
On the left are old forecasts similar to todays
ensemble- mean forecast. The data on the right,
the analyzed precipitation conditional upon the
forecast, can be used to statistically adjust and
downscale the forecast. Analog approaches like
this may be particularly useful for hydrologic
ensemble applications, where an ensemble of
realizations is needed.
10
Downscaled analog probability forecasts
11
Verified over 25 years of forecasts skill
scores use conventional method of calculation
which may overestimate skill (Hamill and Juras
2006).
12
Tornado probability forecasting
forecast wind shear and instability were used as
predictors in an analog approach.
13
ECMWFs reforecast experiments
ECMWF got excited by our results and produced a
test reforecast data set to see if they would get
a big forecast improvement even with their
much-improved ensemble forecast system.

• Model 2005 version of ECMWF model T255
  resolution.
• Initial Conditions 15 members, ERA-40 analysis
  singular vectors
• Dates of reforecasts 1982-2001, Once-weekly
  reforecasts from 01 Sep - 01 Dec, 14 weeks total.
  So, 20y ? 14w ensemble reforecasts 280
  samples.
• Data obtained by NOAA / ESRL T2M and
  precipitation ensemble over most of North
  America, excluding Alaska. Saved on 1-degree lat
  / lon grid. Forecasts to 10 days lead.

14
ECMWF, raw and post-processed
Note 5th and 95th ile confidence intervals very
small, 0.02 or less
15
ECMWF, raw and post-processed
In this metric, calibrated 4-5 day forecasts now
as skillful as uncalibrated 1-day forecast.
Note 5th and 95th ile confidence intervals very
small, 0.02 or less
16
Precipitation 5-mm reliability diagrams(90-km
forecasts verified against 32-km North American
regional reanalysis)
horizontal lines indicate distribution of
climatology
error bars from block bootstrap
Raw forecasts have poor skill in this strict
BSS much improved with calibration
17
Precipitation skill with weekly and 30-day
training data sets
Compared use of the once-weekly ? 20-year
reforecast data set to calibration using only the
past 30 days of forecasts. Substantial benefit
of weekly reforecasts relative to 30-day training
data sets, especially at high thresholds for the
more rare heavy precipitation events, a longer
training data set is needed.
18
ECMWF newsletter, Autumn 2008 Reforecasts
operational at ECMWF
19
NOAA and new reforecasts?

• Climate Forecast System Reanalysis and Reforecast
  (operational 2010-2011)
• 1 reforecast member per day T126 model used for
  climate forecasts.
• GEFS Global ensemble forecast system plan to do
  1 reforecast member per day in real time, e.g.,
  on 1 Dec 2009, do 1 Dec 2008, 2007, etc.

20
Reforecast issues

• How many members?

Results with ECMWF data set suggest additional
benefit from more members.
21
Reforecast issues.Which method of calibration?
NCEP was hopeful that a new calibration method,
Bayesian Processor of Forecasts would allow
them to do improved calibrations with small
training data sets, lessening the need for
reforecasts. Recent ESRL/PSD research
has demonstrated serious problems with the
proposed algorithm relative to ones that have
been previously demonstrated to be effective.
22
Other reforecast issues

• Which calibration method is best for other
  important parameters? (clouds, precipitation
  amount/type, winds, severe weather, etc.)
• What is NOAAs long-term strategy for production
  of probabilistic forecasts? How does
  reforecasting fit with the overall strategy for
  probabilistic forecasting?
• Who does what? ESRL or NCEP to compute
  reforecasts? NCEP or MDL to post-process?
  Technique development at ESRL, NCEP, and/or MDL?
• Best configuration of reforecast data set?
• Reforecasts computed in real-time, with evolving
  model, few members (NCEPs preferred approach for
  GEFS), or
• Fixed model, large reforecast data set, computed
  all at once, and used for many years?
• Does the configuration of the best reforecast
  data set vary with weather parameter? For
  example, do hydrologists need more data than is
  needed for temperature calibration?

23
FY11 Alternative (Climate goal)

• Develop next-generation reforecast with modern
  NCEP model, based on CFSRR reanalyses.
• Examine reforecast ensemble size issues - how
  much of the benefit is possible with reduced size
  data set (fewer members, reforecasts every x
  days).
• Compare against evolving NCEP ensemble system
  with bias corrections from 1-member reforecast.
• Develop suite of new experimental products.
• Etc.
• NWS did not support making this climate activity
  part of FY11 core its above core currently.

24
Conclusions

• Reforecasts shown to aid in calibration of
  forecasts for a wide variety of applications.
• Still a large benefit from forecast calibration,
  even with state-of-the-art ECMWF forecast model.
• Many remaining issues to be explored, and renewed
  importance given recent emphasis on uncertainty
  forecasting in NWS.
• Still working on securing stable funding CPU
  time for next-generation reforecasts.