Title: Using reforecasts for probabilistic forecast calibration
1Using reforecasts for probabilistic forecast
calibration
NOAA Earth System Research Laboratory
- Tom Hamill Jeff Whitaker
- NOAA Earth System Research Lab, Boulder, CO
- tom.hamill_at_noaa.gov
2Reforecasts?
- A hindcast, a numerical prediction for a date in
the past using the model and data assimilation
system that is currently operational. - Uses
- (1) Post-processing of ensemble weather and
climate forecasts correcting for systematic
bias, spread deficiencies, downscaling. Crucial
for implementing reliable uncertainty forecasts
in NWS. - (2) Data assimilation first-guess forecasts
corrected by observations forecasts assumed
unbiased reforecasts can help adjust to make
sure they are. - (3) Diagnosing model errors sometimes model
deficiencies arent obvious without large sample.
3NOAAs 1st-generation reforecast data set
- Model T62L28 NCEP GFS, circa 1998
- Initial States NCEP-NCAR Reanalysis II plus 7
/- bred modes. - Duration 15 days runs every day at 00Z from
19781101 to now. (http//www.cdc.noaa.gov/people/j
effrey.s.whitaker/refcst/week2). - Data Selected fields (winds, hgt, temp on 5
press levels, precip, t2m, u10m, v10m, pwat,
prmsl, rh700, heating). NCEP/NCAR reanalysis
verifying fields included (Web form to download
at http//www.cdc.noaa.gov/reforecast). Data
saved on 2.5-degree grid. - Experimental precipitation forecast products
http//www.cdc.noaa.gov/reforecast/narr .
4Reforecastingadvantages disadvantages
- Advantages
- Very large gains in skill and reliability from
removing systematic errors (demonstrated later). - A real pathway to the NWS providing objective,
reliable, skillful probabilistic forecasts
without a lot of human intervention (no fleet of
forecasters working on probabilistic IFPS). - Disadvantages
- Computationally expensive, especially if model is
changing frequently. - Not getting forecast improvement directly through
improving the model.
5Application NCEP/CPCs 6-10 day outlook
Map of probabilities of above / below / near
normal. 33 percent probability assumed in near
normal unless above or below gt 67 percent.
6Comparison against NCEP / CPC forecasts at 155
stations, 100 days in winter 2001-2002
temperature forecasts
Reforecast calibrated Week-2 forecasts more
skillful than operational NCEP/CPC 6-10 day,
which was based on human blending of NCEP, ECMWF,
other tools.
precipitation forecasts
7Reforecast-based example floods causing La
Conchita, California landslide, 12 Jan 2005
week-2 from reforecast
6-10 day from reforecast
8Reforecast model brought back into production
and used operationally at NCEP/CPC because of
the usefulness of reforecast products. Also
working on transition of products to NCEP/HPC
9Application downscaled precipitation forecasts
using analog technique
On the left are old forecasts similar to todays
ensemble- mean forecast. The data on the right,
the analyzed precipitation conditional upon the
forecast, can be used to statistically adjust and
downscale the forecast. Analog approaches like
this may be particularly useful for hydrologic
ensemble applications, where an ensemble of
realizations is needed.
10Downscaled analog probability forecasts
11Verified over 25 years of forecasts skill
scores use conventional method of calculation
which may overestimate skill (Hamill and Juras
2006).
12Tornado probability forecasting
forecast wind shear and instability were used as
predictors in an analog approach.
13ECMWFs reforecast experiments
ECMWF got excited by our results and produced a
test reforecast data set to see if they would get
a big forecast improvement even with their
much-improved ensemble forecast system.
- Model 2005 version of ECMWF model T255
resolution. - Initial Conditions 15 members, ERA-40 analysis
singular vectors - Dates of reforecasts 1982-2001, Once-weekly
reforecasts from 01 Sep - 01 Dec, 14 weeks total.
So, 20y ? 14w ensemble reforecasts 280
samples. - Data obtained by NOAA / ESRL T2M and
precipitation ensemble over most of North
America, excluding Alaska. Saved on 1-degree lat
/ lon grid. Forecasts to 10 days lead.
14ECMWF, raw and post-processed
Note 5th and 95th ile confidence intervals very
small, 0.02 or less
15ECMWF, raw and post-processed
In this metric, calibrated 4-5 day forecasts now
as skillful as uncalibrated 1-day forecast.
Note 5th and 95th ile confidence intervals very
small, 0.02 or less
16Precipitation 5-mm reliability diagrams(90-km
forecasts verified against 32-km North American
regional reanalysis)
horizontal lines indicate distribution of
climatology
error bars from block bootstrap
Raw forecasts have poor skill in this strict
BSS much improved with calibration
17Precipitation skill with weekly and 30-day
training data sets
Compared use of the once-weekly ? 20-year
reforecast data set to calibration using only the
past 30 days of forecasts. Substantial benefit
of weekly reforecasts relative to 30-day training
data sets, especially at high thresholds for the
more rare heavy precipitation events, a longer
training data set is needed.
18ECMWF newsletter, Autumn 2008 Reforecasts
operational at ECMWF
19NOAA and new reforecasts?
- Climate Forecast System Reanalysis and Reforecast
(operational 2010-2011) - 1 reforecast member per day T126 model used for
climate forecasts. - GEFS Global ensemble forecast system plan to do
1 reforecast member per day in real time, e.g.,
on 1 Dec 2009, do 1 Dec 2008, 2007, etc.
20Reforecast issues
Results with ECMWF data set suggest additional
benefit from more members.
21Reforecast issues.Which method of calibration?
NCEP was hopeful that a new calibration method,
Bayesian Processor of Forecasts would allow
them to do improved calibrations with small
training data sets, lessening the need for
reforecasts. Recent ESRL/PSD research
has demonstrated serious problems with the
proposed algorithm relative to ones that have
been previously demonstrated to be effective.
22Other reforecast issues
- Which calibration method is best for other
important parameters? (clouds, precipitation
amount/type, winds, severe weather, etc.) - What is NOAAs long-term strategy for production
of probabilistic forecasts? How does
reforecasting fit with the overall strategy for
probabilistic forecasting? - Who does what? ESRL or NCEP to compute
reforecasts? NCEP or MDL to post-process?
Technique development at ESRL, NCEP, and/or MDL? - Best configuration of reforecast data set?
- Reforecasts computed in real-time, with evolving
model, few members (NCEPs preferred approach for
GEFS), or - Fixed model, large reforecast data set, computed
all at once, and used for many years? - Does the configuration of the best reforecast
data set vary with weather parameter? For
example, do hydrologists need more data than is
needed for temperature calibration?
23FY11 Alternative (Climate goal)
- Develop next-generation reforecast with modern
NCEP model, based on CFSRR reanalyses. - Examine reforecast ensemble size issues - how
much of the benefit is possible with reduced size
data set (fewer members, reforecasts every x
days). - Compare against evolving NCEP ensemble system
with bias corrections from 1-member reforecast. - Develop suite of new experimental products.
- Etc.
- NWS did not support making this climate activity
part of FY11 core its above core currently.
24Conclusions
- Reforecasts shown to aid in calibration of
forecasts for a wide variety of applications. - Still a large benefit from forecast calibration,
even with state-of-the-art ECMWF forecast model. - Many remaining issues to be explored, and renewed
importance given recent emphasis on uncertainty
forecasting in NWS. - Still working on securing stable funding CPU
time for next-generation reforecasts.