Robin Hogan

1
What is the half-life of a cloud forecast?

• Robin Hogan
• Ewan OConnor
• University of Reading, UK

2
Cloud fraction Bony diagrams

• Winter (Oct-Mar) Summer (Apr-Sep)

ECMWF model Chilbolton
3
with snow
Winter (Oct-Mar) Summer (Apr-Sep)
ECMWF model Chilbolton
4
How good is a forecast?

• Most model comparisons evaluate the cloud
  climatology
• What about individual forecasts?
• Standard measure shows forecast half-life of 8
  days (left)
• But virtually insensitive to clouds!

ECMWF 500-hPa geopotential anomaly correlation

• Overview of talk
• Which skill scores have the most desirable
  properties?
• How does skill depend on spatial scale, lead time
  etc?
• If it has an inverse-exponential decay with
  forecast lead time, what is the half-life of
  the forecast?

5
(No Transcript)
6
Joint PDFs of cloud fraction

• Raw (1 hr) resolution
• 1 year from Murgtal
• DWD COSMO model

7
Desirable properties of skill scores

• Equitable all random forecasts score zero
• This is essential!
• Note that forecasting the right climatology
  versus height but with no other skill should also
  score zero
• Proper not possible to hedge your bets
• Some scores reward under- or over-prediction
  (e.g. hit rate)
• Jolliffe and Stephenson not possible to be
  equitable and proper!
• Independence of how often cloud occurs
• Almost all scores asymptote to 0 or 1 for
  vanishingly rare events
• Dependence on 10x10 joint PDF, not just 2x2 table
• Difference between cloud fraction of 0.9 and 1 is
  as important for radiation as a difference
  between 0 and 0.1
• Linearity so that can fit an inverse exponential
• Some scores (Yules Q) saturate at the
  high-skill end

8
Three quite good scores

• 1. Log of odds ratio LORln(ad/bc)
• Good properness properties
• Unbounded a perfect forecast scores infinity!
• Generalized skill score (x-xrandom)/(xperfect-xr
  andom)
• Where x is any number derived from the joint
  PDF
• Resulting scores vary linearly from random0 to
  perfect1
• 2. Heidke skill score xad
• Monotonically related to the Equitable Threat
  Score, but more linear
• 3. Linear Brier score xmean absolute difference
• Sensitive to cloud fraction errors in model for
  all values of cloud fraction

9
Score versus lead time, Murgtal 2007

• Both scores well fitted by SS0exp(-t/t0)
• Half lifeln(2)t0
• Met Office NAE has higher scores than DWD COSMO
• But apparently a shorter half life (2.7 days
  versus 4.1 days)
• Obviously need longer lead-time forecasts to
  check this!

10
DWD COSMO versus hours averaged

• Skill and lead time both increase with the number
  of hours over which cloud fraction is averaged
• Larger-scale features are easier to forecast

11
Met Office versus hours averaged

• Statistics poorer for larger number of hours
  averaged
• Log of odds ratio and Heidke skill score are
  sensitive to cloud fraction threshold
• Linear Brier score considers all cloud fractions
  so more robust

12
Summary

• Half-life of a cloud forecast is between 2.5 and
  5 days
• Relatively insensitive to skill score (provided a
  good one is used)
• Compare to 8 days for ECMWF 500-hPa geopotential
  height forecast
• Skill at forecasting cloud increases somewhat for
  larger scale features
• Important to assess the merits of various skill
  scores
• At least 5 criteria to judge against, and none
  are good on all
• Plenty of bad ones to use (hit rate, false-alarm
  rate etc)!
• Worth trying Stephensonss Extreme Dependency
  Score, which is good for very rare events
• Wish list
• Obtain Met Office cloud forecasts beyond a lead
  time of 3 days
• Compare skill of the Met Office model at
  different model resolutions, but averaged to the
  same scale
• Can we see what skill comes from global model at
  boundaries, what comes from mesoscale data
  assimilation etc?

13
(No Transcript)
14
Contingency tables
Comparison with Met Office model over
Chilbolton October 2003
Observed cloud Observed clear-sky

• Model cloud
• Model clear-sky

A Cloud hit B False alarm
C Miss D Clear-sky hit
15
Simple skill scoreHit Rate
Met Office short range forecast
Météo France old cloud scheme

• Hit Rate fraction of forecasts correct
  (AD)/(ABCD)
• Consider all Cabauw data, 1-9 km
• Increase in cloud fraction threshold causes
  apparent increase in skill.

16
Scores independent of clearsky hits

• False alarm rate fraction of forecasts of cloud
  which are wrong B/(AB)
• perfect forecast is 0

• Probability of detection fraction of clouds
  correctly forecast A/(AC)
• perfect forecast is 1

• Skill decreases as cloud fraction threshold
  increases

17
More sophisticated scores

• Equitable threat score (A-E)/(ABC-E) where E
  removes those hits that occurred by chance.

• Yules Q (?-1)/(?1) where the odds ratio
  ?AD/BC.
• Advantage little dependence on frequency of cloud

• For both scores, 1 perfect forecast, 0 random
  forecast
• From now on use Equitable threat score with
  threshold of 0.1.

18
Skill versus height

• Model performance
• ECMWF, RACMO, Met Office models perform similarly
• Météo France not so well, much worse before April
  2003
• Met Office model significantly better for shorter
  lead time
• Potential for testing
• New model parameterisations
• Global versus mesoscale versions of the Met
  Office model

19
Monthly skill versus time

• Measure of the skill of forecasting cloud
  fractiongt0.05
• Comparing models using similar forecast lead time
• Compared with the persistence forecast
  (yesterdays measurements)
• Lower skill in summer convective events

20
Skill versus lead time

• Unsurprisingly UK model most accurate in UK,
  German model most accurate in Germany!

• Half-life of cloud forecast 2 days
• More challenging test than 500-hPa geopotential
  (half-life 8 days)