AQF Capability_Operational Readiness Review for Expanded Ozone Predictions

1
Simplifying the fvs statistical calculation
and plotting process Jeff McQueen and Perry
Shafran NCEP/EMC Thursday, November 06, 2014

2
Overview

• core script make-fvs.sh on tempest Dew
• tempest /export-4/tempest/wd22jm/fvs/scripts
• dew /meso/save/wx22mc/jeff/FVS-SCRIPTS
• Summarizes vsdb records for user selected dates,
  regions, variables
• fvs s step
• Computes error statistics (eg rmse, bias) and
  plots error statistics averaged by user selected
  bins
• fvs p step
• Diurnal plots averaged by forecast hour DIU
• Daily Time Series plots for a particular forecast
  hour DTS
• Upper air plots averaged by multiple vertical
  levels UPA
• Plots of statistics averaged by event thresholds
  FHO

3
FVS Scripts Files

• Scripts located at
• tempest /export-4/tempest/wd22jm/fvs/scripts
• dew /meso/save/wx22mc/jeff/FVS-SCRIPTS
• Copy the following files
• make-fvs.sh
• cr8trace.sh ? called by make-fvs.sh
• FVS.CTL ? Control file for user sets (varbs,
  models) or can be created thru cr8cfg.sh and
  runfvs.sh master script
• cr8cfg.sh ? Create FVS.CTL from exported fields
  set in master script (eg runfvs.sh)
• runfvs.sh ? Example master script that calls
  cr8cfg.sh and
• make-fvs.sh
• RLBL.CFG ? Region label configuration file
• SLBL.CFG ? Statistics label configuration file

4
Make-fvs.sh settings

• Command line plot-type and forecast cycle
• make-fvs.sh DTS 12
• creates daily time series plot for 12 UTC cycle
• Plottype DTS, DIU, UPA, FHO
• Forecast cycle 00, 06, 12, 18 or nothing for all
  cycles
• CTL file or export variables from master script
• FVS.CTL
• or
• export TRACESNAM/218 NAMY/218
• export PLOTHRS84
• export stno1101 1102 1 2
• export PLVLSFC
• export RGNG236
• export VARBT
• export appstat1
• scrdir/cr8cfg.sh -? creates an FVS. CTL
  file
• scrdir/make-fvs.sh DTS cycle

5
Key fields for FVS

• Model(s) to verify
• eg NAM/218, GFS/212, SRMEAN/212
• Region(s) to verify
• eg G236, G216, G104/NEC
• Variable(s) to verify
• eg T, VWND, RH, DPT, Z, SLP, APCP/24, TOTCLD,
  AOD, smoke, OZON1, OZON8,OZMX1, OZMX8
• Vertical level(s) for each variable
• eg SFC, P850, P700, P850-P700, ATMOS
• Forecast hour(s) to verify
• eg 36, 48, 84
• Date/Time range
• Eg 200807280000 200808272359
• FVS Statistics code to compute and plot
• Eg 1101 (bias), rmse (1102)

6
DTS DAILY TIME SERIES PLOTS for a Particular
Forecast Hour NAM vs NAM-Y 2 m Temperature
Errors (C)
Error vs Date
36h Forecast for 12 Z cycle runs (Valid 00 Z)
7
UPPER AIR PLOTS for a particular forecast hour
UPANAM vs NAM-Y Temperature Errors (C)
Pressure Level Vs error
36h Forecast (Valid 00 Z)
8
Errors Averaged by Forecast hours for all Dates
DIUrnal plotsNAM-Y vs NAM RH errors (Aug. 2007)
2 m Relative Humidity Error ()
Error vs Forecast hour
9
Errors Averaged by threshold event for all
dates FHONAM vs SREF CSI 48 hour precip errors
Error vs threshold
10
Settings to plot errors for different regions
from two different models (NAM NAMEXPR)

• FVS. CTL file format
• 
  
• NAM/218 NAMEXPR/218 NAM/218 NAMEXPR/218
  NAM/218 NAMEXPR/218
• G236 G236 G104/SWC
  G104/SWC G104/GMC G104/GMC RGN
• SLP SLP SLP
  SLP SLP SLP
  VARB
• SFC SFC SFC
  SFC SFC SFC
  PLVL
• 36 36 36
  36 36
  36 FHR
• 1 1101 1102 appstat statistics codes
• 200708050000 200708182359 Date Range
• 
  
• Will create 6 traces but 12 time-series
• appstat1 put all statistics on one plot
• 0 make separate plot files for
  each stat type (1101 and 1102)
• NAM/218 SLP for CONUS region (G236) at the SFC
  for 36 hour forecasts from 8/5/2007- 8/1/8/2007
  for stats 1101 (rmse) and 1102 (bias)
• NAMEXPR/218 SLP for CONUS region (G236) at the
  SFC for 36 hour forecasts from 8/5/2007-
  8/1/8/2007 for stats 1101 (rmse) and 1102 (bias)
• NAM/218 SLP for SW Coast region (SWC) at the SFC
  for 36 hour forecasts from 8/5/2007- 8/1/8/2007
  for stats 1101 (rmse) and 1102 (bias)
• NAMEPR/218 SLP for SW Coast region (G236) at the
  SFC for 36 hour forecasts from 8/5/2007-
  8/1/8/2007 for stats 1101 (rmse) and 1102 (bias)
• NAM/218 SLP for NW Coast region (NWC) at the SFC
  for 36 hour forecasts from 8/5/2007- 8/1/8/2007
  for stats 1101 (rmse) and 1102 (bias)

11
Resulting trace.ctl file for fvs s/p step

1 2 0 2 1 1 1 0/0/0/0 18 0 0 NAM/218 0 2 2 0 2 1
1 1 0/0/0/0 18 0 0 NAMEXPR/218 0 3 2 0 2 1 1 1
0/0/0/0 18 0 0 NAM/218 0 4 2 0 2 1 1 1 0/0/0/0 18
0 0 NAMEXPR/218 0 5 2 0 2 1 1 1 0/0/0/0 18 0 0
NAM/218 0 6 2 0 2 1 1 1 0/0/0/0 18 0 0
NAMEXPR/218 0 1 3 0 2 1 1 1 0/0/0/0 9 36 0 0 0 2
3 0 2 1 1 1 0/0/0/0 9 36 0 0 0 3 3 0 2 1 1 1
0/0/0/0 9 36 0 0 0 4 3 0 2 1 1 1 0/0/0/0 9 36 0 0
0 5 3 0 2 1 1 1 0/0/0/0 9 36 0 0 0 6 3 0 2 1 1 1
0/0/0/0 9 36 0 0 0 1 15 0 2 1 1 1 0/0/0/0 18 0 0
0000 0 2 15 0 2 1 1 1 0/0/0/0 18 0 0 0000 0 3 15
0 2 1 1 1 0/0/0/0 18 0 0 0000 0 4 15 0 2 1 1 1
0/0/0/0 18 0 0 0000 0 5 15 0 2 1 1 1 0/0/0/0 18 0
0 0000 0 6 15 0 2 1 1 1 0/0/0/0 18 0 0 0000 0 1
14 0 2 1 1 1 0/0/0/0 18 0 0 SFC 0 2 14 0 2 1 1 1
0/0/0/0 18 0 0 SFC 02 2 0 2 1 1 1 0/0/0/0 3 14 0
2 1 1 1 0/0/0/0 18 0 0 SFC 04 14 0 2 1 1 1
0/0/0/0 18 0 0 SFC 0 5 14 0 2 1 1 1 0/0/0/0 18 0
0 SFC 0 6 14 0 2 1 1 1 0/0/0/0 18 0 0 SFC 0 1 8 0
2 1 1 1 0/0/0/0 18 0 0 ONLYSF 0 2 8 0 2 1 1 1
0/0/0/0 18 0 0 ONLYSF 0
4 10 0 2 1 1 1 0/0/0/0 18 0 0 SL1L2 0 5 10 0 2 1
1 1 0/0/0/0 18 0 0 SL1L2 0 6 10 0 2 1 1 1
0/0/0/0 18 0 0 SL1L2 0 1 12 0 2 1 1 1 0/0/0/0 18
0 0 SLP 0 2 12 0 2 1 1 1 0/0/0/0 18 0 0 SLP 0 3
12 0 2 1 1 1 0/0/0/0 18 0 0 SLP 0 4 12 0 2 1 1 1
0/0/0/0 18 0 0 SLP 0 5 12 0 2 1 1 1 0/0/0/0 18 0
0 SLP 0 6 12 0 2 1 1 1 0/0/0/0 18 0 0 SLP 0 1 5 1
2 1 1 1 0/0/0/0 15 0 0 200708050000
200708182359 2 5 1 2 1 1 1 0/0/0/0 15 0 0
200708050000 200708182359 3 5 1 2 1 1 1
0/0/0/0 15 0 0 200708050000 200708182359 4 5 1 2
1 1 1 0/0/0/0 15 0 0 200708050000 200708182359 5
5 1 2 1 1 1 0/0/0/0 15 0 0 200708050000
200708182359 6 5 1 2 1 1 1 0/0/0/0 15 0 0
200708050000 200708182359

3 14 0 2 1 1 1 0/0/0/0 18 0 0 SFC 0 4 14 0 2 1 1
1 0/0/0/0 18 0 0 SFC 0 5 14 0 2 1 1 1 0/0/0/0 18
0 0 SFC 0 6 14 0 2 1 1 1 0/0/0/0 18 0 0 SFC 0 1 8
0 2 1 1 1 0/0/0/0 18 0 0 ONLYSF 0 2 8 0 2 1 1 1
0/0/0/0 18 0 0 ONLYSF 0 3 8 0 2 1 1 1 0/0/0/0 18
0 0 ONLYSF 0 4 8 0 2 1 1 1 0/0/0/0 18 0 0 ONLYSF
0 5 8 0 2 1 1 1 0/0/0/0 18 0 0 ONLYSF 0 6 8 0 2 1
1 1 0/0/0/0 18 0 0 ONLYSF 0 1 9 0 2 1 1 1
0/0/0/0 18 0 0 G236 0 2 9 0 2 1 1 1 0/0/0/0 18 0
0 G236 0 3 9 0 2 1 1 1 0/0/0/0 18 0 0 G104/SWC
0 4 9 0 2 1 1 1 0/0/0/0 18 0 0 G104/SWC 0 5 9 0 2
1 1 1 0/0/0/0 18 0 0 G104/GMC 0 6 9 0 2 1 1 1
0/0/0/0 18 0 0 G104/GMC 0 1 10 0 2 1 1 1
0/0/0/0 18 0 0 SL1L2 0 2 10 0 2 1 1 1 0/0/0/0 18
0 0 SL1L2 0 3 10 0 2 1 1 1 0/0/0/0 18 0 0 SL1L2
0 4 10 0 2 1 1 1 0/0/0/0 18 0 0 SL1L2 0 5 10 0 2
1 1 1 0/0/0/0 18 0 0 SL1L2 0 6 10 0 2 1 1 1
0/0/0/0 18 0 0 SL1L2 0 1 12 0 2 1 1 1 0/0/0/0 18
0 0 SLP 0 2 12 0 2 1 1 1 0/0/0/0 18 0 0 SLP 0 3
12 0 2 1 1 1 0/0/0/0 18 0 0 SLP 0
12
NAM vs NAMEXPR 36 hTemperature Errors by date
for different regions
13
Example FVS.CTL files to plotmodel performance
for multiple FHRs
FVS.CTL
NAM/218 NAM/218 NAM/218 NAM/218 AM/218 NAM/218
G236 G236 G236 G236 G236 G236 T T T T T T
SFC SFC SFC SFC SFC SFC 12 24 36 48 60 72 1
1101 1102 200712010000 200801282359
FVS.CTL created with the following UNIX Shell
ksh calls
export appstat1 export TRACESNAM/218 NAM/218
NAM/218 NAM/218 NAM/218 NAM/218 export RGN
G236 export PLOTHRS 12 24 36 48 60 72 export
PLVLSFC export VARBT export stno1101 1102
" scrdir/cr8cfg.sh scrdir/make-fvs.sh dts cycle
14
NAM T Performance by FHRWinter 07-08 Resulting
Plot
15
Example FVS.CTL files to plotmodel errors for
different variables
FVS.CTL
NAM/218 NAM/218 NAM/218 NAM/218 G236 G236 G236
G236 T DPT VWND SLP SFC SFC SFC SFC 00 00
00 00 0 1101 1102 200712010000 200802012359
FVS.CTL created with the following UNIX Shell
ksh calls
export TRACESNAM/218 NAM/218 NAM/218
NAM/218 export RGN G236 export
PLOTHRS00 export PLVLSFC export VARBT DPT
VWND SLP export stno1101 1102
" scrdir/cr8cfg.sh scrdir/make-fvs.sh DIU cycle
16
NAM T, VWND, SLP, Z Performance by FHRWinter
07-08 Resulting Plot
17
make-fvs.sh notes

• Any combination of fields can be plotted on one
  graph
• models
• regions
• variables
• vertical levels
• forecast hours
• Statistics can be plotted on one plot with
• appstat1 (cmb)
• Or on separate plots
• appstat0 (rmse, bias, etc)
• Output plot name VARBLVL_STATFHR_MDL-TYP
  E.gif
• TSFC_cmb84_mean.gif

18
Make-fvs.sh advanced features

• Region labels edit RLBL.CFG
• Eg G236 CONUS
• G104/NEC North-East-Coast
• Statistics labels edit SLBL.CFG
• Eg 1101 Root Mean Squared Error
• Bias
• Probabilistic plotting is set by Stat. type
• Eg RHET ? Talagrand Ranked Historgram
• RHNT? Equally Likely Ranked Histogram
• ESL1L2 ? Ensemble mean RMSE, Bias, statistical
  consistency

19
SLBL.CFG
2 mean Mean 1001 bias Bias 1002 ets
Equitable-Threat-Score 1003 csi
Critical-Success-Index 1004 far
False-Alarm-Ratio 1007 pod Probability-of-Detect
ion 1021 far False-Alarm-Rate 1034 hit
Hit-Rate 1101 bias BIAS 1102 rmse
Root-Mean-Squared-Error 1103 corr
Correlation 1107 sprd Forecast Spread 1109 stdv
Standard-Deviation 1112 stdv Standard-Deviation 1
116 cons Statistical-Constistancy 1201 bias
BIAS 1202 rmse Root-Mean-Squared-Error 1207 sprd
Forecast Spread 1209 stdv Standard-Deviation 121
2 stdv Standard-Deviation 1216 cons
Statistical-Constistancy 1600 RHNT
Equally-Likely-Histogram 1601 RHET
Talagrand-Ranked-Histogram 1605 OUTL
Outlier-Percent 1606 MRE Missing-Rate-Error
20
RLBL.CFG

• G236 CONUS
• G212 North-America
• G218 North-America
• G245 East-US
• G246 West-US
• G246 Cent-US
• SWC Southwest-Coast
• NWC Northwest-Coast
• NMT Northern-Mountains
• SMT Southern-Mountains
• SPL Southern-Plains
• NPL Northern-Plains
• MDW Midwest
• LMV Lower-Mississippi-Valley
• GMC Gulf-Mexico
• NEC Northeast
• SEC Southeast
• APL Appalachian-Mountains
• WCA Western-Canada

21
Verification Sub-domains
SW Coast includes both LA, SD SJV
22
Models available for analysis tempest
/mmb/wd20er/vsdb/dataDescription
http//www.emc.ncep.noaa.gov/mmb/mmbpll/mmbverif/v
eriflog.html

• akarw dgexx edetnx esatp namcenthw ndas expprem r
  saspbc
• akem eastarw edetp esatpbc nameasthw ndasexpr
  prnmm rsaspx
• Aknmm eastem edetpbc esatpx nameasthw_old ndasr pr
  nmm_old rsmh
• aknmm_old east nmmedetpx gdas namexp ndasx rrasn
  ruc
• arwctl east nmm_old ekfcc gfs namexpr ndasy
  rrasnbc rucx
• arwctlbc ebmjc ekfccbc gfseo namhihw ngm rrasnx
  srmean
• Arwctlx ebmjcbc ekfccx hiarw namhihw_old nmmctl rr
  asp srmeanbc
• arwn1 ebmjcx ekfcn hiem namprhw nmmctlbc rraspbc
  srmeanx
• arwn1bc ebmjn ekfcnbc hinmm namprhw_old nmmctlx rr
  aspx westarw
• arwn1x ebmjnbc ekfcnx hinmm_old
  namr nmmn1 rsasc westem
• arwp1 ebmjnx ekfcp namwesthw nmmn1bc
  rsascbc westnmm
• arwp1bc ebmjp ekfcpbc namwesthw_old nmmn1x rsascx
  westnmm_old
• arwp1x ebmjpbc ekfcpx nam namx nmmp1 rsasn
• centem ebmjpx esatn nam40 namx40 nmmp1bc
  rsasnbc
• centnmm edetn esatnbc namakhw namy nmmp1x rsasnx
• dgexo edetnbc esatnx namakhw_old
  ndas prarw rsasp

23
SLIL2 Statistics types

• 
  
• SL1L2 -- L1L2 norm statistics for scalars
  
• 
  
• Computation Code Action
  
• 
  
• 9002 Trace of data field
  values averaged over all pts
• -n Return raw data for field
  n (undivided sum)
• -1 Contributing record count
  at all trace points
• 1 Return data from field 1
  N, the data count
• 2 Return data from field 2
  forecast mean
• 3 Return data from field 3
  observation mean
• 4 Return data from field 4
  F O mean
• 5 Return data from field 5
  F 2 mean
• 6 Return data from field 6
  O 2 mean
• 
  
• 1101 Compute BIAS (SSAL1L2,
  SL1L2)
• 1102 Compute RMS error
  (SSAL1L2, SL1L2)
• 1103 Compute Correlation
  (SSAL1L2, SL1L2)
• 1104 Compute Covariance
  (SSAL1L2,SL1L2)

24
FHO Statistics types

• 
  
• FHO -- Forecast, hit, observation rate
  statistics
• 
  
• Computation Code Action
  
• 
  
• 9002 Trace of data field
  values averaged over all pts
• -n Return raw data for field
  n (undivided sum)
• -1 Contributing record count
  at all trace points
• 1 Return data from field 1
  N, the data count
• 2 Return data from field 2
  F, fraction forecast
• 3 Return data from field 3
  H, fraction corret
• 4 Return data from field 4
  O, fraction observed
• 
  
• 1001 Compute BIAS from FHO
  values
• 1002 Compute Equitable Threat
  Score from FHO values
• 1003 Compute Critical Success
  Index from FHO values
• 1004 Compute False Alarm Ratio
  from FHO values
• 1005 Compute Postagreement
  from FHO values
• 1006 Compute Prefigurance from
  FHO values

Any of the preceding codes may be computed from
bias adjusted FHO values. To do this,
precede the 4-digit code number with one of
the following bias adjustment specification
prefixes

101 Fedor Mesinger's dH/dF
bias adjustment 102
Conservation of Odds Ratio bias adjustment
103 Fedor Mesinger's dH/dA
bias adjustment

Example 1 -1020003 displays the bias
adjusted hit count Example 2
1021008 displays the bias adjusted threat score

Any of the
preceding 10xx series that are N dependent may be
computed using FO-H from the Mth
preceding threshold. The first M
thresholds use the original value of N, the total
number. Thres- hold must be the
independent variable. To do this, precede the
10xx code with the following prefix


20M The FO-H value from i-M threshold
is used, Mgt0 and Mlt10

25
Probabilistic Stats

• RHNT Ranked Histogram Nearest Truth
• Equally Likely Diagram (1600)
• RHNT5 E-L diagrams for ensemble subsets
• SREF/ETA, SREF/EKF, SREF/RSM, SREF/WRF
• RHET Ranked Histogram Equal to Truth
• Talagrand Diagram (1600)
• RHET5 Talagrand diagrams for ensemble subsets
• Also can plot Outlier Percentage (1605) Missing
  Rate Error(1606)
• ESL1L2 Partial sums for ensemble groups
• rmse (1102) , bias (1101), spread (1107),
  statistical consistency(1116)

26
Ranked Histogram examples
export TRACESSREFX/ETA SREFX/EKF SREFX/RSM
SREFX/WRF export RGN G236 export
PLOTHRS00 export PLVLP850 export VARBT export
sttypRHET5 ? Talagrand diagram export
sttypRHNT5 ? equally likely export stno1601
export
stno1600 scrdir/cr8cfg.sh scrdir/make-fvs.sh
dts cycle
27
ESL1L2 rmse, bias, spread, statistical
consistency
export TRACESSREFX/ETA SREFX/EKF SREFX/RSM
SREFX/WRF export RGN G236 export
PLOTHRS00 export PLVLSFC export VARBT export
sttypESL1L2 export stno1101 1102 1116 1107 ?
bias, rmse, statistical cons., spread scrdir/cr8c
fg.sh scrdir/make-fvs.sh dts cycle
28
Partitioned Brier Score (PBS)
GENERAL USE
CODES

1301 Compute Brier
Score for weather risks



Codes for three risk category PBS statistics


1302 Compute Fraction observed for NO
RISK frcst 1303 Compute
Fraction observed for LOW RISK frcst
1304 Compute Fraction observed for MOD
RISK frcst 1305 Compute
Fraction observed for HIGH RISK frcst
1306 Compute optimal Target Brier
Score


GEN
ERAL USE CODES

For all non-zero
risk/probability categories
1307 Compute BIAS Score
from PBS values 1308
Compute Equitable Threat Score from PBS
values 1309 Compute False
Alarm Rate from PBS values 1310
Compute False Alarm Ratio from PBS
values 1311 Compute
Postagreement from PBS values
1312 Compute Prefigurance from PBS
values 1313
Compute Probability of Detection from PBS
values 1314 Compute Threat
Score from PBS values


Codes
for three risk category PBS statistics

For
moderate high risk categories
1315 Compute
BIAS Score from PBS values
1316 Compute Equitable Threat Score
from PBS values 1317
Compute False Alarm Rate from PBS values
1318 Compute False Alarm
Ratio from PBS values 1319
Compute Postagreement from PBS values
1320 Compute Prefigurance
from PBS values 1321
Compute Probability of Detection from PBS
values 1322 Compute Threat
Score from PBS values

For only high risk category
1323
Compute BIAS Score from PBS values
1324 Compute Equitable
Threat Score from PBS values 1325
Compute False Alarm Rate from PBS values
1326 Compute False Alarm
Ratio from PBS values 1327
Compute Postagreement from PBS values
1328 Compute Prefigurance
from PBS values 1329
Compute Probability of Detection from PBS
values 1330 Compute Threat
Score from PBS values

GENERAL USE
CODES

Overall sums
1331
Total number of forecast events (sum
2-gtM/2)N 1332 Total number
observed (sum M/21-gtM)N,
where M is the number of data field
elements
For all
non-zero risk/probability categories
1333 Compute
Heidke Skill Score
1334 Compute Peirce-Hanssen-Kuipers
Skill Score 1335
Compute ODDS RATIO
1336 Compute log of ODDS
RATIO 1337
Compute Standard Error of log of ODDS RATIO
1338 Compute ODDS RATIO
Skill Score 1339
Compute ODDS RATIO Skill Score Cubed



Codes for three risk category PBS
statistics

For moderate high risk categories
1340
Compute Heidke Skill Score
1341 Compute
Peirce-Hanssen-Kuipers Skill Score
1342 Compute ODDS RATIO
1343 Compute
log of ODDS RATIO
1344 Compute Standard Error of log of
ODDS RATIO 1345 Compute
ODDS RATIO Skill Score
1346 Compute ODDS RATIO Skill Score
Cubed
For
only high risk category
1347 Compute
Heidke Skill Score
1348 Compute Peirce-Hanssen-Kuipers
Skill Score 1349
Compute ODDS RATIO
1350 Compute log of ODDS
RATIO 1351
Compute Standard Error of log of ODDS RATIO
1352 Compute ODDS RATIO
Skill Score 1353
Compute ODDS RATIO Skill Score Cubed



GENERAL USE
CODES

For all PBS categories
1354
Compute sample climatological BS
(uncertainty) 1355 Compute
Reliability term of BS decomposition
1356 Compute Resolution term of BS
decomposition 1357
Compute Uncertainty term of BS decomposition
1358 Compute Brier Skill
Score against sample climo 1359
Compute sample climatological frequency
1360 Attribute Diagram for
time series of PBS data 1361
Compute Extreme Dependency Score (EDS)