Evaluation of Models-3 CMAQ I. Results from the 2003 Release II. Plans for the 2004 Release Model Evaluation Team Members Prakash Bhave, Robin Dennis, Brian Eder, Peter Finkelstein, Rob Gilliam, Alice Gilliland, Steve Howard, John Irwin,

1
Evaluation of Models-3 CMAQI. Results from the
2003 ReleaseII. Plans for the 2004 Release
Model Evaluation Team Members Prakash Bhave,
Robin Dennis, Brian Eder, Peter Finkelstein, Rob
Gilliam, Alice Gilliland, Steve Howard, John
Irwin, Daiwen Kang, George Pouliot, Alfreida
Rankin, Shawn Roselle, Jenise Swall, Shaocai
YuAtmospheric Modeling DivisionNational
Exposure Research LaboratoryU.S. Environmental
Protection AgencyAtmospheric Sciences Modeling
DivisionAir Resources LaboratoryNational
Oceanic and Atmospheric AdministrationResearch
Triangle Park, NC 27711
2
Part I. Results of the 2003 Release Evaluation
The 2003 Release of CMAQ featured numerous
changes - updated science -
corrected implementations - efficiency
enhancements - bug fixes Largest
impacts involved the aerosol modeling,
especially - aerosol nitrate modeling
updated with more recent science Changes
resulted in substantially lower concentrations of
aerosol nitrate bringing them more in line with
observations.
3
Part I. Results of the 2003 Release Evaluation
In order to characterize the impact of these
changes and its overall performance, CMAQ was
evaluated over two simulation periods -
Summer 1999 (four weeks) - Winter 2002
(six weeks) - Against observational data
from four networks - IMPROVE -
CASTNet - STN (only winter 2002) -
SEARCH - Against modeled data from the 2002
Release.
4
Part I. Results of the 2003 Release Evaluation
CMAQ Simulation Attributes 
Emissions Gas-phase emissions are based on the
1999 EPA NEI. Chemistry SAPRC99 gas-phase
chemistry mechanism. Meteorology Derived from
MM5 and processed through MCIP (2.2).
Resolution Horizontal Contiguous U.S.
Domain. 32 km by 32 km grids(Summer) 36
km by 36 km grids (Winter) Vertical Set on a
sigma coordinate. 21 layers
(Summer) 24 layers (Winter)
5
Part I. Results of the 2003 Release Evaluation
Species SO4, NO3, NH4, PM2.5, OC,
EC Statistics Summary statistics and
plots Mean Bias (MB) Normalized Mean Bias
(NMB) Root Mean Square Error (RMSE) Normalized
Mean Error (NME) Results Full evaluation
available on AMDs website www.epa.gov/asmdnerl
/models3
6
Part I. Results of the 2003 Release Evaluation
Sulfate (µg m-3) June 15 July 16, 1999
NME
CASTNet CASTNet IMPROVE IMPROVE
n 264 n 490
R 0.90 R 0.90
MB 1.74 MB 0.75
NMB() 38.0 NMB() 38.4
RMSE 2.89 RMSE 2.37
NME() 46.4 NME () 61.9
NMB
7
Part I. Results of the 2003 Release Evaluation
Sulfate (µg m-3) Jan. 4 Feb. 19, 2002
NME
CASTNet CASTNet IMPROVE IMPROVE STN STN
n 407 n 728 n 1149
R 0.83 R 0.85 R 0.66
MB 0.15 MB 0.29 MB -0.26
NMB() 9.2 NMB() 31.5 NMB() -12.0
RMSE 0.60 RMSE 0.72 RMSE 1.16
NME() 25.7 NME() 53.1 NME() 35.9
NMB
8
Part I. Results of the 2003 Release Evaluation
2002 Release vs. 2003 Release
January 4 to January 20, 2002
June 15 to June 29, 1999



2002 Release
2002 Release
2003 Release
2003 Release
2003 Release produces substantially lower
concentrations of nitrate - bringing them
more in line with observations
9
Part I. Results of the 2003 Release Evaluation
Nitrate (µg m-3) June 15 July 16, 1999
NME
CAST CAST IMP IMP
n 264 N 415
R 0.27 R 0.36
MB -0.15 MB -0.11
NMB() -30.8 NMB() -39.1
RMSE 0.57 RMSE 0.51
NME() 75.7 NME() 95.0
NMB
10
Part I. Results of the 2003 Release Evaluation
Nitrate (µg m-3) Jan. 4 Feb. 19, 2002
NME
NME
CAST CAST IMP IMP STN STN
n 407 N 688 n 1044
R 0.76 R 0.54 R 0.39
MB 0.64 MB 0.20 MB -0.27
NMB() 46.8 NMB() 28.9 NMB() -8.0
RMSE 1.36 RMSE 1.26 RMSE 4.37
NME() 70.4 NME() 96.5 NME() 66.9
NMB
NMB
11
Part I. Results of the 2003 Release Evaluation
June 15 July 16, 1999
Species SO4 SO4 NO3 NO3 PM2.5 NH4 OC EC
Network CAS IMP CAS IMP IMP CAS IMP IMP
n 264 490 264 415 457 264 396 396
r 0.90 0.90 0.27 0.36 0.71 0.86 0.32 0.69
MB 1.74 0.75 -0.15 -0.11 -0.73 0.31 -0.02 0.00
NMB () 38.0 38.4 -30.8 -39.1 -9.8 22.7 -1.5 1.00
RMSE 2.89 2.37 0.57 0.51 4.70 0.70 1.35 0.24
NME () 46.4 61.9 75.7 95.0 40.3 36.5 67.2 52.6
Jan. 4 - Feb.19, 2002
Species SO4 SO4 SO4 NO3 NO3 NO3 PM2.5 PM2.5 NH4 NH4 OC OC EC EC
Network CAS STN IMP CAS STN IMP STN IMP CAS STN IMP STN IMP STN
n 407 1149 728 407 1044 688 927 714 407 1149 731 1106 731 1148
r 0.83 0.66 0.85 0.76 0.39 0.54 0.37 0.68 0.85 0.41 0.56 0.47 0.55 0.40
MB 0.15 -0.26 0.29 0.64 -0.27 0.20 0.51 1.49 0.33 0.07 0.14 -1.64 0.07 0.42
NMB() 9.2 -12.0 31.5 46.8 -8.0 28.9 4.1 40.3 40.0 5.2 17.2 -50.0 31.0 59.1
RMSE 0.60 1.16 0.72 1.36 4.37 1.26 10.4 3.81 0.51 1.46 0.93 3.32 0.48 1.17
NME () 25.7 35.9 53.1 70.4 66.9 96.5 50.0 68.9 47.2 57.7 70.0 60.4 81.0 95.0
12
Part I. Results of the 2003 Release Evaluation
Summary The 2003 Release of CMAQ featured many
changes, the largest of which impacted aerosol
nitrate Other species continue to be
simulated reasonably well - majority of
correlations are greater than 0.60 - majority
of model simulations are within 21 -
majority NMB within 35 - majority NME
less than 70 (depending on the specie and
observational network.)
13
Part II. Plans for the 2004 Release Evaluation
Through collaboration with AMD, OAQPS will soon
perform a full one year simulation (2001) using
the 2004 Release of CMAQ.
CMAQ Simulation Attributes 
Emissions 2001 Extrapolated from 1999 EPA NEI
(Version 3). Chemistry CBIV gas-phase
chemistry mechanism. Meteorology MM5 processed
through MCIP (2.2). Resolution Horizontal Cont
iguous U.S. Domain. 36 km by 36 km
grids Vertical Set on a sigma
coordinate. 14 layers
14
Part II. Plans for the 2004 Release Evaluation
A comprehensive evaluation of this simulation
will characterize the performance of CMAQ as well
as its individual components Meteorology
model Emissions model Chemical Transport
Model This evaluation will include the
statistics presented earlier, along with new
metrics - Shaocai Yus presentation diagnostic
s - Robin Dennis presentation And new,
innovative techniques examining model performance
over space and time John Irwins
presentation
15
Part II. Plans for the 2004 Release Evaluation
Meteorology Model Evaluation

• Surface-Based 2-m Temperature
• 2-m Specific humidity
• 10 m wind (speed, direction)
• Hourly precipitation
• Upper-Air wind profiles
• temperature profiles
• specific humidity profiles
• Space / Time Local, regional, and domain based
  summaries
• Diurnal, synoptic, monthly and seasonal
  summaries

16
Part II. Plans for the 2004 Release Evaluation
Meteorology Model Evaluation
Example 2 m Temperature (K)
1000 GMT 1 July 31 August
17
Part II. Plans for the 2004 Release Evaluation
Example 2 m Temperature (K) Mean
Error Mean Bias Winter
18
Part II. Plans for the 2004 Release Evaluation
Meteorology Model Evaluation
Example Precipitation Bias (cm)
Monthly January July
19
Part II. Plans for the 2004 Release Evaluation
Spatial Techniques
Example Kriging CASTNet SO4 concentrations
20
Part II. Plans for the 2004 Release Evaluation
Spatial Techniques
Example Kriging SO4 concentrations CMAQ -
CASTNet residuals
21
Part II. Plans for the 2004 Release Evaluation
Summary Through collaboration with AMD, OAQPS
will soon perform a full one year simulation
(2001) using the 2004 Release of CMAQ. An
unprecedented, comprehensive evaluation of this
simulation will characterize the performance of
CMAQ as well as its individual components Met
eorology model Emissions model Chemical
Transport Model This evaluation will include
new and innovative techniques examining the
spatial and temporal characteristics of model
performance.