Interpretation of PM2.5 Model Performance Metrics paralysis by analysis

1
Interpretation of PM2.5 Model Performance
Metricsparalysis by analysis

• Kirk Baker
• Lake Michigan Air Directors Consortium
• Midwest Regional Planning Organization
• February 2004

2
PM2.5 Specie Scatter-plot
PSO4 PNO3 SOIL COARSE OC EC
3
Mean (Gross) Bias
PSO4 PNO3 SOIL COARSE OC EC
4
Mean (Gross) Bias
5
Mean Absolute (Gross Error)
PSO4 PNO3 SOIL COARSE OC EC
6
Mean Absolute (Gross Error)
7
Normalized Bias
PSO4 PNO3 SOIL COARSE OC EC
8
Normalized Bias
9
Normalized Gross Error
PSO4 PNO3 SOIL COARSE OC EC
10
Normalized Gross Error
11
Fractional Error
PSO4 PNO3 SOIL COARSE OC EC
12
Fractional Error
13
Fractional Bias
PSO4 PNO3 SOIL COARSE OC EC
14
Fractional Bias
15
Modified Index of Agreement
16
Coefficient of Determination
17
Mean Fractional Bias
PSO4 PNO3 SOIL COARSE OC EC
18
Relative Response Evaluation

• Used CART Analysis to find similar meteorological
  episode to June 2002
• Model June 2002 and past year using 2002
  meteorological inputs
• Construct an approximation of the past episode
  emission inventory
• Examine model response in past episode and
  current episode for O3 and SO2/PSO4 where
  available

19
Multiple Model Diagnostics
20
Regulatory v. Scientific Evaluation

• Performance driven by standard averaging time
• Attainment tests use relative factor approach by
  specie
• Metric guidelines by specie
• Monitor error
• Chemistry uncertainty
• Gross emissions uncertainty
• Regulatory importance

• Mindset of trying to get the right answer
  everywhere, all the time, for all PM2.5 species
  and precursor species
• Middle Ground?