Automated Homogenization of Monthly Temperature Series via Pairwise Comparisons

1
Automated Homogenization of Monthly Temperature
Series via Pairwise Comparisons

• Matthew Menne
• and
• Claude Williams
• NOAA/National Climatic Data Center
• Asheville, North Carolina USA

2
Outline

• Motivation The United States Historical
  Climatology Network (U.S. HCN)
• Overview of the pairwise homogenization
  algorithm
• Some examples
• Impact of inhomogeneities on U.S. temperature
  trends
• A word about GHCN-Daily

3
U.S. Climate Network
Historical Climatology Network
Cooperative Observer
4
U.S. HCN -- Version 1Monthly Data

• 1221 stations selected to comprise the HCN in
  mid-1980s
• Monthly dataset originally released in 1987
• Addressed the following
• Time of observation bias (Karl et al. 1986 Vose
  et al. 2003)
• Station History Changes (Karl and Williams 1987)
• Optimized reference series based on station
  history archives
• Urbanization (Karl et al. 1988)
• LiG to MMTS instrument change (Quayle et al. 1991)

5
U.S. HCN -- Version 2Monthly Data

• 1218 stations in a re-defined network
• Addresses
• Time of observation bias (Karl et al. 1986 Vose
  et al. 2003)
• Station history (documented) and undocumented
  changes (Menne and Williams, Journal of Climate,
  in review)
• Automated pairwise comparison of series

6
Station Siting Example of ratings assigned by
Watts based upon NOAA/NCDC criteria
Class 1 - Flat horizontal ground. Sensors
located at least 100 meters from artificial
heating
Class 2 - Same as Class 1, except no artificial
heating sources within 30 meters.
Class 3 - Same as Class 2, except no artificial
heating sources within 10 meters.
Siting Classification based upon standards for
NOAAs U.S. Climate Reference Network ftp//ftp.nc
dc.noaa.gov/pub/data/uscrn/documentation/program/X
030FullDocumentD0.pdf
Class 5 - Temperature sensor located next
to/above an artificial heating source
Class 4 - Artificial heating sources lt10 meters.
www.surfacestations.org
7
Station Siting Example of ratings assigned by
Watts based upon NOAA/NCDC criteria
Siting Classification based upon standards for
NOAAs U.S. Climate Reference Network ftp//ftp.nc
dc.noaa.gov/pub/data/uscrn/documentation/program/X
030FullDocumentD0.pdf
Class 5 - Temperature sensor located next
to/above an artificial heating source
Class 4 - Artificial heating sources lt10 meters.
8
Why Pairwise?

• Avoid problems associated with reference series,
  e.g.,
• Difficulties in ensuring homogeneity
• Mix of record lengths in climate series
• All temperature series can be evaluated

9
Pairwise Comparison of Series

• Jones et al. (1986)
• Informal examination of paired temperature series
• Cassinus and Mestre (2004)
• Optimal segmentation of paired difference series
• Series causing the change point can be traced
  more directly

10
Basic Steps

• Form combinations of pairwise difference series
• Apply undocumented changepoint tests to the
  difference series
• Unconfound the identified changepoints
• Conflate changepoint dates
• Undocumented changepoints attributed to date of
  metadata event, or
• To most common changepoint date
• Calculate multiple pairwise estimates of step
  change amplitude for each target changepoint

11
Step 1 Formation of difference series

• All series are paired the with most highly
  correlated neighboring series
• First difference correlation used to minimize
  impact of step changes and trends on correlation

12
Simulations

• Simulated 1000 groups of 21 correlated red noise
  series (n1200)
• Imposed between 0 and 10 changepoints at random
  locations and of random magnitude (average 5)

13
Simulated temperature series with random shifts
caused by station moves/site changes
(Annual Averages)
s (C)

• Series in red treated as the target in subsequent
  figures
• All shifts are considered to be undocumented
• True climate trend in all simulated series is
  zero

14
Case 7 unadjusted
Target series (lower panel) and differences with
neighbors
15
Step 2 Breakpoint Testing
SNHT (Alexandersson,1986) - TPR-0
Change in mean with no trend
TPR-1 (Wang 2003)
Change in mean within constant trend
TPR-2 (Lund and Reeves 2002)
Change in mean and/or change in trend

• Multiple breaks resolved via a semi-hierarchical
  splitting algorithm (Hawkins, 1976 Menne and
  Williams 2005)

16
Step 2 Breakpoint Testing

• Use SNHT (TPR-0) with Bayesian Information
  Criterion (S(q)) verification of changepoint

17
Step 2 Changepoint model identification
q 1
q 4
q 4
q 2
q 4
q 3
q 5
q 4
18
Why worry about local trends?

• Determine impact of land use changes (e.g.,
  urbanization)
• Trend changes get confounded with step changes
  (especially at annual resolutions)

19
(No Transcript)
20
Step 3 Attribute Cause of Shifts

• Date by date find station whose target-neighbor
  difference series has failed Ho the most
• Subtract one from tally of total number of shifts
  on corresponding date from each neighbor-target
  difference series
• Iterate for all dates and difference series

21
(No Transcript)
22
(No Transcript)
23
Step 4 Conflation of Changepoint Dates
24
Step 4 Conflation of Changepoint Dates

• Estimate magnitude of changepoint
• Assign cluster of changepoint dates within
  uncertainty window
• to a single event in the target stations history
  or,
• to most common changepoint date if undocumented

25
010011012013014015016017018
01901a01b01c01d01e01f01g01h01i01j01k
1940 6 474 ---------------------------
--------------------- 4------------
1940 7 475 ---------------------------
--------------------- 2------------
1940 8 476 ---------------------------
--------------------- 3------------
1940 9 477 --- 3---------------------
------------------------------------
1940 11 479 ---------------
6------------------------------------
--------- 1941 1 481 ---
4------------------------------------
--------------------- 1941 2 482
-------------------------------------
----------------- 2------ 1941 3 483
--------- 2-------------------------
-------------------------- 1941 4 484
--------- 2--------- 2-------------
-------------------------- 1941 6 486
------ 3 3-------------------------
-------------------------- 1941 11 491
------------------------
4------------------------------------
1941 12 492 ------------------------
2------------------------------------
1942 2 494 ------------------------
3------------------------------------
1942 4 496 ------ 4-----------------
-------------------------------------
1942 7 499 --------------------------
---------- 5------------------------
1943 11 515 ------------------------
2------------------------------------
1944 2 518 --------------------------
---------- 5------------------------
1944 3 519 ------------------
2--------------- 3------------------
------ 1944 4 520 --------------------
---------------- 2------------------
------ 1944 12 528 ------------------
4 5---------------------------------
------ 1945 1 529 --------------------
- 2---------------------------------
------ 1945 4 532 ------
2------------------------------------
------------------ 1945 11 539
-------------------------------------
-- 2--------------------- 1946 2 542
-------------------------------------
-- 3--------------------- 1946 8 548
-------------------------------------
-------------- 2--------- 1948 7 571
-------------------------------------
-------------- 2--------- 1948 8 572
-------------------------------------
-------------- 2--------- 1949 6 582
-------------------------------------
----------- 9------------
26
010011012013014015016017018
01901a01b01c01d01e01f01g01h01i01j01k
1940 6 474 ---------------------------
--------------------- 9------------
1940 11 479 ---------------
6------------------------------------
--------- 1941 1 481 ---
7------------------------------------
--------------------- 1941 2 482
-------------------------------------
----------------- 2------ 1941 4 484
--------------------- 2-------------
-------------------------- 1941 6 486
------ 3 7-------------------------
-------------------------- 1941 11 491
------------------------
9------------------------------------
1942 4 496 ------ 4-----------------
-------------------------------------
1942 7 499 --------------------------
---------- 5------------------------
1943 11 515 ------------------------
2------------------------------------
1944 2 518 --------------------------
---------- 10------------------------
1944 12 528 ------------------ 7
8------------------------------------
--- 1945 4 532 ------ 2--------------
-------------------------------------
--- 1946 2 542 -----------------------
---------------- 5------------------
--- 1946 8 548 -----------------------
----------------------------
2--------- 1948 7 571 ---------------
------------------------------------
4--------- 1949 6 582 ---------------
---------------------------------
9------------
27
Step 5 Estimation of Step Change

• Use remaining metadata
• Step-change magnitude calculated according to
  model appropriate for each target-neighbor
  changepoint or as a simple difference in means
• Median of step estimates is used as adjustment
  significance evaluated by estimating the 5th
  (median gt 0) or 95th (median lt 0) of pairwise
  estimates.

28
Simulated temperature series with random shifts
caused by station moves/site changes
(Annual Averages)
s (C)

• Series in red treated as the target in subsequent
  figures
• All shifts are considered to be undocumented
• True climate trend in all simulated series is
  zero

29
Case 7 unadjusted
Target series and differences with neighbors
before adjustment for undocumented shifts
30
Target series and differences with neighbors
after adjustment for undocumented shifts
31
Simulated temperature series following adjustment
by pairwise algorithm
(Annual Averages)
s (C)

• Original Target Series in Red
• Adjusted Target Series in Green
• Adjusted Neighbor Series in Black

32
Diagnostic

• For the target example and its nine neighbors, 34
  of 43 changepoints were detected and attributed
  to the correct series.
• Of the 9 changepoints not accounted for
• 6 are under 0.3s
• 2 are under 0.5 s
• 1 was equal to 0.696s (but was preceded by an
  unidentified shift of -0.451 10 months earlier)

33
Simulations

• Simulated 1000 groups of 21 correlated red noise
  series (n1200)
• Monthly Case 1 Imposed between 0 and 10
  changepoints at random locations and of random
  magnitude (average 5)
• Monthly Case 2 As in case 1, except with
  random unrepresentative (local) trends (from
  0.001s/month up to about 0.18s/month)

34
Algorithm Results for Step Change Only Case
35
Algorithm Results for Steps and Local Trends
Case
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
Impact of Adjustments on Trends
U.S. annual and seasonal temperature trends (C
dec-1) 1895 to 2006
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
How to conceive of the difference series?
51
An Example Reno, Nevada
From http//wattsupwiththat.wordpress.com/2008/0
1/10/how-not-to-measure-temperature-part-46-renos-
ushcn-station/
52
Reno, Nevada Average Minimum Temperature
53
Reno, Nevada Average Minimum Temperature
54
Reno, Nevada Average Minimum Temperature
55
Reno, Nevada Average Minimum Temperature
56
Reno, Nevada Average Minimum Temperature
57
Reno, Nevada
Move to Airport
Likely urban warming
ASOS Equip. Moves

1. Mean annual TOB and fully adjusted (TOBPairwise)
   minimum temperatures at Reno, Nevada
2. Difference between minimum temperatures at Reno
   and the mean from its 10 nearest neighbors.

58
Conclusions

• Pairwise comparison is the most direct way to
  detect undocumented changepoints
• Changepoint modeling is necessary in changepoint
  testing in order to identify unrepresentative
  trends
• No way to safely pass local (unrepresentative)
  trends through homogenization process
• Aliasing of trend inhomogeneities leds to a
  confused discussion about magnitude of UHI

59
Future

• Adjust trend inhomogeneities as trends
• Homogenize monthly data from Global Historical
  Climatology Network
• Derive daily adjustments for U.S. and
  GSN/GHCN-Daily

60
THANK YOU!