Robert Gilmore Pontius Jr

1
Land Change Modeling Methods calibration,
validation and extrapolation

• Robert Gilmore Pontius Jr
• Associate Professor, Clark University
• Master of Applied Statistics, The Ohio State
  University
• PhD, State University of New York / College of
  Environmental Science and Forestry
• www.clarku.edu/rpontius
• rpontius_at_clarku.edu
• and
• Albert Decatur, Rahul Rakshit, Ting Ting Zhao

2
Why do we model Land-Use Land-Cover Change
(LUCC)?

• In order of importance
• To make better management decisions
• To implement environmental agreements
• To assess what we (do not) know about LUCC
• To increase knowledge about LUCC
• To learn the implications of our assumptions
• To be able to predict with a measured level of
  accuracy
• To be able to predict with a high level of
  accuracy

3
Major Points

• A model can be useful when it
• considers scale.
• is policy-relevant.
• is sufficiently simple.
• is connected to data.
• estimates its certainty.
• communicates results clearly.
• compares results to a null model.
• separates calibration from validation.
• focuses on basic issues of quantity and location.
• addresses basic problems of conventional
  statistics.

4
Gils Mantra
How can I make it more complicated? is a good
question for a juggler, not for a
modeler. www.clarku.edu/rpontius/stardust.html
5
Worcester Massachusettsand nine surrounding towns
6
From Forest To Residential 2003
7
Built 1971
8
Built 1985
9
Built 1999
10
Signal to Calibrate 1971-1985
11
Pattern to Predict 1985-1999
12
Quantity of Built
Validation
Extrapolation
Calibration
13
Slope Lubrication Values
Percent Built
Built map of 1985 for calibration of spatial
pattern.
14
1971 Land Use Lubrication Values
Percent Built
Built map of 1985 for calibration of spatial
pattern.
15
Suitability Map
16
Legal Constraints
17
Model Run Selection
18
Predicted 1985-1999 (Worst)
19
Predicted 1985-1999 (Medium)
20
Predicted 1985-1999 (Best)
21
Observed 1985-1999
22
Observed versus Predicted
23
Null versus Actual versus Predicted
24
Null versus Actual versus Predicted
25
Null (1985) versus 1999
26
Predicted (1999) versus 1999
27
Null versus Predicted
28
Null Resolution Correct
Large Quantity Change
Unconstrained
Small Quantity Change
Null Correct
Geomod Diamond Linear Quantity Blue Large
No Laws CA_Markov Square GainLoss Quantity
Red Small With Laws
29
Step 1 Change Analysis

• Make certain your machine uses a . as a decimal
  symbol and a , as the digit grouping symbol.
  You can check this by going to Settings gt
  ControlPanel gt RegionalAndLanguageOptions gt
  Customize gt DecimalSybmol then restart Idrisi.
• Use the Project Explorer to insert a new project
  with a path to your folder.
• DISPLAY
• s_landuse1985_02
• s_landuse1999_02
• s_towns_02
• CROSSTAB in hard classification mode, where first
  image s_landuse1999_02, second image
  s_landuse1985_02, mask image s_towns_02. Use
  both cross-classification and tabulation. Call
  the result cross1999_1985_01.
• Right click on the legend to describe each
  transition.
• Save a text file of the transition matrix called
  cross1999_1985_01.txt.
• Type the matrix of the text file into
  PontiusMatrix1.xls and save the result using
  Excels feature under DatagtWhat-ifAnalysisgtScenari
  oManager.
• What is the amount of net quantity change and
  swap location change?
• What quantity of Built should we predict for
  1999?
• See sheet QuantitiesFigure in s_crosses_04.xls.

30
Step 1 Result
31
Quantity of Category 1
32
Step 2 Spatial Driver Maps

• DISPLAY
• s_slope_03
• s_landuse1971_01
• What is the meaning of zero in each map?
• What is the meaning of one in each map?
• CROSSTAB in hard classification mode, where first
  image s_landuse1985_02, second image second
  image s_slope_03, mask image s_towns_02. Use
  both cross-classification and tabulation. Call
  the result cross1985_slope_01.
• Which slopes were developed in 1985?
• Humans have a propensity to develop on which
  slopes, according to the 1985 data?

33
Step 2 Result
34
Step 3 Running Geomod

• GISanalysis gt Change/TimeSeries gt Geomod
• Create new parameter file atry01
• Beginning landuse image s_landuse1985_02
• No Mask or Strata image
• Constrained
• Beginning Time 1985
• Ending Time 1999
• Time Step 1
• Driver image s_slope_03
• Do not do environmental impact analysis
• Weights Equal
• Number of pixels State 1 END predicted for 1999
  407234
• Do not output interim time images
• Prefix of final landuse images atry01
• Save As atry01, then OK to run

35
Step 4 Assessing Output atry01

• DISPLAY atry01_1, s_landuse1999_02,
  s_landuse1985_02
• How do the results look? What did Geomod do?
• CROSSTAB hard classification using Both
  cross-classification and tabulation, where first
  image s_landuse1999_02, second image atry01,
  , mask image s_towns_02 , call the result
  cross1999_atry01_01.
• Type the matrix of the text file into
  PontiusMatrix1.xls and save the result using
  Excels scenarios feature.
• CROSSTAB hard classification using Both
  cross-classification and tabulation, where first
  image atry01_1, second image
  s_landuse1985_02, mask image s_towns_02, call
  the result crossatry01_1985_01.
• Type the matrix of the text file into
  PontiusMatrix1.xls and save the result using
  Excels scenarios feature.
• Contrast the three pairwise comparisons among the
  three landuse maps cross1999_1985_01,
  crossatry01_1985_01, cross1999_atry01_01.
• Did Geomod predict the correct quantity of gain?
• Did Geomod predict the correct location of gain?
• Did Geomod predict more correctly than a null
  model of persistence?
• In what ways is it helpful to consider all three
  transitions among 1985, 1999, and predicted?

36
Step 4 Raw Result
37
Step 4 Crosstab Result
38
Step 4 Statistical Result
Change 1985-1999
Change 1985-Prediction1999
Error 1999-Prediction1999
39
Step 4 Statistical Result
Change 1985-1999
Change 1985-Prediction1999
Error 1999-Prediction1999
40
Step 5 Presenting Results atry01

• Make one map that shows the entire story by
  performing CROSSTAB hard classification where the
  first image is the crosstab map from Step 1
  (cross1999_1985_01) and the second image is
  atry01_1, call the result atry01visual01.
• Edit the legend in the metadata to describe the
  status for three maps 1985, 1999, and prediction
  of 1999.
• Display the map with a palette (apersist3.smp)
  where
• white is mask
• gray is correct due to persistence
• red is true change predicted correctly
• blue is true persistence predicted as change
• yellow is true change predicted as persistence

41
Step 5 Result
42
Step 6 Assessing Suitability atry01

• DISPLAY
• s_slope_03 with Quantitative Palette
• atry01_suitability_1 with Quantitative Palette
• Use Text Editor to examine
• atry01.lub
• atry01.gmd
• What is the relationship between s_slope_03 and
  atry01_suitability_1

43
Step 6 Result
44
Extrapolation of Built Quantity
45
Step 7 Running Geomod again

• GISanalysis gt Change/TimeSeries gt Geomod
• Open and parameter file atry01 then Modify
• Save As atry02
• Beginning landuse image s_landuse1985_02
• Stratum image s_towns_01
• Unconstrained
• Beginning Time 1985
• Ending Time 2185
• Time Step 50
• 2 Driver images s_slope_03, s_landuse1971_01
• Weights Equal
• Number of pixels State 1 END to zero for each
  town
• Prefix of final landuse images atry02
• Output all interim times to make several
  prediction maps, but do not display immediate
  results.
• Save As atry02 then OK to run. (This run might
  take several minutes. Wait till the progress bar
  at the bottom disappears. If you want to stop,
  press CntrlAltDel.)

46
Step 8 Assessing Suitability atry02

• DISPLAY
• atry02_suitability_1
• s_slope_03
• s_landuse1971_01
• s_towns_01
• Use Text Editor to examine
• atry02.lub
• atry02.gmd
• How does Geomods suitability map combine drivers
  and handle regions?

47
Step 8 Result
48
Step 8 Result
49
Ladder of Equations

Perfect
Information of Allocation
Simulated
No
No
Simulated
Perfect
Information of Quantity
50
Ladder of Equations

Perfect
Perfect Stratum
Information of Allocation
Simulated Grid Cell
Simulated Stratum
No
No
Simulated
Perfect
Information of Quantity
51
Step 9 Change Analysis

• GISanalysis gt Change/TimeSeries gt VALIDATE using
• Input type images
• Comparison image s_landuse1985_02
• Reference image s_landuse1999_02
• Mask image s_towns_01
• Multiples of base resolution 1116 with
  Geometric Sequence
• Click More gt ViewAsText gt Save As anull01 to
  save a txt file.
• Compare results to those produced by
  PontiusMatrix1.xls.
• What is the amount of net quantity change and
  swap location change
• at the finest resolution?
• at the coarsest resolution?

52
Step 9 Result
53
Step 10 Prediction Analysis

• GISanalysis gt Change/TimeSeries gt VALIDATE using
• Input type images
• Comparison image atry01_1
• Reference image s_landuse1999_02
• Mask image s_towns_01
• Multiples of base resolution 1116 with
  Geometric Sequence
• Click More gt ViewAsText gt Save As atry01valid
  to save a txt file.
• Compare results to those produced by
  PontiusMatrix1.xls.
• What is the amount of net quantity change and
  swap location change
• at the finest resolution?
• at the coarsest resolution?

54
Step 10 Result
55
Step 11 Assess Prediction vs. Null

• Open Assess10.xls
• In pre2007Excel Tools gt Macro gt Security gt
  Medium
• In 2007Excel DevelopergtCodegtMacroSecuritygtDisable
  AllMacrosWithNotification
• Close Assess10.xls, then Open Assess10.xls and
  Enable Macros.
• Enter information on sheet Input in three gray
  cells B2B4.
• B2 path
• B3 anull01
• B4 atry01valid
• Click on Run then look at Charts.

56
Step 11 Result
57
Step 11 Result
58
Step 12 Assessing Suitability Output01 in Idrisi

• GISanalysis gt Change/TimeSeries gt ROC
• Input image atry01_suitability_1
• Reference image s_landuse1999_03
• Mask image s_landuse1985_04
• Do not check Sampling (Stratified)
• Equal Interval Thresholds
• Threshold width 1
• Save to File atry01ROC01 as a text file.

59
Step 12 Result
60
Step 13 Assessing Suitability Output01 with
Figure

• Make certain your machine uses a , as the digit
  grouping symbol. You can check this by going to
  Settings gt ControlPanel gt RegionalAndLanguageOptio
  ns gt Customize gt DigitGroupingSybmol.
• Open ROCfigure03.xls.
• File gt Open gt Find text file gt Delimited gt Space
  only and Treat consecutive delimiters as one gt
  Finish
• Copy entire sheet of text from atry01ROC01, then
  Paste Special the Values from atry01ROC01 into
  ROCfigure03.xls
• Interpret the ROC curve.

61
Step 13 Result
62
Topics for useful discussion

• What are appropriate criteria for model
  performance?
• What are appropriate criteria for validation?
• How can we tell whether the model captures an
  appropriate process?
• What are appropriate methods to communicate the
  level of certainty we have in models?
• What are the appropriate scales at which to
  present results?

63
International Comparison Exercise

• We invited land-change modelers to submit
• Reference Map of Time 1,
• Reference Map of Time 2,
• Prediction Map of Time 2,
• Criterion to evaluate the maps.

64
Contributors Twelve Sites
Haidian,China
Cho Don, Vietnam
Perinet, Madagascar
65
Geomod
There is more error than correctly predicted
change. Most of the error is due to predicting
the wrong location by not more than 4 kilometers.
66
Geomod and SLEUTH
Most of the error is due to predicting too little
quantity of change.
67
Land Use Scanner and Environment Explorer
We need to consider data quality and format,
especially when the amount of change is small.
68
Logistic Regression and SAMBA
Perinet is the only case that has more correctly
predicted change than error. SAMBA and subsequent
models use post-time 1 information for
calibration.
69
Land Transformation Model
Data and model show one-way change from non-urban
to urban. Simulation uses the correct quantity of
urban gain. Prediction of location is better than
random, but not better than null.
70
CLUE-S
All three use the correct quantities from time
2. All three are better than random and better
than null.
71
CLUE
It is possible to model and to measure accuracy
with heterogenous pixels.
72
13 cases
73
Venn Diagrams for 13 cases
74
Rare events are difficult to predict
Cases with less than 15 figure of merit have
less than 10 observed net change.
75
Observed, Predicted, and Error
76
13 Multiple-Resolution Profiles
77
Characteristics of 13 cases

1 The original pixels contain partial
membership to 36 categories, which are then
reassigned the dominant category after the
prediction. 2 The reference maps and the model
are designed to show exclusively a one-way
transition. 3 The pixels contain simultaneous
partial membership to multiple categories.
78
Alternative to conventional statistics

• Conventional statistical methods are not helpful
  to analyze Land-Use -Cover Change (LUCC)
  because
• There is no natural unit of analysis in LUCC, but
  conventional methods place great emphasis on the
  unit of analysis.
• We already know that patterns of LUCC are not
  random, so conventional hypothesis testing is not
  interesting.
• Spatial dependence is important in the analysis
  of LUCC, but conventional statistics treat
  spatial autocorrelation as a problem.

79
How do you measure the agreement between two
mixed pixels?
Reference
Comparison
0.4 0.6
0.1 0.9
Upper number is membership in Green
Category. Lower number is membership in Red
Category.
A good definition of agreement should be
intuitive. Perfect agreement should give
1. Perfect disagreement should give 0.
80
Multiple Choice QuizWhat is the agreement
between these two pixels?
Reference
Comparison
0.4 0.6
0.1 0.9

• 1.00
• 0.58
• 0.70
• 0.50
• none of the above
• some of the above

81
A) Winner-take-all purifying ruleleads to
perfect agreement.
Reference
Comparison
0.0 1.0
0.0 1.0
A) 1.00 This strategy corrupts information.
82
We need a conceptual foundation to analyze mixed
pixels.
Reference
Comparison
0.4 0.6
0.1 0.9

• 1.00
• 0.58
• 0.70
• 0.50
• none of the above
• some of the above

83
The inspiring paint analogy
84
The inspiring paint analogy with many colors
85
B) Multiplication Rule gives a possible measure
of agreement.
Reference
Comparison
0.4 0.6
0.1 0.9
Green in North. Red in South.
Distribution is random.
B) Reference vs. Comparison (0.40.1)
(0.60.9) 0.58
Reference vs. Reference (0.40.4) (0.60.6)
0.52 lt 0.58 lt 1.00 The agreement between a pixel
and itself is less than 1. The agreement between
a pixel and itself is less than the agreement
between a pixel and a different pixel.
86
C) Minimum Rule gives mostpossible agreement.
Reference
Comparison
0.4 0.6
0.1 0.9
Green in North. Red in South.
Distribution gives most possible agreement.
C) Reference vs. Comparison MIN(0.4, 0.1)
MIN(0.6, 0.9) 0.7
Reference vs. Reference MIN(0.4, 0.4) MIN(0.6,
0.6) 1.0 gt 0.7 The agreement between a pixel
and itself is 1. The agreement between a pixel
and itself is greater than the agreement between
a pixel and something different.
87
D) Maximum Rule gives leastpossible agreement.
Reference
Comparison
0.4 0.6
0.9 0.1
Green in North. Red in South.
Distribution gives least possible agreement.
D) Reference vs. Comparison MAX(0, 0.40.1-1.0)
MAX(0, 0.60.9-1.0) 0.0 0.5 0.5 This
gives the least possible overlap of identical
colors.
88
Multiple Choice QuizIf you believe in the paint
analogy, then there is a range of possible
answers.
Reference
Comparison
0.4 0.6
0.1 0.9

• 1.00
• 0.58
• 0.70
• 0.50
• none of the above
• some of the above

89
Highway in Pará, Brazil
90
Regional Strata
91
Household Sub-Strata
92
Reference with 26 deforestation in 1999
93
Simulation with 15 deforestation
target intercept 40 frontier speed
0.8 maximum duration 6
94
Simulation with 24 deforestation
target intercept 60 frontier speed
0.8 maximum duration 3
95
Simulation with 49 deforestation
target intercept 80 frontier speed
1.6 maximum duration 3
96
Prediction with 15 deforestation
target intercept 40 frontier speed
0.8 maximum duration 6
97
Prediction with 24 deforestation
target intercept 60 frontier speed
0.8 maximum duration 3
98
Prediction with 49 deforestation
target intercept 80 frontier speed
1.6 maximum duration 3
99
BLM versus Highway Model
100
Multiple Resolution Assessment
101
Components of agreement and disagreement
102
Ladder of Equations
103
Major Points

• Conventional methods of map comparison often
  produce misleading results.
• I propose methods of map comparison that are
  useful for
• accuracy assessment
• change analysis
• model validation

104
2003 Vegetation Deviation from the average of the
previous 18-years Predicted (Y) versus Observed
(X)
Null model would predict zero everywhere, as seen
in the ocean.
105
Which comparison map is more similar to the
reference map?
Comparison 1 Mean Absolute Error 0.48
Comparison 2 Mean Absolute Error 0.46
Reference
106
Predicted (Y) versus Observed (X)at 8 km by 8 km
pixel resolution
107
Conventional presentation of Observed versus
Predicted

• There is a significant correlation between
  observed and predicted (p-value lt 0.01).
• R-squared is 0.03.
• Root Mean Squared Error is 0.6.

108
Components of Information forObserved versus
Predicted

• Positive agreement due to quantity shows that
  prediction is better than null.
• Zero agreement due to location shows that a
  uniform spatial allocation would have been more
  accurate.
• Decreasing disagreement due to allocation
  indicates distance of errors of location.
• Disagreement due to quantity shows the model
  predicted less vegetation than there actually was.

Allocation
109
Coarsening resolution can reduce disagreement due
to allocation
Reference
Comparison
Fine Resolution Mean Absolute Error 2
Reference
Comparison
Coarse Resolution Mean Absolute Error 0
110
The inspiring paint analogy
111
What is the general principal?
112
How do two paintings compare?
Masterpiece
Forgery
Masterpiece is slightly lighter than Forgery.
113
How do two maps compare?
Map X
Map Y
-2
-1
7
8
2
0
8
8
-3
-4
5
6
0
-2
6
6
-6
-5
3
4
-4
-4
2
6
2
-8
-7
1
-2
-4
-2
-4
X is slightly lighter than Y.
114
Y versus X
X
Y
115
Null Quantity,Uniform Allocation
X
Y
116
Medium Quantity,Uniform Allocation
X
Y
117
Medium Quantity,Medium Allocation
X
Y
118
Medium Quantity,Perfect Allocation
X
Y
119
Perfect Quantity,Perfect Allocation
X
Y
120
Information Space for Maps
INFORMATION OF ALLOCATION Perfect Medium Uniform
X
Perfect Medium
Null INFORMATION OF QUANTITY
121
Information Space for Fine Plots
INFORMATION OF ALLOCATION Perfect Medium Uniform
Perfect Medium
Null INFORMATION OF QUANTITY
122
Information Space for Coarse Plots
INFORMATION OF ALLOCATION Perfect Medium Uniform
Perfect Medium
Null INFORMATION OF QUANTITY
123
Information Space for MAE
INFORMATION OF ALLOCATION Perfect Medium Uniform
Perfect Medium
Null INFORMATION OF QUANTITY
124
Information Space for RMSE
INFORMATION OF ALLOCATION Perfect Medium Uniform
Perfect Medium
Null INFORMATION OF QUANTITY
125
Components of InformationObserved versus
Predicted

• Positive agreement due to quantity shows that
  prediction is better than null.
• Zero agreement due to location shows that a
  uniform spatial allocation would have been more
  accurate.
• Decreasing disagreement due to location indicates
  distance of errors of location.
• Disagreement due to quantity shows the model
  predicted less vegetation than there actually was.

Allocation
Allocation
126
FEEDBACKWhat was most helpful?What should be
changed?
127
The End