Assessing Uncertainty in FVS Projections Using a Bootstrap Resampling Method

1
Assessing Uncertainty in FVS Projections Using a
Bootstrap Resampling Method
by Tommy F. Gregg Region 6 NR/FID and Susan
Stevens Hummel PNW Research Station
2
Objective

• Develop process for assessing uncertainty in
  model projections.
• Create a program compatible with SUPPOSE and FVS
  to implement the process.

3
Criteria for the Process

• Must be statistically valid.
• Must be feasible given current technology.

4
Why is this important?
The oldest and simplest device for misleading
folks is the barefaced lie. A method that is
nearly as effective and far more subtle is to
report a sample estimate without any indication
of its reliability (Frank Freese 1967)
5
Available variance estimators

• Simple random sample
• Stratified random sample
• Double sampling
• Multi-stage or cluster sample
• ..and many more!

6
Problem with available variance estimators

• They do not apply to model projections over time.
• They can not be used for making inferences about
  model complex results.

7
Means Confidence Limits from a set of
independent samples may look like this
m
8
Sample data are often projected through time
without regard to sampling uncertainty
Time
m
x
9
Bootstrap Resampling Method

• Developed in the 1980s (Efron), based on
  classical statistical theory from the 1930s.
• Computer intensive method for assessing
  uncertainty.
• Used for complex problems in many fields.

10
Why Bootstrap Model Projections?

• Bootstrapping allows us to substitute
  computational power for classical statistical
  analysis.
• Bootstrapping may be the only technical method
  for assessing uncertainty in model projections.
• Bootstrapping is doable.

11
Stand with 31 stand-exam inventory plots
What is the Bootstrap Resampling Method?
12
The Process for Generating Monte Carlo Bootstrap
Samples

• (1) Randomly select a sample of size n with
  replacement from the original empirical
  distribution (where n is the sample size for that
  original sample).
• (2) Compute a bootstrap mean using the
  bootstrap sample.
• (3) Repeat steps 1 and 2 k times.

13
An Example of a Bootstrap Sampling
14
Generating a set of Nonparametric Bootstrap
Confidence Intervals

• Confidence intervals are obtained from the Monte
  Carlo bootstrap distribution.
• They are taken at appropriate percentiles from a
  sorted list of the k bootstrap means.
• For example, a two-sided approximate 95
  confidence interval about mean would be
  extracted at the 2.5 and 97.5 percentile.

15
LIST OF 200 Sorted BootStrap Means
( 0) ( 1) ( 2)
( 3) ( 4) ( 5) ( 6)
( 7) ( 8) ( 9) 0
- 1207.0 1204.0 1191.0 1181.0
1178.0 1173.0 1159.0 1117.0 1116.0 10
1082.0 1079.0 1072.0 1071.0 1067.0
1066.0 1065.0 1056.0 1051.0 1047.0
20 1042.0 1040.0 1032.0 1031.0 1031.0
1030.0 1018.0 1017.0 1013.0 1010.0
30 1010.0 1008.0 1004.0 1003.0
995.0 994.0 992.0 985.0 984.0
983.0 40 982.0 981.0 981.0
975.0 975.0 969.0 968.0 967.0
967.0 965.0 50 962.0
962.0 960.0 960.0 956.0 955.0
953.0 951.0 950.0 949.0 60
948.0 945.0 943.0 942.0 942.0
941.0 936.0 933.0 928.0
927.0 70 926.0 925.0 924.0
924.0 924.0 923.0 922.0 919.0
919.0 919.0 80 917.0 916.0
913.0 913.0 911.0 905.0 899.0
898.0 898.0 896.0 90 894.0
892.0 892.0 892.0 891.0
890.0 889.0 889.0 887.0 885.0
100 885.0 885.0 884.0 884.0
883.0 880.0 879.0 879.0 878.0
878.0 110 877.0 876.0 876.0
873.0 872.0 872.0 870.0 867.0
865.0 864.0 120 863.0 861.0
858.0 858.0 857.0 855.0 854.0
853.0 852.0 852.0 130 852.0
851.0 847.0 846.0 846.0 845.0
844.0 841.0 840.0 840.0 140
840.0 837.0 835.0 834.0
833.0 830.0 830.0 827.0 823.0
823.0 150 823.0 818.0 817.0
813.0 810.0 809.0 806.0 805.0
805.0 805.0 160 803.0 801.0
800.0 799.0 796.0 795.0 789.0
781.0 778.0 777.0 170 777.0
770.0 769.0 767.0 765.0 761.0
757.0 755.0 749.0 749.0 180
748.0 747.0 747.0 745.0 741.0
739.0 734.0 721.0 720.0
713.0 190 709.0 709.0 703.0
702.0 702.0 688.0 680.0 670.0
654.0 623.0 200 588.0
16
Program Description ( FVS2Boot.exe )

• Builds bootstrap files from existing FVS input
  treelist files.
• Interfaces seamlessly with SUPPOSE.
• Processes FVS output
• .sum files
• .out files
• Displays results.

17
There are two sources of variation that can be
used to characterize uncertainty in FVS

• The stochastic components in FVS (accessible to
  the user through the RANNSEED keyword). We call
  this FVS-Mean and FVS Prediction Interval
  (FVS-PI).
• Variation among sampling units. We call this
  Sampling Error Prediction Interval (SE-PI).

18
FVSBoot Main Menu
19
Open FVS Files and Directories
20
Bootstrap Option Screen
21
Suppose Main Menu
22
Suppose Select Stands
23
Run FVS on all Bootstrap Samples
24
Select FVS Variables for Display
25
FVS Bootstrap Output Screen
26
DISPLAY OF FVS MODEL STOCASTIC BEHAVIOR Model
output data from FVS Cycle( 5), TPA
FVS-PI Mean 467.55 Number of FVS
runs 200 Standard Deviation
1.48 Median 468.00
Max 471.00 Min 463.00
Range 8.00 Frequency distribution
for ( 201 ) bootstrap samples --------
--------- --------------------------------------
------------------------- 1 463.50
2 II 2 464.50 14 IIIIIIIIIIIIII
3 465.50 32 IIIIIIIIIIIIIIIIIIIIIIIIII
IIIIII 4 466.50 49
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
5 467.50 53 IIIIIIIIIIIIIIIIIIIIIII
IIIIIIIIIIIIIIIIIIIIIIIIIIIIII 6 468.50
34 IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 7
469.50 11 IIIIIIIIIII 8 470.50
6 IIIIII -------- ---------
-------------------------------------------------
--------------
27
FVS PREDICTION INTERVAL FVS-PI CAUSED BY SAMPLE
VARIATION Data from FVS Variable
Cycle( 5), TPA FVS-PI Mean 467.55
Sampling Error PI Number of samples
500 Bootstrap Mean
464.53 Standard Deviation
52.86 Bootstrap Median 461.00
Max outcome 657.00 Min
outcome 318.00 Range of
outcomes 339.00
28
Frequency distribution for ( 500 ) bootstrap
samples for "Cycle(5), TPA" from FVS. Interval
Midpoints Counts -------- ---------
-------------------------------------------------
-------------------------------- 1
326.48 3 III 2 343.42 4
IIII 3 360.38 8 IIIIIIII
4 377.33 16 IIIIIIIIIIIIIIII 5
394.27 28 IIIIIIIIIIIIIIIIIIIIIIIIIIII
6 411.23 34 IIIIIIIIIIIIIIIIIIIIIIIIIIIII
IIIII 7 428.18 57
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
IIIIIIII 8 445.13 65
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
IIIIIIIIIIIIIIII 9 462.08 67
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
IIIIIIIIIIIIIIIIII 10 479.02 59
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
IIIIIIIIII 11 495.98 52
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
III 12 512.92 35 IIIIIIIIIIIIIIIIIIII
IIIIIIIIIIIIIII 13 529.88 31
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 14
546.83 20 IIIIIIIIIIIIIIIIIIII 15
563.78 10 IIIIIIIIII 16 580.73
4 IIII 17 597.67 3 III 18
614.63 2 II 19 631.58 0
20 648.53 2 II -------- ---------
-------------------------------------------------
---------------------------------
29
BOOTSTRAP SAMPLING ERROR PREDICTION INTERVALS
(SE-PI) Variable Mean
Percent Lower Upper ----------------
-------- ----- --------
-------- Cycle( 5), TPA 467.55
68 414.00 518.00
80 398.00 531.00
90 381.00
553.00 95
364.00 571.00
99 329.00 648.00
--------------------------------------------------
-------------------