Sensitivity Analysis in GEM-SA

1
Sensitivity Analysis in GEM-SA

• Jeremy Oakley

2
Example

• ForestETP vegetation model
• 7 input parameters
• 120 model runs
• Objective conduct a variance-based sensitivity
  analysis to identify which uncertain inputs are
  driving the output uncertainty.

3
Exploratory scatter plots
4
Sensitivity Analysis Walkthrough

• ? Project ? New
• Select the Files tab. Click on Browse on the
  Inputs File row
• GEM-SA Demo Data / Model1 / emulator7x120inputs.tx
  t
• Click on Browse on the Outputs File row
• GEM-SA Demo Data / Model1 / out11.txt
• Return to the Options tab

5
Sensitivity Analysis Walkthrough

• Change the Number of Inputs to 7.
• Tick the calculate main effects and sum effects
  boxes only
• Leave the other options unchanged
• Input uncertainty options All unknown, uniform
• Prior mean options Linear term for each input
• Generate predictions as function realisations
  (correlated points)
• Click OK
• ? Project ? Run

6
Sensitivity Analysis Walkthrough
7
Main effect plots
8
Main effect plots
Fixing X6 18, this point shows the expected
value of the output (obtained by averaging over
all other inputs).
Simply fixing all the other inputs at their
central values and comparing X610 with X640
would underestimate the influence of this
input (The thickness of the band shows emulator
uncertainty)
9
Variance of main effects
Main effects for each input. Input 6 has the
greatest individual contribution to the variance
Main effects sum to 66 of the total variance
10
Interactions and total effects

• Main effects explain 2/3 of the variance
• Model must contain interactions
• Any input can have small main effect, but large
  interaction effect, so overall still an
  important input
• Can ask GEM-SA to compute all pair-wise
  interaction effects
• 435 in total for a 30 input model can take some
  time!
• Useful to know what to look for

11
Interactions and total effects

• For each input Xi
• Total effect main effect for Xi all
  interactions involving Xi
• Total effect gtgt main effect implies interactions
  in the model
• NB main effects normalised by variance, total
  effects normalised by sum of total effects
• Look for large total effects relative to main
  effects

12
Interactions and total effects
Total effects for inputs 4 and 7 much larger than
its main effect. Implies presence of interactions
13
Interaction effects

1. ? Project ? Edit
2. Tick calculate joint effects
3. De-select all inputs under inputs to include in
   joint effects, select 4,5,6,7
4. Click OK
5. ? Project ? Run

14
Interaction effects
15
Interaction effects
Note interactions involving inputs 4 and
7 Main effects and selected interactions now
sum to 91 of the total variance
16
Exercise

• Set up a new project using SAex1_inputs.txt for
  the inputs and SAex1_outputs.txt for the output
• 8 input parameters (uniform on 0,1)
• 100 model runs
• Estimate the main effects only for this model and
  identify the influential input variables
• By comparing main effects with total effects, can
  you spot any interactions?
• Estimate any suspected interactions to test your
  intuition!