Syllabus for a Graduate Course in Sensitivity Analysis

1
Syllabus for a Graduate Course in Sensitivity
Analysis

• by Terry Andres
• Computer Science Department
• University of Manitoba
• Winnipeg, Canada

2
Why a course?

• Old saying
• Those who can, do those who cant, teach.
• Saying for the 21st Century
• Those who can, do. Those who believe others can
  also, teach."
• John E. King in Captive Notions

3
Syllabus for a Graduate Course in Sensitivity
Analysis

• What is Sensitivity Analysis (SA)
• Which grad students need to know about it?
• What do they need to know, specifically?
• How do we meet their varied needs?

4
Sensitivity Analysis

• The scientific development of a simple empirical
  model for the output variation of a complex
  system
• It typically uses
• experimental design
• simulation
• statistical analysis
• modelling of the output
• It is often based on partitioning variance

5
Complex system?

• What human changes to the environment most affect
  global climate?
• What would be the economic impacts of increasing
  average lifespan to 100 years?
• How come my simulations take so long to run?
  (Variation, not uncertainty)

6
Which Grad Students Need Sensitivity Analysis?

• Students in technical disciplines
• computer science, engineering, economics,
  environmental studies
• who deal with complex systems
• computer models, networks, large programs,
  economic models, environmental models

7
Who are the students?

• A diverse group
• different fields of knowledge
• affects examples and projects
• different levels of preparation
• in math, statistics, programming, writing,
  presenting
• different expectations
• of how the course will be presented

8
What do they need to know?

• How to
• produce quantitative results from a complex
  system
• perform each step of sensitivity analysis
• assess the significance of results

9
Process of Sensitivity Analysis
10
Process of Sensitivity AnalysisElicit
distributions

• Probability distributions
• normal, lognormal
• poisson, exponential
• Elicitation
• calibrating experts
• resolving differences
• building consensus
• Law of requisite variety Ashby, 1956
• Only variety can destroy variety
• limited number of influential parameters

11
Process of Sensitivity AnalysisDesign
experiments

• The step that separates SA from Data Mining
• Simple random sampling (Monte Carlo)
• pseudo-random
• quasi-random
• Stratified sampling
• factorial, fractional factorial
• latin hypercube
• orthogonal designs
• Group designs
• supersaturated

12
Process of Sensitivity AnalysisGenerate Sample

• Inverse CDF transform
• Truncate distributions
• Assume independence
• Maintain order

13
Process of Sensitivity AnalysisRun Simulations

• Use a simulation manager
• OR FOR AN EXISTING 1-SHOT MODEL
• Retrieve a simulation
• Set up input file(s)
• Run simulation
• Harvest results
• Update database

14
Process of Sensitivity AnalysisAnalyze Results

• For stratified samples
• analysis of variance (ANOVA)
• For continuous variables
• linear and nonlinear regression
• For specialized samples
• Supersaturated group sampling ?
• group analysis
• stepwise analysis
• Goal create a simple model to explain results

15
How do we meet their needs?

• Provide some references
• Introduce basic concepts in a standard computing
  environment
• Give them incentives to research and teach some
  advanced techniques
• Give them an opportunity to apply what they have
  learned

16
Suggested References

• Sensitivity Analysis, edited by
• Saltelli, Chan and Scott, 2000.
• New book Global sensitivity
• analysisGauging the worth of
• scientific models, by Saltelli et al.
• Handbook of Simulation Principles, Methodology,
  Advances, Applications, and Practice, edited by
  Jerry Banks, 1998.

17
a standard computing environment What
Environment to Use?

• Sensitivity analysis requires the manipulation of
  data. How?
• Statistical package like S-Plus / R
• Common programming language like Java or C
• Dedicated SA tool like SimLab
• Spreadsheet package like Excel or OpenOffice

18
a standard computing environment What
Environment to Use?

• Spreadsheet package because
• generally familiar to students
• built-in management, access, and display of data
• built-in functions (e.g., inverse normal cdf)
• built-in statistical methods (ANOVA, regression)
• built-in charting
• gradual improvements
• pseudo-random generator
• larger grid size
• Executable specifications

19
incentives to research and teach Student
Evaluation

• Presenting an existing SA method
• e.g. from an approved paper
• Implementing a SA method
• new or from the literature
• Applying sensitivity analysis
• student's own model

20
incentives to research and teach Presenting
existing method

• Rated by peers
• Who must ask question

21
incentives to research and teach Implementing
a Method

• Experimental design
• Statistical analysis method
• Interface
• decorate a
• webpage with
• a sensitivity
• analysis panel

22
Give them opportunity Applying Sensitivity
Analysis

• Determine videogame settings that maximize frame
  rate
• Analyze multi-national network flow problem
• Analyze gate current in a MOSFET simulator
• Analyze contributors to error in estimating
  object locations from two photographs
• Analyze published nuclear fuel
• waste management study

23
Scope for the Future

• Parallel processors (GPUs)
• Novel experimental designs
• Genetic / evolutionary algorithms
• Sequential analysis of results
• More powerful statistical analysis techniques

24
Scope for the Future

• Sensitivity analysis is currently bound by the
  paradigm
• discrete simulations
• experimental design
• statistical analysis
• But what if uncertainty analysis is done some
  other way?

25
Scope for the futureVariateTools

• VariateTools a software package that carries out
  math operations on entire distributions at once
• E.g. Suppose you start out with 1000
• Your investment grows by a uniformly distributed
  factor fj between 1 and 1.2 each year
• How much money do you have after 7 years?

26
Scope for the futureVariateTools
27
Scope for the futureVariateTools

• The problem statement remains the same
• Having a new software package changes the
  uncertainty analysis method
• What happens to Sensitivity Analysis?

28
Conclusion

• Grad students in SA could come from many fields,
  such as engineering
• The course must cover enough background so that
  each student understands basic steps / approaches
• Grad students need to develop skills in research
  and presentation
• New techniques are needed to match advances in
  uncertainty analysis

29

• Teaching of Psychology
• 2001, Vol. 28, No. 4, Pages 295-298
• Microsoft Excel(tm) As a Tool for Teaching Basic
  Statistics
• C. Bruce Warner?
• Anita M. Meehan?