JPL Cost Risk Analysis Approach That Incorporates Engineering Realism Updated again for AIAA meeting

1
JPL Cost Risk Analysis Approach That Incorporates
Engineering Realism - Updated again for AIAA
meeting

• AIAA Economics Technical Committee Meeting -
  January 24, 2007
• Leigh Rosenberg
• Corey Harmon
• Keith Warfield
• (Previous versions given at 2006 NASA Cost
  Symposium, 2006 ISPA Annual Conference, JPL Risk
  Workshop)

2
Presentation Outline

• Introduction
• Use of S-Curves
• NASA Cost Estimating Handbook Approach
• JPL Cost Engineering Group Methodology
• Case Study
• Current Issues/Future Work

3
Introduction

• Why are we interested in cost risk analysis?
• Decision makers want to know the likelihood of
  achieving a particular cost estimate
• Single point cost estimates fail to capture
  uncertainty in the estimate and cannot provide
  proper context for comparison to other estimates
• The methodology needs to be well understood and
  justifiable in order to generate likelihood
  statistics (usually, an S-curve) that reflects
  reality

4
Introduction - Continued

• What is cost risk? Risk due to economic factors,
  rate uncertainties, cost estimating errors, and
  statistical uncertainty inherent in the estimate.
  
• What is an S-curve? An S-curve is a statistical
  tool (cumulative density function -- CDF) that
  can portray cost risk.
• The S-curve should make engineering sense and be
  able to provide realistic inputs to decision
  makers.

5
Introduction - Continued

• There are 3 elements contributing to risk that
  should be evaluated
• Uncertainty in cost estimation algorithms
  (accounted for by using validation statistics as
  a basis for standard deviations),
• Uncertainty in the inputs to the algorithms
  (technical risk),
• Correlation between WBS elements (cost increase
  in 1 subsystem usually equates to an increase in
  other subsystems).
• All 3 are important when adding distributions
  through Monte Carlo simulation. If they are not
  assessed, then the resultant S-curve can
  understate the variance of the final cost
  estimate. The S-curve will be too steep
  (unrealistically low variance).

6
Convolution of Different Cost Risk Factors
From NASA Cost Estimating Handbook p.158
7
S-Curves Need to be Realistic
8
2004 NASA Cost Estimating Handbook (CEH)
Approach(Cost Estimating Tasks, pgs. 95-96)

• CEH discusses method 1 and method 2 for
  generating S-curves.
• Method 1 Three-point cost distribution
  (typically triangular (min, max, most likely))
  assigned to each WBS element identified as a
  significant risk. Monte Carlo tool then
  transforms individual distributions into a
  project cost S-curve.
• Method 2 For each WBS element identified as a
  significant risk,worst case (99th percentile)
  costs are elicited instead. These values are then
  combined to form the S-curve by the Monte Carlo
  tool.
• JPL Cost Engineering Group (CEG) does not exactly
  conform to either.
• CEG uses a log-normal distribution instead of
  three-pointed. Log-normal eliminates problem of
  guessing at highest possible cost for missions
  that usually have significant unknown unknowns
  and that are using new technologies. Log-normal
  also closely resembles distributions of actual
  costs that were studied at JPL.
• 99th percentile (and other high percentages) are
  not used by CEG because project designers are
  usually far too optimistic in the early design
  stages.

9
CEG Methodology - Distributions

• CEG uses log-normal distributions
• Eliminates problem of guessing at highest
  possible cost
• Fits historical cost growth of JPL missions
• Assumes point estimate is the mean of distribution

Cost Growth of JPL Missions
Cost Growth within a JPL Project
10
Cost Growth History

• Used to validate average variance to be used in
  cost risk analysis
• Recent study based on 15 JPL projects
• Tracked cost growth from proposal/Phase A
  estimate, to actuals and/or estimates through
  2005.

11
Overrun vs. Original Project Budget for JPL
Missions (w/Res, no LV)
The variance in overruns has been large.
12
Cost Engineering Group Methodology

• Point estimates are usually generated at the
  subsystem level by PMCM or PCAT.
• Distributions are applied to subsystem-level
  point estimates.
• Standard deviation is typically 25-35
• Based on comparisons of model costs to actual
  costs with the observed uncertainty used as the
  variance.
• For WBS items that are not well defined, a higher
  variance may be used
• Log-normal distribution assumed
• Log-normal eliminates problem of guessing at
  highest possible cost for unique space missions
• Log-normal closely resembles observed cost
  distributions.
• Correlation applied prior to adding distributions
  together
• Correlation has been found to be 40-60 between
  all WBS items
• Correlation is present since subsystem designs
  are usually related to each other to some degree.
• Examples dual redundancy on one subsystem
  generally means dual redundancy for the other
  subsystems on the same spacecraft. A schedule
  delay on one subsystem is a schedule delay for
  all the other subsystems.
• Such a correlation gives a variance that is
  consistent with results found in comparisons to
  the results of JPL missions/proposal cost
  studies.
• PMCM - JPL Parametric Mission Cost Model
• PCAT - JPL Project Cost Analysis Tool

13
Cost Engineering Group Methodology - Continued

• Total WBS cost is calculated through Monte Carlo
  simulation
• Original project estimate, and 70 (or other
  chosen) level of confidence estimate are then
  found on generated S-curve
• Reserves are recommended to move estimate to 70
  level of confidence, in accordance with NASA
  requirements (2004 NASA CEH, pg. 93)
• Note Since input technical ranges are not
  usually included the resultant curve could
  understate the resulting standard deviation. PMCM
  does allow for input ranges and distributions.

14
Case Study Methodology

• Used _at_Risk software package.
• Variances assigned according to Input Reserve
  Risk provided in example.
• Monte Carlo simulation is run on each cost
  element.
• 5000 iterations
• Correlation matrix set to 60 between all WBS
  items
• S-Curve Analysis
• Original project estimate is at a 39 level of
  confidence (LOC)
• 16 reserves recommended for project estimate to
  achieve 70 LOC
• Standard deviation of S-curve is 16
• Caveats
• Case study includes launch vehicle. This cost is
  usually outside the control of the project.
• Model or technical uncertainty not determinable
  in case study.
• Deriving risk from project budget omits technical
  evaluation of risk. Technical risk should be
  based on the design should be determined before
  budget is compiled. Purpose of cost risk
  assessment is to determine risk of the budget
  proposed, not from the budget.

15
Case Study Inputs
16
Case Study Cost Risk S-Curve (CDF)
17
S-Curves with Same Assumptions

• Using identical assumptions, results are nearly
  identical for various methods
• Difference between results is from underlying
  assumptions, not the methodology used to
  calculate the s-curve

18
Current Issues

• Use of other distributions (i.e. triangular,
  normal)
• Triangular requires upper lower bounds which
  are usually not calculable on outputs and may put
  an artificial upper bound on an estimate.
• Normal may allow for unusual/impossible case of a
  below zero cost.
• Inclusion of model technical variance needs to
  be more widely better implemented.
• Variance ranges need to be defined for all WBS
  elements.
• Example New technology or instrument 50
  variance
• Better correlation statistics are needed.
• Start with determining general correlations
  between level 1 WBS items (PM, MA, S/C, etc.)
• Can use historical data gathered from CADRe
  and/or Team X to generate a JPL-specific
  correlation matrix.
• In FY 2007, there is a NASA HQ CAD task at JPL
  to study this.
• Additional improvements
• Develop statistics on the growth uncertainty of
  the input parameters to PMCM from historic actual
  data. This will provide a factual basis for
  estimating uncertainty on technical inputs.