NAFCOM Probabilistic Cost Analysis Module presented by Sharon Winn NASA Cost Symposium March 10, 200

1
NAFCOMProbabilistic Cost Analysis
Modulepresented bySharon WinnNASA Cost
Symposium March 10, 2004
2
NAFCOM Description

• NAFCOM is a parametric estimating tool for space
  hardware.
• It is intended to be used in the very early
  phases of a development project.
• NAFCOM can be used at the subsystem or component
  levels and estimates development and production
  costs.
• NAFCOM is applicable to various types of missions
  (manned spacecraft, unmanned spacecraft, and
  launch vehicles).
• There are two versions of the model a government
  version that is restricted and a contractor
  releasable version.

3
NAFCOM Evolution

• Since 1990, eight versions of NAFCOM have been
  developed and distributed across NASA and other
  government agencies.

NASCOM Books
NASCOM Automated DB
NASCOM Ver. 3.0
NASCOM Ver. 4.0
NAFCOM Ver. 2002
NAFCOM Ver. 2004
NAFCOM96
NAFCOM99
1990
1992
1994
1996
1998
2000
2002
2004

• NASCOM database in hardcopy
• Estimators hand-entered data into spreadsheets
• Database contained 65 data points

• Allowed online searching copying of data
• Cost estimates developed in spreadsheets with
  CERs created by individuals
• Database contained 70 data points

• Fully functional cost model with user defined WBS
  and data access
• CERs built within NASCOM using 1st Pound method
• Database contained 91 data points

• Combined NASA and Air Force data
• Enhanced search and filtering of data
• Standardized WBS elements created
• Database contained 102 data points

• First non-weight based CERs for five subsystems
  (Complexity Generators)
• Gov and contractor versions distributed
• Database contained 114 data points

• Total re-write of all NAFCOM program code
• Complexity Generators for all subsystems
• Major user interface improvements
• Database contains 122 data points

• Complexity Generators reworked
• Fully integrated version of SOCM
• Probabilistic Cost Risk Analysis module

4
NAFCOM Probabilistic Cost AnalysisRationale

• NAFCOM provided a point estimate as opposed to a
  probabilistic range estimate.
• The addition of a probabilistic cost risk
  analysis module
• provides the ability to identify, analyze, and
  quantify risk/uncertainty
• Expands NAFCOM from a static analysis tool to one
  that provides probabilistic analysis of uncertain
  values

5
NAFCOM Probabilistic Cost AnalysisDevelopment

• SAIC was tasked with developing a Probabilistic
  Cost Analysis capability for NAFCOM.
• Experts in the risk field, including
  representatives from MCR, Aerospace, NASA, IPAO,
  and Mitre participated in the methodology
  development.
• A meeting was held at NASA HQ to discuss issues
  including sampling approaches, incorporation of
  correlation and model implementation.
• Dr. Steve Book has been working directly with
  SAIC and NASA to ensure that the best possible
  approaches are considered for integration into
  NAFCOM and to consult on the methodology
  implementation.

6
NAFCOM Probabilistic Cost AnalysisModel Selection

• Choice between Analytic approach (FRISK) or Monte
  Carlo-based sampling approach
• Analytic Approach
• Pros Simple for summing WBS elements,
  instantaneous results, simple development
• Cons Difficult to apply risk to algebras beyond
  summation (i.e., inflation, phasing)
• Monte Carlo Approach
• Pros Allows probability analysis on different
  algebras
• Cons Difficult to develop or interface w/COTS,
  time intensive to run (I.e., 5,000 to 10,000
  samples)
• Decision Use Analytic Approach

7
NAFCOM Probabilistic Cost AnalysisImplementation
At Estimating Level
B
Best-Estimate Cost
0
SEE
Distribution of Estimating Error
L
H
B
Cost Implications of Complexity Drivers
(Technical, Management and New Design)
8
NAFCOM Probabilistic Cost AnalysisImplementation
At Summation Levels
WBS Element 1

Total Cost

• Approximate by lognormal distribution
• Compute analytically
• Mean
• Mode (Most likely)
• Confidence percentiles

WBS Element 2

.
.
.
9
NAFCOM Probabilistic Cost AnalysisModel Interface

• Users can elect to run NAFCOM in Risk On or Risk
  Off mode via toolbar

10
NAFCOM Probabilistic Cost AnalysisModel Interface

• NAFCOMs Risk On Mode provides capability to
  define triangular distributions for all cost
  driver inputs and for cost thruputs
• These distributions are combined with CER errors
  to produce distributions for nonrecurring and
  recurring cost for each subsystem

11
NAFCOM Probabilistic Cost AnalysisModel Interface

• The distributions for each subsystem are then
  aggregated, taking correlation and System
  Integration into account
• Both intra- and inter-subsystem correlation is
  taken into account
• Users can elect to modify the default correlation
  matrix
• The default correlation matrix was developed
  based on Dr. Steve Books recommendation of 0.2
  correlation across all elements (knee of the
  curve)

12
NAFCOM Probabilistic Cost AnalysisModel Interface

• The final result is uncertainty distributions for
  DDTE, Flight Unit, Production and Total Cost.
• Result data includes summary statistics,
  probability densities, and cumulative
  distributions for each major estimating element
  (i.e. stage, bus)

13
NAFCOM Probabilistic Cost AnalysisSummary

• Uses well-documented analytical method to
  calculate risk
• User can input low, most likely, and high values
  for all NAFCOM complexity generator and
  conventional CER inputs
• Incorporates both technical and estimating
  uncertainties
• Incorporates correlation between subsystem costs
• Results displayed to the user are summary
  statistics, probability densities, and cumulative
  distributions for DDTE, Flight Unit, Production,
  and Total Costs for each major estimating element
  (i.e. stage, bus, etc.)
• User can select either the Normal distribution or
  the Lognormal distribution to approximate the
  final results

14
NAFCOM Probabilistic Cost AnalysisCurrent Status

• Model is in final test and debug phase
• A beta version of the model was delivered to MSFC
  last week
• Roll out of model to cost estimating community is
  planned for early April
• NAFCOM can be downloaded from the NAFCOM home
  page at http//nafcom.saic.com
• winns_at_saic.com
• 256-971-7275 256-705-8539

15
NAFCOM Probabilistic Cost AnalysisPlanned
Enhancements

• Initial delivery of NAFCOM probabilistic cost
  analysis module does not address engine CERs, the
  allocation of risk dollars, or an assessment of
  the accuracy of the sampling method used.
• The the cost risk analysis capability will be
  expanded to include all engine CERs including
  liquid, combined cycle, and air breathing
  estimating algorithms allowing the user to
  perform risk analysis on all current NAFCOM CER
  types.
• The ability to determine which elements have the
  most cost risk associated with them and to
  allocate risk dollars back to those WBS elements
  will be added.
• User will be allowed to select a point estimate
  (most likely,mean, median, mode) and a level of
  confidence above the point estimate to determine
  risk dollars.
• There is no unique optimal way of allocating
  the risk dollars, but NAFCOM will implement a
  technique acceptable to the NASA estimating
  community that takes into account inter-element
  correlations and other features of the estimate.
• The accuracy of the analytical versus Monte Carlo
  approaches to sampling will be studied to
  determine the most cost effective capability.
• Determine if there is any value added in using
  the Monte Carlo sampling approach over the
  statistical method.
• Document findings to support/refute the need to
  use one approach over the other.