Recent NAFCOM Extensions and Improvements

1
Recent NAFCOM Extensions and Improvements
Presented by SAIC NASA Cost Symposium April 14,
2005
2
Agenda

• NAFCOM Introduction
• Benchmarking
• CG Improvements
• Latest NAFCOM Enhancements

3
Introduction

• The Systems Technologies Operation of SAIC, under
  contract to NASA MSFC, develops cost estimating
  tools and databases.
• Development and maintenance of NAFCOM and REDSTAR
  are funded under the Engineering Cost Analysis
  Techniques Development (ECATD) contract.
• Versions of NAFCOM produced under this contract
  are disseminated agency-wide to NASA and widely
  within the Air Force.

4
NAFCOM Description

• NAFCOM is a parametric estimating tool for space
  hardware.
• Uses cost estimating relationships (CERs) which
  correlate historical costs to mission
  characteristics to predict new project costs.
• It is based on historical NASA and Air Force
  space projects.
• It is intended to be used in the very early
  phases of a development project.
• NAFCOM can be used at the subsystem or component
  levels and estimates development and production
  costs.
• NAFCOM is applicable to various types of missions
  (manned spacecraft, unmanned spacecraft, and
  launch vehicles).
• There are two versions of the model a government
  version that is restricted and a contractor
  releasable version.

5
NAFCOM Evolution

• Since 1990, ten versions of NAFCOM have been
  developed and distributed across NASA and other
  government agencies.

2002

• Fully functional cost model with user defined WBS
  and data access
• CERs built automatically within NASCOM using 1st
  Pound method
• Database contained 91 data points

• Total re-write of all NAFCOM program code
• Complexity Generators for all subsystems
• Major user interface improvements
• Database contains 122 data points

• Cost Risk Analysis Module
• CER Improvements
• SOCM
• Component level Complexity Generator

• First non-weight based CERs for five subsystems
  (Complexity Generators)
• Government and contractor versions distributed
• Database contained 114 data points

• Allowed online searches and copying of data
• Cost estimates developed in spreadsheets with
  CERs created by individuals
• Database contained 70 data points

• Combined NASA and Air Force data
• Enhanced search and filtering of data
• Standardized WBS elements created
• Database contained 102 data points

• NASCOM database in hardcopy only
• Estimators hand-entered data into spreadsheets
• Database contained 65 data points

6
Benchmarking Overview

• At the direction of MSFC, SAIC worked with
  Lockheed, Boeing, and Northrop Grumman to
  benchmark NAFCOM with relevant and recent
  completed missions.
• Atlas V, Delta IV, RS-68, EOS-Aqua, Genesis, and
  GOES were benchmarked in three separate phases of
  analysis.

7
Benchmarking

• Results showed close comparisons to actual costs
  at top level, except for Atlas V, which is
  attributed to new engine (RD-180).
• Estimates 6 above actual cost for Delta IV, for
  example.
• At lower levels, NAFCOM consistently
  underestimated avionics costs in the first phase
  of benchmarking.

8
Benchmarking

• The results gleaned from benchmarking were used
  to modify NAFCOM estimating algorithms and
  approaches in order to increase the models
  estimating accuracy.
• Subsequent comparisons for missions benchmarked
  in prior phases supported evidence for improved
  estimating capability.
• Delta IV estimates improved from 11 higher than
  actual cost to 6 higher than actual cost.

9
Assessment of Model Validity

• SAIC was tasked to assess the validity of
  acquisition estimating techniques, specifically
  NAFCOM.
• An iterative validation process was developed to
  indicate areas of improvement and then track the
  impact of those improvements at both the CER
  level and the total estimating capability level.

10
Assessment of Model ValidityBaselining NAFCOM

• SAIC developed sixteen estimates using NAFCOM
  including manned, unmanned, and launch vehicles.
• These estimates were compared to actuals at both
  the subsystem and total cost levels.
• Identified potential trends in estimating
  inaccuracies (consistent over or under).

11
NAFCOM Baseline Estimates
12
NAFCOM Baseline Estimates
13
NAFCOM Baseline Estimates
14
Assessment of Model ValidityNAFCOM Complexity
Generators

• SAIC has conducted a statistical analysis of
  actual costs versus estimateds cost for each
  Complexity Generator CER.
• This analysis provides an overall measure of
  predictability including standard error, R2,
  coefficient of variance, and other metrics
  related to the estimated versus actual.
• The data was then stratified to highlight
  potential trends in estimating inaccuracies, e.g.
  mission type, weight, etc.

15
CER Segregation CCDH
16
Improvement MethodologyMission Class
Stratification

• Stratified subsystem databases using attribute
  variables.
• Earth Orbiting Lite, Earth Orbiting, Planetary,
  Launch Vehicles, and Manned.
• Previously, NAFCOM used a single variable with
  differing values for each mission class.
• The use of attribute or dummy variables is more
  statistically correct, allows for a better fit,
  and does not mask the potential counter-balancing
  effects of a single mission class variable.

17
Improvement MethodologyRemoval of Outliers

• In a few cases, outliers were removed from the
  CERs.
• A t-test was used to remove outliers the
  statistic
• follows a t-distribution with n-k-1 degrees
  of freedom,
• where n is the number of data points, k is
  the number of variables, ln y is the natural log
  of the actual cost of the potential outlier,
  est(ln y(l)) is the estimate of the potential
  outlier y using the model without y, and se(l) is
  the standard error of the model without y.
• Using a 95 critical value, any data point with
  p-value less than 1 in 1,000 is rejected as an
  outlier and removed.

18
Improvement MethodologyRemoval of Outliers

• Removal of data points was infrequent, at most
  three data points were removed from a CER because
  they were outliers.
• No large missions (e.g., Shuttle) were removed.
• Almost all missions removed were small satellites.

19
Improvement MethodologySubsystem Specific

• Other subsystem-specific improvement methods were
  applied on a case-by-case basis.

20
Attitude Control Improvements

• Stratified the Attitude Control database using
  mission class dummy variables.
• Microsat was proven to be a statistical outlier
  for the Attitude Control subsystem and removed.
• Adjusted database for old technology.
• Removed all missions with launch dates prior to
  1976.
• Assumption based on introduction year for first
  Intel 4004 microchip, with a four-year lag.

21
Attitude Control ImprovementsAdjustment for New
Technology

• Missions removed

Hawkeye INTELSAT-IV LANDSAT-1 Lunar
Orbiter Mariner-10 Mariner-4 Mariner-6 Mariner-8 M
odel-35 OSO-8
AE-3 Apollo CSM Apollo LM ATS-1 ATS-5 ATS-6 Centau
r-D DSCS-II Gemini
Pioneer-10 S3 S-IVB SMS-1 Surveyor TACSAT TIROS-M
VELA-IV
22
CCDH Improvements

• Stratified the CCDH database using mission class
  dummy variables.
• Microsat was proven to be a statistical outlier
  for the CCDH subsystem and removed.
• Also looked at REX but data point was not a
  significant outlier.
• Adjusted database for old technology.
• Removed all missions with launch dates prior to
  1976.
• Assumption based on introduction year for first
  Intel 4004 microchip, with a four-year lag.

23
Electrical Power Improvements

• Stratified the Electrical Power database using
  mission class dummy variables.
• Microsat, Hawkeye, and Radcal were proven to be
  statistical outliers for the Electrical Power
  subsystem and removed.

24
Reaction Control Improvements

• Stratified the Reaction Control database using
  mission class dummy variables.
• Also tried the following variables as inputs, but
  the fit was not improved
• Propellant Type
• ISP
• Launch Year

25
Structures Improvements

• Stratified the Structures database using mission
  class dummy variables.
• Added a dummy variable for Large, Inert
  Structure.
• Added a Year of Technology variable.
• SME was removed due to its being built primarily
  from spare parts.
• UFO and GPSMYP were removed due to them being
  follow-ons of previous missions.
• Mariner-4 was removed due to old technology
  (oldest data point in the Structures database).

26
SRM Kick Motors Improvements

• Stratified the SRM Kick Motors database using
  mission class dummy variables.
• Unmanned
• Manned
• Retro Rockets
• Upper Stage
• P-78 was proven to be statistical outlier for the
  SRM Kick Motor subsystem and removed.

27
Thermal ControlImprovements

• Improvement in fit due to incorporation of dummy
  variables for mission stratification, and a
  refinement of technical rating.
• Several missions were removed.
• SME Thermal control was built from spare parts.
• GPSMYP Follow-on to GPS-1.
• Removed NATO-III, Intelsat-III, MACSAT, DSP, and
  Voyager for being outliers.

28
Crew Accommodation and ECLSSImprovements

• Added data points to the CERs.

Propulsion and OMSImprovements

• Added attribute variables for environment and
  reusability.

29
NAFCOM CG Improvement Summary

• Reviewed all subsystem CERs.
• Modified all the complexity generator equations.
• Added new cost drivers (e.g., weight/volume
  ratio added to
• ECLS CER).
• - Stratified database according to mission type.
• - Removed a few statistical outliers.
• - Incorporated launch year (year of
  technology).
• - Changed technical rating definitions.
• Removed old missions (pre-1976) for some
  subsystems
• (e.g., CCDH).
• Changes improved goodness-of-fit statistics.

NAFCOM is continually improved to increase
estimating credibility.
30
Summary of Improvements
Additional data points added to Crew
Accommodations and ECLSS CERs.
31
Example CER Estimates for COBE, Before
Improvements and After
32
New NAFCOM Features

• New Template Wizard Options (e.g. CTV Only, Probe
  Only, etc.)
• WBS Generator
• User Defined Elements
• Stage/System Insert and Delete
• Risk Dollar Allocation Reports
• Engine Risk (LRE, CCP, and Turbojet)
• Pop-Up Notes Feature

New version of NAFCOM is now available.
33
Backup Slides
34
New Template Wizard Options
Additional templates have been developed and
incorporated into the model (1) CTV Only (2)
Upper Stage Only (3) Inner Planetary Orbiter
Only (4) Inner Planetary Flyby Spacecraft
Only (4) Inner Planetary Flyby Probe Only (5)
Inner Planetary Flyby Lander Only (6) Outer
Planetary Orbiter Only (7) Outer Planetary Flyby
Spacecraft Only (8) Outer Planetary Flyby Probe
Only
35
WBS Generator

• The WBS Generator is an interface to provide the
  user the flexibility of starting from scratch
  along with the automated functions and
  convenience of the modern NAFCOM.
• To begin users are prompted to enter the number
  of stages/systems to include in their WBS.
• The stages/systems are created automatically with
  a total roll-up, hardware rollup, one blank
  element, and the system integration elements.
• The user can drag and drop, cut, copy, and paste
  to populate the stages/systems.
• The user can also add instruments, operations
  cost, or launch services to the WBS from the
  Wizard interface.

36
User Defined Element

• The User Defined Element Type allows the user to
  bypass the NAFCOM methodology.
• It provides an additional estimating option.
• Known costs or custom equations can be defined to
  estimate a particular element.
• When using the User Defined Element Type the user
  will be allowed to use simple mathematical
  commands to enter costs or equations.
• The User Defined input screen allows the user to
  input equations for DD, STH, and Flight Unit.
  The user can also create an equation that
  contains any combination of the variables W, X, Y
  and Z.
• The User Defined Element Type is accessible via
  the Input Subsystem menu item.

37
Risk WBS Allocation Report

• Risk Dollars can now be allocated by WBS element.
• Risk Dollars is the amount of additional dollars
  (above the mean) required to fund a program at
  some appropriate level of confidence.
• The two new Risk Reports are
• Risk Allocation Report Xth Percentile
• Provides a WBS allocation of Risk Dollars only by
  WBS element calculated by the 70th, 80th, or 90th
  Percentile.
• Risk Allocation Report Xth Percentile Total
• Provides a WBS allocation of Risk Dollars and
  Mean costs by WBS element calculated by the 70th,
  80th, or 90th Percentile.