PostColumbia Budget Proposal: FY 2003 Supplemental FY 2004 AmendmentSupplemental Integrated OMB Brie

1
NASA Cost Estimating Initiatives/Activities
Joe Hamaker NASA HQ Cost Analysis Division March
2004
2
Thank You!

• Thanks to Richard Whitlock and his staff, and to
  Rey Carpio for organizing this years symposium
• Thanks to Gigi Hackford and Awards Committee
  establishing Rutledge Award and selecting this
  years winner
• And thanks to all yall for coming too!

All yall yall for larger groups
3
Message From Steve Isakowitz

• Degradation of cost function during the 1990s has
  turned around
• Administrator is very cost conscious
• Tom Young Committee and ISS ICE clear evidence of
  renewed commitment to quality estimating
• Congress, GAO, IG are calling for better cost
  estimating and overall financial credibility)
• We are being asked to execute on our promises of
  performance, schedule and budget with the latter
  perceived as our greatest challenge
• Changes in place reflect our commitment to
  improve
• Code BC, IPAO, Code D, IFMP, full cost, etc.
• Now we have the Presidents vision for
  exploration
• Cost played a huge role in the deliberations
  leading to this vision--sand chart has been key
  to the arguments
• Underlying assumptions assumed no heroic miracles
  
• Teams are hard at work establishing requirements
  and top level architectures
• Code T spinning up
• The challenge for the cost community is to cost
  these architecture elements and constructively
  critique any innovative business practices for
  real substance
• To do our job we need good cost estimating at all
  levels from the Centers, through the IPAO to HQ
• Independent multi level review is healthy (like
  peer reviews in science)
• Cost function needs to fully participate in
  systems engineering of these architectures using
  CAIV and other approaches
• While staunchly maintaining real credibility in
  the cost estimates
• Cost played a huge role in the formulation of the
  vision and will play an even larger role in the
  implementation

4
Status of Cost Analysis Division at NASA HQ

• Cost function/presence firmly re-established at
  HQ
• Actively participating in other mainline
  activities (next chart)
• Code BC
• 4 FTEs on board in Code BC
• 1 rotational on board through May
• 1 rotational coming on board in June
• Job announcement forthcoming for 2 GS14/15
  analysts

5
Current Cost Activities at NASA HQ

• Fielding questions relative to Exploration
  Initiative sand chart
• Providing cost support to space architect
  activities
• Providing Agency interface to the GAO as they
  complete their report on NASA cost estimating
• Working with BAH to finalize Cost Estimating
  Handbook
• Working with new Exploration Systems Office (Code
  T)
• Serving on Core and Blue Team/Red Team Reviews of
  requirements and top level architectures
• Coordinating with EVM Focal Point Council
• Coordinating with IFMP
• Continued close interface with Office of Chief
  Engineer (Code D)
• Completed looking back cost growth study
• Cost estimating initiatives (balance of briefing)

WBS
Cost
EVM
IFMP
6
Ongoing NASA Cost Estimating Initiatives

• Cost Analysis Data Requirement (CADR)
• One NASA Cost Estimating (ONCE) Database
• Cost Readiness Levels
• Cost Growth Study
• Full cost estimating methods
• Cost risk analysis methods and best practices
• Cost Risk Feedback Continuum
• NASA Inflation Index
• Cost Estimating Handbook
• NPR 7120.5C
• Cost Estimator Career Guide
• Process Based Cost Estimating
• Improved Operations Cost Modeling
• Use of COTs tools

7
Ongoing NASA Cost Estimating Initiatives
Joe Hamaker

• Cost Analysis Data Requirement (CADR)
• One NASA Cost Estimating (ONCE) Database
• Cost Readiness Levels
• Cost Growth Study
• Full cost estimating methods
• Cost risk analysis methods and best practices
• Cost Risk Feedback Continuum
• NASA Inflation Index
• Cost Estimating Handbook
• NPR 7120.5C
• Cost Estimator Career Guide
• Process Based Cost Estimating
• Improved Operations Cost Modeling
• Use of COTs tools

David Graham Panel Sharon Winn
David Graham
Kelley Cyr
Jill Hardash
Dr. Chris Smart
Rich Greathouse, Dr. Martin Steele, Edgar Zapata
et al
Galorath, Mainstay, Price
8
Cost Analysis Data Requirement (CADR)
9
Cost Analysis Data Requirement (CADR)

• Cost estimating touches several DRs in a typical
  flight hardware development project
• There is considerable overlap in the cost data
  requirements
• Especially the WBS, CARD and Life Cycle Cost, and
  Cost Report DRs
• Also considerable integration needed between
  these DRs
• One DR can integrate many of the cost
  requirements
• One stop shopping for project cost data
• Can serve PM as a valuable management tool

XYZXYZ
XYZXYZ
Risk Plan
EMV
Schedule
Cost Data Report
Life Cycle Cost Estimate
CARD
WBS
NASA DRs DOD CDRLs
10
Cost Analysis Data Requirement (CADR)

• Most similar to a CARD but
• Provides standardized WBS and WBS dictionary
• Includes not only cost estimating inputs but cost
  estimates as well (as a separable appendix)
• Contractor cost reporting
• Project Office (full cost) cost reporting
• Increased focus on KPPs
• Mandatory HQ DR on Cat 1 and high risk Cat 2
  flight projects
• The CADR must be a PM responsibility
• Prime contractor can build most of it
• Cost shops at Centers can support by providing
  definitions/interpretations of what is being
  asked for
• What is a SLOC?
• CADR scope/focus
• Basic and applied research
• Technology development
• Flight projects
• Operations
• Functional products services

CADR only intended as a DR on these 7120.5C
categories
11
CADR Contents

• Standardized WBS structures and dictionaries to
  level 5 (major component level)
• Earth science (bus/payload)
• Space science (bus/payload)
• Launch systems
• In-space transportation systems
• Cost Input Report
• Data sheets for KPP cost parameters by WBS
  element (see next chart)
• Cost Estimate Report
• Data sheets for cost estimates by WBS element
• Contractor cost reporting
• Project Office (full cost) cost reporting

By WBS Element
Cost Estimate Report
Cost Input Report
12
CADR As A Flight Data Recorder For Management

• The inputs for parametric cost models include a
  comprehensive set of KPPs as well as
  programmatic, schedule and management metrics
  that describe a project
• In the concise and precise language of parametric
  cost estimating
• Metrics are rigorously defined by the models
• CADRs are living documents
• Snapshots at least annually
• Provides the PM with basis for updated EACs
  (performed by Agency parametricians)
• Independent of contractors opinion
• Provides the PM with predicted cost effects of
  remedial actions
• CADRs are encased into the One NASA Cost
  Estimating Data Base
• Automatic lessons learned and cost autopsy
  document

SLOC Count
Time
13
Cost Analysis Data Requirement (CADR)

• CADR status
• CADR document being drafted to be ready for
  review at April 6-7 CASG meeting
• Completed document to be vetted through HQ
  Enterprises for buy-in
• CADR is proposed as a RFP DR on Category I and
  high risk Category II flight projects
• To be included in 7120.5C and the NASA Cost
  Estimating Handbook (CEH)
• CADR template downloadable from CEH website
• To be included in NASA FAR Supplement
• Heads up via Code H Procurement Information
  Circulars (PICs)
• To be reviewed at Acquisition Strategy Meetings
  (ASMs)

NASA FAR Sup
NPR 7120.5C CEH
ASMs
PICs
14
Cost Analysis Data Requirement (CADR)

• Possible extension of CADR
• CADR could be tied to a USG cost model (NAFCOM,
  Price, Seer, other)
• As CADRs are submitted, the prime contractor and
  project office would be required to process the
  CADR project description (the KPPs) through the
  provided cost model
• With the help of Agency parametricians
• This would occur at end of Phase A, again at the
  end of Phase B/PDR
• At least annually thereafter
• Working with the Agency parametricans, the
  project would be required to calibrate the
  provided model to match the prime
  contract/Project Office estimate
• The contractor and Project Office would
  encourages to maintain their own indigenous
  estimating method
• Would provide exposure of what underlying
  assumptions have to be made to calibrate to the
  prime contractor/project office position on cost
• Up and down the WBS
• 80 code reuse? Simpliest materials/tolerances?
  25 confident schedule? Truckloads of GFE? Lake
  Wobegone Team?

Where all the children are above average
15
Once NASA Cost Engineering (ONCE) Data Base
16
Once NASA Cost Engineering (ONCE) Data Base
From One NASA Website
Recommendation 6 Tools and Business
Practices Increase NASA-wide cross-collaboration
through common tools and business
practices. Enhance cross-Agency collaboration by
putting in place standard engineering and
collaborative tools and databases, processes, and
knowledge-sharing structures. These tools and
methods need to enhance the ability to
collaborate, reduce inefficiency, and create time
and resources for collaboration.
6d. Review Common Databases. Assess the current
state of databases, such as the Life Sciences
Data Archive, for their ability to span multiple
NASA Centers. Recommend adoption of common
databases as appropriate. Develop a plan to
increase awareness of databases and to provide
database-specific guidelines for use.
6f. Knowledge Management System. Establish a
formal Knowledge Management (KM) pilot project,
in which KM communities of practice (COPs) can
occur for a small set of engineering, research,
and administrative specialized disciplines that
span multiple NASA Centersfor example, thermal
test engineers. Identify and train these COPs in
KM principles of operation.
17
One NASA Cost Engineering (ONCE) Data Base

• A major tenet of One NASA is to share information
  and, where it makes sense, utilize common data
  bases
• Since 1971, NASA has had the REDSTAR Agency wide
  cost data base and the corresponding NAFCOM data
  base of normalized data
• Many other cost data bases exist in pockets
  across the Agency
• Often estimators are unaware of or have
  difficulty quickly obtaining data
• Data normalizations are not standard
• Some Center data is undoubtedly unique to that
  Centers mission (e.g. presumably only KSC, MSFC
  and SSC are interested in rocket engines)
• But there remain convincing reasons to share cost
  data
• A battery is a battery (i.e. not Center unique)
• Will serve to standardize arbitrary WBS usage and
  cost normalization processes across Centers
• Facilitates CoSTER data sharing with other USGs
  active in space activities
• More data and more current data in the hands of
  estimators is always the direction of good

18

One NASA Cost Engineering (ONCE) Data Base

• Some goals
• A One NASA product, with data availability across
  Centers (and via CoSTER, across USG space
  organizations)
• Protect contractor proprietary data rights
• Protect NASA Center competitive issues (bids on
  AOs and NRAs)
• Support all Center cost methodologies and
  numerous models (NAFCOM, PRICE, SEER, other NASA
  Center Models)
• Use a common WBS and common set of cost
  normalization ground rules
• Which may be re-mapped to suit particular user
  needs
• Collect cost and key performance parameters and
  programmatic cost drivers in one integrated data
  base
• Utilize the new Code BC CADR as a data collection
  instrument

19
NASA Cost Data Base Improvements

• Some issues
• Obtaining initial Center buy-in and long term
  continued support
• Protecting Center competitive stance in future AO
  and NRA competitions
• Devising a common data normalization procedure
  that is usable by all Centers
• Defining cost data
• Quotes versus actuals
• When to begin collection for a particular project
  (PDR? CDR? FRR?)
• Sorting out in-house versus prime contractor
  efforts
• How to book wraps?
• Are we collecting OS costs?
• What data base software to use? What server? What
  is the role of our support contractors?

20
One NASA Cost Engineering (ONCE) Data Base Concept
Cost Shops at NASA Centers

• Historical data
• From existing Center data bases
• MSFC REDSTAR/NAFCOM
• GSFC
• JPL
• etc.
• Data for Completed Projects
• Future data
• Collected with CADR at major milestones (i.e.
  ATP, PDR, CDR, FRR, etc.), at least annually

Database Support Contractor
ONCE DB
21
Once NASA Cost Engineering (ONCE) Data Base

• Concept
• Data collected at WBS level 4/5
• In early project phases, higher level OK
• Data base resident on server at IPAO/LaRC
• Data base update/access controlled by IPAO
• NDAs required for non-Government users
• WBS is a common framework for management and
  estimating
• If data is to be useful (for improving
  estimating), a standard WBS must be implemented

22
ONCE Database Level One Requirements

• Data Attributes
• Compatible with NASA Full-Cost initiative
• Include technical, schedule and cost data and
  narrative descriptions
• Include cost as either or hours, or both
• Data normalized prior to entry into database
• Identify recurring and non-recurring costs
• Identify actuals vs. estimated vs. budgeted costs
• Record project data at periodic intervals (major
  milestones)
• Identify OS costs
• Identify costs by project phase
• Incorporate existing NAFCOM data
• Incorporate cost-risk data
• Incorporate remarks/comments

• Data Base Functionality
• Prototype database operational by end of CY2004
  fully operational by mid-CY2005
• Web-accessible, simultaneous multiple users
• Secure access for authorized individuals (User ID
  and password)
• Flexible data storage at all appropriate WBS
  levels
• Database easily searched
• Store and retrieve data by project or mission as
  well as equipment type
• Allow data manipulation (e.g. change economic
  base year)
• Allow for manual data entry (in addition to data
  derived from CADR)
• Flexible, custom report formats

23
NASA Cost Growth A Look at Recent Performance

• Matt Schaffer
• NASA HQ Cost Analysis Division

24
Analytic Approach

• Compared initial and final budget estimates of
  development costs for 45 recent projects and
  computed percent budget growth as a surrogate for
  cost growth.
• Selected projects are predominantly Space Science
  projects, since these are the easiest projects
  for which one can develop a budget trace.
• Recent generally means missions flown since
  1996.
• All projects were initiated in 1990 or later.
• Four-fifths of the projects were initiated in the
  last ten years.
• Data come mostly from historical NASA budget
  documents other data come from other NASA
  archives and Internet documents.

25
Are the results truly cost growth?

• The analysis did not correct for content change.
• Many NASA projects were and are able to constrain
  budget growth by eliminating science
  requirements.
• Thus, budget growth is not a true representation
  of cost growth.
• But adjusting for these changes is difficult
  without some knowledge of the details, and these
  details are not found in budget documents.
• A partial reconstruction of the projects history,
  through analysis of project documentation, would
  be necessary before one could estimate the
  necessary adjustments.
• Such project documentation can be difficult, if
  not impossible, to find, as Code BC discovered in
  an attempt to gather documentation from recently
  launched systems.
• These difficulties not withstanding, the
  following results are probably fairly
  representative of actual cost growth in recent
  NASA projects.
• The analysis did not adjust for inflation.
• Cost growth is usually calculated using same-year
  constant-dollar estimates and actuals.
• Budget documents do not provide constant-dollar
  estimates, and inflation assumptions change
  yearly, making suspect any attempts to adjust
  budget dollars to constant dollars a posteriori,
  especially the initial budget estimate.
• The study attempted to capture only
  implementation budgets.
• In general, the budget data should be only for
  the implementation phase, although not all budget
  documents were clear on the content in budget
  lines.
• It is possible some growth represents an increase
  in content, e.g., including launch costs in
  development budgets.

26
Overall Statistics

• Average growth
• 36 arithmetic mean
• This statistic is the average of percent budget
  growth on individual projects.
• 45 dollar-weighted mean
• This statistic is the average of percent budget
  growth on individual projects weighted by final
  budget amount.
• Thus, some of the more expensive projects
  experienced considerable budget growth.
• Median growth 26
• 35 of 45 projects exceeded the initial budget
  estimate
• Total growth 28
• This statistic is the relative change from the
  total of the 45 initial budgets to the total of
  the 45 final budgets.

Recent NASA cost growth is about 30.
Schaffer study goes into much more detail
27
Full Cost Estimating
28
The Wethington Full Cost Factor

• Mike Wethington, on detail to Code BC, is
  developing a simplified approach for estimating
  the effect of full cost
• Suitable for Pre-Phase A, Phase A, (maybe) early
  Phase B estimates
• Under the assumption that sometime in Phase B,
  definitive full cost services agreements will be
  negotiated between the project and the Centers
  providing full cost services
• Combining GRC (Sefcik/Parkey) model, internal
  Code B data, and Creech civil service oversight
  curve

Ratio of CS to Contractor
Level 0 - No Oversight Accept performing
organizations tasks at face value (based on
assessment that no penetration is required) Level
1 - Low Oversight Participate in reviews and
Technical Interchange Meetings and assess only
the data presented Perform periodic audits on
pre-defined process(es) Chair board or serve as
board member, or RID writer, at a formal review
Participate in resolution and closure of
issues Level 2 - Intermediate Oversight
Includes low penetration with addition of
Daily or weekly involvement to identify and
resolve issues Level 3 - In-depth Oversight
Includes intermediate penetration with addition
of Methodical review of details Independent
models to check and compare vendor data, as
required Level 4 - Total Oversight Perform a
complete and independent evaluation of each task
0.4
1 to 3
0.35
0.3
1 to 5
0.25
1 to 8
0.2
1 to 10
0.15
0.1
0.05
0
1
2
3
4
Level of Insight
29
The Wethington Full Cost Factor
30
Cost Analysis Steering Group

• Next meeting scheduled for Tu/We April 6-7 at
  NASA HQ
• Agenda
• Discussion of Cost Analysis Data Requirement
  (CADR)
• Discussion of One NASA Cost Estimating (ONCE)
  Data Base
• Discussion of Cost Risk Analysis
• Discussion of Cost Risk Feedback Continuum
  (CRIFCON)

31
End