Early Systems Costing

1
Early Systems Costing
Prof. Ricardo Valerdi Systems Industrial
Engineering rvalerdi_at_arizona.edu Dec 15,
2011 INCOSE Brazil Systems Engineering
Week INPE/ITA São José dos Campos, Brazil
2
Take-Aways

• Cost f(Effort) f(Size) f(Complexity)
• Requirements understanding and ilities are the
  most influential on cost
• Early systems engineering yields high ROI when
  done early and well
• Two case studies
• SE estimate with limited information
• Selection of process improvement initiative

3
Cost Commitment on Projects
Blanchard, B., Fabrycky, W., Systems Engineering
Analysis, Prentice Hall, 2010.
4
Cone of Uncertainty
4x
2x
Relative Size Range
x
0.5x
Initial Operating Capability
OperationalConcept
Life Cycle Objectives
Life Cycle Architecture
0.25x
Feasibility
Plans/Rqts.
Design
Develop and Test
Phases and Milestones
Boehm, B. W., Software Engineering Economics,
Prentice Hall, 1981.
5

• The Delphic Sybil
• Michelangelo Buonarroti
• Capella Sistina, Il Vaticano (1508-1512)

6
Systems Engineering Effort vs. Program Cost
NASA data (Honour 2002)
7
COSYSMO Origins
Systems Engineering
(Warfield 1956)
1950
Software Cost Modeling
COSYSMO
(Boehm 1981)
1980
CMMI
(Humphrey 1989)
1990
Capability Maturity Model Integrated (Software
Engineering Institute, Carnegie Mellon
University) Warfield, J. N., Systems Engineering,
United States Department of Commerce PB111801,
1956. Boehm, B. W., Software Engineering
Economics, Prentice Hall, 1981. Humphrey, W.
Managing the Software Process. Addison-Wesley,
1989.
8
How is Systems Engineering Defined?

• Acquisition and Supply
• Supply Process
• Acquisition Process
• Technical Management
• Planning Process
• Assessment Process
• Control Process
• System Design
• Requirements Definition Process
• Solution Definition Process

• Product Realization
• Implementation Process
• Transition to Use Process
• Technical Evaluation
• Systems Analysis Process
• Requirements Validation Process
• System Verification Process
• End Products Validation Process

EIA/ANSI 632, Processes for Engineering a System,
1999.
9
COSYSMO Data Sources
Boeing Integrated Defense Systems (Seal Beach, CA)
Raytheon Intelligence Information Systems (Garland, TX)
Northrop Grumman Mission Systems (Redondo Beach, CA)
Lockheed Martin Transportation Security Solutions (Rockville, MD) Integrated Systems Solutions (Valley Forge, PA) Systems Integration (Owego, NY) Aeronautics (Marietta, GA) Maritime Systems Sensors (Manassas, VA Baltimore, MD Syracuse, NY)
General Dynamics Maritime Digital Systems/AIS (Pittsfield, MA) Surveillance Reconnaissance Systems/AIS (Bloomington, MN)
BAE Systems National Security Solutions/ISS (San Diego, CA) Information Electronic Warfare Systems (Nashua, NH)
SAIC Army Transformation (Orlando, FL) Integrated Data Solutions Analysis (McLean, VA)
L-3 Communications Greenville, TX
10
Modeling Methodology
11
Results of Bayesian Update Using Prior and
Sampling Information
12
COSYSMO Scope

• Addresses first four phases of the system
  engineering lifecycle (per ISO/IEC 15288)
• Considers standard Systems Engineering Work
  Breakdown Structure tasks (per EIA/ANSI 632)

Conceptualize
Operate, Maintain, or Enhance
Replace or Dismantle
Transition to Operation
Oper Test Eval
Develop
13
COSYSMO Operational Concept
Requirements Interfaces Scenarios
Algorithms
Size Drivers
COSYSMO
Effort (schedule, risk)
Effort Multipliers

• Application factors
• 8 factors
• Team factors
• 6 factors

Calibration
WBS guided by EIA/ANSI 632
14
Software Cost Estimating Relationship
MM Man months a calibration constant S size
driver E scale factor c cost driver(s) KDSI
thousands of delivered source instructions
Boehm, B. W., Software Engineering Economics,
Prentice Hall, 1981.
15
COSYSMO Model Form
Where PMNS effort in Person Months (Nominal
Schedule) A calibration constant derived from
historical project data k REQ, IF, ALG,
SCN wx weight for easy, nominal, or
difficult size driver quantity of k
size driver E represents diseconomies of
scale EM effort multiplier for the jth cost
driver. The geometric product results in an
overall effort adjustment factor to the nominal
effort.
16
Size Drivers vs. Effort Multipliers

• Size Drivers Additive, Incremental
• Impact of adding a new item inversely
  proportional to current size
• 10 -gt 11 rqts 10 increase
• 100 -gt 101 rqts 1 increase
• Effort Multipliers Multiplicative, system-wide
• Impact of adding a new item independent of
  current size
• 10 rqts high security 40 increase
• 100 rqts high security 40 increase

17
4 Size Drivers

1. Number of System Requirements
2. Number of System Interfaces
3. Number of System Specific Algorithms
4. Number of Operational Scenarios

Weighted by complexity and degree of reuse
18
Number of System Requirements This driver
represents the number of requirements for the
system-of-interest at a specific level of design.
The quantity of requirements includes those
related to the effort involved in system
engineering the system interfaces, system
specific algorithms, and operational scenarios.
Requirements may be functional, performance,
feature, or service-oriented in nature depending
on the methodology used for specification. They
may also be defined by the customer or
contractor. Each requirement may have effort
associated with is such as VV, functional
decomposition, functional allocation, etc.
System requirements can typically be quantified
by counting the number of applicable
shalls/wills/shoulds/mays in the system or
marketing specification. Note some work is
involved in decomposing requirements so that they
may be counted at the appropriate
system-of-interest.
Easy Nominal Difficult
- Simple to implement Familiar - Complex to implement or engineer
- Traceable to source - Can be traced to source with some effort - Hard to trace to source
- Little requirements overlap - Some overlap - High degree of requirements overlap
19
Counting Rules Example

• COSYSMO example for sky, kite, sea,
• and underwater levels where

Sky level Build an SE cost model Kite level
Adopt EIA 632 as the WBS and ISO 15288 as the
life cycle standard Sea level Utilize size and
cost drivers, definitions, and counting
rules Underwater level Perform statistical
analysis of data with software tools and
implement model in Excel
Source Cockburn 2001
20
Size Driver Weights
Easy Nominal Difficult
of System Requirements 0.5 1.00 5.0
of Interfaces 1.7 4.3 9.8
of Critical Algorithms 3.4 6.5 18.2
of Operational Scenarios 9.8 22.8 47.4
21
Cost Driver Clusters

• UNDERSTANDING FACTORS
• Requirements understanding
• Architecture understanding
• Stakeholder team cohesion
• Personnel experience/continuity
• COMPLEXITY FACTORS
• Level of service requirements
• Technology Risk
• of Recursive Levels in the Design
• Documentation Match to Life Cycle Needs
• OPERATIONS FACTORS
• and Diversity of Installations/Platforms
• Migration complexity

• PEOPLE FACTORS
• Personnel/team capability
• Process capability
• ENVIRONMENT FACTORS
• Multisite coordination
• Tool support

22
Stakeholder team cohesion Represents a
multi-attribute parameter which includes
leadership, shared vision, diversity of
stakeholders, approval cycles, group dynamics,
IPT framework, team dynamics, trust, and amount
of change in responsibilities. It further
represents the heterogeneity in stakeholder
community of the end users, customers,
implementers, and development team.
1.5 1.22 1.00 0.81 0.65
Viewpoint Very Low Low Nominal High Very High
Culture Stakeholders with diverse expertise, task nature, language, culture, infrastructure Highly heterogeneous stakeholder communities Heterogeneous stakeholder community Some similarities in language and culture Shared project culture Strong team cohesion and project culture Multiple similarities in language and expertise Virtually homogeneous stakeholder communities Institutionalized project culture
Compatibility Highly conflicting organizational objectives Converging organizational objectives Compatible organizational objectives Clear roles responsibilities Strong mutual advantage to collaboration
Familiarity and trust Lack of trust Willing to collaborate, little experience Some familiarity and trust Extensive successful collaboration Very high level of familiarity and trust
23
Technology Risk The maturity, readiness, and
obsolescence of the technology being implemented.
Immature or obsolescent technology will require
more Systems Engineering effort.
Viewpoint Very Low Low Nominal High Very High
Lack of Maturity Technology proven and widely used throughout industry Proven through actual use and ready for widespread adoption Proven on pilot projects and ready to roll-out for production jobs Ready for pilot use Still in the laboratory
Lack of Readiness Mission proven (TRL 9) Concept qualified (TRL 8) Concept has been demonstrated (TRL 7) Proof of concept validated (TRL 5 6) Concept defined (TRL 3 4)
Obsolescence - Technology is the state-of-the-practice - Emerging technology could compete in future - Technology is stale - New and better technology is on the horizon in the near-term - Technology is outdated and use should be avoided in new systems - Spare parts supply is scarce
24
Migration complexity This cost driver rates the
extent to which the legacy system affects the
migration complexity, if any. Legacy system
components, databases, workflows, environments,
etc., may affect the new system implementation
due to new technology introductions, planned
upgrades, increased performance, business process
reengineering, etc.
Viewpoint Nominal High Very High Extra High
Legacy contractor Self legacy system is well documented. Original team largely available Self original development team not available most documentation available Different contractor limited documentation Original contractor out of business no documentation available
Effect of legacy system on new system Everything is new legacy system is completely replaced or non-existent Migration is restricted to integration only Migration is related to integration and development Migration is related to integration, development, architecture and design
25
Cost Driver Rating Scales
Very Low Low Nominal High Very High Extra High EMR
Requirements Understanding 1.87 1.37 1.00 0.77 0.60   3.12
Architecture Understanding 1.64 1.28 1.00 0.81 0.65   2.52
Level of Service Requirements 0.62 0.79 1.00 1.36 1.85   2.98
Migration Complexity     1.00 1.25 1.55 1.93 1.93
Technology Risk 0.67 0.82 1.00 1.32 1.75   2.61
Documentation 0.78 0.88 1.00 1.13 1.28   1.64
and diversity of installations/platforms     1.00 1.23 1.52 1.87 1.87
of recursive levels in the design 0.76 0.87 1.00 1.21 1.47   1.93
Stakeholder team cohesion 1.50 1.22 1.00 0.81 0.65   2.31
Personnel/team capability 1.50 1.22 1.00 0.81 0.65   2.31
Personnel experience/continuity 1.48 1.22 1.00 0.82 0.67   2.21
Process capability 1.47 1.21 1.00 0.88 0.77 0.68 2.16
Multisite coordination 1.39 1.18 1.00 0.90 0.80 0.72 1.93
Tool support 1.39 1.18 1.00 0.85 0.72   1.93
26
Cost Drivers Ordered by Effort Multiplier Ratio
(EMR)
27
Effort Profiling
Transition to Operation
Operational Test Evaluation
Conceptualize
Develop
ISO/IEC 15288
ANSI/EIA 632
Acquisition Supply
Technical Management
System Design
Product Realization
Technical Evaluation
28
Benefits of Local Calibration
Before local calibration
After local calibration
29
Prediction Accuracy
PRED(30)
PRED(25)
PRED(20)
PRED(30) 100 PRED(25) 57
30
(No Transcript)
31
Human Systems Integration in the Air Force
Joint HSI Personnel Integration Working Group.
Human System Integration (HSI) Activity in DoD
Weapon Acquisition Programs Part II Program
Coverage and Return on Investment. April 16,
2007. U.S. Air Force Scientific Advisory Board.
Report on Human Systems Integration in Air Force
Weapon Systems Development and Acquisition.
July, 2004.
32
HSI Early in F119 Development
AF Acquisition Community-led requirements
definition studies 40 fewer parts than previous
engines
33
Parametric Cost Estimation
34
Parametric Cost Estimation
Architecture Understanding
Operational Scenarios
Technology Risk
Algorithms
Interfaces
Tool Support
Requirements
Personnel/team capability
and diversity of installations/platforms
35
Illustrative Example
Nominal System
100 Difficult Requirements
200 Medium Requirements
200 Easy Requirements
SE Effort 300 Person-months
36
Illustrative Example
Size impact
Nominal System
110 Difficult Requirements
5th/95th Percentile Manpower
20 minute component replacement
210 Medium Requirements
Tools reduction
Interactive Technical Manuals
200 Easy Requirements
SE Effort 327 Person-months
37
Illustrative Example
Cost impact
Nominal System
effect of effort multipliers High Level of
service requirements High HSI Tools support
SE Effort 368 Person-months
38
ROI Calculation

• Cost 68 Person Months _at_ 10,000/person month
• 680,000 of Human Systems Integration
• Benefit assuming 30 year life cycle
• Manpower (one less maintainer) 200,000 30
  6,000,000
• Human factors (40 fewer parts) 300,000 30
  9,000,000
• Safety (one less repair accident)
  50,000
• Survivability (one less engine failure)
  500,000
• 15,550,000
• Return on Investment

Note time value of money excluded
39
Relative ROI
Activity ROI
CMMI 173
ISO 9001 229
Systems Engineering 1,700
Human Systems Integration 2,186
Software inspections 3,272
Personal Software Process 4,133
Brodman, J. G., Johnson, D. L., Return on
Investment from Software Process Improvement as
Measured by U.S. Industry, Crosstalk The
Journal of Defense Software, 1996. Rico, D. F.,
ROI of Software Process Improvement Metrics for
Project Managers and Software Engineers, J. Ross
Publishing, Boca Raton, FL, 2004. Boehm, B. W.,
Valerdi, R. and Honour, E., The ROI of Systems
Engineering Some Quantitative Results for
Software-Intensive Systems, Systems Engineering,
11(3), 221-234, 2008.
40
(No Transcript)
41
Case Study I Albatross Budgetary Estimate

• You, as the Albatross SE lead, have just been
  asked to provide your program manager a systems
  engineering budgetary estimate for a new,
  believed-to-be critical function to the current
  baseline
• This new functionality would add some new,
  nearly-standalone, capability to your Albatross
  program, your best educated guess by looking at
  the emailed requirements provided by your
  customer is that it adds about 10 to the
  requirements baseline of 1,000 and two new
  interfaces, you guess we need at least one new
  Operational Scenario.
• The customer also stated that they really need
  this capability to be integrated into the next
  delivery (5 months from now)
• The PM absolutely has to have your SE cost
  estimate within the next two hours, as the
  customer representative needs a not-to-exceed
  (NTE) total cost soon after that!
• Information that may prove useful in your
  response
• Albatross is well into system IT, with somewhat
  higher than expected defects
• Most of the baseline test procedures are nearing
  completion
• The SE group has lost two experienced people in
  the past month to attrition
• So far, the Albatross customer award fees have
  been excellent, with meeting of schedule
  commitments noted as a key strength

42
Case Study I In-Class Discussion Questions

• What are some of the risks?
• What additional questions could you ask of your
  PM?
• What additional questions could the PM (or the SE
  Lead) ask of the Customer Representative?
• What role could the Albatross PM play in this
  situation?
• Is providing only a number appropriate in this
  situation?
• What additional assumptions did you make that can
  be captured by COSYSMO?

43
Case Study II Selection of Process Improvement
Initiative

• You are asked by your supervisor to recommend
  which process improvement initiative should be
  pursued to help your project, a 5-year
  100,000,000 systems engineering study. The
  options and their implementation costs are
• Lean 1,000,000
• Six Sigma 600,000
• CMMI 500,000
• Which one would you chose?

44
Case Study II Selection of Process Improvement
Initiative

• Assumptions
• Implementing Lean on your project will greatly
  improve your teams documentation process. Using
  COSYSMO, you determine that the Documentation
  cost driver will move from Nominal to Low
  yielding a 12 cost savings (1.00 0.88 0.12).
• Implementing Six Sigma will greatly improve your
  teams requirements process. Using COSYSMO, you
  determine that the Requirements Understanding
  cost driver will move from Nominal to High
  yielding a 23 cost savings (1.00 0.77 0.23)
• Implementing CMMI will greatly improve team
  communication. Using COSYSMO, you determine that
  the Stakeholder Team Cohesion cost driver will
  move from Nominal to High yielding a 19 cost
  savings (1.00 0.81 0.19)

45
Cost Driver Rating Scales
Very Low Low Nominal High Very High Extra High EMR
Requirements Understanding 1.87 1.37 1.00 0.77 0.60   3.12
Architecture Understanding 1.64 1.28 1.00 0.81 0.65   2.52
Level of Service Requirements 0.62 0.79 1.00 1.36 1.85   2.98
Migration Complexity     1.00 1.25 1.55 1.93 1.93
Technology Risk 0.67 0.82 1.00 1.32 1.75   2.61
Documentation 0.78 0.88 1.00 1.13 1.28   1.64
and diversity of installations/platforms     1.00 1.23 1.52 1.87 1.87
of recursive levels in the design 0.76 0.87 1.00 1.21 1.47   1.93
Stakeholder team cohesion 1.50 1.22 1.00 0.81 0.65   2.31
Personnel/team capability 1.50 1.22 1.00 0.81 0.65   2.31
Personnel experience/continuity 1.48 1.22 1.00 0.82 0.67   2.21
Process capability 1.47 1.21 1.00 0.88 0.77 0.68 2.16
Multisite coordination 1.39 1.18 1.00 0.90 0.80 0.72 1.93
Tool support 1.39 1.18 1.00 0.85 0.72   1.93
46
Case Study II Selection of Process Improvement
Initiative

• Assumptions
• The systems engineers on your project spend
• 5 of their time doing documentation
• 30 of their time doing requirements-related
  work
• 20 of their time communicating with
  stakeholders
• What is the financial benefit of each process
  improvement initiative?
• How does the financial benefit vary by project
  size?

47
Return-on-Investment Calculation(benefit
cost) / cost
Best option is Six Sigma!
48
Return-on-Investment vs. Project Size
Six Sigma
CMMI
Lean
49
Reuse is a Universal Concept
50
Reusable Artifacts

• System Requirements
• System Architectures
• System Description/Design Documents
• Interface Specifications for Legacy Systems
• Configuration management plan
• Systems engineering management plan
• Well-Established Test Procedures
• User Guides and Operation Manuals
• Etc.

51
Reuse Principles

• The economic benefits of reuse can be described
  either in terms of improvement (quality, risk
  identification) or reduction (defects,
  cost/effort, time to market).
• Reuse is not free, upfront investment is required
• (i.e., there is a cost associated with design for
  reusability).
• Reuse needs to be planned from the
  conceptualization phase of program

52
Reuse Principles (cont.)

1. Products, processes, and knowledge are all
   reusable artifacts.
2. Reuse is as much of a technical issue as it is an
   organizational one
3. Reuse is knowledge that must be deliberately
   captured in order to be beneficial.

53
Reuse Principles (cont.)

1. The relationship between project size/complexity
   and amount of reuse is non-linear
2. The benefits of reuse are limited to closely
   related domains.
3. Reuse is more successful when level of service
   requirements are equivalent across applications.

54
Reuse Principles (cont.)

1. Higher reuse opportunities exist when there is a
   match between the diversity and volatility of a
   product line and its associated supply chain.
2. Bottom-up (individual elements where make or buy
   decisions are made) and top-down (where product
   line reuse is made) reuse require fundamentally
   different strategies.
3. Reuse applicability is often time dependent
   rapidly evolving domains offer fewer reuse
   opportunities than static domains.

55
Reuse Success Factors

• Platform
• Appropriate product or technology, primed for
  reuse
• People
• Adequate knowledge and understanding of both the
  heritage and new products
• Processes
• Sufficient documentation to acquire and capture
  knowledge applicable to reuse as well as the
  capability to actually deliver a system
  incorporating or enabling reuse

56
Background on Software Reuse

• Main size driver KSLOC
• Adapted Source Lines of Code (ASLOC)
• Percent of Design Modification (DM)
• Percent of Code Modification (CM)
• Percent of Integration Required for Modified
  Software (IM)
• Percentage of reuse effort due to Software
  Understanding (SU)
• Percentage of reuse effort due to Assessment and
  Assimilation (AA)
• Programmer Unfamiliarity with Software (UNFM)

AAF
From COCOMO II Model Definition Manual (p. 7-11)
57
Example COSYSMO 2.0 Estimate
Estimated as 129.1 Person-Months by COSYSMO
(without reuse) a 30.4 difference
58
COSYSMO 2.0 Implementation Results

• Across 44 projects at 1 diversified organization
• Using COSYSMO
• PRED(.30) 14
• PRED(.40) 20
• PRED(.50) 20
• R2 0.50
• Using COSYSMO 2.0
• PRED(.30) 34
• PRED(.40) 50
• PRED(.50) 57
• R2 0.72
• Result 36 of 44 (82) estimates improved

59
Reuse Terminology

• New
• Items that are completely new
• Managed
• Items that are incorporated and require no added
  SE effort other than technical management
• Adopted
• Items that are incorporated unmodified but
  require verification and validation
• Modified
• Items that are incorporated but require tailoring
  or interface changes, and verification and
  validation
• Deleted
• Items that are removed from a legacy system,
  which require design analysis, tailoring or
  interface changes, and verification and validation

• Notes
• New items are generally unprecedented
• Those items that are inherited but require
  architecture or implementation changes should be
  counted as New

60
Reuse Continuum
1.0
New
Modified vs. New Threshold
Modified
0.65
Deleted
Adopted
Reuse weight
0.51
0.43
Managed
0.15
0
61
(No Transcript)
62
Impact
Commercial Implementations
COSYSMO Model
10 Academic Theses
Academic Curricula
63
Take-Aways

• Cost f(Effort) f(Size) f(Complexity)
• Requirements understanding and ilities are the
  most influential on cost
• Early systems engineering yields high ROI when
  done early and well
• Two case studies
• SE estimate with limited information
• Selection of process improvement initiative