Nancy S. Eickelmann, PhD Motorola Labs 1303 E. Algonquin Rd. Annex2 Schaumburg, IL 60196 Phone: 847

1
Nancy S. Eickelmann, PhDMotorola Labs1303 E.
Algonquin Rd.Annex-2Schaumburg, IL 60196Phone
[phone]Fax [phone]Nancy.Eickelman
n_at_motorola.com
2
Developing Risk-Based Financial Analysis Tools
and Techniques to Aid IVV Decision-Making
FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL
(CSIP) for the NASA Independent Verification and
Validation Facility COTR Ken McGill PI Nancy
Eickelmann S-54493-G September 5, 2001
3
PROBLEM STATEMENT

•  
• This research addresses NASAs need to evaluate
  the ROI and cost/benefit of applying IVV
  technologies.
• A prototype is to be developed that will provide
  financial valuation of IVV for a given program.
• The prototype will be developed using an
  iterative process that will incrementally
  implement the models and methodology researched
  and developed during prior years of this effort.
• The tool will be evaluated for usability,
  accuracy, and consistency through limited use
  scenarios with NASA program managers.

4
Return on Investment - Status

• This project was funded July 20, 2001
• Evaluation of data sets is in progress
• Benchmarking for key factor target value ranges
  in progress
• Model integration and interface to existing
  programs in progress, Ask Pete, ARRT

5
RESEARCH APPROACH

• Phase 1
• Reduce the models we developed earlier to
  actionable guidelines for practice
• Phase 2
• Introduce these models, processes and support
  tools to a small group of carefully selected
  pilot projects
• Evaluate the results of applying the tools and
  methods
• Phase 3
• Use the feedback from step 3 to adapt the tools
  and methods for widespread dissemination, if
  warranted within the software project
  decision-making community at NASA.

6
HYPOTHESES/OBJECTIVE

•  
• The IVV valuation methodology will be
  iteratively refined based on feedback from NASA
  program managers and statistical evaluation of
  the methodology and results.
• Specific factors to be evaluated
•  
• Hypothesis 1 The cost relative to the potential
  benefits of IVV is inversely proportional to key
  organizational factors, such as the capability
  maturity of the development organization.
•   Hypothesis 2 The realization of potential IVV
  benefits is directly related to the development
  organizations acceptance of IVV.
•   Hypothesis 3 The cost/benefit ratio for IVV
  is directly related to the criticality of the
  application (and its individual subsystems).
•  

7
IVV YIELD

• Ultimately, the yield of an IVV program is based
  upon the difference between the net resource flow
  with IVV and without IVV.
• If the resources saved (e.g., reduced rework) or
  returns gained (e.g., improved customer
  satisfaction or increased safety) are greater
  than the resources consumed to save/gain these
  resources, we have a net benefit.
• Should the resources saved be less than the
  resources consumed, we have a net cost.

8
IVV Yield

• Cost of Quality
• Key components
• Cost of Poor Quality
• Key components

9
What we already know3 issues of empirical
studies...

• June 5-6, 1986 the 1st Workshop on Empirical
  Studies of Programmers, Washington, D.C.
• Need scientific rigorA Plan for Empirical
  Studies Victor Basili
• Need to look at real world variable valuesBy
  the Way, Did Anyone Study Real Programmers Bill
  Curtis
• Need to study PITLMeeting the Challenge of
  Programming in the Large (PITL) Elliot Soloway

10
Why is it Difficult to Apply Quantitative
Management Principles for Software Engineering?

• SE domain has a large number of key variables
  that have different degrees of significance
  depending on the environment
• SE domain has key variables that have extreme
  variance within the same environment (i.e.,
  programmer productivity 101)
• SE domain variables in combination may create a
  critical mass not present when variables are
  studied in isolation
• 1986 IEEE TSE, Basili, Selby and Hutchins,
  Surveyed software engineering empirical studies
  published to date. Cited 116 published studies.

11
Iterative IVV Methodology
Inputs
Outputs
Activities
Software IVV Plan
IVV Planning - Activities - Organization -
CARA - Schedules - Tools - WBS
Critical/High Risk Functions List
IVV Technical Reports
Software Problem Reports
Software Requirements Analysis
Software Interface Analysis
IVV Traceability Matrix
Iterative Per Software Release
Software Design Analysis
TRACEABILITY ANALYSIS
CHANGE IMPACT ANALYSIS
DELIVERABLES VALIDATION
TECHNICAL REVIEWS AND AUDITS
SPECIAL STUDIES
Software Code Analysis
Findings and Recommendations
Developer Test Analysis
IVV Metrics
Monthly Progress/Status Reports
Phase Dependent IVV Tasks
Phase Independent IVV Tasks
12
IVV Technologies - COQ
13
Empirical Research Summary

• Experimental Simulation
• Qualitative and quantitative results based on
  non-deterministic or hybrid simulation model
• Math Modeling quantitative results based on a
  deterministic model

• Mirrors a segment of the real world, control of
  variables is high, supports testing of causal
  hypothesis, results can be replicated, high
  internal validity and generalizability
• Captures real world context in which to isolate
  and control variables

• Researcher bias can be introduced through
  selection of variables, parameters and
  assumptions concerning the model. Modeling
  requires high degree of analytical skill, and
  interdisciplinary knowledge
• Results are not typically generalizable to other
  populations or environmental contexts, researcher
  bias is common,

14
Process Modeling and Simulation

• Managed, measured, productivity
• gains through
• process improvement
• data driven decision-making
• technological innovation
• Quantitative valuation of
• COQ vs COPQ

15
COQ versus COPQ
16
Process Simulation Models

• Experimental Simulation
• Qualitative and quantitative results based on
  non-deterministic or hybrid simulation model
• mirrors a segment of the real world
• control of variables is high
• supports testing of causal hypothesis
• results can be replicated
• high internal validity
• high external validity, generalizability

17
IVV Yield

• Organizational context factors for cost
• Key components

18
Independent Verification and Validation

• An organization independent from the developers
  study the artifacts of software production.
• This requires
• Technical independence. Members of the IVV team
  may not be personnel involved in the development
  of the software.
• .Managerial independence. The responsibility for
  IVV belongs to an organization outside the
  contractor and program organizations that develop
  the software.
• Financial independence. Control of the IVV
  budget is retained in an organization outside the
  contractor and program organization that develop
  the software.
• IVV is often perceived as testing the code after
  the development is completed NASA IVV is full
  life cycle activities

19
State of the Practice Process Maturity
Source SEI Web Site SEMA Report for March 2000
20
Measuring IVV Effectiveness
21
IndustryBenchmarking
Source for US Data Capers Jones (2000) Software
Assessments, Benchmarks, and Best
Practice, Addison-Wesley, p 339, System Software
Baseline.
22
IVV Yield

• System factors for cost and gain

23
Prior Empirical ROI Studies
ROI Independent VV Benefits
IVV applied early in the lifecycle has the
greatest ROI. Source Jet Propulsion Laboratory
TR.
24
IMPACT of Major Air Space Software Problems
DS-1
Galileo
Lewis
Poseidon
Pathfinder
Galileo
NEAR
93
96
97
98
99
Aggregate Cost
640 million
116.8 million
255 million
1.6 billion
Loss of Life
3
160
Loss of Data
99 NASA IVV presentation
25
Tracing Impacts to Causes Cause-Effect
Graphing

Mission Success at Reduced Cost

Safety Objective
Quality Objective
Reliability Objective
Cost Objective
Identify and Manage Risks
Identify and Eliminate Hazards
Defect Prevention
Defect Detection
IT Infrastructure, Web-based reporting, DSS, ARM,
PITS, RMS, Ask Pete, ARRT Communication Channels
Reporting
Process Improvement Avoid Rework Eliminate
Redundancy Efficient Resource Allocation
Skilled Workforce Domain Experts Engineers VV
Experts
PL Reuse Technologies Domain Engineering Knowledge
Maintenance VV Models and Methods
Information Analysis Information Management,
Product Certification
Skills training program
26
BSC Cause and Effect Graphing

Strategic and Financial Goals
Competitive Objective
Quality Objective
Reliability Objective
Cost Objective
Identify and Manage Risks
Optimize resource allocation utilization
Defect Prevention
Defect Detection
IT Infrastructure, Web-based reporting, COMPASS,
TIGERS, TeamPlay, Communication Channels
Reporting
Process Improvement Avoid Rework Eliminate
Redundancy Efficient Resource Allocation
Skilled Workforce Black Belts
Engineers Telecom Experts
SIX SIGMA Performance Excellence Knowledge
Maintenance Communications Models and Methods
Information Analysis Information Management
Product Certification
Skills training program - Motorola University
27
Filter Attributes
28
DTE Rule Based
29
NEURAL NETWORK
30
Intelligent update of rule structure
31
STATISTICAL ANALYSIS
32
BENEFITS

• The benefits of this proposed Center Initiative
  would be applicable to all NASA software
  development organizations for whom IVV is an
  option. The formalization of an objective
  decision-making process, along with enabling
  support tools would provide key capabilities to
  make rational budgetary decisions that impact
  safety and mission critical aspects of all NASA
  software systems. This is significant in enabling
  NASA to engage in effective administrative and
  managerial control based on objective, quantified
  information.
• The techniques proposed under this initiative
  will also provide NASA participants increased
  visibility into their process improvement
  efforts. The ISO-9001 certification requires that
  managers be able to document the benefits
  contributed to the organization by specific
  processes and process improvement effort 8. A
  formalized, well-defined decision-making process
  would therefore make a significant contribution
  to NASAs overall quality strategy.

33
MILESTONES

• StartJuly 20, 2001 3 mo
• IVV Process Description Product
  Characterization
• Based on prior CSIP results
• StartJuly 20, 2001 6 mo
• Information Analysis
• Data gathering for methodology
• StartJuly 20, 2001 6 mo
• Initial Prototype Demonstration(s) Iteration(s)
  Delivered
• GSFC IVV interface required

34
Nancy S. Eickelmann, PhDMotorola Labs1303 E.
Algonquin Rd.Annex-2Schaumburg, IL 60196Phone
[phone]Fax [phone]Nancy.Eickelman
n_at_motorola.com