Overview of GAOs Implementation of Earned Value Management

1
Overview of GAOs Implementation of Earned Value
Management

• Karen Richey
• Jennifer Echard
• GAO Cost Analysts
• Applied Research Methods Team
• April 28, 2005

2
Table of Contents

• Using EVM as an Audit Tool at GAO
• Summary of GAO Report Findings using EVM analysis
• FAA STARS (GAO-03-343)
• Missile Defense (GAO-04-409)
• Airborne Laser (GAO-04-643R)
• NASA Cost Estimating Process (GAO-04-642)
• Polar-Orbiting Environment Satellite
  (GAO-04-1054)
• Navy Shipbuilding (GAO-05-183)
• Next steps for GAO and EVM

3
Using EVM as an Audit Tool

• GAO began using EVM analysis in 2002 for selected
  audits
• Since that time GAO has issued 6 reports using
  EVM analysis as a basis for findings
• Programs are far ranging covering major Automated
  Information Systems, Shipbuilding, Missile
  Defense and NASA
• Results vary among programs--Examples include
• Realism of the Performance Measurement Baseline
  (PMB)
• Integrated Baseline Reviews (IBRs) data
  validation
• Number of rebaselines
• Optimistic Estimates at Completion (EACs)
• Frequency of reporting

4
Summary of GAO Report Findings using EVM
analysis FAA STARS (GAO-03-343)

• Background On November 17, 2002, FAA began using
  the Standard Terminal Automation Replacement
  System (STARS) to control air traffic at the
  Philadelphia air route traffic control center.
• STARS is a joint procurement with DOD, which is
  purchasing up to 199 units for its facilities.
• STARS will replace controller workstations with
  new color displays, processors, and computer
  software at FAA and DOD terminal air traffic
  control facilities.
• STARS provides an automation platform that will
  enable efforts to optimally configure the
  terminal airspace around the country, exchange
  digital information between pilots and
  controllers, as well as introduce new position
  and surveillance capabilities for pilots.

5
FAA STARS (GAO-03-343) Website
www.gao.gov/new.items/d03343.pdf

• FAA was not using CPR data to manage its contract
  because the data did not reflect current effort
  and had not been validated by an IBR after
  changes
• Major modifications changed the Performance
  Measurement Baseline (PMB) curve
• PMB never updated when STARS implementations fell
  from 188 to only 74 systems
• PMB did not reflect 179M of new work authorized
  since contract award

6
FAA STARS (GAO-03-343)

• Conclusions FAA was unable to effectively
  manage the contract
• CPR showed program to be on schedule and within 1
  percent of budgeted costs
• In reality, FAA was contracting to modernize
  fewer than half the systems at more than twice
  the original cost
• Recommendations GAO recommended that FAA
• Maintain current baselines,
• Negotiate all authorized work within 3 months,
  and
• Conduct IBRs after any major contract
  modifications within 6 months
• Result FAA concurred with recommendations.

7
Summary of GAO Report Findings using EVM
analysis Missile Defense (GAO-04-409)

• Background GAO has an annual congressional
  mandate to assess the extent to which MDA
  achieves program goals.
• In 2002 two key events transformed DODs approach
  to missile defense
• The Secretary of Defense consolidated existing
  missile defense elements into a single
  acquisition program and placed them under the
  management of the Missile Defense Agency (MDA)
• The President directed MDA to begin fielding an
  initial configuration, or block, of missile
  defense capabilities in 2004.

1 Airborne Laser 2 Ground Based
Midcourse Defense
8
Missile Defense (GAO-04-409) (cont) Website
www.gao.gov/new.items/d04409.pdf

• Prime contractors for 4 system elements completed
  work at or near budgeted costs, but contractors
  for two system elements overran budgeted costs by
  a total of 380 million during fiscal year 2003
• ABL1 underestimated the complexity of integrating
  subcomponents into a flight-worthy
  configuration causing major cost overruns
• Initial PMB was highly optimistic given the
  cutting edge technology
• Unrealistic schedule contributed to both cost and
  schedule problems
• Developmental and delivery problems with the GMD2
  interceptor were the leading contributors to cost
  overruns and schedule slips
• CPRs for Aegis system reflected only the
  interceptor portion and not the entire Aegis
  element
• MDA had no insight regarding the Aegis
  interceptor divert system which was experiencing
  development problems at the time

1 Airborne Laser 2 Ground Based
Midcourse Defense
9
Missile Defense (GAO-04-409) (cont)

• Conclusions GAO found that MDAs program goals
  do not serve as a reliable and complete baseline
  for accountability purposes and investment
  decision making because
• they can vary year to year,
• do not include life cycle costs,
• are based on assumptions about performance not
  explicitly stated,
• does not explain some critical assumptions such
  as an enemys type and number of
  decoysunderlying its performance goals
• As a result, decision makers in DOD and Congress
  do not have a full understanding of the overall
  cost of developing and fielding the Ballistic
  Missile Defense System and what the systems true
  capabilities will be.

1 Airborne Laser 2 Ground Based
Midcourse Defense
10
Missile Defense (GAO-04-409) (cont)

• Recommendations GAO recommended that MDA
• Establish cost, schedule, and performance
  measurement baselines (including full life cycle
  costs) for each system being fielded
• Result DoD concurred with baseline
  recommendations.

1 Airborne Laser 2 Ground Based
Midcourse Defense
11
Summary of GAO Report Findings using EVM
analysis Airborne Laser (GAO-04-643R)

• Background From 1996 to 2001, the ABL program
  was an Air Force major defense acquisition
  program.
• In October 2001, the DOD transferred
  responsibility for the program to the Ballistic
  Missile Defense Organization (now MDA) where ABL
  became one element of the Ballistic Missile
  Defense System (BMDS).
• ABL's mission is to destroy enemy ballistic
  missiles in the boost phase as part of the
  layered defense strategy.
• ABL is being developed incrementally in a series
  of 2-year blocks.

1 Airborne Laser
12
Airborne Laser (GAO-04-643R) (cont) Website
www.gao.gov/new.items/d04643.pdf

• Prime contractors cost for developing ABL1 had
  nearly doubled from Air Force original estimate
  and additional cost growth expected
• Costs increased from 1B in 1996 to 2B in 2003
• Unrealistic PMB at contract award prompted
  numerous rebaselines
• GAO forecasted an overrun between 431M - 943M
  for first ABL demo
• Cumulative Cost Variance at that time was -242M
  with a schedule variance of -28M
• Contractor reported no variance at completion
• The cost growth occurred because the program did
  not adequately anticipate the complexities
  involved in developing the system.
• MDA continues to face significant challenges in
  developing the ABL's revolutionary technologies
  and in achieving cost and schedule stability.

1 Airborne Laser
13
ABL Cost and Schedule Variance in FY 2003 ( in
Millions)
14
ABL Actual Performance Vs. Projected Performance
15
Airborne Laser (GAO-04-643R) (cont)

• Conclusions Program did not adequately plan for
  or anticipate complexities involved in developing
  ABLs revolutionary technologies
• Rapid prototyping forced program to integrate
  components before they were fully tested causing
  unexpected rework
• Fabrication of ABLs unique components and
  developing its complex software proved more
  costly and time consuming than anticipated
• Bow wave of uncompleted work from prior years
  competed for current resources despite an
  increase in staff and additional work shifts
• Recommendations GAO recommended that MDA
• complete an uncertainty analysis on the
  contractors new cost estimate to quantify level
  of confidence
• Result DOD concurred.

1 Airborne Laser
16
Summary of GAO Report Findings using EVM
analysis NASA Cost Estimating Process
(GAO-04-642)

• Background NASA's programs encompass a broad
  range of complex and technical activities--from
  investigating the composition and resources of
  Mars to providing satellite and aircraft
  observations of Earth for scientific and weather
  forecasting.
• Our review focused on 27 of 68 NASA programs in
  the development phase as of April 2003 or that
  completed development in fiscal year 2001 or
  2002.
• To assess NASA's cost-estimating processes and
  methodologies, we conducted a more in-depth
  review of 10 of the 27 programs
• We chose programs using the following critieria
• The highest development cost estimate within five
  of NASA's seven Enterprises.

17
NASA Cost Estimating Process (GAO-04-642) (cont)
Website www.gao.gov/new.items/d04642.pdf

• We compared cost estimating documentation from 10
  programs against SEI cost estimate criteria to
  determine whether NASA followed good cost
  estimating practices.
• We found that NASA cost estimates for 8 out of
  the 10 programs we reviewed in detail increased
  over time
• NASA attributed cost growth to technical problems
  and funding shortages
• Variability in cost estimates indicated that
  programs lacked sufficient knowledge needed to
• Establish priorities,
• Quantify risks,
• Make informed decisions, and
• Thus predict costs

18
NASA Cost Estimating Process (GAO-04-642)
(cont)Overview of 10 Selected Programs
19
NASA Cost Estimating Process (GAO-04-642) (cont)

• Conclusion NASAs basic cost estimating
  processes lacked discipline needed to ensure that
  program estimates are reasonable
• None of the 10 programs GAO reviewed met all of
  GAOs cost estimating criteria
• Only 3 of the 10 programs provided a breakdown of
  the work to be performed
• Only 2 programs used EVM for measuring cost and
  performance to identify risks
• The other 8 programs compared planned versus
  actual obligations and expenditures
• NASA had no data to determine ahead of time
  whether these 8 programs were at risk of cost and
  schedule overruns
• Recommendations GAO recommended that NASA
• develop rebaselining guidance to ensure
  consistency among programs,
• implement EVM as a tool for bringing cost to the
  forefront,
• and ensure EVM policies are enforced
• Result NASA concurred.

20
Summary of GAO Report Findings using EVM
analysis Polar-Orbiting Environment Satellite
(GAO-04-1054)

• Background The United States operates
  polar-orbiting meteorological satellite systems
  to
• Provide graphical weather images and specialized
  weather products that are inputs into weather
  prediction.
• These images, products, and models are all used
  by weather forecasters, the military, and the
  public for both weather forecasting and climate
  monitoring.
• NPOESS1 is being developed to combine two current
  satellite systems into a single state-of-the-art
  environment monitoring satellite system.
• NPOESS is a tri-agency endeavor involving DOD,
  NASA, and the National Organic and Atmospheric
  Administration

1 National Polar-Orbiting Environmental
Satellites
21
Polar-Orbiting Environment Satellite
(GAO-04-1054) (cont) Website www.gao.gov/new.ite
ms/d041054.pdf

• Recently the NPOESS program office developed a
  new cost and schedule baseline.
• Congress asked us to identify any cost or
  schedule changes as a result of the revised
  baseline and determine what factors contributed
  to these changes.
• We found that the program office increased
  NPOESS1 cost estimate by 1.2B and delayed the
  availability of the first NPOESS satellite by 20
  months.
• Funding cuts stretched out NPOESS program causing
  an increase in cost
• Other cost increases were due to problems with
  sensor development and additional program office
  costs to mitigate risks after program rebaseline.
• Lack of schedule slack increased risk for
  managing development issues.

1 National Polar-Orbiting Environmental
Satellites
22
Polar-Orbiting Environment Satellite
(GAO-04-1054) (cont)

• Conclusion NPOESS could experience further cost
  and schedule increases
• Contractors continued schedule slippage in both
  cost and schedule targets indicated the NPOESS
  contract would likely overrun by 500M
• Program risks with developing critical sensors
  could also add costs
• Since contract rebaseline, cost overruns have
  more than doubled and the schedule was slipping
  again
• We estimated an overrun between 372M and 891M
  at completion compared to the contractor which
  estimated an overrun of only 130M
• Cumulative cost variance at that time was 55M at
  only 20 percent complete
• Recommendations We concluded that continued
  oversight is more critical than ever.
  Accordingly, GAO plans to continue the review of
  this program.
• Result DOD, NASA, and NOAA officials generally
  agreed with the report

1 National Polar-Orbiting Environmental
Satellites
23
Summary of GAO Report Findings using EVM
analysis Navy Shipbuilding (GAO-05-183)

• Background The U.S. Navy invests billions of
  dollars each year to maintain technological
  superiority of its warships.
• Our study reviewed the top four ship classes
  involved in new ship construction to determine
  the reasons behind significant cost overruns.

24
Navy Shipbuilding (GAO-05-183) (cont)website
www.gao.gov/new.items/d05183.pdf

• For the 8 ships assessed (2 ships in 4 classes),
  we found that Congress had appropriated 2.1
  billion in additional funding to cover in cost
  increases through 2004
• GAO EVM analysis indicated cost increases of
  3.1B or more for all 8 ships could be expected
• Cost increases for CVN 77 and LPD 17 were the
  most pronounced
• Virginia Class submarine was expected to also
  require additional costs
• Increases in labor hour and material costs
  together account for 77 percent of the cost
  growth on the eight ships. Shipbuilders cited the
  following as key causes
• Design modifications
• Additional and more costly materials
• Changes in employee pay and benefits

25
Navy Shipbuilding (GAO-05-183) (cont)

• Conclusion Navy practices for estimating costs,
  contracting, and budgeting for ships resulted in
  unrealistic funding of programs
• PMB costs for CVN, SSN, and LPD 17 were all
  unrealistic increasing the likelihood of
  increased costs
• Navy does not account for probability of cost
  growth when estimating costs
• No uncertainty analysis
• No management reserve
• 3 out of 4 ship classes received CPRs on a
  quarterly basis
• LPD 17 switched to monthly CPRs after
  unprecedented cost growth
• CVN 77 is switching to monthly CPRs within the
  year
• SSN 774 class is staying with quarterly CPRs
  despite expected cost growth
• Recommendation GAO recommended the Navy
• perform uncertainty analysis,
• establish realistic budgets for measuring
  performance,
• and include more timely and complete reporting on
  program costs.
• Results Navy partially concurred.

26
Next steps for GAO and EVM

• Select members of GAO have been extensively
  trained to use EVM for audits
• These members delivered in-house training to
  various GAO teams to get the word out about the
  benefits of using EVM
• Ability to quantify historical cost and schedule
  performance
• Provide a means for forecasting future costs and
  schedules
• Identify areas of poor performance and technical
  concern
• Ability to forecast potential problem areas and
  report to Congress
• GAO has ordered Winsight software and will be
  trained in the next few months
• Next area of focus includes scheduling techniques
  and tools, with an emphasis on the critical path
  method