CS445 CS645 ECE451 Software Requirements Specification and Analysis

1
CS445 / CS645 / ECE451Software Requirements
Specification and Analysis

• Lecture 16
• Cost estimation

2
Readings

• Required
• Heemstra (course pack)

3
Overview

• Software Cost Estimation (SCE)
• Why estimate?
• State of the practice
• Techniques
• Delphi model (use experts)
• Function Points
• LOC
• COCOMO

4
Software cost estimation

• Recall that one part of requirements engineering
  is feasibility analysis of the proposed
  requirements.
• Given an early description of the system, we want
  to determine as early as possible if a proposed
  system or requirement is technically and
  economically feasible.
• Economically feasibility means that the client is
  willing to pay what it will cost to develop the
  project, and the developer is willing to devote
  the resources to the project.
• Both of these questions have to be answered based
  on estimates
• estimates of the cost of the project, and
• estimates of the development effort.

5
Estimation

• One job of the requirements analyst is to
  estimate
• time to develop
• cost
• number of developer/months
• Related Questions
• Dominant cost factors?
• Risk factors?

6
Why estimate software cost and effort?

• To provide a basis for agreeing to a job
• You must make a business case for taking on a job
  or set thresholds for negotiating a price for
  performing the job.
• To make commitments that you can meet
• Cost overruns can cause customers to cancel
  projects.
• Alternatively, to avoid passing on all extra
  costs to customer, the project team may work
  without full financial compensation, swallowing
  the cost overrun.
• To help you track progress
• As Tom Demarco famously noted, You cannot manage
  what you cannot measure.
• If you dont know how long it will take to
  develop a system, you wont know if youre
  falling behind.

7
State of the practice

• Typically
• no estimates are done,
• software is delivered late,
• software is over budget,
• software doesnt meet its specification.
• When estimates are done, 100 errors is
  estimation are normal.
• Boehm (who is considered an optimist!)
• Today 1981, a software cost estimation model
  is doing well if it can estimate within 20 of
  the actual costs, 70 of the time, and on its own
  turf (that is, within the class of projects to
  which it has been calibrated).

8
FAA Advanced Automation System

• FAAs major modernization project, the Advanced
  Automation System (AAS), was originally estimated
  to cost 2.5 billion with a completion date of
  1996.
• The program, however, experienced numerous delays
  and cost overruns, which were blamed on both FAA
  and the primary contractor, IBM.
• In 1994, FAA cancelled part of the program and
  split the remaining systems into three phases,
  and in several cases, re-bid the contracts. . . .
  
• According to the General Accounting Office (GAO),
  almost 1.5 billion of the 2.6 billion spent on
  AAS was completely wasted.

9
FAA Advanced Automation System

• Robert Britcher, now a systems engineering
  professor at Johns Hopkins University who worked
  on AAS for IBM and wrote about it in his book,
  The Limits of Software People, Projects, and
  Perspective
• "For example, they wanted the system to have only
  3 seconds of downtime a year. But to get the
  data to prove that requirement had been met would
  have taken about 10 years."

10
We cant estimate very well

• Unfortunately, it is very difficult to estimate
  the cost and effort to build a project when you
  dont know very much about that project.
• Were not estimating repeatable, objective
  phenomena.
• The earlier the estimate, the less is known about
  the project.
• Unlike, say, building bridges, most of the time
  and effort in software development is in creating
  new design
• When we know how to do something well in
  software, we automate it and bump up the
  abstraction level one notch!
• Estimates can be biased by business and other
  pressures.
• The desire to take a job may prompt you to
  believe an unrealistic estimate.
• A goal to estimate within 10 of actual cost is
  unrealistic.
• Experience shows that by the time we know enough
  about a project to estimate its cost to within
  10 of its actual cost, the product is almost
  complete.

11
Software estimation and reality

• Estimation make sense only when you have
  something well understood to compare with!
• Often, software
• has changing requirements
• has changing platforms
• has changing developers
• has changing methodology and tools
• is trying new technology
• is one-time throw away
• And therefore has no data on past related
  projects for comparison.

12
Software estimation and reality

• There are many different kinds of software
• embedded systems, old fashioned compilers, COTS
  products, web apps, etc.
• Data about costs from one kind of system isnt
  necessarily applicable to another kind of
  software.
• Data from one company isnt necessarily
  applicable to another company, even in the same
  domain!
• Software engineering is a young enough discipline
  that we dont yet know how to estimate very well.

13
SCE techniques Heemstra

• Delphi model (use experts)
• Estimation based on the requirements
• Estimate function points from requirements
• Estimate LOC from function points
• Estimate resources needed from LOC (COCOMO)

14
Bad estimators

• Price-to-win
• Pricing to win means bidding as low as possible
  to beat the competition and win the contract.
• But then you will not get enough money to pay for
  the resources needed for development, and you
  will not show a profit.
• This can hardly be called an estimation
  technique.
• The idea is to price low enough to win the
  contract, but high enough to show a profit.
• In practice, there are lots of sneaky ways to get
  this to work (financially) but in such a case,
  its not really being used as an estimation
  technique.

15
Bad estimators

• Available capacity
• Estimate based on the resources available
• e.g., 5 people are available to work for 3
  months, so the project will take 15
  person-months.
• This is not an estimate based on the
  requirements!
• Could easily suffer from over- or
  under-estimation
• Parkinsons Law
• Works expands to fill the time allotted for it.

16
Delphi method

• Delphi methods are based on expert judgment.
• Each expert submits a secret prediction, using
  whatever process (s)he chooses.
• The average estimate is sent to the entire group
  of experts.
• Each expert revises his / her prediction
  privately.
• In some variations of the Delphi method, the
  experts discuss their rationales before new
  estimates are made, justifications are circulated
  anonymously, or no discussion is allowed.
• Repeat until no expert wants to revise his / her
  estimate, i.e., until a fixed point is reached.
• Recall the Oracle of Delphi of ancient Greece
  who would make mystical predictions of the future

17
Critical points about Delphi

• Its success depends on the experts abilities to
  determine which past projects are similar and in
  what ways.
• An experts experience cannot easily be
  transferred to junior developers.
• An scientist would complain that a methodology
  based on experts making gut-feeling predictions
  is hardly scientific!
• Thats correct, but it may be a reasonable
  solution (maybe the best / only one) in a given
  situation.
• Engineers / scientists prefer (repeatable)
  predictable, scientific models based on
  discernable properties of the problem /
  requirements.

18
KLOC

• LOC lines of code
• KLOC thousand LOC
• Problems
• How do you measure them?
• How do you count one line that has several
  statements?
• How do you count a statement that is over several
  lines?
• How do you count constructs, e.g., conditionals?
• One persons line may be anothers several lines
• But they are used as the unit of code size with
  care and with standards that answer these
  questions.

19
Estimating resources from requirements

• During requirements analysis, we do not have code
  with which to make estimates (!)
• So we want to be able to estimate cost based on
  what we know at requirements analysis time, i.e.,
  the requirements.
• We break the problem down into three parts
• estimating the number of function points from the
  requirements,
• estimating code size from function points, and
• estimating resources required (time, personnel,
  money) from code size.

20
Function points (FPs)

• Probably a better measure than KLOC ... but
  caveat emptor
• Makes some real sense in industrial settings,
  producing well-understood backend types of
  systems
• Lots of tools and methods exist that use FPs.

21
Estimating FPs from reqs

• Function points are used to predict the size of a
  system ( functions)
• The idea is to predict the complexity of the
  system in terms of the various functions to be
  written, without being as specific as lines of
  code (which is often highly prog lang dependent).
• Thus, we are counting the various types of
  functions, and weighting them according to their
  types and complexities.
• The basic model is
• FP a1P1 a2P2 ... aNPN
• ai is weighted factor for function type i
• Pi is of instances of ith function type

22
FP weightings
23
Estimating code size from FPs

• There are tables that list for each programming
  language, the number of statements in it that are
  required to implement one function point.
• These tables are calibrated for each shop, for
  each domain, etc.

24
Estimating code size from FPs

• It was observed a long time ago that a
  programmers productivity in terms of debugged,
  documented, lines of code per day is pretty
  constant and is independent of the language (s)he
  is using.
• This is why the higher the level the language,
  the more productive is the programmer (s)he does
  much more per line (s)he writes.
• Frameworks, libraries, hard interfaces are Good
  Things.
• Reusable infrastructure means fewer bugs are
  likely
• You dont have to re-invent the wheel yet again.

25
Estimating resources from LOC

• There are several methods.
• The best known, and most validated, is COCOMO.

26
COCOMO

• COnstructive COst MOdel
• Barry Boehms 1981 book Software Engineering
  Economics is revered by many,
• Others remains somewhat skeptical
• COCOMO is a set of models for performing software
  estimation, based on Boehms (and others)
  experiences building software systems for the US
  DoD.
• Three levels basic, intermediate, and advanced.
• The difference is in how detailed you want to be.

27
Basic COCOMO model

• Input
• KLOC estimated size of system
• Outputs
• E effort in person-months
• D development time in months
• Types of projects
• organic smaller project, requirements not
  rigid, experienced developers
• semi-detached intermediate in size
  complexity, mix of rigid and flexible
  requirements
• embedded tight constraints/requirements on
  hardware, software, environment

28
Basic COCOMO model

• E a ? KLOCb ? X
• D c ? Ed
• a, b, c, d are empirically observed coefficients
• X project attribute multipliers

29
Basic COCOMO model

• KLOC is the estimated size of code, adjusted to
  account for any reuse of design or code.
• a, b, c, and d are empirically observed
  coefficients.
• Both are adjusted at each shop for each domain
  based on historical data.
• The X stands for multipliers for 15 project
  attributes that have been observed to be critical
  (cost drivers).

30
Project attributes

• Product attributes
• Required reliability, complexity database size
• Resource attributes
• Execution time, memory constraints, hardware
  volatility, tight response time
• Personnel attributes
• Overall experience / quality of of developers
  analysts,
• Experience with particural product, hardware,
  prog lang, tools
• Project attributes
• Use of modern techniques, prog lang (i.e., not
  COBOL / FORTRAN / PL/1), and tools
• Schedule constraints

31
Estimating time

• Recall the formula
• T c x E d
• Effort is measured in person-months time is
  measured in months.
• Why is this formula not linear?

32
The mythical person-month

• Fred Brooks examined this in his famous book
• The Mythical Man-Month
• The problem with person- months as unit of
  measure is that it implies this graph is accurate
• but it isnt

of developers
Time
33
Mythical person-month

• But it does not work for software development!
• Certain tasks require a certain minimum amount of
  time and throwing more personnel at the tasks
  does not reduce the time needed.
• It works for painting a fence.
• Why? Because little communication, planning, and
  co-ordination is required.
• It does not work for software development teams.
  Why?

34
Two eternal quotes by Fred Brooks

• The bearing of a child takes nine months, no
  matter how many women are involved.
• i.e., development time isnt fungible
• Adding people to a late project makes it later.
• Brooks law
• because you have to stop what youre doing,
  bring the newbies up to speed, deal with their
  questions and mistakes this takes a lot more
  time and effort than you might imagine.

35
A few quick notes on Fred Brooks

• He was part of the development team for IBMs
  OS-360 in the 1960s.
• Much of The Mythical Man Month comes from those
  experiences
• MMM is a joy to read. Brooks is a great writer,
  and the book is accessible to you.
• MMM was first published in 1975 its still very
  widely read today
• How many other computer books published in 1975
  still exist in hard copy?
• In 1995, a 20th anniversary edition of MMM was
  published, adding his famous 1986 paper 1986 No
  Silver Bullet Essence and Accident in Software
  Engineering
• Brooks has been a prof at UNC for many years
• In 2000, Brooks won the Turing Award.

36
Should we estimate?

• It sounds like voodoo mathematics, doesnt it?
• It doesnt seem repeatable in the grand
  scientific tradition
• It sometimes relies on the opinions of experts
  (magical wisewomen)
• Constants are peculiar to your company / domain /
  team, they arent universal
• Some quotes for discussion
• You cannot control what you cannot measure.
  Tom DeMarco
• Measuring is better than not measuring.

37
Should we estimate?

• While it may not be great science, ya gotta do
  it.
• You need this information
• to plan the project,
• to negotiate the cost with the client,
• to determine how many developers to hire or to
  assign to this project, and
• to know how long theyll be dedicated to this
  project and not to others.
• For these reasons, poor estimates are usually
  better than no estimates.

38
Should we estimate?

• Your ability improves with experience!
• Steve McConnells example
• Dont get too caught up in an estimate. Its
  wrong. Youll get better, but youll never master
  the problem.
• Accept that you will never win this battle, but
  that you may still win the war.

39
Why do we underestimate?

• Often, we underestimate by an order of magnitude.
• Fred Brooks observes
• Everybody thinks program when he or she should be
  thinking software system product.
• The program what you write for yourself (and
  thus what you know)
• The system program that interfaces with other
  programs, directly or indirectly, costs 3 times
  as much as central program (more stuff to write)
• The product program written for others, that
  must therefore be robust, costs 3 times as much
  as central program
• The software system product program that is
  system and product, costs 9 times as much as
  central program

40
In summary

• Estimates improve when we
• Keep data on past projects.
• Focus on particular types of systems, development
  methods, group dynamics.
• Practice estimating!

41
Caveats

• But
• Models have to be calibrated to an organization
• Accuracy is perturbed by local factors, such as
  expertise, process, product type, definition of
  LOC.
• Be careful of model parameters based on old
  projects/technology
• Weights and coefficients are based on empirical
  studies of past projects using old technology,
  and may be completely unlike new projects.

42
Caveats

• Predictions can become self-fulfilling
• If estimates are used to generate the project
  plan, which is used by managers to manage the
  project, the project ends up having to conform to
  the estimate!
• Parkinsons Law!
• Some people will be better at estimating than
  others.
• Cost estimation is not a science. Its an art,
  based on intuition and experience.
• Be wary of any method or tool vendor that claims
  to predict cost or effort to unrealistic
  precision, i.e., more than one significant digit!.

43
Some closing quotes

• There is no point in using exact methods where
  there is no clarity in the concepts and issues to
  which they are to be applied.
• von Neumann and Morganstern,
• Theory of Games
• An estimate is the most optimistic prediction
  that has a non-zero probability of coming true.
  Accepting this definition leads irrevocably
  toward a method called whats-the-earliest-date-by
  -which-you-cant-prove-you-wont-be-finished
  estimating.
• Tom DeMarco

44
CS445 / CS645 / ECE451Software Requirements
Specification and Analysis

• Lecture 16
• Cost estimation