Software Engineering Process I Measurement

1
Software Engineering Process IMeasurement

• INFO 636
• Glenn Booker

2
Measurement

• Measurements are critical, because the act of
  measuring something focuses peoples attention
  on it
• We have already looked at several basic measures
  in the PSP
• Now expand to a more general process for
  choosing measurements and how to present them

3
Measurement

• The approach presented here is a variation on the
  GQM approach cited in the text, called GQ(I)M
• The (I) adds Indicators, which are the means
  used to present the measurements graphically
• This approach is also used in INFO 630

4
Why Care About Measurement?

• To quote Albert Einstein, "Not everything that
  counts can be counted and not everything that can
  be counted counts."
• We are seeking to identify things
• 1) which can be counted, and
• 2) which count toward achieving our goals

5
Reasons for Measurement

• Measurements are required by all major process
  models (CMM, ISO 9000, etc.)
• To Characterize or understand the current status
  of activities or products
• To compare that understanding to our objectives,
  and Evaluate whether the current status is good
  or bad

6
Reasons for Measurement

• To Predict future performance, based on past
  trends
• To form the basis for measuring Improvement
• You wont know if you improved, if you dont
  know where you started!

7
Where do we get Information?

• Metrics are often used to support decision making
  (the Evaluate step on a previous slide)
• Decisions should be based on quantified
  information
• To get that information, we calculate measures
  from raw data called data elements
• The data elements each come from a data source

8
How do we Choose what to Measure?

• A commonly used method for selecting measurements
  is called GQ(I)M for Goal, Question, Indicator,
  and Measurement
• It is based on GQM work by Victor Basili (first
  reported circa 1988-89)
• GQ(I)M was published by Robert Park et al

9
How do we Choose what to Measure?

• The GQ(I)M method uses ten steps to describe
  measurements systematically
• The steps dont have to be followed in the order
  presented their main purpose is to help ensure
  that measurements have been fully thought out
• You can start at the top, or the bottom, or in
  the middle

10
1. Identify Business Goals

• Business goals are the big, vague, lofty desired
  accomplishments for an organization
• Think of what youd find bragged about in a
  companys annual report
• Reduce cycle time
• Improve customer satisfaction

11
1. Identify Business Goals

• Develop detailed process history
• Respond to changing business environment
• Reduce overhead
• Improve competitive position
• Increase market share
• Improve product quality

12
2. Identify Desired Knowledge

• Break down each goal into products, resources,
  and activities needed to meet those goals
• Think of questions like
• What activities do I manage or execute?
• What do I want to achieve or improve?
• To do this, I will need to

13
3. Identify Subgoals

• Set subgoals (objectives) for each of the areas
  you manage
• What do you want to know about the results of
  step 2? What kind of information is important to
  you?
• How big/fast/expensive/complex/much time will a
  process/product/tool/resource take?

14
4. Identify Entities and Attributes

• Formulate questions to identify entities
  (documents, work products) created by your
  process, and the attributes of them you are
  interested in (size, quality, duration, cost)
• Dont get too detailed at this point - just
  identify the type of information desired (what
  do you want to understand), and the subject of
  that information (what about it do you want to
  know)

15
4. Identify Entities and Attributes

• Entities that you want to measure can come from
  any of
• A process inputs
• The activities within a process
• The process outputs
• Then decide what characteristics of each entity
  are of interest

16
5. Define Measurement Goals

• First choose the type of measurement goals
• Active goals reduce or improve something
• Passive goals identify, assess, understand
• Lower maturity organizations start with passive
  goals, then work on active goals after some
  history is known

17
5. Define Measurement Goals

• Form structured statements of the measurement
  goals for each attribute
• This step defines a metric in the form of a
  sentence

18
5. Define Measurement Goals

• The general format or syntax is
• ltverbgt ltmeasuregt ltqualifier(s)gt objective
• where
• ltverbgt is active or passive, as chosen earlier
• ltmeasuregt is the actual measure to be collected
• ltqualifiersgt indicate the scope or time frame of
  the measurements (a.k.a. independent variables),
• objective is only given for active measurement
  goals what value is desired?

19
5. Define Measurement Goals

• Passive measurement goal examples
• Measure the number of requirements which changed
  each month.
• Identify the voluntary turnover rate per month
  for programmers and software engineers.
• Active measurement goal examples
• Reduce the defect rate of developed code over
  time to under 20 defects/KLOC
• Improve the percent of satisfied customers after
  30 days of product use to 95 or more

20
5. Define Measurement Goals

• DO NOT report traits of individual people (for
  fear of judgment)
• Unless thats part of their job description
  (e.g. sales people)
• Also need to balance how often measurements are
  made
• More frequent measurement gives finer control,
  but excessive measurement wastes time and slows
  the process being measured

21
6. Quantify Questions and Indicators

• Pose questions to address your measurement goals
  (quantifiable ones, if addressing active goals)
• Active Can we resolve customer emergencies, on
  average, in under 24 hours?
• Passive How many requirements do we have at the
  end of the Requirements Definition phase?

22
6. Quantify Questions and Indicators

• Sometimes the search for a meaningful metric
  starts with a question, and that leads to filling
  out the GQ(I)M from the middle
• Identify indicators to show the results
  effectively (see later this lecture), such as
• Pie charts
• Bar graphs
• Scatter plots, etc.

23
7. Identify Data Elements

• Identify the data elements needed to prepare
  (calculate) each indicator
• e.g. defect rate by module each month needs a
  list of defects found, with the module each came
  from, for the last month, and the size of each
  module in kSLOC

24
7. Identify Data Elements

• Determine the source for each data element from
  where do you get it?
• E.g. defect data will come from our change
  request database
• Size data will be generated by the development
  environment

25
8. Define Measures

• Define exactly what you mean by each measure
• Use a checklist if needed to show what is and
  isnt included in its definition (such as we used
  for definition of LOC)
• Cite the source if an unusual measure is used or
  if you made it up, explain why
• New measures are allowed!

26
8. Define Measures

• Include any rules, assumptions, constraints, and
  environment which are part of the measures
  definition
• E.g. Defect rate ( of defects whose origin was
  in each module)/( of kSLOC in that module)

27
9. Identify Measurement Implementation

• Deciding how to implement a measurement program
  is generally done in three steps
• Analyze what measures are currently collected (if
  anything) and how theyre used
• Diagnose how well the current measurements meet
  your goals find whats missing?

28
9. Identify Measurement Implementation

• Act on implementing new measurements, possibly in
  a phased approach based on project or
  organizational priorities
• Implementing measurements is a cultural change,
  so like any other change, its best done in small
  steps
• Add a few measurements now, wait six months, add
  a few more, etc.

29
10. Prepare Measurement Plan

• Take all of the aforementioned information and
  create a complete plan to identify and implement
  measurement for your organization or project
• This is generally called a Metrics Plan or a
  Measurement Plan

30
Summary of GQ(I)M Steps

• Goal
• Subgoal (objective why collect this metric)
• Question (answered by this metric)
• Indicator (how display metric)
• Measurement (the actual metric and its
  definition)
• Data Elements (used to calculate metrics)
• Source (of each data element)

31
Indicators

• Indicators are the means used to present
  measures, such as charts, graphs, etc.
• Consider how your data will be presented - in
  color or B/W, live or printed
• Will your audience see pristine originals, or
  will it be copied and faxed a zillion times?

32
Indicators

• To choose the right indicator, consider
• The Amount of Data to be presented for each
  interval (e.g. one measure at a time or five
  different survey responses at once)
• The Number of Intervals to be shown, such as time
  units, modules of code, etc.

33
Indicators

• Different indicators are better at different
  Amount or Number characteristics

34
Indicators

• For most graphs
• The X-axis of a graph (the horizontal line) is
  the independent variable often a ltqualifiergt,
  such as time, severity, etc.
• If the X axis is Time, it should show how often
  measurements are made (weekly, monthly, etc).

35
Indicators

• The Y-axis of a graph (the vertical line) is the
  dependent variable the thing you are measuring
  (the Measure).

36
Pie Chart

• The lowly pie chart is good for presenting a
  small amount of data
• of customers satisfied and not satisfied
• of defects by severity at this moment
• Amount of Data Shows a few data points (2-10)
• Number of Intervals One - it generally shows
  only one moment in time

37
Sample Pie Chart
38
Ishikawas Basic Tools

• Developed circa 1989 for manufacturing
  production
• Used widely in quality control
• Focuses on project level concerns is this batch
  good enough to accept?
• Not very useful for research has little
  theoretical basis

39
1. Check sheet

• Used to gather data easily, consistently, and in
  a standard format
• A check sheet used to help the quality of a
  process or product is a checklist
• Helps to define key parts of a process
• Examples include code inspection checklist,
  detailed test procedures

40
2. Pareto Diagram

• Used to identify problem areas - where to fix
  first what are the biggest fires to put out
• Defects tend to cluster in buggy portions of code
• Use Pareto to plot a column graph of the defect
  rate by the module it came from
• Could add a line graph for the cumulative of
  defects above it (not shown)
• Pareto diagram must list X axis categories in
  descending order of value

41
2. Sample Pareto Diagram
42
3. Histogram

• A bar chart is used to break down data by an
  ordered category (e.g. defect severity,
  satisfaction rating)
• Can choose to put bars next to each other
  (clustered), or stack them.
• Stack when they add to a constant (e.g. 100), or
  when the total is also a useful measure (total
  number of defects)

43
3. Histogram

• Can show limited time spans, e.g. a few time
  intervals

Simple bar graph ?
44
3. Histogram
? Stacked
Clustered ?
45
3. Histogram

• A true histogram can group ranges of values in
  the X axis, and show counts or average Y values
  for each group
• E.g. average salary (Y axis) for employees aged
  18-29, 30-39, 40-49, etc. (X axis groups)
• Good for hiding individual values, and looking at
  larger trends in the data

46
4. Run Charts

• Plots some measure versus time using a line graph
• Often compare values to a desired or target
  value, especially at higher process maturity
  levels (CMM Level 3 and up)
• Special case The S curve plots (cumulative
  completion of something) versus time

47
4. Sample Run Chart
48
5. Scatter Diagram

• Plot two measures against each other to see if
  theres a correlation between them
• E.g. Defect rate per module versus module
  complexity, or productivity versus experience
• When we say plot Blah versus Ick, usually Blah
  is the Y axis, and Ick is the X axis

49
5. Scatter Diagram
50
5. Scatter Diagram

• Once theres enough data, can add curve fitted
  lines, and /- variances
• The scatter diagram is the basis for regression
  analysis
• Curve fitting to the data, to help define a
  statistically significant connection between the
  two measures

51
6. Control Chart

• Statistical Process Control tool
• Rare in software development since
• Specifications poorly defined
• Each project takes a long time
• Too many uncontrolled process variances
• Process-quality relationship not well defined

52
6. Control Chart

• Too many processes used
• Rapidly changing technology

53
6. Control Chart

• Pseudo-control charts include
• The u chart for defect rates by component, BMI,
  etc. versus time
• The p chart for percentages, such as inspection
  effectiveness or customer satisfaction rating
• There are many other varieties of control charts

54
6. Control Chart

• Usual control limits are /- 3 sigma from the
  mean, which define the upper and lower control
  limits (UCL and LCL)
• Can add a warning limit at /- 2 sigma
• Control Chart tend to be used in very mature (CMM
  level 4 or 5), long term projects

55
Fishbone Diagram

• Technically, the fishbone diagram is Ishikawas
  seventh tool, but I doubt anyone will use it in
  conjunction with the PSP

56
Measurement Basics

• Given all that on GQ(I)M and indicators, the most
  essential types of measurements are still pretty
  basic
• Process how long does it take?, how much does
  it cost?, are we starting it on time?, etc.
• Tools (rare) how well are our development tools
  being used?

57
Measurement Basics

• Resources how much does it cost to keep our
  people trained? How much does it cost to find new
  personnel?
• Product how big is our product (LOC)? What is
  its quality? How many defects have been found?
  How happy is the customer with it?

58
Gathering Data

• Critical from the PSP perspective is defining our
  processes to gather data consistently and
  efficiently
• How do you record time spent?
• How do you collect measurements from various
  people and merge them? (A critical concern for
  INFO 637)

59
Gathering Data

• Our forms have been designed to help collect key
  aspects of the processes
• Product size
• Time spent in each part of the process
• Defect injection and removal phases
• Defect type and fix time
• Bad fixes (the infamous Fix Error field)
• Productivity (LOC/hour)

60
Gathering Data

• The result of this is a large body of data that
  can be collected across many projects
• While the data gathering and analysis takes time,
  it also produces real answers about how quickly
  and well you perform various tasks
• This is your personal process baseline