SE 468 Software MeasurementProject Estimation

1
SE 468 Software Measurement/Project Estimation

• Dennis Mumaugh, Instructor
• dmumaugh_at_cdm.depaul.edu
• Office Loop, Room CDM 430, X26770
• Office Hours Monday, 400-530

2
Administrivia

• Comments and feedback
• Midterm Examination this coming week October
  16-22
• Will use the Blackboard system https//oll.depaul
  .edu/
• Using Blackboard http//condor.depaul.edu/dmumaug
  h/common/Quizzes in Blackboard.html See note
  below.
• Details
• Things to avoid

3
Term Papers

• Function Point papers Remember we already have
  had a lot of discussion. Look at Object Points as
  a alternative. Also Use Case points. Look at the
  criticisms of FP analysis.
• COCOMO Remember we already have had a lot of
  discussion on COCOMO, fill in the missing parts,
  do not tell me what we have already covered. Look
  at the analysis of the base data and how the
  constants are derived. How does COCOMO II change
  things? Does COCOMO work? How reliable is it?

4
Assignment 3

• Project Estimation
• Using homework data spreadsheet, what is the
  total person months required to complete this
  project?
• Assume every artifact is written/executed by one
  author
• Assume every review is carried out by 5
  headcounts including the author
• Calculate the total number of defects found and
  defect removal rate per KLOC.
• Due October 26, 2009

5
SE 468 Class 5

• Topics
• Quality
• Metrics
• Software Quality (Redux)
• Software Quality Metrics 
• Basic Quality Tools
• Reading
• Kan chapters 1, 4-5
• Articles on the Class Page

6
Thought(s) for the Day

• If you've found 3 bugs in a program, best
  estimate is that there are 3 more.
• 60 of product cost comes after initial shipment.
• Quality is not an abstraction its a
  measurable, manageable business issue. John
  Guaspari

7
Why Do We Measure?

• Assess the status of an ongoing project
• Track potential risks
• Uncover problem areas before they go critical,
• Adjust work flow or tasks,
• Evaluate the project teams ability to control
  quality of software work products.

8
Process Measurement

• We measure the efficacy of a software process
  indirectly.
• That is, we derive a set of metrics based on the
  outcomes that can be derived from the process.
• Outcomes include
• Measures of errors uncovered before release of
  the software
• Defects delivered to and reported by end-users
• Work products delivered (productivity)
• Human effort expended
• Calendar time expended
• Schedule conformance
• Other measures.
• We also derive process metrics by measuring the
  characteristics of specific software engineering
  tasks.

9
Metrics Guidelines

• Use common sense and organizational sensitivity
  when interpreting metrics data.
• Provide regular feedback to the individuals and
  teams who have worked to collect measures and
  metrics.
• Dont use metrics to appraise individuals.
• Work with practitioners and teams to set clear
  goals and metrics that will be used to achieve
  them.
• Never use metrics to threaten individuals or
  teams.
• Metrics data that indicate a problem area should
  not be considered negative. These data are
  merely an indicator for process improvement.
• Dont obsess on a single metric to the exclusion
  of other important metrics.

10
Process Metrics

• Quality-related
• Focus on quality of work products and
  deliverables
• Productivity-related
• Production of work-products related to effort
  expended
• Statistical SQA data
• Error categorization analysis
• Defect removal efficiency
• Propagation of errors from process activity to
  activity
• Reuse data
• The number of components produced and their
  degree of reusability

11
Software Quality

• Conformance to customers requirements

12
Quality

• The American Heritage Dictionary defines quality
  as
• a characteristic or attribute of something.
• For software, two kinds of quality may be
  encountered
• Quality of design encompasses requirements,
  specifications, and the design of the system.
• Quality of conformance is an issue focused
  primarily on implementation.
• user satisfaction compliant product good
  quality delivery within budget and schedule

13
Cost of Quality

• Prevention costs include
• Quality planning
• Formal technical reviews
• Test equipment
• Training
• Internal failure costs include
• Rework
• Repair
• Failure mode analysis
• External failure costs are
• Complaint resolution
• Product return and replacement
• Help line support
• Warranty work

14
Saving Time and Money

• Even experienced software engineers inject a
  defect about every ten lines of code
• The cost of finding and fixing defects increases
  at every step in the development process
• The defect find fix times range from 3 minutes
  in code reviews to 25 minutes in inspections and
  1400 minutes in system testing
• For accurate plans and reliable commitments, you
  must insist on what?

15
Discipline and Training

• Engineers are not incompetent or lazy, theyre
  just human, and being human, we all make mistakes
• SEI data show that the most efficient way to find
  and fix defects is for engineers to read and
  carefully analyze their programs
• What can you do?
• Code reviews
• Defect data
• Training

16
Role of the Customer

• Cant be its the customers perceived value
  that sells products
• Its based on a variety of variables such as
  price, performance, reliability, and satisfaction
• Customer satisfaction is the ultimate validation
  that a product conforms to requirements
• What are the most basic measures for software
  quality?

17
Two-level Concept

• Intrinsic product quality is often limited to
  product defect rate and reliability (small q)
• The broader level includes product quality,
  process quality, and customer satisfaction (the
  big Q)
• In the past, product requirements were often
  generated without customer input
• Whats a better approach?

18
Customers Expectations

• Whats wrong with performance to customers
  expectations rather than requirements?
• Often hear people say We must exceed the
  customers expectations!
• Whats the basic problem with this?
• The result is?

19
Software Quality

• Conformance to explicitly stated functional and
  performance requirements, explicitly documented
  development standards, and implicit
  characteristics that are expected of all
  professionally developed software.
• Quality must be defined and measured if
  improvements are to be achieved
• In the narrowest sense, it is commonly recognized
  as the lack of bugs in the product
• Also, the most basic meaning of conformance to
  requirements because if the software contains too
  many functional defects, the basic requirement of
  providing the desired function is not met
• How is this usually expressed?

20
Application to Software

• Simplistically, software product quality is lack
  of bugs in the product
• Why is this problematical for software systems?
• Correct operation is not sufficient
  performance?
• Usability by the end-user
• Software specifications

21
Software metrics

• Any type of measurement which relates to a
  software system, process or related documentation
• Lines of code in a program, the number of
  staff-days required to develop a component
• Allow the software and the software process to
  be quantified
• Measures of the software process or product
• Should be captured automatically if possible

22
Measurement analysis

• Not always obvious what data means. Analyzing
  collected data is very difficult
• Professional statisticians should be consulted if
  available
• Data analysis must take local circumstances into
  account

23
Software Quality Metrics

• A subset of software metrics that focus on the
  quality aspects of the product, process and
  project

24
Topics Covered

• Product Quality Metrics
• In-Process Quality Metrics
• Phase-Based Defect Removal
• Maintenance Metrics
• Fix Backlog
• Fix Response Time
• Fix Quality
• Examples of Metrics Programs

25
Software Metrics

• Classified into three categories
• Product Metrics describe the characteristics of
  the product such as size, complexity, design
  features, performance, and quality level
• Process Metrics can be used for improving the
  software development and maintenance process such
  as defect removal effectiveness, defect arrival
  patterns, and fix response time
• Project Metrics describe the project
  characteristics and execution such as resource
  profiles, schedule, cost and productivity metrics
• In general, software metrics are more closely
  associated with process and product metrics than
  project metrics

26
Product quality metrics

• A quality metric should be a predictor of product
  quality.
• Most quality metrics are design quality metrics
  and are concerned with measuring the coupling or
  the complexity of a design.
• Whats the problem with the relationship between
  these metrics and quality?

27
Product Quality Metrics

• Two key metrics for intrinsic product quality are
  Mean time to failure (MTTF) and Defect density
• MTTF is most often used with safety critical
  systems such as air traffic control systems,
  avionics, and weapons
• Defect density metric is used in many commercial
  software systems
• Both are correlated, but different in the same
  way that failures and defects are different

28
Terminology

• We will use the term defect to refer to an error,
  fault or failure.
• Errors are defects in the human thought process
  made while trying to understand given
  information, to solve problems, or to use methods
  and tools.
• Faults are concrete manifestations of errors
  within the software. One error may cause several
  faults and various errors may cause identical
  faults.
• Failures are departures of the operational
  software system behavior from user expected
  requirements. A particular failure may be caused
  by several faults and some faults may never cause
  a failure.

29
Product Quality Metrics

• For all practical purposes, a fault and a defect
  are the same
• When an error occurs a fault or defect is caused
  in the software
• In an operational mode, failures are caused by
  faults or defects, but defect and failure dont
  have a one-to-one correspondence
• Sometimes one fault can cause multiple failures
  or it may take several faults to cause one failure

30
Software Reliability

• A simple measure of reliability is
  mean-time-between-failure (MTBF), where
• MTBF MTTF MTTR
• The acronyms MTTF and MTTR are mean-time-to-failur
  e and mean-time-to-repair, respectively.
• Software availability is the probability that a
  program is operating according to requirements at
  a given point in time and is defined as
• Availability MTTF/(MTTF MTTR) x 100
• Consider 5 nines reliability (99.999) what does
  this mean?
• 15 minutes of down time per year

31
MTTF Metric

• Defects that cause higher failure rates are
  usually encountered and removed early
• For special-purpose software systems like an ATC
  system or Telecom system, the operations profile
  and scenarios are better defined so MTTF is an
  appropriate metric
• For general-purpose computer systems or
  commercial-use software, there is no typical user
  profile so it is more difficult to implement an
  MTTF metric

32
Defect Density Metric

• In general, defect rate is the number of defects
  over the opportunities for errors during a
  specified time frame
• Because failures are defects materialized, we can
  approximate the of defects by the of observed
  failures, but how do you quantify the denominator
  opportunities for error during a time period
• Most companies use the size of the software
  product expressed in KLOC and various
  operational definitions for Life of Product (LOP)
• In operating systems generally 95 of all defects
  are found within four years of release
• For applications, most defects are found within
  two years of release

33
Defect Density Metric

• There can be significant differences in LOC
  counting even when two groups use the same
  language huge variations exist when using
  different programming languages
• Consequently, extreme caution must be exercised
  when comparing defect rates of two products if
  they were developed using different programming
  languages
• Also, there is always the question of whether you
  count just new and added LOC or the total LOC in
  the finished product
• Examples see Kan, pp. 91-93

34
Defect Density Metric

• In determining the relative quality of several
  releases of a software system, it is best to look
  at both the defect density based on modified LOC
  and the defect density based on total LOC
• Motorola measured both and the defect density per
  modified LOC varied far more than it did per
  total code the later was typically 10 times the
  former
• Why is this true?

35
Customers Perspective

• The defect rate measures code quality on a per
  unit basis, but from the customers point of
  view, the defect rate is not as relevant as the
  total number of defects he/she is going to
  encounter
• To improve the customers perception of quality,
  we need to ensure that the total number of
  defects decreases from release to release
• Examples?

36
Defect Rate Vs. Total Defects
37
Defect Rate Vs. Total Defects
38
Customer Problems Metric

• Measures the problems reported by customers when
  using the product
• From the customers viewpoint, all problems that
  they encounter, not just valid defects, are
  problems with the software
• Examples of non-defect problems are usability
  problems, unclear documentation, duplicates of
  valid defects, or even user errors
• Usually expressed as problems per user month
  (PUM)

39
Customer Problems Metric

• PUM total of problems reported by customers
  (including non-defect problems) over the total
  of license-months
• of license-months of installed licenses
  times of months in the calculation period
• To achieve a low PUM, one can
• Improve the development process reducing
  product defects
• Reduce the non-defect-oriented problems by
  improving all aspects of the product such as
  customer education support
• Increase the sale of the product

40
Defect Rate Vs. Customer Problems
Kan Table 4.1
41
Customer Satisfaction Metrics

• The customer problems metric is an intermediate
  measurement between defects measurement and
  customer satisfaction
• If you improve the customer problems metric, you
  should improve customer satisfaction
• Improving customer satisfaction also involves
  factors of a broader scope such as timing and
  availability of the product and services

42
Customer Satisfaction

• Often measured by customer survey data via a
  five-point scale
• Very satisfied
• Satisfied
• Neutral
• Dissatisfied
• Very dissatisfied
• Satisfaction with the overall quality and its
  specific dimensions is usually obtained through
  various survey techniques

43
Customer Satisfaction

• Motorola measured
• Product Features
• Reliability
• Availability
• Billing
• Documentation
• Based on the five-point scale, several metrics
  can be constructed
• Percent of very satisfied customers
• Percent of satisfied customers (very satisfied)
• Percent of dissatisfied customers
• Percent of non-satisfied (neutral, dissatisfied,
  very)

44
Customer Satisfaction

• Usually the percent of satisfied customers metric
  is used unless your focus is on reducing the of
  non-satisfaction, then the fourth metric is used
• Motorola also measured through their surveys what
  the customers perceived as best in class
  performance in the various specific parameters
• The best in class measure was shown as a goal
  to strive for in developing improvement plans
• I will cover more in lecture 9.

45
In-Process Quality Metrics

• Compared to product quality metrics, in-process
  quality metrics are less formally defined and
  their practices vary greatly among SW
  organizations
• Some organizations only track defect arrival
  during formal machine testing
• Others track various parameters during each phase
  of the development cycle
• Several metrics are needed as minimum for
  in-process quality management

46
Defect Density

• Defect rate found during formal machine testing
  is usually positively correlated with defect rate
  experienced in the field
• One reason that they wouldnt be positively
  correlated is if the higher testing defect rate
  was due to an extraordinary testing effort
• What would happen to testing defect rate if there
  was a more effective defect appraisal activity
  added to the development process such as formal
  document and code inspections?

47
Defect Density

• So the simple metric of defects per KLOC is a
  good indicator of quality
• Especially useful to subsequent releases of the
  same product because release-to-release
  comparisons are not contaminated by other
  extraneous factors
• However, this is a summary indicator and the
  pattern of defect arrivals (time between
  failures) gives more information

48
Defect Arrival

• Even with the same overall defect rate during
  test, different patterns of defect arrivals
  indicate different quality levels in the field
• Motorola also monitored this defect arrival rate
  at which defects were being found to determine if
  additional testing was required the rate had to
  level off before a product could ship
• Also, tracked the defect removal backlog (defects
  found, but not fixed) by severity and would not
  ship product unless all Severity 1 problems were
  fixed

49
Phase-Based Defect Removal

• An extension of the test defect density metric
• Requires tracking of all defects at all phases of
  the development cycle starting with requirements
• The pattern of phase-based defect removal
  reflects the overall defect removal ability of
  the software development organization
• The following figure shows two patterns of defect
  removal one was front-end loaded and the other
  was heavily testing-dependent for removing
  defects
• The field quality for the front-end loaded
  project significantly outperformed the
  testing-dependent project

50
Defect Removal Pattern
51
Defect Removal Effectiveness

• Defined as defects removed during a development
  phase over total number of latent defects times
  100
• Since the total latent defects is not known, the
  denominator is approximated by summing the
  defects found in phase and defects found in later
  phases that are attributed to that phase
• High values mean that the phase is more effective
  at preventing defects from escaping to the next
  phase
• More next time.

52
Maintenance Metrics

• When a product has completed development and is
  released, it enters a maintenance phase
• During this phase, there are two important
  metrics
• Defect arrivals by time interval
• Customer problem calls by time interval
• However, these two metrics dont reflect the
  quality of software maintenance
• During maintenance, fixing problems quickly will
  improve customer satisfaction
• The following metrics are very important to
  measure the effectiveness of the maintenance
  process
• Fix Backlog and backlog management index
• Fix response time
• Percent delinquent fixes
• Fix Quality

53
Fix Backlog

• Work load statement for maintenance
• Simple count of reported problems that remain
  open at the end of each month or week
• Also, useful to form a trend line to judge
  whether the process is under control
• The Backlog Management Index (BMI) is the ratio
  of of problems closed to of problems opened
  during a given time interval times 100

54
Example of BMI Chart
55
Fix Response Time

• Frequently set guidelines on the time limit
  within which fixes should be available for
  reported defects by the severity of the problem
• The fix response metric is usually calculated by
  severity level as the mean time that all problems
  of a given severity are open
• Frequently, low severity problems remain open for
  so long that this metric becomes meaningless
• Percent Delinquent Fixes
• More sensitive metric than the mean response time
  is the of delinquent fixes
• Assuming that an organization has set some
  guidelines for fix response time, this metric
  counts those fixes that exceed the guideline by
  severity level
• delinquent fixes is the ratio of fixes that
  exceed the limit to the total fixes delivered in
  a given time interval times 100

56
Fix Quality

• Number of defective fixes is a very important
  metric especially with respect to customer
  satisfaction
• Customers get very upset when they install a fix
  that doesnt fix a problem or worse causes more
  problems
• Simply the of all fixes in a time interval that
  are defective
• Prefer just a count rather than a because the
  goal is to deliver ZERO defective fixes

57
Examples

• See detailed example of Motorolas software
  metrics program on pages 110 through 115 of Kan.
• See detailed example of Hewlett-Packards
  software metric program on pages 115 through 116
  of Kan.
• See overview of IBM Rochesters metrics on pages
  116 117 of Kan.

58
Collecting Data

• Must carefully consider what data to collect
• Collection must be based on well-defined metrics
• Keep the amount of data and number of metrics to
  a minimum
• More importantly, the information extracted from
  the data must be focused, accurate, and useful
• Avoid over-collection by clearly defining the
  data collection methodology

59
Basili and Weiss Methodology

• Consists of the following six steps
• Establish the goal of the data collection
• Develop a list of questions of interest
• Establish data categories
• Design and test data collection forms
• Collect and validate data
• Analyze data
• Important to perform data validation concurrently
  with data collection
• Whats another name for this method?

60
Testing Vs. Inspection Defects

• Kan states that testing defects are generally
  more reliable than inspection defects Why?
• Important to have a clear definition of an
  inspection defect
• Ill cover Inspections in a separate lecture, but
  There is an important advantage to inspections
  that I want you to be aware of
• What frequently happens when you encounter a
  defect while testing software?
• What happens when you encounter a defect while
  inspecting software?

61
Small Organizations

• Although there is no correlation between the
  existence of a metrics program and the size of an
  organization, Kan presents some good
  recommendations for small organizations on pages
  124 125
• Implement a defect arrival metric
• Need to distinguish field reported defects so BMI
  is a good measurement
• Normalize data to product size using FP or LOC

62
Basic Quality Tools

• Application of Ishikawas seven basic tools for
  quality control to software development

63
Topics Covered

• Application Notes
• Checklist
• Pareto Diagram
• Histogram
• Run Charts
• Scatter Diagram
• Control Chart
• Cause and Effect Diagram

64
Application Notes

• Statistical tools for project and quality control
  at the project level
• Useful to project leaders and product managers
  especially in medium and large development
  organizations
• Dont provide specific information to software
  developers on how to improve the quality of their
  designs or implementation
• Also, less useful for small projects
• Not widely recognized as beneficial to software
  development, but Kan believes that there are
  benefits to using these tools correctly and I
  agree

65
The seven basic tools of quality

• Checklist
• Pareto Diagram
• Histogram
• Run Charts
• Scatter Diagram
• Control Chart
• Cause and Effect Diagram aka Fishbone or Ishikawa
  diagram

66
Checklist

• For software, we use a confirmation check sheet
  commonly called a checklist
• Used by pilots before take-off
• Used by NASA before Shuttle flights
• Distinguishes it from a data gathering check
  sheet
• Mainly concerned with the quality characteristics
  of a process or product
• There are many examples such as a code inspection
  checklist, a process checklist, a entry condition
  checklist or a requirements complete checklist
• See example in the book of a Program Temporary
  Fix (PTF) checklist figure 5.2 on pages
  132-133.

67
Example of Checklist
68
Pareto Diagram

• A frequency bar chart in descending order by
  types of problems or defects
• X-axis is usually the defect cause and Y-axis is
  the defect count
• Identifies the few causes that account for the
  majority of defects
• Commonly see a 80-20 pattern 80 of the defects
  from 20 of the causes

69
Pareto Analysis of Software Defects
70
Another Sample Pareto Diagram
71
Histogram

• A graphic representation of frequency counts of a
  sample or population
• X-axis lists the unit intervals of a parameter
  like defect severity level and Y-axis contains
  frequency counts
• Purpose is to show the distribution
  characteristics of the parameter

72
SEI L2 KPA Rating Histogram
73
Sample Resource Histogram
74
Run Charts

• Tracks the performance of the parameter of
  interest over time
• X-axis is time and Y-axis is the value of the
  parameter
• Best used for trend analysis
• Especially useful if historical data is available
  for comparisons with the current trend
• Frequently used for project management

75
Phase Containment Effectiveness
76
Scatter Diagram

• Vividly portrays the relationship of two
  interval variables
• For a cause-effect relationship, the X-axis is
  the independent variable and the Y-axis is for
  the dependent variable
• Each point represents an observation of both
  variables
• Aid in looking for relationships between two
  variables
• Can clearly see outliers that may distort
  correlation coefficient calculations
• Compared to other tools, more difficult to apply
• Requires precise data and investigative work

77
High-, Medium-, and Low-Risk Projects
78
Project Duration Scatter Diagram
79
Control Chart

• Advanced form of a run chart for monitoring
  process capability
• Consists of a central line and a pair of upper
  and lower control limits
• X-axis is time and Y-axis is the value of the
  parameter of interest
• If all values are within the control limits, the
  process is regarded as being under control
• Look for a trend or out of limit condition to
  take corrective action

80
Control Chart

• Almost impossible to estimate the process
  capability of a software development organization
• Control charts are useful for software quality
  improvement when used in a more relaxed manner
  used as a tool for improving consistency and
  stability
• The most appropriate for software applications
  are the p chart (proportion non-conforming) for
  percentages and the u chart (non-conforming per
  unit) when defects are used

81
Control Chart
82
Quality Control Charts

• A control chart is a graphic display of data that
  illustrates the results of a process over time.
• The main use of control charts is to prevent
  defects, rather than to detect or reject them.
• Quality control charts allow you to determine
  whether a process is in control or out of
  control.
• When a process is in control, any variations in
  the results of the process are created by random
  events processes that are in control do not need
  to be adjusted.
• When a process is out of control, variations in
  the results of the process are caused by
  non-random events you need to identify the
  causes of those non-random events and adjust the
  process to correct or eliminate them.

83
The Seven Run Rule

• You can use quality control charts and the seven
  run rule to look for patterns in data.
• The seven run rule states that if seven data
  points in a row are all below the mean, above the
  mean, or are all increasing or decreasing, then
  the process needs to be examined for non-random
  problems.

84
Sample Quality Control Chart
85
Cause and Effect Diagram

• Also known as the fishbone diagram or Ishikawa
  diagram
• Shows the relationship between a quality
  characteristic and factors affecting that
  characteristic
• Quality characteristic is labeled at the fish
  head position
• Factors affecting the characteristic are placed
  where the bones are located
• Identifies all causal factors of a quality
  characteristic on one chart

86
Sample Fishbone Diagram
87
Fishbone Diagram
88
Small Organizations

• Focus on
• Checklist
• Simple and based on teams experience
• Start with a couple and update and review them
  periodically
• Gradually will develop into a self-learning team
• Pareto diagram
• Most useful for quantitative parameters such as
  defects and customer reported problems
• Cause and effect diagram
• Use for a team oriented approach to problem
  solving
• Does not recommend control charts unless you have
  someone with a strong statistical background

89
Summary

• The professional view is that quality must be
  defined and measured for improvement
• It is best defined as conformance to customers
  requirements
• In software the operational definition is at two
  levels the intrinsic product quality (small q)
  and customer satisfaction (big Q)
• Metrics gather information about both process and
  product
• Quality metrics should be used to identify
  potentially problematical components

90
Summary

• Basic tools not appealing from a researchers
  standpoint
• Real value lies in the consistent and persistent
  use by development teams
• Impact can be enormous, especially when they are
  automated and become ingrained in the software
  development process
• Used with great success by customer satisfaction
  teams at Motorola

91
Next Class

• Topic
• Quality
• Defect Removal Effectiveness
• Process and Project metrics
• Evaluate Measurements
• Analysis and models
• Analysis Techniques
• Normal curve and six sigma
• The Rayleigh Model Reliability Models Quality
  Management Models
• Reading
• Kan, Chapters 6-9, pp. 66-70
• Articles on the Class Page

92
Journal Exercises

• Testing defects are generally more reliable than
  inspection defects.
• Why?
• How can one improve inspections to generate more
  reliable defects?

93
Midterm Examination

• Nobody expects the Spanish Inquisition!
• Monty Python

94
Mid-term Examination

• Midterm Examination will be on the Blackboard
  system starting Friday, October 16, through
  Thursday, October 22
• See important information about Taking Quizzes in
  Blackboard see notes below as well
• On Blackboard
• Login to the Blackboard system https//oll.depaul
  .edu/
• Take the Mid-term examination.
• It will be made available Friday, October 16.
• You must take the exam by COB Thursday, October
  22
• Allow 3 hours (should take about one and a half
  hours if you are prepared) note books or notes
  should not be used, but are allowed.
• Midterm study guide