ICS 52: Introduction to Software Engineering

1
ICS 52 Introduction to Software Engineering

• Lecture Notes for Summer Quarter, 2003
• Michele Rousseau
• Topic 12
• Partially based on lecture notes written by
  Sommerville, Frost, Van Der Hoek, Taylor
  Tonne. Duplication of course material for any
  commercial purpose without the written permission
  of the lecturers is prohibited

2
Verification and Validation
Verification is implementation consistent with
requirements specification? Validation does the
system meet the customers/users needs?
3
Software Quality assessment by VV

• Software process must include verification and
  validation to measure product qualities
• correctness, reliability, robustness
• efficiency, usability, understandability
• verifiability, maintainability
• reusability, portability, interoperability,
• real-time, safety, security, survivability,
  accuracy
• Products can be improved by improving the process
  by which they are developed and assessed

4
Testing Terminology

• Failure Incorrect or unexpected output, based
  on specifications
• Symptom of a fault
• Fault Invalid execution state
• Symptom of an error
• May or may not produce a failure
• Error Defect or anomaly or bug in source code
• May or may not produce a fault

5
Examples Faults, Errors, and Failures

• Suppose node 6 should be X C(A2B)
• Failure/Fault-less error
• - Suppose the inputs are (A2,B1)
• the executed path will be (1,2,4,5,7,8)
  which will not reveal this fault because 6 is
  not executed
• - Suppose the inputs are (A-2,B-1)
• the executed path will be (1,2,3,5,6,8)
  which will not reveal the fault because C 0
• Need to make sure proper test cases are selected
• the definitions of C at nodes 3 and 4 both affect
  the use of C at node 6
• - executing path (1,2,4,5,6,8) will reveal the
  failure, but only if B ! 0
• - (e.g. Inputs (A1,B-2) )

6
Software Testing

• Exercising a system component
• on some predetermined input data
• capturing the behavior and output data
• comparing with test oracle
• for the purposes of
• identifying inconsistencies
• verifying consistency between actual results and
  specification
• to provide confidence in consistency with
  requirements and measurable qualities
• to demonstrate subjective qualities
• validating against user needs
• Limitations
• only as good as the test data selected
• subject to capabilities of test oracle

7
Definition to execution (Levels of testing)

• System Testing
• Defined at Requirements -gt Run after integration
  testing
• Integration Testing
• Defined at Design -gt Run after Module Testing
• Module Testing
• Defined at Design -gt Run after Unit Testing
• Unit Testing
• Defined at Implementation -gt Run after
  Implementation of each unit
• Regression Testing (testing after Change)
• Defined throughout the process -gt Run after
  modifcations

8
Testing-oriented Life Cycle Process
9
Fundamental Testing Questions

• Test Criteria What should we test?
• Test Oracle Is the test correct?
• Test Adequacy How much is enough?
• Test Process Is our testing effective?

How to make the most of limited resources?
10
Practical Issues

• Purpose of testing
• Fault detection
• High assurance of reliability
• Performance/stress/load
• Regression testing of new versions
• Conflicting considerations
• safety, liability, risk, customer satisfaction,
  resources, schedule, market windows and share
• Test Selection is a sampling technique
• choose a finite set from an infinite domain

11
Test Criteria

• Testing must select a subset of test cases that
  are likely to reveal failures
• Test Criteria provide the guidelines, rules,
  strategy by which test cases are selected
• actual test data
• conditions on test data
• requirements on test data
• Equivalence partitioning is the typical approach
• a test of any value in a given class is
  equivalent to a test of any other value in that
  class
• if a test case in a class reveals a failure, then
  any other test case in that class should reveal
  the failure
• some approaches limit conclusions to some chosen
  class of errors and/or failures

12
Test Adequacy

• Coverage metrics
• when sufficient percentage of the program
  structure has been exercised
• Empirical assurance
• when failures/test curve flatten out
• Error seeding
• percentage of seeded faults found is proportional
  to the percentage of real faults found
• Independent testing
• faults found in common are representative of
  total population of faults

13
Functional and Structural Testing

• Functional Testing
• Test cases selected based on specification
• Views program/component as black box
• Structural Testing
• Test cases selected based on structure of code
• Views program /component as white box(also
  called glass box testing)

14
Black Box vs. White Box Testing
15
Structural (White Box) Testing

• Use source code to derive test cases
• Build flow graph of program
• State test requirements in terms of graph
  coverage
• Various types of coverage identified
• Choose test cases that guarantee different types
  of coverage
• All-Statements coverage
• All-Branches coverage
• All-Edges
• All-Paths

16
Flow Graphs

• Control Flow
• The partial order of statement execution, as
  defined by the semantics of the language
• Data Flow
• The flow of values from definitions of a variable
  to its uses

Graph representation of control flow and data
flow relationships
17
A Sample Program to Test
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
function P return INTEGER is begin X, Y
INTEGER READ(X) READ(Y) while (X gt 10) loop X
X 10 exit when X 10 end loop if (Y lt
20 and then X mod 2 0) then Y Y
20 else Y Y 20 end if return 2X
Y end P
18
Ps Control Flow Graph (CFG)
6
10
7
F
T
T
T
9
F
2,3,4
5
14
T
9
9a
9b
F
F
12
function P return INTEGER is begin X, Y
INTEGER READ(X) READ(Y) while (X gt 10) loop X
X 10 exit when X 10 end loop
1 2 3 4 5 6 7 8
9 10 11 12 13 14 15
if (Y lt 20 and then X mod 2 0) then Y
Y 20 else Y Y 20 end if return 2X
Y end P
19
All-Statements Coverage

• Select test cases such that every node in the
  graph is visited
• Also called node coverage
• Guarantees that every statement in the source
  code is executed at least once
• Selects minimal number of test cases

1
3
7
8
2
4
5
6
9
10
20
All-Statements Coverage of P
At least 2 test cases needed
7
6
10
F
T
T
T
F
5
9
2,3,4
14
F
Example all-statements-adequate test set (X
20, Y 10) lt2,3,4,5,6,7,9,10,14gt (X 20, Y
30) lt2,3,4,5,6,7,9,12,14gt
12
21
All-Branches Coverage

• Select test cases such that every branch in the
  graph is visited
• Guarantees that every branch in the source code
  is executed at least once
• More thorough than All-Statements coverage
• More likely to reveal logical errors

1
3
7
8
2
4
5
6
9
10
22
All-Branches Coverage of P
At least 2 test cases needed
6
10
7
F
T
T
T
F
2,3,4
5
14
9
F
Example all-branches-adequate test set (X 20,
Y 10) lt2,3,4,5,6,7,9,10,14gt (X 15, Y
30) lt2,3,4,5,6,7,5,9,12,14gt
12
23
All-Edges Coverage

• Select test cases such that every edge in the
  graph is visited
• Takes complex statements into consideration
  treats them as separate nodes
• More thorough than All-Branches coverage
• More likely to reveal logical errors

24
All-Edges Coverage of P
At least 3 test cases needed
6
10
7
F
T
T
T
2,3,4
5
9b
14
F
9a
T
F
F
Example all-edges-adequate test set (X 20, Y
10) lt2,3,4,5,6,7,9a,9b,10,14gt (X 5, Y
30) lt2,3,4,5,9a,12,14gt (X 21, Y
10) lt2,3,4,5,6,7,5,6,7,5,9a,9b,12,14gt
12
25
All-Paths Coverage

• Path coverage
• Select test cases such that every path in the
  graph is visited
• Loops are a problem
• 0, 1, average, max iterations
• Most thorough
• but is it feasible?

26
All-Paths Coverage of P
Infinitely many test cases needed
6
10
7
F
T
T
T
F
T
2,3,4
5
9b
14
9a
F
F
Example all-paths-adequate test set (X 5, Y
10) (X 15, Y 10) (X 25, Y 10) (X 35, Y
10)
12
27
Ps Control and Data Flow Graph
X
Y
X
6
10
7
X
Y
F
X
X
X
T
T
T
T
F
2,3,4
5
9b
14
9a
F
X
F
Y
12
Y
X
Y
X
28
Subsumption of Criteria

• C1 subsumes C2 if any C1-adequate testT is also
  C2-adequate
• But some T1 satisfying C1 may detect fewer faults
  than some T2 satisfying C2

29
Structural Subsumption Hierarchy
all-paths
all-DU-paths
boundary-interior loop testing
min-max loop testing
all-uses
all-defs
all-p-uses
all-c-uses
all-edges
C1
C2
subsumes
all-branches
all-statements
30
Equivalence partitioning

• Identify the set of all inputs
• Identify possible bases for subdividing the input
  Domain
• size, order, structure
• correctness
• stated requirements your smarts
• Divide input set into (sub)domains using the
  basis
• Equivalence partitions
• Subdomains may overlap (may not be a partition)
• Select representative test cases from each
  subdomain
• one test case may suffice, more is better

31
Example
1 float homeworkAverage(float scores)
2 float min 99999
3 float total 0
4 for (int i 0 i lt scores.length i)
5 if (scoresi lt min)
6 min scoresi
7 total scoresi
8
9 total total min
10 return total / (scores.length 1)
11
32
Possible Bases

• Array length
• empty array
• one element
• 2 or 3 elements
• 4 or more elements

subdomains, eq. partitions
Input domain floatBasis array length
small
one
empty
large
33
Possible Bases

• Position of minimum score
• Smallest element first
• Smallest element in middle
• Smallest element last

Input domain floatBasis position of minima
somewhere in middle
first
last
34
Possible Bases

• Number of minima
• Unique minimum
• A few minima
• All minima

35
Equivalence partitions white-box testing
The basis - equivalence partition - test case
approach works for structural testing, too. A
basis could be nodes, edges, or paths. Each
node, edge, or path defines a partition of the
input.
36
Challenges

• Structural testing can cover all nodes or edges
  without revealing obvious faults
• No matter what input, program always returns 0
• Some nodes, edges, or loop combinations may be
  infeasible
• Unreachable/unexecutable code
• Thoroughness
• A test suite that guarantees edge coverage also
  guarantees node coverage
• but it may not find as many faults as a
  different test suite that only guarantees node
  coverage

37
Limitations of Structural Testing

• Interactive programs
• Listeners, event-driven programs
• Concurrent programs or tasks threads
• Exceptions
• Self-modifying programs (assembly language)
• Objects loaded over the network
• Super-class constructors
• Asynchronous garbage collection

38
Summary of Structural Testing

• Conceptually simple
• Relies on access to source
• Not applicable in all situations
• Acts in ignorance of specifications

39
Regression Testing

• Permanent suite of test cases
• Saves effort creating test cases
• Provides record of existing functionality
• Add new test cases and delete obsolete ones when
  necessary

Ensure that changes made during maintenance do
not destroy existing functionality
40
Selective Regression Testing

• Analyze relationship between the test cases and
  the software entities they cover
• Use information about changes to select test
  cases for new version

Cost reduction Select minimal subset
of regression test suite that tests the changes
41
Simple Example
modified entity
42
Selective Regression Testing
System Under Test
entity
exercises relation
modified entity
43
The Cost-Effectiveness Tradeoff
Retest All
Cost of running necessary tests
Cost of running unnecessary tests
Cost-Effective Selective Retest
Cost of running necessary tests
Cost of additional analysis
Cost-Ineffective Selective Retest
Cost of running necessary tests
Cost of additional analysis
Cost
44
The Reality of Software Testing

• One of the most widely-used, practical class of
  techniques for revealing faults in software
• An area of software engineering showing the
  greatest gap between theory and practice