Software Testing

1
Software Testing

• Dr Z He
• University of Ulster

2
Lecture 6 Software Testing

• Objectives
• Issues of software testing
• Software testing, Software testing principles,
  Need for testing
• Verification and validation
• Testing stages
• unit, integration, system, acceptance
• Test Planning
• Test plan
• Techniques for testing
• static reading, walkthrough and inspections,
  correctness proofs, stepwise abstraction
• dynamic functional, structural
• Testing Coverage
• Further Testing Stages

3
Software Testing

• once source code has been generated, software
  must be tested to uncover(and correct) as many
  errors as possible before delivery to the
  customer
• the goal is to design a series of test cases that
  have a high likelyhood of finding errors
• software testing techniques provide systematic
  guidance for designing tests that
• exercise the internal logic of software
  components, and
• exercise the input and output domains of the
  program to uncover errors in program function,
  behaviour and performance
• is a process of executing a program with intent
  of finding an error
• a good test case is one that has a high
  probability of finding an uncovered error
• a successful test is one that uncovers an
  undiscovered error
• cannot show the absence of errors and defects, it
  can only show that software error and defect are
  present

4
Software Testing Principles

• All tests should be traceable to customer
  requirements
• Tests should be planned before testing begins.
  Test planning can begin as soon as the
  requirements model is complete. Detailed
  definition of test cases can begin as soon as the
  design model has been solidified
• Pareto principle applies to software testing (80
  of all errors uncovered during testing will
  likely to be traceable to 20 of all program
  components)
• Testing should begin in small (components) and
  progress toward testing in large(ultimately the
  entire system)
• Exhaustive testing is not possible
• To be most effective, testing should be conducted
  by an independent third party

5
Need for Testing

• 30-85 errors per 1000 lines of code (Boehm 1981)
• After testing, 0.5-3 errors per 1000 lines of
  code (Myers 1986)
• Errors are expensive
• The later errors are found the more expensive
  they are to fix

Cost to fix
Phase
Requirements Operation
6
Need for Testing

• Testing shows presence of errors but cannot prove
  their absence
• In most cases it is impossible to test
  exhaustively
• E.g.
• for i from 1 to 100 do
• print (if ai true then 1 else 0 endif)
• has 2100 different outcomes - approx. 3x1014
  years to exhaustively test. Given a machine can
  execute 10 million print instructions per second.

7
Verification and Validation

• Verification - Are we building the product
  right?
• Validation - Are we building the right product?
• 1. At the requirements analysis stage
• cheapest to correct at this stage
• Test criteria
• completeness - are all the requirements gathered
• consistency - no contradictions within the system
  or with external components
• feasibility - benefits should outweigh costs
• testability - requirements should be unambiguous,
  otherwise they cannot be tested

8
Verification and Validation

• 2. At the Design stage
• criteria as for requirements stage.
• Elements from the requirements should be traced
  to the design
• simulation, walkthroughs, inspections to test
  quality

9
Verification and Validation

• 3. Implementation stage
• Programs can be checked by reading through them
  (not by the author)
• Stepwise abstraction - determine the function of
  code from a number of its steps
• Tools are available to support the testing of
  code
• Static analysis - test the code without running
  it - E.g. have all the variables been declared?
• Dynamic analysis - test by executing the code

10
Testing Stages

• Unit testing
• testing of individual components
• Integration testing
• testing collections of modules integrated into
  sub-systems
• System testing
• testing the complete system prior to delivery
• Acceptance testing
• testing by users to check that the system
  satisfies requirements. Sometimes called
  alpha(?) testing

11
Testing Stages
U
n
i
t
t
e
s
t
i
n
g
S
u
b
-
s
y
s
t
e
m
t
e
s
t
i
n
g
S
y
s
t
e
m
t
e
s
t
i
n
g
A
c
c
e
p
t
a
n
c
e
t
e
s
t
i
n
g
12
Test Planning

• Test Plan
• The testing process
• Requirements traceability
• Tested items
• Testing schedule
• Test recording procedures
• Hardware and software requirements
• Constraints

13
Techniques for Testing - Static

• 1. Reading, Walkthroughs and inspections
• All involve someone else looking at the code
• Look for
• inappropriate usage of data. E.g. uninitialised
  variables, arrays out of bounds, dangling
  pointers
• errors in declarations. E.g. undeclared
  variables, repeated names
• Faults in computations. E.g. division by zero,
  overflow, type mismatches, erroneous operator
  order (,/,,- etc.)
• faults in logical operators. E.g. lt instead of gt
• faults in control. E.g. infinite loops, loop
  executing the wrong number of times
• faults in interfaces. E.g. Wrong number of
  parameters
• Many of these are detected by the compiler
• Can be applied to documents
• Can be applied to all stages of life cycle

14
Techniques for Testing - Static

• 2. Correctness Proofs
• Does a program meet its specification.
• If specification is expressed formally and proved
  the program can be proved
• Difficult to use.
• Cannot prove every aspect.
• Validation can only be done by testing.
• 3. Stepwise Abstraction
• bottom-up process
• start with code and derive functions
• check these with the requirements

15
Techniques for Testing - Dynamic

• Dynamic Testing Techniques
• Black Box Testing
• also called functional testing, or behavioural
  testing
• focuses on the functional requirements of the
  software
• enables the software engineer to derive sets of
  input conditions that will fully exercise all
  functional requirements for a program (test cases
  derived from specifications)
• tends to be applied during later stages of
  testing
• purposely disregards control structure, attention
  is focused on the information domain
• exhaustive functional testing is normally not
  feasible
• use characteristics of input domain to find the
  minimum set of test cases to reveal most defects
• we will study
• Equivalence Partitioning
• Boundary value analysis

16
Techniques for Testing - Dynamic

• Dynamic Testing Techniques
• White Box Testing
• also called glass box testing, or structural
  testing
• uses the control structure of the procedural
  design to derive test cases
• exhaustive structural testing is normally not
  feasible
• to find the minimum set of test cases to exercise
  most important logical paths
• We will study
• Basis Path Testing
• Loop Testing
• Testing Coverage

17
Techniques for Testing - Dynamic

• Equivalence Partitioning
• a black box /functional testing technique
• in determining a set of test cases, this
  technique is to partition the program domain into
  a (small) number of equivalent classes
• the assumption is that if a representative
  element from a class is tested and passed, all
  other elements in the class need not to be tested
  assuming they would pass the test
• How to define equivalent classes
• if a input condition specifies a range, one valid
  and two invalid equivalent classes are defined
• if an input condition requires a specific value,
  one valid and two invalid equivalent classes are
  defined
• if an input condition specifies a member of a
  set, one valid and one invalid equivalent classes
  are defined
• if an input condition is Boolean, one valid and
  one invalid class are defined

18
Techniques for Testing - Dynamic

• Equivalence Partitioning
• Example 1 Online banking
• The user can access the bank using his personal
  computer, providing a 6-digit password, and
  follow with a series of typed commands that
  trigger various banking functions. The software
  supplied for baking application accepts data in
  the form
• area code blank or three digit number not
  beginning with 0 or 1
• password six digit alphanumeric string
• commands check, deposit, or bill pay
• Equivalent classes can be identified for each
  field
• area code(boolean, range, specific value)
• ec1 blank ec2 3-digit numbers (lt200) ec3
  3-digit numbers (200-999) ec4 less than 3-digit
  numbers ec5 more than 3-digit numbers (at least
  one value is needed for test case design)

19
Techniques for Testing - Dynamic

• Equivalence Partitioning
• password (boolean, specific value, set)
• ec6 blank ec7 6-digit alphanumeric
  strings(valid passwords) ec8 6-digit
  alphanumeric strings(invalid passwords) ec9
  less than 6-digit alphanumeric strings ec10
  more than 6-digit alphanumeric strings
• commands(boolean, set)
• ec10 blank ec11 valid (check, deposit,
  bill pay) ec12 invalid
• Example 2 an input condition each book has a
  6-digit number (specific value) can be
  classified as either valid or invalid equivalent
  classes
• one valid class 6-digit numbers
• two invalid classes more or less than 6 digits
• test cases can be constructed to cover the three
  equivalent classes
• Weak point partitioning may be inadequate so
  that the assumption may not hold

20
Techniques for Testing - Dynamic

• Boundary Value Analysis (BVA)
• complements the equivalent class partitioning
  technique
• Assumes software usually fails at boundaries.
• concentrate on the extreme (boundary) values from
  each class (rather than random values from each
  class)
• BVA leads to a selection of test cases that
  exercise bounding values
• output can also be partitioned into equivalent
  classes, and extreme values from each class are
  tested as well
• Guidelines for BVA
• if an input condition specifies a range bounded
  by value a and b, test cases should be designed
  with values a and b, just above and just below a
  and b, respectively (a, b, --a, a, --b, b)
• if an input condition specifies a number of
  values, test cases should be developed that
  exercise the minimum and maximum numbers. Values
  just above and below minimum and maximum are also
  tested

21
Techniques for Testing - Dynamic

• Boundary Value Analysis (BVA)
• Apply guidelines 1 and 2 to output conditions.
  For example, assume that a temperature vs.
  pressure table is required as output from an
  engineering analysis program. Test cases should
  be designed to create an output report that
  produces the maximum (and minimum) allowable
  number of table entries.
• If internal program data structures have
  prescribed boundaries (e.g., an array has a
  defined limit of 100 entries), be certain to
  design a test case to exercise that data
  structure at its boundary.

22
Techniques for Testing - Dynamic

• Basis Path Testing
• a white-box testing technique, proposed by Tom
  McCabe, 1976
• to derive a logical complexity measure of a
  procedural design, and use this measure as a
  guide for defining a basis set of execution paths
• test cases derived to exercise every statement
  and branch in the program at least once during
  testing (statement/branch coverage)
• if every condition in a compound condition is
  considered, condition coverage can be achieved
• Steps
• Draw a (control) flow graph, using the flowchart
  or the code
• Calculate the cyclomatic complexity, using the
  flow graph
• Determine the basis set of linearly independent
  paths
• Design test cases to exercise each path in the
  basis set

23
Basis Path Testing

• Flow Graph
• used to depict program control structure
• can be drawn from a flowchart (a procedural
  design representation)
• can be drawn from a piece of source code
• Flow Graph Notation
• a flow graph composed of edges and nodes
• an edge starts from a node and ends to another
  node
• Sequence if-then-else While
  Repeat-until Case

24
Basis Path Testing

• Flow Graph
• Draw a flow graph from source code
• 1 procedure insert(a, b, n, x)
• 2 begin bool foundfalse
• 3 for I1 to n do
• 4 if aIx
• 5 then foundtrue goto leave endif
• 6 enddo
• 7 leave
• 8 if found
• 9 then bIbI1
• 10 else nn1 anx bn1 endif
• 11 end insert

25
Basis Path Testing

• Flow Graph
• Draw a flow graph from a flowchart

26
Basis Path Testing

• Cyclomatic Complexity
• a software metric that provides a quantitative
  measure of the logical complexity of a program
• Basis set is a maximal linearly independent set
  of paths through a graph
• An independent path is any path through a
  program that introduces at least one new set of
  processing statements or a new condition (I.e. at
  least one new edge in a flow graph)
• Cyclomatic complexity defines the number of
  independent path in the basis set of a program
• gives an upper bound for the number of tests that
  must be conducted to achieve statement/branch/cond
  ition coverage
• How to calculate cyclomatic complexity
• cc e - n 2p
• e - number of edges n - number of nodes p -
  number of components
• if all nodes in a graph are connected, then p
  1, thus
• cc e - n 2

27
Basis Path Testing Example 1

• 1. Draw a flow graph
• 0 /Finding the maximum of three integers/
• 1 include ltstdio.hgt
• 2 intmaximum(int, int, int)
• 3 main()
• 4 int a,b,c
• 5 printf((Enter three integers )
• 6 scanf(ddd, a,b,c)
• 7 printf(Maximum is d\n, maximum(a,b,c))
• 8
• 9 int maximum(int x, int y, int z)
• 10 int maxx
• 11 if(ygtmax)
• 12 maxy
• 13 if(zgtmax)
• 14 maxz
• 15 return max
• 16

gt
28
Basis Path Testing Example 1

• 2. Calculate cyclomatic complexity
• e7, n6, p1 so that cc7-623
• 3. Identify a basis set of independent paths
• p1 a-b-c-d-e-f (ygtx, zgty)
• p2 a-b-d-e-f (yltx, zgtx)
• p3 a-b-c-d-f (ygtx, zlty)
• 4. Design test cases

29
Basis Path Testing Example 2

• 1. Draw a flow graph
• see slide 6-24 source code, flow graph
• 2. Calculate cyclomatic complexity
• e 12 n 10 p 1
• cc 12 - 10 2 x 1 4
• 3. Determine a basis set of independent paths
• expect to specify 4 independent paths
• p1 1-2-3-7-8-9-11
• p2 1-2-3-4-5-7-8-9-11
• p3 1-2-3-4-5-7-8-10-11
• p4 1-2-3-4-6-3-7-8-10-11 (1 or more times)
• HOWEVER by reading source code, we found
• 3-7 gt 10 5 gt 9
• p1 and p3 must be modified

30
Basis Path Testing Example 2

• 3. Determine a basis set of independent paths
• if p3 modified, it would be the same as p2. Thus
  p3 should be deleted.
• But the new paths introduced by p3 (8-10-11) must
  be covered by other paths! We found p4 covers
  them.
• Modify p1, delete p3, we can have three
  independent paths
• p1 1-2-3-7-8-10-11
• p2 1-2-3-4-5-7-8-9-11
• p3 1-2-3-4-6-3-7-8-10-11
• if you study the program carefully, you will find
  the following is better
• p1 1-2-3-7-8-10-11 (insert x when a is empty)
• p2 1-2-3-4-5-7-8-9-11(insert x when a1x)
• p3 1-2-3-4-6-3-4-5-7-8-9-11 (insert x when
  aix,igt1, ngti)
• p4 1-2-3-4-6-3-7-8-10-11 (insert x when a is
  not empty and x is not in a p4 does not
  introduce any new edge but it exercises a new
  combination of the program logic!)

31
Basis Path Testing Example 2

• 4. Design test cases
• Path 1 test case 1-2-3-7-8-10-11 (insert x when
  a is empty)
• input data n0 x8 a10 b10
• expected results a18 b11 n1
• Path 2 test case 1-2-3-4-5-7-8-9-11(insert x
  when a1x)
• input data n3 x9 a19 a22
  a33b12b25b38
• expected results b13
• Path 3 test case 1-2-3-4-6-3-4-5-7-8-9-11
  (insert x when aix,igt1, ngti)
• input data n3x3a19a22a33b13b
  22b38
• expected results b39
• Path 4 test case 1-2-3-4-6-3-7-8-10-11 (insert x
  when a is not empty and x is not in a)
• input data n3x6a19a22a33b13b
  22b38
• expected results a46 b41 n4

32
Techniques for Testing - Dynamic

• Loop Testing
• a white box/structural testing technique
• focuses exclusively on the validity of loop
  constructs
• four different classes of loops can be defined
• simple loops nested loops
  concatenated loops unconstructed loops

33
Techniques for Testing - Dynamic

• Loop Testing
• simple loops
• if n is the maximum number of allowable passes
  through the loop
• the following set of test cases (7 cases) can be
  applied
• skip the loop entirely (0)
• only one pass through the loop (1)
• two passes through the loop (2)
• m passes through the loop where mltn (m)
• n-1, n, n1 passes through the loop (n-1, n, n1)
• Class Exercise
• use the example for the basis path testing to
  derive 7 test cases
• for real testing, you need to write a program to
  test the function

34
Techniques for Testing - Dynamic

• Loop Testing
• nested loops
• start at the innermost loop. Set all other loops
  to minimum values
• conduct simple loop tests for the innermost loop
  while holding the outer loops at their minimum
  iteration parameter values.
• Work outward, conducting tests for the next loop,
  but keeping all other outer loops at minimum
  values and other nested loops to typical values
• continue until all loops have been tested
• concatenated loops
• if concatenated loops are independent, use the
  approach defined for simple loops
• if not independent(the loop counter for loop 1 is
  used as the initial value for loop 2), then the
  approach applied to nested loops is recommended

35
Techniques for Testing - Dynamic

• Testing Coverage
• Statement coverage - Run a series of tests which
  ensure that every statement is tested at least
  once. A tool is used to keep track of how many
  times a statement has been executed. BUT - not
  every situation is dealt with!
• Branch coverage - Run a series of tests that
  ensure that every possible branch is tested at
  least once. Again a tool is required.
• Condition coverage - Run a series of tests that
  ensure that every possible outcome of each
  condition is tested at least once.
• Path Coverage - Run a series of tests that cover
  all possible combination of paths. This may be
  impossible so paths have to be selected
• All-definition-use-path coverage - Create a
  test case for all paths that lead from a
  definition e.g. x1 to a variable x x 1.

36
Further Test Stages

• So far we have dealt with module testing or unit
  testing
• Integration Testing
• Modules need to be tested when combined
• Start with low level modules and move up
• difficult to get an impression of the overall
  system
• Start at the top and work down to subsystems
• may prove repetitive and laborious
• A combination of both approaches can be used
• System Test Testing the whole system
• Acceptance Test Testing the usability
• Installation Test If the operating environment
  is different from the development environment