A SURVEY OF SOFTWARE TESTING METHODOLOGY

1
A SURVEY OF SOFTWARE TESTING METHODOLOGY
Tao Feng
Kasturi Bidarkar
2
Overview

• Motivation
• Different Classification of Testing methods
• Testing methods introduction
• Comparison
• Conclusion

3
Motivation

• Designed with the aim to detect the faults
• Estimate the operational effectiveness and
  suitability of the program being developed
• Can be classified in several ways based on
• 1) The Stage of the software lifecycle
• 2) whether the software execution is
  needed or not
• 3)Design Diversity
• 4)Structural or Functional Testing

4
Based on the Stage of the software lifecycle

• Unit Testing
• Unit Testing tests softwares basic
  component.
• Integration Testing
• Integration Testing is performed to test a
  larger software structure when two or more
  components are being integrated into a larger
  structure.
• System Testing
• System Testing is performed to test the end to
  end quality
• of the entire system.

5
Based on whether the software execution is needed
or not

• Static Analysis
• Static Analysis refers to the methods that
  determine the software quality without reference
  to the actual execution. Three steps Review,
  walkthrough, inspection
• The typical examples are syntax testing,
  inspection.
• Dynamic Analysis
• Dynamic Analysis deals with specific methods
  that ascertain quality through actual executions.
  The typical examples are statement coverage
  testing, state transition testing.
• 1. know how code works 2. Know what users
  expect

6
Based on Design Diversity

• Recovery Block Method
• Algorithm
• ensure Acceptance Test
• by Primary Alternate
• else by Alternate 2
• else by Alternate 3
• ...
• else by Alternate n
• else failure exception

7
N-self Checking Algorithm

• Algorithm Compare Results 1 and 2 compare
  Results 3 and 4
• if not (match) if not (match)
• set NoMatch1 set NoMatch2
• else set Result else set Result
  Pair 1 Pair 2
• if NoMatch1 and not NoMatch2, Result Result
  Pair 2
• else if NoMatch2 and not NoMatch1, Result
• Result Pair 1
• else if NoMatch1 and NoMatch2, raise exception
• else if not NoMatch1 and not NoMatch2
• then compare Result Pair 1 and 2
• if not (match), raise exception
• if (match), Result Result Pair 1 or 2
• return Result

8
N-Version Programming

• Algorithm
• General syntax
• run Version 1, Version 2, ..., Version n
• if (Decision Mechanism (Result1,
  Result2,...,?Result n))?
• return Result
• else failure exception

9
Based on the procedure of Testing

• Structural Testing
• Structural Testing mainly consider the
  programming implementation details such as
  programming style, control methods, coding
  details.
• Functional Testing
• The software program is treated as a black box
  which means that it is subjected to inputs and
  its outputs are verified for conformance to
  specified features and regardless of the
  programs implementation details

10
Structural Methods

• Mutation Testing
• Select vectors that distinguish between the
  faulty and correct versions of the program.
• Killed Mutant
• Equivalent Mutant
• Mutation Score
• Compare different mutation sets with Mutation
  Score

11
Path Testing

• .

• Algorithm
• 1. Start with the simplest, most direct paths
  from the entry to the exit. Add paths as needed
  to achieve higher fault coverage.
• 2. Add paths that could execute some extreme
  cases for loops or combinations of loops or
  forbidden cases.
• 3. Find the input-data sets that are
  path-sensitizing for each selected path.
• 4. Design test cases and path from the design
  flowgraph .
• 5. Treat each path as a subroutine and document
  the outcome for the stated inputs and the path
  trace

12
Structural Methods(Contd)

• Fault Injection Testing
• Introduce faults into system either by modifying
  code or by inserting code
• Can be either compile time injection or run time
  injection

13
Structural Methods(Contd)

• Code Coverage Analysis
• inspects the code directly
• describes the degree to which the source code of
  a program has been tested
• Statement coverage
• identifies which statements in a method or class
  have been executed.However it has problems for
  example
• if (condition) without a case
  that evaluates to false,
• statements statement
  coverage rates this code as 100
• else covered and
  the fault is undetected
• statements

14
Structural Methods(Contd)

• Decision Coverage
• reports whether boolean expressions tested in
  control structures evaluated to both true and
  false.
• metric ignores branches within boolean
  expressions which occur due to short-circuit
  operators
• Illustrating the problem with an Example
• if (condition1 (condition2 function1()))
• statement1
• else
• statement2

15
Structural Methods(Contd)

• Condition Coverage
• reports the true or false outcome of each Boolean
  sub-expression, separated by logical-and and
  logical-or if they occur
• similar to decision coverage but full condition
  coverage does not guarantee full decision
  coverage
• For example
• bool f(bool e) return false
• bool a2 false, false
• if (f(a b)) ...
• if (aint(a b)) ...
• if ((a b) ? false false) ...

16
Structural Methods(Contd)

• Combinatorial Testingdetects faults generated by
  interactions among various factors in software.
• Algorithm
• 1.start with an empty test suite
• 2.While uncovered pairs remain do
• 3.For each factor, Compute the interaction
  weights,Initialize test with all factors not
  fixed
• 4.While a factor remains whose value is not
  fixed
• 5.Select the factor with largest interaction
  weight using factor tie-breaking rule
• 6.Compute factor level interaction weights for
  each level of factor f
• 7.Select a level l for f which offers the largest
  increase in weighted density.using a level tie
  breaking rule
• 8.Fix factor f to level l
• 9.End whileAdd test to test suite
• 10.End while

17
Structural Methods(Contd)

• Evolutionary Algorithms
• Initialize the population
• Evaluate initial population
• Repeat
• Perform competitive selection
• Apply genetic operators to generate new
  solutions
• Evaluate solutions in the population
• Until some convergence criteria is satisfied
• Recombination and mutation are used to generate
  new solutions that are biased towards regions
  expected to give good outcomes.

18
Advantage Based Comparison

• The Design Diversity based techniques require
  additional hardware which increases cost.
• They also require development of N versions of
  code so repetitive and non-feasible.
• Statement Coverage is insensitive to logical
  operators and cant distinguish between switch
  labels.
• Decision Coverage includes coverage of switches,
  exception handlers but doesnt evaluate each
  part of boolean expression

19
Advantage Based Comparison

• Condition Coverage has better sensitivity to
  control flow and evaluates each part of
  expression in a conditional statement.
• Mutation methods and Path testing take the
  longest time to execute.
• Combinatorial testing requires evaluation of the
  interaction between N factors which are difficult
  to evaluate.

20
Functional Methods

• State Testing
• State testing strategies are based on the use of
  finite-state machine models.
• 1. The present state is fetched from memory
• 2. The present input value is fetched. If it is
  already numerical, it can be used directly
  otherwise, it may have to be encoded into a
  numerical value
• 3. The present state and the input code are
  combined to yield a pointer of the transition
  table and its logical image.
• 4. The output table, via a case statement,
  contains a pointer to the routine to be executed
  for that state-input combination.
• 5. The same pointer is used to fetch the new
  state value, which is then stored.

21
Functional Methods

• State Testing
• a program that detects the character sequence
  ZCZC can be in the following states
• Neither ZCZC nor any part of it has been
  detected.
• Z has been detected.
• ZC has been detected
• ZCZ has been detected.
• ZCZC has been detected.
• The state transition could be represent as state
  graph

22
Functional Methods

• Domain testing
• Domain testing, a special case of
  partition-testing, can be based on specifications
  or equivalent implementation information. It is a
  functional testing method when it is focused on
  specifications.
• Domain testing,as practiced,is usually applied to
  one input variable or to simple combinations of
  two variables,based on specifications.
• In domain testing,predicates are assumed to be
  interpreted in terms of input vector variables.
• If domain testing is applied to structure ,
  predicate interpretation must be based on
  implementation control flowgraph.

23
Functional Methods

• Domain testing
• Testing strategy
• 1. Identify input variables
• 2. Identify variables which appear in
  domain-defining predicates
• 3. Interpret all domain predicates in terms of
  input variables The path interpreted here is
  specified data flowgraph.
• 4. For p binary predicates which are defined by a
  linear inequality, there are at most 2P
  combinations, hence TRUE-FALSE values which
  indicates domains.
• 5. Solve these inequalities to find all the
  extreme points of each domain.

24
Functional Methods

• Equivalence Class Testing
• Equivalence class testing is designed with the
  assumption that a programs input and output
  domains can be partitioned into a finite number
  of classes that all cases in a single partition
  exercise the same functionality or exhibit the
  same behavior.

25
Functional Methods

• Equivalence Class Testing
• Test cases
• 1. Special char in numerical field
• 2.alpha chars in numeric fields
• 3.max,min values in all fields
• 4.valid inputs in required fields
• 5. valid inputs in all fields

26
Functional Methods

• Equivalence Class Testing
• Main Advantage
• 1. Only one test case from each partition is
  required, which reduces the numbers of test cases
  necessary to achieve functional coverage and
  eliminates the need for exhaustive testing.
• 2. It guides a tester in selecting a subset of
  test inputs with a high probability of detecting
  a defect.
• 3.It allows a tester to cover a larger domain of
  inputs/outputs with smaller subset selected from
  an equivalence class.

27
Functional Methods

• Equivalence Class Testing
• Main Disadvantage
• It was hard to find the test data chosen for an
  equivalence class to represent that partition in
  terms of the behavior of the program function
  that is being tested.

28
Functional Methods

• Boundary Value Testing
• Boundary value analysis is performed by creating
  test sets that exercise the boundary of the input
  and output classes identified in the
  specification so that upper and lower edges of an
  equivalence class are covered by test cases.

29
Functional Methods

• Boundary Value Testing
• Testing strategy
• 1. If an input condition for the software is
  specified as a range of values, the valid test
  cases could be developed for the ends of the
  range while invalid cases could be just above and
  below the ends of the range
• 2. If an input condition for the software is
  enumerated as a number of values, valid test
  cases could be developed for the minimum and
  maximum numbers
• 3. If the input or output of the software is an
  ordered set such as a table, the first and last
  elements of the set could be chosen for the valid
  test cases

30
Functional Methods

• Boundary Value Testing
• Main disadvantage
• It is not a systematic method due to the fact
  that it requires to find out the most extreme
  value of inputs that the program can take.

31
Functional Methods

• Random Testing
• In general random testing picks a random thread
  at every point where a thread switch can occur.
• Main disadvantages
• 1. Many sets of values may lead to the same
  observable behavior and are thus redundant.
• 2. The probability of selecting particular
  inputs that cause buggy behavior may be small.

32
Functional Methods

• Decision table technique
• The decision table technique is employed to
  represent combinations of actions for the
  respective set of decisions with each column
  comprising a test case.
• The decision table method is categorized as
  dynamic functional method.

33
Functional Methods

• Decision table technique
• Each column of the table is a rule that specifies
  the conditions under which the actions named in
  the action stub will take place. The condition
  stub is a list of names of conditions. A rule
  will indicate whether a condition should or
  should not be met for the rule to be satisfied.
• For example Action 2 will take place if
  condition 1,2,3,4 are all not met.

34
Functional Methods

• Decision table technique
• Testing Strategy
• 1. Obtain program conditions, condition states
  and actions of the decision situation. When
  conditions are defined, a list of conditions
  values are expected to know.
• 2.Specify the problem in terms of decision
  rules, which means that we have to establish a
  connection between condition states and actions
  that have to be executed.
• 3. Fill in the decision table based on the
  decision rules
• 4. Check for completeness, exclusivity, and
  correctness and simplify it.

35
Functional Methods

• Decision table technique
• The use of decision-table model to design tests
  is only warranted when
• 1. The specification can be easily converted into
  a decision table
• 2. The order in which the conditions are
  evaluated does not affect interpretation of the
  rules or the resulting action
• 3.The order in which the rules are evaluated
  does not affect the resulting action
• 4. Once a rule is satisfied and an action
  selected, no other rule need be examined

36
Functional Methods

• Cause and effect graphing
• Testing Procedure
• 1. The test subjects must decompose the
  specification of a complex software component
  into lower-level units.
• 2. For each specification unit, the tester needs
  to identify causes and their effects.
• 3.From the cause and effect information, a
  Boolean cause and effect graph is created.
• 4. The graph may be annotated with constraints
  that describe combinations of causes and effects
  that are not due to syntactic constraints.
• 5.The graph is converted to a decision table.
• 6. The columns in the decision table are
  transformed into test cases.

37
Functional Methods

• Cause and effect graphing

38
Functional Methods

• Cause and effect graphing
• Advantage
• Development of the rules and the graph from the
  specification could detect omissions,
  inaccuracies, or inconsistencies and faults.
• Weakness
• Develop the graph, especially for the situation
  that there are many causes and effects to
  consider, is difficult.

39
Advanced testing methods

• UML-based integration testing
• Unified Modeling Language(UML) have been used as
  a basis for the implementation of the distributed
  component-based application software. This
  testing method is introduced as a state-based
  testing method.
• The Siemens Corporate Research has developed a
  Test Development Environment(TDE) for generating
  a recursive, directed graph that has a root
  partition and contains all paths for plain data
  choices. An instantiation of a partition is a
  random selection of a choice from the possible
  set of choices of a test case. With a partition,
  a new choice is selected at random with every new
  instantiation which covers full branching within
  the graph and influences the test data
  generation.

40
Methods Comparison

• 1. In software bugs detection with experienced
  subjects, functional testing method performs
  better effectiveness than structural method while
  the effectiveness seems to be identical with
  inexperienced subjects.
• 2. Functional testing methods reveals more
  observable faults than structural testing
  methods
• 3. For experienced subjects the fault detection
  cost like fault detection time and fault rate are
  the same with function and structural methods
• 4. For omission initialization faults detection
  functional methods appear better than structural
  methods, while for interface faults and
  observable faults their behavior is identical.

41
CONCLUSION

• Testing Procedural know what user ask for
• 1. Plan test------what should be tested
• 2. Come up test plan------whats the test
  strategy
• 3. design test cases------use which method to
  test
• A lot of testing methods have been developed for
  achieving software testing effectiveness and
  fault coverage.
• The testing method should be carefully chosen by
  the customer based on their requirement of the
  software reliability.

42

• THANK YOU