CS 8532: Advanced Software Engineering

1
Chapter 14 CS 8532 Advanced Software
Engineering Dr. Hisham Haddad
Class will start momentarily.
Please Stand By
2

• Software Testing Techniques
• Discussion of Software
• Testing Techniques
• Chapter 14

3

• Review - What is Testing?
• - Testing is the process of exercising/examining
  a program with the intent of finding errors
  prior to delivery to the end user.
• - Errors may belong to a variety of classes
  (functional, behaviour, performance).
• - Among others, the objectives of testing are
  to
• gt find uncovered errors,
• gt demonstrate conformance to requirements,
• gt get insight on performance, and
• gt get indication of quality and reliability.
• - A testing strategy is a plan that outlines
  detailed testing activities (steps, test case
  design, test execution, effort, time, resources),
  and it results in Test Specification document.
• Note 30 - 40 of development effort is spent on
  testing!

4

• Testing Principles
• Among others, some basic testing principles
  include
• - Test cases should be traceable to
  requirements.
• - Design of test cases should starts as early
  as completing software requirements and before
  testing starts.
• - Testing should begin in the small and
  progress toward testing in the large.
• - Exhaustive testing is not possible. Test
  selective execution paths.
• - Testing should be conducted by independent
  team.
• Pareto Principle 80 of all uncovered errors
  during testing is
• likely be traceable to 20 of all program
  components.
• See section 5.5.2, page 114.

5

• Software Testability
• As testing become integral part of development,
  software is developed with testability in mind!
  (i.e., how easy to test the program) (evolving
  mind set)
• Testability results from the design (good design
  leads to better testability)
• Testability attributes (next slides) are mapped
  back to the quality of design and implementation.
• When testing is performed by an independent
  tester, the tester must learn the system, try to
  break it, and keep the testing quality-driven.
• Q what attributes make software testable?

6

• Testability Attributes (1)
• Characteristics that lead to a testable software
• Operability (the program operates cleanly)
• - no or few known bugs
• - no serious bugs to prevent execution of test
  cases
• - ability to conduct simultaneous testing on
  components
• Observability (the results of each test case are
  readily observed)
• - distinct output for each input
• - ability to see system state and variables
• - ability to easily identify incorrect outputs
• - ability of detect internal errors and report
  them
• - access to source code

7

• Testability Attributes (2)
• Controllability (the degree to which testing can
  be automated
• and optimized)
• - ability to generate all possible outputs (via
  inputs)
• - all code is executable using combination of
  inputs
• - ability to control SW/HW states and variables
• - consistent input/output format
• - tests may be automated and re-produced
• Decomposeability (component targeted testing)
• - modular design of the software
• - ability to test modules independently

8

• Testability Attributes (3)
• Simplicity (reduce complex architecture and
  logic to simplify
• testing)
• - functional simplicity (specific functions of
  requirements)
• - structural simplicity (modularized
  architecture)
• - code simplicity (following coding and
  documentation
• standards)
• Stability (few changes are required during
  testing)
• - changes are infrequent, controlled, dont
  invalidate test cases
• - the program can recover from failures

9

• Testability Attributes (4)
• Understandability (good knowledge of the design)
• - understanding component design
• - understanding of component hierarchy
• - understanding of design changes
• - accessibility to technical documents
• gt readily available
• gt well organized
• gt detailed and accurate content
• Q how about the test cases themselves?

10

• Test Case Attributes
• What is a good test?
• - It has high probability of finding errors.
  This requires good understanding of the software.
• - It is not redundant. Each test case must have
  a well-defined and unique purpose and objective.
• - It is neither too simple nor too complex to
  avoid error masking in case of combining test
  case together.
• - It should be the best of breed (select the
  best from among similar test cases)

11

• Test Case Design

12

• Exhaustive Testing is not Possible!

13

• Testing Selected Paths

14

• Testing Methods

15

• White-Box Testing (1)

The goal is to ensure that all statements and
conditions have been executed at least once
16

• White-Box Testing (2)
• With white-box testing, test
• - all independent paths within the
  module/component
• - all decision structures for their true and
  false values
• - all loop structures for their boundaries and
  operations
• - all data structures (arrays, sets, maps,
  stacks, etc)

17

• Why White-Box Testing?
• - Logical errors and incorrect assumptions are
  inversely proportional to a path's execution
  probability.
• - It is often believed that a logical path is
  not likely to be executed when it may be executed
  on regular basis!
• - It is likely that untested paths will contain
  some typographical errors.
• - Others?

18

• Basic Path Testing (1)
• Basic path method is white-box techniques that
  enables the designer to derive a logical
  complexity measure (number of independent paths)
  that can be used for defining basic set of
  execution paths.
• Steps
• - Construct flow graph (nodes and edges)
• - Determine the cyclomatic complexity value
• - Identify the basic set of execution paths
• - Design test cases for basic set paths

19

• Flow (Program) Graph

20

• Cyclomatic Complexity

Cyclomatic complexity V(G) is a metric that
provides quantitative measure of the logical
complexity of code (that is, number of
independent paths). Computed as 1. V(G) E -
N 2 2. V(G) P 1 3. V(G) of regions on
the graph where E is of edges N is of
nodes P is of predicate nodes (red
nodes) (for combined conditions, see page 395
figure)
21

• Basic Set of Paths

Independent path Is a path that includes a new
edge(s) that have not need visited by other
paths. Since V(G) 4, there are 4 independent
paths (called Basic Set) Path 1
1,2,3,6,7,8,9,10 Path 2 1,2,3,5,7,8,9,10
Path 3 1,2,4,8,9,10 Path 4
1,2,4,8,9,2, ... ,9,10 What is next?
22

• Design Test Cases
• Each test case must force the execution of the
  intended path.
• The goal is to ensure that each statement is
  tested at least once.
• Note A path my not be tested stand-alone. It
  must be
• tested as part of another path(s)
  (e.g., loop paths)
• See complete example on page 397, figure 14.4,
  Procedure average.

23

• Basic Paths - Notes

24

• Cyclomatic Complexity - Note

A number of industry studies have indicated that
the higher the cyclomatic complexity V(G) of a
module, the higher the probability of errors in
that module.
modules
V(G)
modules in this range are more error prone
25

• Graph Matrices
• A graph matrix is a square matrix whose size
  (i.e., number of rows and columns) is equal to
  the number of nodes on a flow graph (see page
  399, figure 14.6)
• Each row and column corresponds to an identified
  node, and matrix entries correspond to
  connections (an edge) between nodes.
• Adding a link weight to each matrix entry, the
  graph matrix can be used to calculate useful info
  such as
• Probability of execution
• Required memory
• Processing time
• Other resources

26

• Other Testing Techniques (concepts)
• Other White-Box based testing techniques include
• 1. Condition Testing Focus on testing logical
  conditions
• 2. Data Flow Testing Focus on testing variables
• 3. Loop Testing Focus on testing loops
• Note these techniques are used to test suspect
  modules
• and not necessarily the entire software.

27

• Condition Testing Concept
• Conditions are potential places for errors.
  Condition testing is designing test cases to
  detect errors in conditional structures. That is,
• - Arithmetic expressions
• - Relational operators
• - Operator precedence
• - Boolean operator (with compound conditions)
• - Boolean variables (with simple conditions)
• - Formation of parenthesis
• A condition may require several test cases for
  complete coverage.

28

• Data Flow Testing Concept
• Data flow testing is designing test cases based
  on the definition and use of a variable (called
  DU testing). The selected path includes the
  module that defines the targeted variable and all
  modules that make use of that variable.
• Data flow testing may be used with paths that
  include nested loops and if-structures.
• The difficulty with this approach is measuring
  test coverage and selecting test paths.

29

• Loop Testing Concept
• Loop testing is designing test cases to validate
  a loop structure and it boundaries.
• A loop structure can be
• - simple
• - nested
• - concatenated
• - unstructured
• Each structure requires different test cases.

30

• Loop Structures

31

• Testing Simple Loops
• Typical strategy to test a simple loop that
  iterates n times
• 1. Skip the loop entirely
• 2. Test only one pass through the loop
• 3. Test two passes through the loop
• 4. Test m passes through the loop m lt n
• 5. Test iterations n-1, n, and n1

32

• Testing Nested Loops
• Typical strategy to test nested loop
• 1. Set all outer loops to their minimum
  iteration parameter values.
• 2. Test the the innermost for min1, typical,
  max-1, and max values, while holding the outer
  loops at their minimum values.
• 3. Move out one loop and test it as in step 2,
  holding all other inner loops at typical values
  and outer loops at minimum values. Continue this
  step until the outermost loop has been tested.

33

• Testing Concatenated Loops
• If the loops are independent of one another then
  test them as simple loop, otherwise, test them as
  nested loops.
• Two loops are dependent is when the loop counter
  value of first loop is used to initialize the
  second loop.
• Avoid unstructured loop! They can be re-designed
  using other loop structures.

34

• Black-Box (behavior) Testing (1)

35

• Black-Box (behavior) Testing (2)
• Black-Box testing focuses on testing functional
  requirements of program components.
• Test cases should be designed to fully exercise
  the outlined functions of the component.
• It complements White-Box testing and uncovers
• - Incorrect/missing functions
• - Interface errors
• - Module initialization and termination errors
• - External data errors
• - Performance errors

36

• Black-Box (behavior) Testing (3)
• Black-Box testing is applied in later stages of
  the testing process.
• Test cases are designed to answer these
  questions
• How is functional validity tested?
• How are system behavior and performance tested?
• What classes of input will make good test cases?
• Is the system particularly sensitive to certain
  input values?
• How are the boundaries of a data class isolated?
• What data rates and data volume can the system
  tolerate?
• What effect will specific combinations of data
  have on system operation?

37

• Black-Box Testing Techniques (1)
• Black-Box based testing techniques include
• 1. Graph-Based methods Focus on testing
  relationship between modules. Different graphs my
  be utilized (Transition Flow, Finite State, Data
  Flow, Execution Time)
• 2. Equivalence Partitioning Focus on dividing
  input data into equivalence classes from which
  test cases are derived to test different classes
  of errors.
• (classes may include user queries, mouse
  click,
• prompts, output format, inputs, etc)
• (valid and invalid values (or ranges of
  values) for each
• class are defined)

38

• Black-Box Testing Techniques (2)
• 3. Boundary Value Analysis Extends Equivalence
  partitioning and focus on testing values at the
  edges of each partitioning (ranges of valid and
  invalid values)
• 4. Comparison Testing Focus on testing software
  systems with redundancy. (multiple instances of
  the program)
• 5. Orthogonal Testing Focus on minimizing number
  of test cases by designing test cases with
  maximum coverage. (testing one item/parameter at
  a time vs. multiple items at a time) (see
  figure 14.10)
• See textbook for details.

39

• OO Testing
• By nature, OO design results in layered
  subsystems of collaborating classes
• OO testing focuses on uncovering errors in class
  collaborations.
• OO testing starts in the small (testing classes
  and their collaborations) toward testing in the
  large (testing interfacing among subsystems)
• OO testing is derived by class state changes
  (sequence of operations to test class states)
• Encapsulation can be an obstacle for testing!
  How?
• Inheritance can be an obstacle for testing! How?
• White-box techniques can be applied to class
  operations!
• or shouldn't be?

40

• OO Testing Methods (1)
• 1. Fault-based testing
• - The tester looks for plausible faults
  (part of the implementation
• that may result in defects). To
  determine whether these faults
• exist, test cases are designed to
  exercise the design or code.
• - Test cases focus on class methods.
• Class hierarchy testing
• - Inheritance complicates testing since
  testing base classes
• does not preclude the need to test
  derived classes.
• - Test cases for a base class may be used
  to test a derived
• class, but expect different results.

41

• OO Testing Methods (2)
• 3. Scenario-based testing
• - It focuses on user (actor) interaction
  (what the user does
• not what the system does)
• - It uncovers interaction errors among
  subsystems
• - Its test cases are more complex to design
  (than other methods)
• 4. Surface and deep structure testing
• - Surface structure testing is black-box
  testing (focus on
• what the user see in (and can do with)
  the system)
• - Deep structure testing is white-box
  testing (focus on the
• technical (analysis and design) details
  of the system, such
• as interactions, behavior, and
  dependencies)

42

• Class-Level Testing Methods (2)
• 1. Random testing at class-level
• - identify operations applicable to a
  class.
• - define constraints on their use.
• - identify a minimum test sequence (random
  sequence of
• operation) that defines the minimum life
  history of the class
• (object).
• - generate a variety of random (but valid)
  test sequences to
• exercise other (more complex) class
  instance life histories.
• See textbook for more details and examples.

43

• Class-Level Testing Methods (2)
• 2. Partition testing (grouping of class
  operations)
• - It is similar to equivalence partitioning for
  conventional
• software.
• - It reduces the number of test cases required
  to test a class.
• - State-based partitioning groups and test
  operations based on
• their ability to change the state of the
  class.
• - Attribute-based partitioning groups and tests
  operations based
• on the attributes that they use in the
  class.
• - Category-based partitioning groups and tests
  operations
• based on the generic function each
  operation performs.
• See class Account testing sequences page 416.

44

• Inter-Class Testing Methods (1)
• 1. Inter-Class testing (multi-class integration
  testing)
• - For each class, use the list of class
  operations to generate a
• series of random test sequences.
• - Use the operations to send messages to other
  classes.
• - For each generated message, determine the
  collaborator class
• and the corresponding operator in that
  class.
• - For each operation in the called class,
  determine the messages
• that it transmits.
• - For each of the messages, determine the next
  level of
• operations that are invoked and
  incorporate these into the test
• sequence.

45

• Inter-Class Testing Methods (2)
• 2. Behavior-based integration testing (state
  diagram based)
• - Test sequences are derived based on the class
  behavior and
• the classes it collaborates with.
• - The tests should achieve all state coverage
  (for examples,
• the operation sequences should cause the
  Account class
• to make transition through all allowable
  states)
• (see figure 14.12 for Account class
  states)
• - In some cases, multiple state diagrams may be
  need to
• derive test sequences.

46

• Specialized Testing
• Black and white box testing may be specialized
  for different environments, architectures and
  applications.
• See textbook for testing guideline for (page
  420)
• 1. GUI development environment
• 2. Client/Server architecture
• 3. Documentation and help application
• 4. Real-Time system

47

• Testing Patterns
• Test patterns may be developed to define generic
  templates for designing test cases, similar to
  analysis and design patterns.
• Many testing pattern have been proposed in the
  literature. See textbook, page 424, for three
  example patterns

48

• Suggested Problems
• Consider working the following problems from
  chapter 14, page 427
• 3, 9, 10, 11, and
  12.
• No submission is required for practice
  assignments. Work it for yourself!