Software Testing

1
Software Testing

• Test is the most popular quality improvement
  activity
• Focus of testing in this course
• Unit testing by developer
• Glass-box testing, black box (or functional
  testing)
• Testing and debugging are two different
  activities
• Testing means of detecting faults through
  failures
• Debugging
• Fault isolation
• Fault removal

2
BlackBox versus GlassBox

• BlackBox (also known as functional) testing views
  component under test as a black box.
• Test cases derived from functional requirements
  and signature of component only
• GlassBox (also known as whitebox) testing views
  component internals as visible
• Test cases derived from internal structure of
  component in addition to the functional
  requirements

3
GlassBox Functional Requirements

• Doesnt glassbox testing simply test that the
  code does what it does?
• No! Glassbox testing still requires the
  specification (I.e., functional requirements) to
  determine what the expected output should be
• GlassBox testing helps determine what are good
  inputs not what the expected outputs should be

4
Black GlassBox
Functional Requirements
Component Internals
Generate BB Test Cases
Generate Expected Outputs
Generate GB Test Cases
Run Test Cases
Compare Outputs
Pass/Fail
5
Abstract versus Concrete

• Abstract test case defines the what of the test
  case
• Concrete test case defines the actual inputs (how
  of the test case)
• Example
• (Abstract) Test lower case initialisms in middle
  of sentence
• Concrete
• He was going 55 m.p.h through Ballard
• The tires were spinning at 300 r.p.m when he hit
  the lutefisk

6
Why Testing is Hard

• Testings goal is to find errors, development
  goal is to prevent errors
• Testing can never prove the absence of errors
  only their presence
• Testing (by itself) does not improve quality
• Testing an indicator of quality
• To improve SW, dont test more, develop better
• Testing requires you assume you will find error
  in the code (if you assume you wont, you
  probably wont)

7
Resources and Benefits

• Resources depend on size and complexity of
  project
• Unit testing should take between 8 to 35 of
  total project time
• Roughly ½ the time taken to develop
• What is unit testing good for
• Guide corrections to the software
• Assess the reliability of the product
• If recorded, can reveal the kinds of error that
  are most common

8
Testing During Construction

• Two views
• Design a routine as a black-box
• Test a routine as a glass-box
• If you know whats inside, you stand a better
  chance of detecting a fault
• Steps for each routine
• Design/Write
• Check mentally
• (Inspect)
• Test
• Combine with other tested routines
• Why??

9
A General Outline

• Test for each relevant (derived) requirement
  allocated to routine
• Plan for this as early as possible
• Possibly write test cases before you write the
  routine
• Test each relevant design concern
• Augment above with glass-box testing
• Control
• Control Flow (aka Path Testing)
• Data Flow
• Remember, you cant test every combination of
  inputs

10
Testing and Risk

• Level A Software resulting in a catastrophic
  failure condition for the system.
• Modified Condition/Decision Coverage, Decision
  Coverage Statement Coverage
• Level B Software resulting in a hazardous or
  severe-major failure condition for the system.
• Decision Coverage Statement Coverage
• Level C Software resulting in a major failure
  condition for the system.
• Statement Coverage
• Level D Software resulting in a minor failure
  condition for the system.
• None Required
• Level E Software resulting in no effect on the
  system for the system.
• None Required

From RTCA/DO-178B
11
Kinds of GlassBox Testing

• Control - focuses on statements that define
  control flow
• E.g., Statement, Decision, Condition,
  Condition/Decision, Modified Condition/Decision
  Coverage (MCDC)
• Control Flow - focuses of flow of control through
  program
• E.g., Simple Paths, Basis Paths, Multiple
  Condition Paths
• Data Flow - focuses on data usage
• Reach, All uses, all du-paths
• Mutation - Focuses on detecting mutant programs

12
Control (Flow) Testing

• Path testing--family of test techniques based on
  selecting particular paths through the code
• Oldest of all structural test techniques (as
  early at 1964)
• Fundamental to all structural testing techniques
• Based on idea of a control flow graph (as opposed
  to data flow graph, or control flowchart)

13
Control Flowgraph Elements
Process A
Process
true
If X
false
Decision
Junction
1
Case _ of _
Case 1
Case Stmt
Case 2
14
Element Definitions

• Process block-sequence of uninterruptable
  statements (I.e., if one executes, they all
  execute)
• Decision-point at which control flow can diverge
  (mostly binary except Fortran)
• Case Statement-multi-branch decision
• Junction-point at which control flow merges
  (e.g., goto endif, continue)

15
Example Flowgraph
If X
Process A
Process B
1
Process C
16
Path
Path is a sequence of instructions that starts
at an entry, junction, of decision, and ends at
another (possible same) or exit.
If X
Process A
Process B
1
Process C
17
Entry / Exit (or Complete Path)
Complete path-denote a path that starts at entry
and goes to exit. Its hard to test anything
other than a complete path
If X
Process A
Process B
1
Process C
18
Selection Criteria

• Too many paths in a typical routine to test them
  all
• note each pass through loop (1,2,3) creates a
  different path
• What are the possibilities
• Path Coverage-test all paths through the code
• Basis Path Coverage-test a specific subset of
  basis paths
• Statement Coverage-execute each statement at
  least once
• Decision Coverage-execute each decision true and
  false at least once

19
Path Coverage
If X
Process A
If Y
Process B
Process C
2

• Denoted by Process names
• A,B,C
• A,B
• AC
• A

20
Basis Path Coverage
If X
Process A
If Y
Process B
Process C
2

• Denoted by Process names
• A,B,C
• A,C
• A

21
Basis Path Coverage

• First proposed by Tom McCabe (both a testing
  technique and a complexity measure)
• McConnell Technique
• Start with 1 for the straightline path through
  the code
• Add 1 for each of the following keywords (or
  equivalents) if, while, repeat, for, and, or
• Add one for each case in case statement, if it
  doesnt have a default add one more
• Cyclomatic Complexity
• Defines the number of independent paths in the
  basis set

22
Cyclomatic Complexity
A
X
B
Y
C
Cyclomatic Complexity Edges - Nodes
2 Edges 7Nodes 6 Cyclomatic Complexity 7
- 6 2 3
23
Picking Basis Paths

• Pick path through the code that covers the most
  edges
• Pick a new path that covers at least one new edge
  
• Continue until the number of paths equals the
  cyclomatic complexity (I.e., number of basis
  tests)
• Note The basis paths through the code are not
  unique

24
Basis Paths
Test case 1) ABC 2) AB 3) A
25
Compound Logic
Note Basis paths cover compound logic
If (A or B) X() else Y()
A
B
X
Y
X