Testing Strategies CFICSE VV08 October 1999

1
Testing StrategiesCFICSE VV08 October
1999

• Dr. Terry Shepard
• Royal Military College of Canada
• Electrical and Computer Engineering
• 613-541-6000 ext. 6031

2
Testing Strategies Topics

• Possible emphases
• Exhaustive testing is impossible
• Why does testing work at all?
• What do testing strategies depend on?
• Some testing strategies
• Are all testing strategies driven by coverage
  considerations?
• Mode transition code is critical
• Strategy Ideas from Marick 4
• Beizers Classification
• Cause-effect graphing
• Basis path coverage

3
Three possible emphases in selecting test cases

• Safety
• Frequency of Use
• High Error Rates

4
Only a small part of the state space can be
explored

• exhaustive testing is impossible
• Example
• 1 KB of memory is 21024 8 possible states.
• Visit 1 state per 10-9 seconds
• takes 1082.5 seconds, about 1075 years
• effectively, only about 10-75 of the state space
  can be explored.

5
Why does testing work at all? - Slide 1

• Hoare 1 defects are bimodal
• reasonableness and predictability in most
  circumstances
• has not been expressed in quantitative terms
• not the same as measures of reliability.
• number of meaningful and feasible paths is much
  less than the size of the state space
• most of the state space is unreachable

6
Why does testing work at all? - Slide 2

• partitioning the state space does work in
  practice, in spite of Hamlets arguments
• Hamlet 2 and others have argued that random
  values drawn from the whole input space are as
  effective in estimating reliability or at finding
  defects remaining as are random values chosen to
  be typical of subdomains of the input space

7
What do testing strategies depend on? - Slide 1

• size of program being tested unit/module,
  component, subsystem, system
• development strategy/process e.g. incremental
  vs. big bang
• costs of developing stubs, drivers and other
  harnesses.
• testing purpose debugging, reliability
  measurement, usability, load, stress,
  performance, installation, acceptance,

8
What do testing strategies depend on? - Slide 2

• new system or old (regression testing is always
  present!)
• type of system window interface, real-time,
  distributed, ...
• nature of market beta test of shrink-wrapped
  software, not of military software
• degree of certainty needed
• availability of specifications
• top-down or bottom-up integration

9
What do testing strategies depend on? - Slide 3

• available budget and schedule
• point in schedule at which test planning starts
• contractual arrangements and politics
• market timing considerations
• customer tolerance for failures
• extent of automated testing that is possible

10
Kit 3 proposes 4 levels of testing

• Full testing - starts with requirements or before
  (200 payback)
• Partial testing - starts with functional design
  (150 payback)
• Endgame testing - validation oriented
• Audit-level testing - audit of plans, procedures,
  products checking after the fact
• 3 also suggests that 40 to 70 of costs are for
  error detection and defect removal.

11
Some testing strategies - Slide 1

• identify clues and requirements Marick 4
• debug remove (severe) errors that cause
  (frequent) crashes
• find faults on most frequently executed paths
• test scenarios that are most important to the
  customer/user of the software
• guerilla attacks try to break it, using any
  means available, often in a competition

12
Some testing strategies - Slide 2

• meet some coverage target (e.g. 80 statement
  coverage)
• measure a mix of coverage indicators and apply
  judgement as to which to pay attention to
• observe/inspect by simulated execution of a
  representation of the design (e.g. ObjecTime)
• Microsoft tester buddies (is this a strategy?)

13
Choice of testing strategy is partly determined
by the style used to write the code

• roughly three main divisions
• pre-structured programming
• structured code
• object oriented code

14

• Aside Beizer 5 adds a twist All data
  declarations should be global, and should be kept
  in a data dictionary, on the basis that over the
  life of a piece of software, data tends to
  migrate toward being more global, so why not
  recognize this tendency right up front and make
  all data global? He thinks the jury is still out
  on the effects of object-oriented programming on
  this phenomenon. (p. 46)

15
Are all testing strategies driven by coverage
considerations?

• Coverage may be applied to any kind of
  documentation
• A software project contains many different
  structures
• Many different forms of traversal can be measured
• It is common wisdom among testers that choosing
  tests to satisfy some code coverage measure is a
  very poor strategy

16
Mode transition code is critical

• Example 1 dual mission computers, can operate
  with only one, code that makes the transition is
  rarely used, but is critical
• Example 2 aircraft spends most of its time on
  the ground or in level flight, but code for
  takeoff and landing is critical
• Example 3 two modes of operation for a printer
  pool jobs printed on arrival, or jobs queued.
  Transition code between modes is critical.

17
Strategy Ideas from Marick 4 - Slide 1

• clues lead to test requirements, which lead to
  test specifications
• sources of clues code, specs, earlier testing
  results, experience
• e.g. uncovered code may lead to new clues
• most errors are not creative checklists work
  well
• faults of omission are harder to find, and more
  important

18
Strategy Ideas from Marick 4 - Slide 2

• a single test specification usually satisfies
  multiple test requirements
• unit testing is too expensive
• Marick expresses controllability and
  observability in terms of
• reachability
• necessity (fault must produce a different result
  from the correct program)
• propagation

19
Strategy Ideas from Marick 4 - Slide 3

• These are ideal fault conditions
• must be approximated, since fault locations are
  not known
• Marick has built a Generalized Coverage Tool
  (GCT)
• from cs.uiuc.edu or from Testing Foundations
• Test Requirement Catalog 4 Appendix B
• really a black box testing checklist

20
A Classification of Black and White Box Testing
techniques

• From 5 Software Testing Techniques, 2nd Edn,
  Van Nostrand Reinhold, 1990, by Boris Beizer)
• Called strategies by Beizer - to some extent,
  they are.

21
Beizers Classification

• Flowgraphs and Path testing
• Transaction-flow testing
• Data-flow testing
• Domain testing
• also called Partition or Equivalence testing
• includes boundary value analysis
• Syntax testing
• Logic-based testing
• States, state graphs and Transition testing
• Mutation Testing

22
Flowgraphs and Path testing

• includes block/statement (C1) and branch coverage
  (C2)
• hard to achieve 100
• may be worth testing parts beyond 100

23
Flowgraph and Path testing Strategies

• keep paths short and simple,
• make small incremental changes to paths
• start with loop free paths
• include paths that test loop limits
• then decide how extensive to make loop testing
• may help to artificially lower loop upper limits

24
Transaction-flow testing

• equivalent to path testing, but at the level of
  interfaces among software components rather than
  interfaces among individual statements.
• might also be called scenario based or use-case
  based testing.
• If scenarios or use cases were part of the
  requirements, they can be used to generate tests

25
Data-flow testing

• primarily investigates sequences of k,d,u,-
  kill, define (note that declared and initialized
  are not distinguished), use, no mention. For
  example, -u is anomalous, in the sense that the
  variable may have been defined globally, but if
  it has not been, this is an error.

26
Domain testing

• partition the input space, and test using
  representative values from each partition
• Hamlet 2 questions its effectiveness

27
Syntax testing

• hidden languages often exist for describing input
  and output. Testing is driven by explicitly
  defining the syntax (e.g. in BNF), and trying to
  validate (and break) instances of the hidden
  language. If the language is not hidden, so that
  a clear syntax exists, syntax testing is less
  productive.

28
Logic-based testing

• Uses decision tables, path expressions
• Addresses the issue of how to combine a variety
  of different conditions in an orderly fashion to
  create multiple test cases

29
States, state graphs and Transition testing

• Some aspects of a design may be represented as
  finite state machine diagrams
• Testing these parts of a design can be done by
  undertaking a transition tour of the state
  diagram (state graph)
• This is not useful for state represented in the
  form of integers, floats, arrays, records, etc.

30
Mutation Testing

• generate mutations of program under test
  (preferably automatically), run tests on all
  mutations (expensive), examine all cases where
  mutants and original program give the same result
  - either add tests so they give different
  results, or conclude that the original program is
  in error.

31
Cause-effect graphing

• Idea from the 70s
• not practical
• By transforming a written specification into a
  set of cause-effect graphs, the tester is
  replacing one complex representation with
  another 6

32
Basis path coverage

• Select test cases that follow the control flow
  paths on which the McCabe complexity measure
• McCabes conjecture was that such test cases
  would provide adequate coverage
• Beizer 5 disagrees

33
References 1

• 1 C.A.R. Hoare, How did software get so
  reliable without proof?, Formal Methods Europe
  96, Keynote speech
• 2 D. Hamlet and R. Taylor, Partition Testing
  does not Inspire Confidence, IEEE Transactions
  on Software Engineering, Dec. 1990, pp. 1402-1411
• 3 Edward Kit, Software Testing in the Real
  World Improving the Process, Addison Wesley, 1995

34
References 2

• 4 Brian Marick, The Craft of Software Testing,
  Prentice Hall, 1995
• 5 Boris Beizer, Software Testing Techniques,
  2nd Edn, Van Nostrand Reinhold, 1990
• 6 T.J. Ostrand and M.J. Balcer, The
  Category-Partition Method for Specifying and
  Generating Functional Tests, Communications of
  the ACM 31, 6, June 1988, pp. 676-686