TESTING

1
TESTING

• Dr. Riggs
• 10/21/2003

2
Background

• Code errors 30-85/KLOC
• After testing 0.5-3 /KLOC
• Not much theory
• BUT
• Earlier detection gt Lower cost to fix (398)
• Testing reveals presence (not absence) of errors

3
Varieties of Testing

• Coverage Test all of x (reqs, lines)
• Fault-Based test on constructed faults
• Error-based test for common errors
• Black-box test from spec
• White box test from code

4
Test Objectives

• Determine
• Error human action
• Fault improper code
• Failure improper behavior

5
VV

• Verification complies with spec
• Built it right?
• Validation complies with requirements
• Right thing built?

6
Global View of Test Process

• Test Strategy
• Oracle
• System
• Comparator
• Output Failures

Cf. Figure 13.2 P 401
7
Test Adequacy Criteria

• defines tests required
• stop rule
• measure
• generator
• Linked to test strategy
• Coverage example (p402 code, p. 404 fig 13.3)

8
Fault Detection vs Confidence Building

• Fault Detection
• Partition of cases for test
• (complete ? no faults)
• Confidence in Reliability
• Load simulation
• Random testing
• Cleanroom

9
Fault Detection ? Prevention

• Phase models
• Demonstration check against spec
• Destruction check implementation for faults
• Life Cycle Models
• Evaluation detect req., design implementation
  faults
• Prevention Prevent req., deign implementation
  faults

10
Testing in SW Life Cycle

• Determine test strategy
• Test requirements specification
• Generate functional test data
• Check consistency of design / req. s specs
• Evaluate software architecture
• Test design
• Generate structural and functional test data
• Execute tests
• Consistency of design implementation
• Test implementation
• Generate structural and functional test data
• Execute tests
• Repeat above as applies to change

• Req. Engr.
• Design
• Implementation
• Maintenance

11
Requirements

• Essential tests (Boehm)
• Completeness
• Consistency
• Feasibility
• Testability
• Likely errors (Poston)
• Missing info
• Unusable info
• Extra info
• Standard Formats (IEEE 830)

12
Design

• Criteria like reqs
• Likely faults like reqs
• Methods
• simulation, walkthroughs, inspections
• Standard IEEE 1016

13
Implementation

• Manual Techniques
• Code-inspection
• Stepwise abstraction
• Tool techniques
• Static Analysis of code
• Parsing
• Formal
• Dynamic Analysis of Code
• Execution
• Devices
• Drivers
• Data generators
• Stubs

14
Maintenance

• Retest-All
• Selective retest
• Regression test selection
• ???

15
VV Plannign Doc

• Waterfall
• IEEE Standard 1012
• IEEE 86a

16
Manual Test Techniques

• Static
• Reading
• Peer review
• ego-less programming
• Walkthroughs Inspections
• Moderator inspectors author
• Checklist of faults
• Test data as guide
• Specialized roles (fault sets) Parnas

17
Scenario-based Evaluation

• At design architecture phase
• (SAAM Software Architecture analysis Method)
• Choose scenarios (use cases)
• Select candidate Architectures
• Find support in architecture (direct/indirect)
• Find needed changes to architecture(s)
• high change for similar cases ? poor arch
• Overall evaluation

18
Correctness Proofs

• Formal techniques
• Floyd-Hoare
• P S Q
• Precondition
• Code
• Post-condition
• Very different from programming!
• Less different from programming
• Z, VDM, B, ASM etc

19
Stepwise Abstraction

• Example p 419
• Comment
• Loop Bottom
• (search ?bottom ?mid ?top)
• (found ?x ?mid)(search ?bottom)(search ?top)
• Loop Termination
• (found ?x ?mid) (search NULL)
• Abstraction
• (found ?x ?min(lt 1 ?mid ?n)) (not (? ?x ?))

20
Coverage-based techniques

• Cover x (lt 100) of
• Control-flow coverage
• Data flow coverage
• Requirements
• Graph model of requirements

21
Control Flow Coverage

• All-paths
• Exhaustive testing
• All-nodes
• Weak
• Branch coverage
• Effective (at branch) combinations of predicates
• All possible combinations
• Multiple condition Extended Brach Coverage
• Cyclomatic number criterion
• Linearly-independent paths
• All nodes, all connected, all paths, there is gt
  1 unique edge

22
Data Flow Coverage

• Data item
• Defined(state) given a value in the state
• Alive(state) there is a flow from def to state
• Definition-clear Definition?Definition or ?
• Usages
• P use part of a Boolean
• C use part of calculation
• Strategies
• Degree of P/C uses

23
Requirements Spec Coverage

• Develop graph of requirements
• Assume 1 test per module (node)
• Apply some control-flow technique
• Problems
• Modularization may not be accurate for all data

24
Fault-Based Techniques

• Naïve Statistical Method
• Select errors, Seed code (N)
• Test code (m errors, m1 from N)
• Estimate errors (m-m1)N/m1
• Selecting faults
• Experience
• Fault exchange between groups
• Linear relation between early and late faults!

25
Mutation Testing

• Mutate program
• Replace
• constant ? constant, variable
• Variable?variable
• Type.operator?type.operator
• Insert unary operator
• Delete statement
• Better test set kills mutants faster (D/M)
• Assumptions
• Good code close to correct
• Simple faults ? complex faults

26
Error-based techniques

• Focus on error-prone points
• Requirements ? data partition by attributes
• Test from each partition all paths coverage
• ON points on sub-domain border
• OFF point just inside border
• Nx1 domain adequate N ONs gt 1 OFF

27
(No Transcript)