A Theory of Predicate-complete Test Coverage and Generation

1
A Theory of Predicate-complete Test Coverage and
Generation

• Thomas Ball
• Testing, Verification and Measurement
• Microsoft Research
• May 2004

2
Unit Test Generation
3
Unit Test Generation
4
Unit Test Generation
5
Unit Test Generation
6
MSIL Unit Test Tool a hybrid helper

• Goal
• capture developer knowledge ASAP
• via a strong set of unit tests
• to form a specification of the codes behavior
• How
• generate tests based on analysis of MSIL
• symbolic execution constraint satisfaction
• runtime analysis to check complicated invariants
• Facets
• complements specification-based test generation
• positive feedback cycle with programmer

7
What criteria should guide unit test generation?
8
Control-flow Coverage Criteria

• Statement/branch coverage widely used
• 100 coverage ? a bug-free program!!
• More stringent criteria
• modified-condition-decision, predicate,
  data-flow, mutation, path,

9
Predicate-complete Testing

• Predicates
• relational expression such as (xlt0)
• the expression (xlt0) (ygt0) has two predicates
• predicates come from program and safe runtime
  semantics
• Consider a program with m statements and n
  predicates
• predicates partition input domain
• m x 2n possible observable states S
• Goal of Predicate-complete Testing
• cover all reachable observable states R ? S

10
PCT Coverage

• L2 if (A B) S else T
• L3 if (C D) U else V
• PCT requires covering all logical combinations
  over A,B,C,D at
• L2 and L3
• S, T, U and V
• Some combinations may not be reachable

11
PCT Coverage does not imply Path Coverage
L1 if (xlt0) L2 skip else L3 x
-2 L4 x x 1 L5 if (xlt0) L6 A
12
PCT Coverage does not imply Path Coverage
L1 if (xlt0) L2 skip else L3 x
-2 L4 x x 1 L5 if (xlt0) L6 A
13
PCT Coverage does not imply Path Coverage
L1 if (xlt0) L2 skip else L3 x
-2 L4 x x 1 L5 if (xlt0) L6 A
14
PCT Coverage does not imply Path Coverage
L1 if (xlt0) L2 skip else L3 x
-2 L4 x x 1 L5 if (xlt0) L6 A
15
Path Coverage does not imply PCT Coverage
L1 if (p) L2 if (q) L3 x0 L4
ypq
16
Path Coverage does not imply PCT Coverage
L1 if (p) L2 if (q) L3 x0 L4
ypq
17
Denominator Problem

• Coverage metrics require a denominator
• e.g. statements executed / total statements
• Easy to define for observable states
• executed observable states / (m x 2n)
• But (m x 2n) is not a very good denominator!
• most observable states will not be reachable
• R ltltlt S

18
Upper and Lower Bounds

• Bound reachable observable states
• modal transition systems and predicate
  abstraction
• L / U defines goodness of abstraction
• Test generation using lower bound L
• Refinement to increase L / U ratio

19
Overview

• Upper and lower bounds
• Example
• Test case generation
• Refinement
• Discussion
• Conclusions

20
Abstraction Construction
21
Predicate Abstraction

• if Q ? SP(P,s)
• then (P,Q) ? onto

22
Example
23
Upper Bound May-Reachability
24
Upper Bound May-Reachability
25
Pessimistic Lower Bound
a
onto
b
may
c
total
d
26
Pessimistic Lower Bound
a
onto
b
may
c
total
d
27
Pessimistic Lower Bound
a
onto
b
may
c
total
d
28
Postdominance

• pd(a) a ? ? pd(b) (a,b) ? may

29
Postdominance

• pd(a) a ? ? pd(b) (a,b) ? may

30
Postdominance

• pd(a) a ? ? pd(b) (a,b) ? may

31
Postdominance

• pd(a) a ? ? pd(b) (a,b) ? may

32
Postdominance

• pd(a) a ? ? pd(b) (a,b) ? may

33
Optimistic Lower Bound
34
Overview

• Upper and lower bounds
• Example
• Test case generation
• Refinement
• Discussion
• Conclusions

35
Example
void partition(int a) assume(a.length()gt2)
int pivot a0 int lo 1 int hi
a.length()-1 while (lolthi) while
(aloltpivot) lo while
(ahigtpivot) hi-- if (lolthi)
swap(a,lo,hi)
void partition(int a) assume(a.length()gt2)
int pivot a0 int lo 1 int hi
a.length()-1 while (lolthi) while
(aloltpivot) lo while
(ahigtpivot) hi-- if (lolthi)
swap(a,lo,hi)
36
Observation Vector

• lolthi, lolthi, aloltpivot, ahigtpivot
• lolthi ? lolthi
• ?lolthi ? lolthi ? (aloltpivot ? ?ahigtpivot)
  
• ? (?aloltpivot ?
  ahigtpivot)
• Only 10/16 observations possible

37
void partition(int a) assume(a.length()gt2)
int pivot a0 int lo 1 int hi
a.length()-1 L0 while (lolthi) L1
L2 while (aloltpivot) L3 lo
L4 L5 while (ahigtpivot) L6
hi-- L7 L8 if (lolthi) L9
swap(a,lo,hi) LA LB LC
13 labels x 10 observations 130 observable
states
But, program constrains reachable observable
states greatly.
38
Boolean Program
void partition() decl lt, le, al, ah
enforce ( (ltgtle)
((!ltle)gt(al!ah)(!alah)) )
lt,le,al,ah T,T,, L0 while (le) L1
L2 while (al) L3 lt,le,al
(!lt ? F), lt, L4 L5 while (ah)
L6 lt,le,ah (!lt ? F), lt, L7
L8 if (lt) L9 al,ah
!ah,!al LA LB LC
39
State Space of Boolean Program
lolthi, lolthi, aloltpivot, ahigtpivot
Upper Bound 49 states
40
(No Transcript)
41
Overview

• Upper and lower bounds
• Example
• Test case generation
• Refinement
• Discussion
• Conclusions

42
Test Generation

• DFS of Lp generates covering set of paths
• Symbolically execute paths to generate tests
• Run program on tests to find errors and compute
  coverage of observable states

43
(No Transcript)
44
(No Transcript)
45
Generated Inputs
void partition(int a) assume(a.length()gt2)
int pivot a0 int lo 1 int hi
a.length()-1 L0 while (lolthi) L1
L2 while (aloltpivot) L3 lo
L4 L5 while (ahigtpivot) L6
hi-- L7 L8 if (lolthi) L9
swap(a,lo,hi) LA LB LC

• (L0TTTT,L4FTFT) 0,-8,1
• (L0TTTT,L4TTFT) 0,-8,2,1
• (L0TTTT,L4TTTT) 0,-8,-8,1
• (L0TTTF,L4TTFF) 1,-7,3,0
• (L0TTTF,L4FTTF) 0,-7,-8
• (L0TTTF,L4TTTF) 1,-7,-7,0
• (L0TTFT,L7TTFF) 0,2,-8,1
• (L0TTFT,L7FTFT) 0,1,2
• (L0TTFT,L7TTFT) 0,3,1,2
• (L0TTFF,L0TTTT) 1,2,-1,0

46
Results

• Buggy partition function
• U49, L43, Tested42
• Fixed partition function
• U56, L37, Tested43
• What about the remaining 13 states?

47
Overview

• Upper and lower bounds
• Example
• Test case generation
• Refinement
• Discussion
• Conclusions

48
Unreachable State
L7FFFT (logthi) (alogtpivot)
(ahigtpivot) But code maintains invariant
that ?0ltiltlo, ailtpivot
void partition(int a) assume(a.length()gt2)
int pivot a0 int lo 1 int hi
a.length()-1 L0 while (lolthi) L1
L2 while (aloltpivot) L3 lo
L4 L5 while (ahigtpivot) L6
hi-- L7 L8 if (lolthi) L9
swap(a,lo,hi) LA LB LC
49
Refinement
50
New Observation Vector

• lolthi, lolthi, lohi1,
• aloltpivot, ahigtpivot,
• alo-1ltpivot, ahi1gtpivot
• Only 48/128 observations possible
• For this set of predicates, Lp U

51
Overview

• Upper and lower bounds
• Example
• Test case generation
• Refinement
• Discussion
• Conclusions

52
Discussion

• Comparison to bisimulation
• Completeness of abstractions
• Related work

53
Bisimulation
54
Bisimulation
55
Abstraction Completeness
56
Abstraction Completeness
57
Related Work

• Test coverage criteria
• Symbolic execution/constraint satisfaction
• Abstraction-guided test generation
• Three-valued model checking
• Modal transition systems

58
Conclusions

• PCT coverage
• new form of state-based coverage
• similar to path coverage but finite
• Upper and lower bounds
• computed using predicate abstraction and modal
  transitions
• use lower bound to guide test generation
• refine bounds

59
Directions

• How does PCT work in practice?
• MUTT implementation underway
• Other problems
• the oracle problem
• specification inference
• compositional test generation
• object construction

60
For More Information

• http//research.microsoft.com/tvm/
• A theory of predicate-complete test coverage and
  generation
• MSR-TR-2004-28, April 2004
• to appear Third International Symposium on Formal
  Methods for Components and Objects (FMCO 2004),
  November 2004