Efficient Instrumentation for Code Coverage Testing

1
Efficient Instrumentation for Code Coverage
Testing

• Mustafa M. Tikir
• Jeffery K. Hollingworth
• ISSTA 2002
• 2002. 10. 07

2
Contents

• Introduction
• Background
• Dominator tree
• Code coverage algorithm
• Experiments and results
• Conclusion

3
Introduction

• Evaluation of code coverage
• The problem of identifying the parts of a program
  that did not execute in one or more runs of a
  program
• Traditional code coverage measurement tools
• Use static code instrumentation
• Have below problem
• Instrumentation all functions
• Increase instrumentation overhead
• instrumentation code remains throughout the
  execution.
• increase execution time

4
Our approach

• Dynamically insert code and remove
  instrumentation code
• Use and expand dyninst
• Reduce the number of places instrumentation code
  needs to be inserted
• Use dominator tree and CFG information
• Insert the necessary instrumentation code when a
  function is called for the first time during the
  program execution

5
Background (1/2)

• Dynunst Post-complier program
  manipulation tool

Mutator
Application
doWork(int a, b) for(I0 I 1) bar(a,1)
Mutator App
Points
API
Machine Dependent Code
Snippets
Dyninst code
Ptrace/procfs
Run-time Library
6
Background(2/2)
Implementation code insertion into a program
Program
Base Tramp
Mini-Tramp
Save Registers
Pre
Set up Args
Foo()
Relocated Instruction
Snippet
Restore Registers
Post
7
Extended dyninst API

• Provide information about CFG
• Add arbitrary instrumentation points
• Originally, dyninst only supported function
  level instrumentation
• Involve its memory allocator
• Optimization(de-allocate and compacts) may cause
  a significant overhead

8
Dominator tree (1/2)

• A node u dominates a node v.
• If every path from the entry node to v contains
  u.
• Denoted as u v
• Dominator tree expresses dominator relationships.
• u v iff there is a path from u to v in
  the dominator tree.

9
Dominator tree(2/2)

• Covered(u,t)
• If the control reaches the corresponding basic
  block at least once when the program is executed
  on t
• If u v and covered(v,t) then covered(v, t)
• u, v nodes in dominator tree
• t a test case
• If all leaves in the dominator tree are covered
  then all other nodes in the CFG are automatically
  covered

10
A CFG and its dominator tree
CFG
Dominator tree
0
0
1
1
2
3
2
3
4
4
11
Reducing instrumentation points

• Leaf node instrumentation
• Instrument all leaf node in dominator tree
• Necessary but not sufficient to produce correct
  code coverage results
• When there exists a cross edge originating from
  internal node
• Non-leaf node instrumentation
• Can capture correctly above problem
• Instrument basic block n if n has at least one
  outgoing edge to a basic block m that n does not
  dominate.

12
Example insufficient case
Dominator tree
CFG
0
0
1
2
1
2
3
4
3
4
Exit
13
Code coverage algorithm

• Pre-instrumentation
• Create the CFG
• Create dominator tree
• Choose basic block to be instrumented
• Attach a Boolean variable to basic block
• Insert instrumentation code
• On-demand instrumentation
• Insert breakpoints at the beginning of functions
• When a breakpoint is reached
• Generate CFG and insert instrumentation code

14
Deletion instrumentation code

• Deletion code includes
• Restoring original instructions
• De-allocating base and mini-trampoline
• Checking what is already executed
• Checking what can be deleted
• Deletion code has trade-off
• ?Delete instrumentation code at fixed time
  intervals

15
Experiments

• Execute the test programs with
• Pre-instrumentation without/with dominator tree
• On-demand instrumentation without/with dominator
  tree
• Measure the execution time and the number of
  instrumentation points varying the dynamic code
  deletion interval.
• Compare to purecov
• A commercial code coverage tool that uses static
  code editing

16
Reduction in instrumentation points

• Pre-instrumentation

17
Reduction in instrumentation points

• On-demanding instrumentation

18
Coverage percentage curves
19
Execution time slowdown ratios

• Slowdown for SPEC/compress

20
Execution time slowdown ratios

• Slowdown for PosterSQL

21
Comparison of slowdown ratios
22
Conclusion

• Using dominator tree info.
• Reduces the number of instrumentation points
  needed
• Dynamic deletion of instrumentation code
• Reduces the runtime overhead
• Combining dominator tree and on-demand
  instrumentation
• Reduces code coverage costs
• Consider code coverage as part of production code