Code Coverage Testing Using Hardware Performance Monitoring Support

1
Code Coverage Testing Using Hardware Performance
Monitoring Support

• Alex Shye, Matthew Iyer, Vijay Janapa Reddi and
  Daniel A. Connors
• University of Colorado at Boulder
• Department of Electrical and Computer Engineering
• DRACO Architecture Research Group

2
Introduction

• Code coverage is simple but useful for software
  testing
• Common method of code coverage analysis is
  through program instrumentation
• Insertion of software probes statically or
  dynamically
• Incurs a high overhead! (50-200 overhead)
• Modern processors support contain a hardware
  Performance Monitoring Unit (PMU)
• Itanium, Pentium 4, Power PC
• Allow for low overhead sampling of low level
  information
• PMU represents a low-overhead alternative to full
  instrumentation

3
PMU
Itanium-2 PMU Features

• PMUs are becoming more advanced
• Coarse-grained and fine-grained features
• DCPI, Oprofile- PC sampling
• But PMU can do more
• For example, branch vectors on Itanium
• Obstacles to PMU profiling
• Non-deterministic (sampling)
• Sample aliasing
• Sampling Less information
• Offline analysis can extend PMU information!

Goal Explore PMU-based code coverage by sampling
branch vectors and performing offline compiler
analysis
4
Code Coverage Framework
Configured to sample only taken branches
Online
Annotated Binary
Terminology Branch Vector Series of addresses
from BTB Partial Path Path of ops in compiler IR
PMU
Branch Vectors
Offline
Kernel Buffer
Branch Vectors
Address Map
Interrupt on kernel buffer overflow
Dominator Analysis
Partial Paths
Intermediate File
Branch Vector Hash Table
Code Coverage
5
Dominator Analysis

• Dominator Analysis
• Finds all blocks guaranteed to execute
• Cannot be performed effectively online
• But is standard in any compiler infrastructure

BTB Branch Vector
1-2-3-4
1
Partial Path from Branch Vector
2
Basic Blocks added with Dom. Analysis
3
4
Terminology Dominator u dominates v if all paths
from Entry to v include u Post Dominator u
post-dominates v if all paths from v to Exit
include u
6
Methodology

• Experiments run on Itanium-2 with 2.6.10 kernel
• Developed tool using perfmon kernel interface and
  libpfm-3.1 to interface with PMU
• Only sample taken branches to elongate branch
  vectors
• Set of SPEC2000 benchmarks
• Compiled with the OpenIMPACT Research Compiler
• With annotations
• OpenIMPACT module for offline analysis
• Compared to full code coverage information from a
  Pin code coverage tool

Number of Instructions and Actual Code Covered
Coverage percentage is the percent of actually
covered code discovered with PMU sampling and
offline analysis
7
Effect of Sampling Period

• Sampling Overhead due to
• Copy BTB to kernel buffer, interrupt on kernel
  buffer overflow, copy from kernel buffer into
  hash table

8
PMU vs Actual Instruction Distribution

• Kullback-Leibler Divergence
• Relative entropy of p with respect to q
• d ?k0 pk log2(pk/qk)

9
Code Coverage
10
Multiple Runs

• Regular Sampling 1) gzip, parser, twolf improve
  greatly
• Randomized Sampling may discover code regular
  sampling cannot

11
Conclusion

• Motivates and presents initial results and
  rational for PMU-based code coverage
• An example of using advanced PMU feature with
  branch vectors
• Illustrates how simple offline analysis can
  extend PMU information
• Indicates PMU could be very useful for low
  overhead profiling and program understanding
• Could be promising for profiling of released
  software

Questions?