RUGRAT: Runtime Test Case Generation using Dynamic Compilers

1
RUGRATRuntime Test Case Generation using
Dynamic Compilers

• Ben Breech
• NASA Goddard Space Flight Center

Lori Pollock John Cavazos University of Delaware
2
Motivating Example
if ((sptr malloc (size 1)) NULL)
findmem ()
if ((sptr malloc (size 1)) NULL)
xlfail (insufficient string space)
How do I test this callsite?

• Make the machine run out of memory?

• Flip the conditional, recompile, flip back?

• Pretend it doesnt exist during testing?

3
Generalizing the Problem

• Code to handle uncommon situations
• Difficult to test
• May need external environment event to trigger
• Examples
• Error handling code
• Testing program security mechanisms

4
Observation

• Hard to reach code executes when program thinks
  something uncommon has occurred

if ((sptr malloc (size 1)) NULL)
findmem ()
if ((sptr malloc (size 1)) NULL)
xlfail (insufficient string space)

• Could test findmem() by simulating error
• e.g., could add instructions to program so
  program believes malloc failed

5
RUGRAT Approach
Use Dynamic Compilers to generate test cases for
hard to reach code.
Automatically add instructions to program during
execution to simulate uncommon situation.
6
Dynamic Compilers

• Dynamic compilers perform compilation tasks
  during program execution

Analysis transformation optimization
Dynamic Compiler
Basic block
Mod. Basic block
Create basic block
Executeon CPU
code
translate
7
RUGRAT Architecture
Dynatest Generator
Test spec
Analysis transformation optimization
Dynamic Compiler
Basic block
Mod. Basic block
Create basic block
Executeon CPU
Create basic block
code
translate
Test Report
Test Oracle
8
Test Spec

• Details where/how for inserting tests
• Current prototype limited (environment vars).
  Can express
• Function locations
• test all calls to function x
• test only second call to x in function y
• Failure value (e.g., 0, -1, etc)
• Some side effects

9
Dynatest Generator

• Scans instructions for location to insert test
  (e.g., call to function X)
• Allows function X to execute
• Adds instructions to simulate error
• Instructions added after function X
• Program thinks error happened, reacts

10
Example
if ((sptr malloc (size 1)) NULL)
findmem ()
if ((sptr malloc (size 1)) NULL)
xlfail (insufficient string space)
L1
Dynatest Generator
11
The Good, the Bad and the Ugly
The Bad

• Not a perfect simulation

The Good

• Adequate simulation
• Can target system or appl calls
• Saves quite a lot of tester effort

The Ugly

• Still a prototype

12
Security Mechanism TestingEncrypting Function
Pointers

• Protects progs against func pointer attacks
• Difficult to test (need vulnerable program and
  attack)
• RUGRAT can simulate attack by adding instructions
• Very different from error handling code case

RUGRAT can be used for variety of testing tasks.
13
Current Implementation Notes

• Used DynamoRIO1 dynamic compiler
• Some limitations (but new version is available)
• Test spec from env. vars
• Nothing fancy for oracle

1 Bruening, et al., CGO 2003
14
Experiments

• Ran variety of programs with RUGRAT
• space, SPEC, MiBENCH
• Tested handling of errors in
• malloc / fopen / write
• Application calls

15
Experiments Summary

• Can RUGRAT generate tests to cover error handling
  code?

YES! RUGRAT tested error handling code at 120
callsites (missed one because DynamoRIO incurred
a segfault)
16
Experiments Summary

• Can RUGRAT increase statement coverage for error
  handling code?

• YES! RUGRAT increased code coverage 50 (on
  average) of error handling code
• Not all statements executed b/c of different
  options
• RUGRAT detected cases of omission errors

17
Fault Detection

• Could RUGRAT help detect failures in error
  handling code?
• Grad students seeded faults into error handling
  code for space program
• Changed assignments, loops, conditionals,etc
• Seeded total of 34 faults

18
Fault Detection Summary

• RUGRAT detected 15 / 34 faults
• Of 19 undetected faults
• 6 changed return values, but callers only checked
  certain vals (e.g., if (func () ! 0))
• 2 allocated too little memory (malloc may
  allocate more memory than requested anyway)
• 2 unknown
• 1 caused space to quit
• 8 instances were caller performed same code as
  callee (e.g., any fault in callee was undid by
  caller)

19
Some related work

• Holodeck1, FIG2
• Require tester provide alternative stub
  functions to do testing
• Miss application calls
• Dynamic branch switching3
• Not originally intended for testing error code
• Need to know which branch to change
• Far less accurate simulation

1 Thompson et al., SAC 2002 2 Broadwell et al.,
SHAMAN 2002 3 Zhang et al., ICSE 2006
20
Conclusions and Summary

• Presented RUGRAT architecture
• Can test hard to reach (and seldom tested) code
  by using dynamic compilers
• Saves tester effort
• RUGRAT is a general tool

21
RUGRAT Architecture
Dynatest Generator
Test spec
Dynamic Compiler
Basic block
Mod. Basic block
Executeon CPU
Create basic block
code
Test Report
Test Oracle
22
Experiments Summary

• Tested variety programs with RUGRAT
• 120 error code handling callsites covered
• Both application and system calls
• Increased error code coverage 50 over regular
  test cases
• Not all error code statements could be covered
• Different options, etc
• Reasonable time overhead