Test Input Generation with Java PathFinder

1
Test Input Generation with Java PathFinder

• Author Willem Visser, Corina S. Pasareanu,
  and Sarfraz Khurshid (ISSTA 2004)
• Presented by Jongbhin Park
• University of Illinois at Urbana-Champaign
• Some slides from Willem Visser

2
Test input generation

• Test input generation for code that manipulates
  complex data structures is not easy (Java
  TreeMap)
• Test Input generation technique
• Strait model checking
• model checking used in a black-box test input
  generation
• model checking used in during white-box test
  input generation
• Contribution to show how efficient white-box
  test input generation can be done for code
  manipulating complex data

3
Red-Black Tree (for case study)
Self-balancing Binary Search Trees Java TreeMap
Implementation
repOk() conditions (1)-(4)
(2) The root is BLACK
(4) All paths from a node to its leaves contain
the same number of black nodes.
(3) If a node is red, then both its children are
black.
(1) Every node is either red or black
4
Test input generation for treemap My experience

• Statement coverage is just 2 for testing TreeMap
  library using AgitarOne. Vendor declare 80 code
  coverage or better
• (AgitarOne is a commercial tool for automated
  test case generation tool )
• Test cases for put and remove method generated by
  AgitarOne

public void testPut() throws Throwable
TreeMap treeMap new TreeMap() Integer
key new Integer(-100) Object result
treeMap.put(key, key) assertEquals("treeM
ap.size()", 1, treeMap.size())
assert something public void
testRemove() throws Throwable Object
result new TreeMap().remove("")
assertNull("result", result)
Simple Test Input
Just Once
5
Java PathFinder

• explicit state model checker for Java bytecode

6
Symbolic Execution

• Using symbolic values, instead of actual data
• representing the values of program variables as
  symbolic expressions
• As a result, the outputs computed by a program
  are expressed as a function of the symbolic inputs

7
Symbolic Execution Tree
x 1, y 0 1 gt? 0 x 1 0 1 y 1 0 1 x
1 1 0 0 1 gt? 0
Concrete Execution
Symbolic Execution Tree
8
Generalized Symbolic Execution

• model checker generates and explores symbolic
  execution tree
• off-the-shelf decision procedures check path
  conditions
• Lazy Initialization
• it initializes the components of the method
  inputs on an as needed, on demand basis
• Conservative Precondition
• For evaluating partially initialized structures
• Returning false only if the initialized fields of
  the input structure violate a constraint in the
  precondition

9
Case Study

• What to test
• Class java.util.TeeMap
• Method Some helper method for put and remove
• repOK class invariant
• Checks if the input is red-black tree
• Returns true for valid structures, otherwise
  false
• assume this definition can be encoded as a Java
  predicate, called repOk()

10
Test Input Generation for Complex Structures

• Test adequacy measured by branch coverage
• Model Checking and Symbolic Execution
• Black-box over repOk()
• White-box over implementation code

11
Testing by Model Checking

• Encode environment of code under test as a
  nondeterministic program and model check
• Testing for TreeMap methods

public static int N 5 public static
TreeMap t new TreeMap() public static
Integer elems static elems new
IntegerN public static void main(String
args) while (true) if
(Verify.randomBool()) t.put(elemsVerify.
random(N-1)) else
t.remove(elemsVerify.random(N-1))
analyzing all sequences of put and remove calls
on a set with maximally N elements
12
Black-box TIG Symbolic Execution

• Symbolic execution of repOk()
• Generate new structures only when repOk() returns
  true
• Limit the size of the structures generated
• Only correct structures will be generated
• repOk() returns true after all nodes in the tree
  have been visited, hence they must all be
  concrete
• symbolic (partial) structures can fail repOk()

13
White-box TIGSymbolic Execution

• Consider code coverage criterion when generating
  test inputs
• Use conservative repOk as a precondition during
  symbolic execution of source code
• Use repOk to concretize input

14
repOk() x 2abstract and concrete
During Lazy Initialization check conservative
repOk
Symbolic Execution of Code
Concretize inputs by symbolic execution of
(Concrete ) repOk() over symbolic structures -
as with Black-box TIG -
When coverage is achieved, solve the symbolic
constraints to create concrete inputs
Conservative repOk() Symbolic Structure
true,false,dont know
Concrete repOk() Symbolic Structure
Concrete Structure
15
Conservative repOk()

• Eliminate symbolic structures that cannot be
  converted to a concrete structure that satisfy
  repOk()
• Can accept symbolic structures that could lead to
  illegal concrete structures, i.e. it is
  conservative
• Conservative repOk() can return TRUE, FALSE or
  Dont Know
• if FALSE, eliminate structure
• if TRUE or Dont Know, continue ...
• Example (2) Red nodes have only black children.

FALSE
TRUE
Dont Know
16
White-box TIG cover branches in
deleteEntry(Entry p)
/ precondition p.repOk() / private void
deleteEntry(Entry p) if (p.left ! null
p.right ! null) Entry s successor(p)
swapPosition(s, p) Entry replacement
(p.left ! null ? p.left p.right) if
(replacement ! null) replacement.parent
p.parent if (p.parent null) root
replacement else if (p p.parent.left)
p.parent.left replacement else
p.parent.right replacement p.left
p.right p.parent null if (p.color
BLACK) fixAfterDeletion(replacement) ...
17
Symbolic Execution for white-box TIG
18
Experimental Results

• Three helper method (called by put or remove
  method)
• deleteEntry(del)-Optimal BC(branch coverage) 86
• fixAfterDeletetion(fixD)-Optimal BC 100
• fixAfterInsertion(fixI) 88
• All results were obtained using the JPF model
  checker

19
Model Checking
20
Black Box
21
White-box
22
Conclusions
Showed how to use symbolic execution and method
preconditions to do efficient black- and
white-box test input generation for code
manipulating complex data

• Current limitations
• Code instrumentation for symbolic execution by
  hand
• Cannot detect dead-code
• Need to deal with loops

23
JPF Status

• developed at the Robust Software Engineering
  Group at NASA Ames Research Center
• currently in its fourth development cycle
• v1 Spin/Promela translator - 1999
• v2 backtrackable, state matching JVM - 2000
• v3 extension infrastructure (listeners, MJI) -
  2004
• v4 symbolic execution, choice generators - 4Q
  2005
• open sourced since 04/2005 under NOSA 1.3
  licenseltjavapathfinder.sourceforge.netgt
• its a first no NASA system development hosted
  on public site before
• 11100 downloads since publication 04/2005

24
Getting and Running JPF

• Getting JPF
• svn co https//svn.sourceforge.net/svnroot/javapat
  hfinder
• Compiling JPF
• Running JPF generally speaking like a VM
  (java replacement)
• The Eclipse
• Using Subclipse
• Or import downloaded JPF

25
Sample Output (TreeMap)