A Framework for Testing Concurrent Programs

1
A Framework for Testing Concurrent Programs

• COMP 600
• Mathias Ricken
• Rice University
• August 27, 2007

2
Unit Testing
Program
Sub- program
Sub- program
Sub- program
?
?
?
?
?
Difficult
Less difficult
Even less difficult
3
Unit Testing

• Unit tests
• Test a part, not the whole program
• Occur earlier
• Automate testing
• Keep the shared repository clean
• Serve as documentation
• Prevent bugs from reoccurring
• Effective with a single thread of control

4
Foundation of Unit Testing

• Unit tests depend on deterministic behavior
• Known input, expected outputSuccess ? correct
  behaviorFailure ? flawed code
• Outcome of test is meaningful

5
Problems Due to Concurrency

• Thread scheduling is nondeterministic and
  machine-dependent
• Code may be executed under different schedules
• Different schedules may produce different results
• Known input, expected outputSuccess ? correct
  behavior in this schedule, may be flawed in
  other scheduleFailure ? flawed code
• Success of unit test is meaningless

6
Timeliness of the Problem

• Many programs already use concurrency
• Often hidden, as part of GUI
• Most speed increases are due to multiple cores on
  one chip
• Clock speeds have not increased much
• Increased use of concurrency in the future

7
Possible Solutions

• Programming Language Features
• Ensuring that bad things cannot happen
• May restrict programmers
• Lock-Free Algorithms
• Ensuring that if bad things happen, its ok
• May limit data structures available
• Comprehensive Testing
• Testing if bad things happen in any schedule
• Does not prevent problems, but does not limit
  solutions either

details
8
Tractability of Comprehensive Testing

• Deciding whether any given program contains an
  error is undecidable
• Reduction to the halting problem
• Does not imply undecidable for all programs

9
Number of Schedules

• Test all possible schedules
• Concurrent unit tests meaningful again
• Number of schedules (N)
• t of threads, s of slices per thread

Exponential in both s and t
details
10
Critical Points

• If program is race-free, we do not have to
  simulate all thread switches
• Threads interfere only at critical points lock
  operations, volatile variables, etc.
• Code between critical points cannot affect
  outcome
• Simulate all possible arrangements of blocks
  delimited by critical points

11
Critical Points Example
Local Var 1
All accesses protected by lock
lock access unlock
lock access unlock
Thread 1
Shared Var
Lock
Local variables dont need locking
All accesses protected by lock
All accesses protected by lock
Thread 2
lock access unlock
Local Var 1
12
Fewer Schedules

• Fewer critical points than thread switches
• Reduces number of schedules
• Example Two threads, but no communication ?
  only one schedule required
• Unit tests are small
• Reduces number of schedules
• Hopefully comprehensive simulation is tractable
• If not, heuristics are still better than nothing

13
Parts of the Framework

• Improvements to JUnit
• Detect exceptions and failed assertions in
  threads other than the main thread
• Annotations for Concurrency Invariants
• Express complicated requirements about locks and
  threads
• Tools for Schedule-Based Execution
• Record, deadlock monitor
• Random delays, random yields

14
Concurrency Invariants

• Has to be called in event thread
• TableModel, TreeModel
• May not be called in event thread
• invokeAndWait()
• Have to acquire readers/writers lock
• AbstractDocument
• DrJavas documents

15
Invariants Difficult to Determine

• May be found in
• Javadoc comments
• Only in internal comments
• Whitepapers
• Often not documented at all
• Errors not immediately evident
• Impossible to check automatically

16
Java Annotations

• Add invariants as annotations_at_NotEventThreadpub
  lic static void invokeAndWait( Runnable r)
  
• Process class files
• Find uses of annotations
• Insert bytecode to check invariants at method
  beginning

17
Advantages of Annotations

• Java Language constructs
• Syntax checked by compiler
• Easy to apply to part of the program
• e.g. when compared to a type system change
• Light-weight
• Negligible runtime impact if not debugging
  (slightly bigger class files)
• Automated Checking

18
Predicate Annotations

• In annotation definition, specify static boolean
  Java method
• Method must be callable from every context?
  completely static and public
• Data in annotation, method arguments and value of
  this passed when method invoked

19
Predicate Annotation Example

• public class TestCode
• _at_TestAllowed(allowedtrue)
• public void test(String param)
• TestCode t new TestCode()
• t.test("xxx")

_at_PredicateLink( valuePredicates.class,
method"example", argumentstrue) public
_at_interface TestAllowed boolean allowed
public class Predicates public static
boolean example( Object this0,
String param, boolean allowed)
return (allowed)? // this0t
(param.equals("test")) // param"xxx"
(!param.equals("test")) //
allowedtrue
zoom
20
Provided Annotations

• _at_NotEventThread
• _at_NotThreadWithName
• _at_NotSynchronizedThis
• _at_NotSynchronizedArgument
• _at_NotNullArgument
• _at_DistinctArguments
• Inverses, conjunctions, disjunctions

21
Invariant Inheritance

• Invariants apply to the method and all overriding
  methods in subclasses
• ? Methods can have invariants defined elsewhere
• All annotations describe requirements for the
  client (and, due to subclassing, for subclasses)
• Allows frameworks to describe requirements

22
Invariant Subtyping

• To maintain substitutability, subclasses may not
  strengthen invariants
• Invariants can be modeled as special input
  parameter
• Tuple of invariants (record in ? calculus
  Pierce)
• Subtyping rules for records declare the wider
  record as subtype
• In function types, parameter types are
  contravariant

23
Invariant Subtyping

• Analyze methods with invariants
• Invariants subtyping A lt_at_
  B lt_at_ C
• IA , IB inv1, IC inv1,inv2 IC lt IB
  lt IAFA IA ? , FB IB ? , FC IC ? FA
  lt FB lt FC
• Java subtyping C lt B lt A

class A void f()
class B extends A _at_Inv1 void f()

class C extends B _at_Inv2 void f()

24
Detection of Subtyping Problems

• If Java subtyping and invariant subtyping
  disagree (A lt B but B lt_at_ A)
• Substitutability not maintained
• Statically emit warning
• Detect if client subclasses do not use framework
  classes as prescribed
• Safer multithreaded frameworks

25
Java API Annotations

• Started to annotate methods in Java API
• Community project at http//community.concutest.or
  g/
• Browse and suggest annotations
• Annotations can be extracted into XML
• Share annotations
• Add checks without needing source code

26
Testing Invariant Checker

• Annotated two DrJava versions
• 3/26/2004
• 9/2/2006
• Ran test suite, logged invariant violations
• 2004 18.83 failed
• 2006 11.03 failed
• 2006 version easier to annotate
• Better documentation of invariants

details
27
Conclusion

• Improved JUnit now detects problems in other
  threads
• Annotations ease documentation and checking of
  concurrency invariants
• Support programs for schedule-based execution

28
Future Work

• Schedule-Based Execution
• Replay given schedule
• Generate possible schedules
• Dynamic race detection
• Probabilities/durations for random yields/sleeps
• Extend annotations to Floyd-Hoare logic
• Preconditions, postconditions
• Representation invariants

29
Many Thanks To

• My advisor
• Corky Cartwright
• My committee members
• Walid Taha
• Bill Scherer
• My friends
• JavaPLT, CS and Rice
• NFS and Texas ATP
• For partially providing funding

30
Extra Slides
31
Possible Solutions

• Programming Language Features
• Race Freedom
• Deadlock Freedom, Safe Locking
• Atomicity, Transactions
• Usually require changes in type system
• Fundamental change
• C standards 1998, 2003, 200x
• Java major changes 1997 (1.0), 2002 (1.4), 2004
  (1.5)

back
32
Possible Solutions

• Lock-Free Algorithms
• Work on copy and assume no concurrency is present
  (or current thread will finish first)
• If there was interference, threads that dont
  finish first redo their work
• Require some system support (e.g.
  compare-and-swap), then done in libraries
• Not all common data structures are practical and
  efficient to implement as lock-free

back
33
Extra Number of Schedules
Product of s-combinations For thread 1 choose
s out of ts time slices For thread 2 choose s
out of ts-s time slices For thread t-1 choose
s out of 2s time slices For thread t-1 choose s
out of s time slices
Writing s-combinations using factorial
Cancel out terms in denominator and next numerator
Left with (ts)! in numerator and t numerators
with s!
back
34
Improvements to JUnit

• Uncaught exceptions and failed assertions
• Not caught in child threads

35
Sample JUnit Tests

• public class Test extends TestCase
• public void testException()
• throw new RuntimeException("booh!")
• public void testAssertion()
• assertEquals(0, 1)

Both tests fail.
Both tests fail.
if (0!1) throw new AssertionFailedError()
36
Problematic JUnit Tests
Main thread

• public class Test extends TestCase
• public void testException()
• new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• ).start()

new Thread(new Runnable() public void run()
throw new RuntimeException("booh!")
).start()
throw new RuntimeException("booh!")
Child thread
37
Problematic JUnit Tests
Main thread

• public class Test extends TestCase
• public void testException()
• new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• ).start()

new Thread(new Runnable() public void run()
throw new RuntimeException("booh!")
).start()
throw new RuntimeException("booh!")
Child thread
Uncaught exception, test should fail but does not!
38
Improvements to JUnit

• Uncaught exceptions and failed assertions
• Not caught in child threads
• Thread group with exception handler
• JUnit test runs in a separate thread, not main
  thread
• Child threads are created in same thread group
• When test ends, check if handler was invoked

39
Thread Group for JUnit Tests
Test thread

• public class Test extends TestCase
• public void testException()
• new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• ).start()

new Thread(new Runnable() public void run()
throw new RuntimeException("booh!")
).start()
throw new RuntimeException("booh!")
Child thread
invokes
checks
TestGroups Uncaught Exception Handler
40
Thread Group for JUnit Tests
Test thread

• public class Test extends TestCase
• public void testException()
• new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• ).start()

new Thread(new Runnable() public void run()
throw new RuntimeException("booh!")
).start()
throw new RuntimeException("booh!")
Child thread
spawns and waits
resumes
Main thread
failure!
check groups handler
spawns
end of test
Test thread
invokes groups handler
uncaught!
Child thread
41
Improvements to JUnit

• Uncaught exceptions and failed assertions
• Not caught in child threads
• Thread group with exception handler
• JUnit test runs in a separate thread, not main
  thread
• Child threads are created in same thread group
• When test ends, check if handler was invoked
• Detection of uncaught exceptions and failed
  assertions in child threads that occurred before
  tests end

42
Child Thread Outlives Parent
Test thread

• public class Test extends TestCase
• public void testException()
• new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• ).start()

new Thread(new Runnable() public void run()
throw new RuntimeException("booh!")
).start()
throw new RuntimeException("booh!")
Child thread
spawns and waits
resumes
Main thread
failure!
check groups handler
spawns
end of test
Test thread
invokes groups handler
uncaught!
Child thread
43
Child Thread Outlives Parent
Test thread

• public class Test extends TestCase
• public void testException()
• new Thread(new Runnable()
• public void run()
• throw new RuntimeException("booh!")
• ).start()

new Thread(new Runnable() public void run()
throw new RuntimeException("booh!")
).start()
throw new RuntimeException("booh!")
Child thread
check groups handler
spawns and waits
resumes
Main thread
success!
spawns
Too late!
Test thread
end of test
uncaught!
invokes groups handler
Child thread
44
Improvements to JUnit

• Child threads are not required to terminate
• A test may pass before an error is reached
• Detect if any child threads are still alive
• Declare failure if test thread has not waited
• Ignore daemon threads, system threads (AWT, RMI,
  garbage collection, etc.)
• Previous schedule is a test failure
• Should be prevented by using Thread.join()

45
Testing ConcJUnit

• Replacement for junit.jar or as plugin JAR for
  JUnit 4.2
• Available as binary and source at
  http//www.concutest.org/
• Results from DrJavas unit tests
• Child thread for communication with slave VM
  still alive in test
• Several reader and writer threads still alive in
  low level test (calls to join() missing)

46
Limitations

• Improvements only check chosen schedule
• A different schedule may still fail
• Requires comprehensive testing to be meaningful
• May still miss uncaught exceptions
• Specify absolute parent thread group, not
  relative Cannot detect uncaught exceptions in a
  programs uncaught exception handler (JLS
  limitation)

details
47
Extra Limitations

• May still miss uncaught exceptions
• Specify absolute parent thread group, not
  relative (rare)
• Koders.com 913 matches ThreadGroup vs. 49,329
  matches for Thread
• Cannot detect uncaught exceptions in a programs
  uncaught exception handler (JLS limitation)
• Koders.com 32 method definitions for
  uncaughtException method

back
48
Generic Annotations?

• Write _at_And as generic annotation?
• public _at_interface AndltTgt
• T terms()
• public _at_interface OnlyThreadWithName
• String name()
• Generics not allowed in annotations

49
Extra DrJava Statistics
2004 736 610 36 90 5116 4161 965 18.83 107
1
Unit tests passed failed not
run Invariants met failed failed KLOC event
thread
2006 881 881 0 0 34412 30616 3796 11.03 129
99
back
50
Predicate Annotation Example

• _at_PredicateLink(valuePredicates.class,
• method"example",
• argumentstrue)
• public _at_interface TestAllowed
• boolean allowed

Refers to Predicates.example
Definition
back
51
Predicate Annotation Example
Usage

• public class TestCode
• _at_TestAllowed(allowedtrue)
• public void test(String param)
• TestCode t new TestCode() t.test("xxx")

Call
back
52
Predicate Annotation Example
Predicate Method

• public class Predicates
• public static boolean example(
• Object this0,
• String param,
• boolean allowed)
• return (allowed)? (param.equals("test
  "))
• (!param.equals("test"))

back
53
Predicate Annotation Example
back
54
Problem Multiple Annotations

• Java does not allow the same annotation class
  multiple times
• _at_NotThreadWithName("foo")
• _at_NotThreadWithName("bar") // error
• void testMethod()
• Conjunctions, disjunctions and negations?

55
Annotation Subclasses?

• Let annotation extend a supertype?
• public _at_interface Invariant
• public _at_interface OnlyThreadWithName
• extends Invariant String name()
• public _at_interface And extends Invariant
• Invariant terms()
• Subtyping not allowed for annotations

56
Work-Around

• Different meta-annotation, _at_Combine
• _at_Combine(Combine.Mode.AND)
• public _at_interface SeveralNames
• OnlyThreadWithName value()
• _at_SeveralNames(_at_NotThreadWithName("foo"),
• _at_NotThreadWithName("bar"))
• void testMethod()

57
Combine Annotations

• May only contain invariant annotations
• Predicate annotations
• Combine annotations
• Arrays of the above
• Predicate method automatically generated
• Calls member predicate methods
• Performs AND, OR, XOR, NOT or IMPLIES

58
Invariants As Class Annotation

• Short-hand for annotating all methods
• What about methods already introduced in a super
  class, e.g. Object.notify()?
• What about methods introduced in subclasses?

_at_Invariant class A void foo() void
bar()
class A _at_Invariant void foo()
_at_Invariant void bar()
59
Invariants As Class Annotation

• Apply invariants on a class only to methods
  introduced in the class or subclasses
• class A
• public void foo()
• _at_Invariant // only applies to bar()
• class B extends A
• public void bar()

60
Bytecode Rewriting

• Except for JUnit, all tools use bytecode
  rewriting
• Class files easier to parse than source
• Works for classes without source (Java API)
• Can perform changes on-the-fly
• Adding code to detect properties
• instrumentation
• Offline and on-the-fly
• Offline Class file ? class file tool
• On-the-fly Custom class loader

61
Local vs. Global

• Changes can be done by
• modifying one method (local)
• all code that calls the method (global)
• Local instrumentation usually better
• Fewer changes, less bytecode
• Harder to make a partial instrumentation if
  global
• Not all instrumentations can be done locally
• If method is native, no class file exists

62
Other Contributions

• Recording schedules
• thread ID/type
• thread ID/type/object ID/class/method/PC
• Deadlock detector
• thread ID/type/object ID recording required
• Creates wait graph for each thread and lock
• Cycle in graph implies deadlock

63
Random Sleeps/Yields

• Randomly insert sleeps or yield before or after
  critical pointsExample If a notify() is
  delayed, a wait() may time out.
• Can detect a number of sample problems
• Have not studied probabilities or durations for
  sleeps/yields
• One inserted delay may negatively impact a second
  inserted delayExample If both notify() and
  wait() are delayed.