Efficient Software Model Checking of Soundness of Type Systems

1
Efficient Software Model Checking of Soundness of
Type Systems

• Michael Roberson, Melanie Harries,
• Paul T. Darga, Chandrasekhar Boyapati
• University of Michigan

2
Type Soundness

• All Possible Programs
• defined by the syntax

• Good Programs
• do not cause certain kinds of runtime errors
• e.g. memory errors, race conditions, etc.
• defined by the semantics

3
Type Soundness

• Type System
• designed to statically prevent errors
• must accept only good programs

• Well Typed Programs
• defined by the type system

4
Type Soundness

• Type Soundness
• well typed implies good
• entire class of errors eliminated at compile time

5
Type Soundness

• Much research in new type systems
• Proving type soundness is important
• Becoming its own research area
• POPLmark workshop
• Proving type soundness is tedious
• Manual proofs
• Machine checked proofs
• Both require intensive human effort
• Our technique is completely automatic
• Our technique provides counterexamples

6
Outline

• Motivation
• Example
• Approach
• Experimental Results
• Related Work
• Conclusions

7
Example

• Syntax
• e true false 0 succ e pred e iszero
  e if e1 then e2 else e3

Semantics
e ? e
succ e ? succ e
e ? e
pred e ? pred e

pred 0 ? 0

pred succ nv ? nv
e ? e
iszero e ? iszero e

iszero 0 ? true

iszero succ nv ? false
e1 ? e1
if e1 then e2 else e3 ? if e1 then e2 else e3

if true then e2 else e3 ? e2

if false then e2 else e3 ? e3
Final States v true false nv nv
0 succ nv
Good program iszero 0
Bad program iszero true
8
Example

• Type system
• T Nat Bool
• Typing rules

true Bool

false Bool

0 Nat
e Nat
succ e Nat
e Nat
pred e Nat
e Nat
iszero e Bool
e1 Bool e2 T e3 T
if e1 then e2 else e3 T
Type soundness Every well typed program is a
good program.
9
Soundness Proof

• Parts to a Soundness Proof
• Progress Theorem
• If wellTyped(t) then t is a final state or t
  transitions to t
• Preservation Theorem
• If wellTyped(t) and t transitions to t then
  wellTyped(t)
• Together, these two theorems imply Type Soundness

10
Software Model Checking

• Software Model Checking
• For each well typed program state
• Check Progress
• Check Preservation
• If a check fails, show a counterexample

11
Challenges

• Search space is infinite!
• Impose finite bounds on the search space
• Small Scope Hypothesis justifies this approach
• Jackson96Offutt00Marinov03
• Search space is huge!
• Find similarities and eliminate redundancy

12
State Space Reduction

• Many software model checkers
• Verisoft, JPF, CMC, SLAM, Blast, Magic,
• Many state space reduction techniques
• Partial order reduction
• Predicate abstraction
• Not effective for checking type soundness

13
Our Key Idea
if iszero 0 then true else false
PRUNED
14
Outline

• Motivation
• Example
• Approach
• Defining the language
• Searching the state space
• Representing the state space
• Reducing the state space
• Experimental Results
• Related Work
• Conclusions

15
Language Definition
class ExpressionLanguage implements Language
static class Expression int kind //
TRUE, FALSE, ZERO, SUCC, PRED, ISZERO, IF
_at_Tree Expression e1, e2, e3
_at_Declarative boolean wellTyped() return
syntaxOk() (kind TRUE gt true
) (kind
FALSE gt true )
(kind ZERO gt true
) (kind SUCC gt
e1.wellTyped() e1.type() INT)
(kind PRED gt e1.wellTyped() e1.type()
INT) (kind ISZERO gt
e1.wellTyped() e1.type() INT)
(kind IF gt e1.wellTyped() e1.type()
BOOL
e2.wellTyped() e3.wellTyped()
e2.type() e3.type())
Expression smallStep() throws StuckException
if (isValue()) return this if
(!e1.isValue()) e1 e1.smallStep() return
this if (kind PRED
e1.kind ZERO ) return e1 if (kind
PRED e1.kind SUCC ) return e1.e1
if (kind ISZERO e1.kind ZERO ) return
True() if (kind ISZERO e1.kind
SUCC ) return False() if (kind IF
e1.kind TRUE ) return e2 if (kind
IF e1.kind FALSE) return e3
throw new StuckException() // Helper
Functions Expression root
_at_Declarative public boolean wellTyped() return
root.wellTyped() public void smallStep()
throws StuckException root root.smallStep()
public boolean isFinalState() return
root.isValue()
16
Language Definition
class ExpressionLanguage implements Language
static class Expression int kind //
TRUE, FALSE, ZERO, SUCC, PRED, ISZERO, IF
_at_Tree Expression e1, e2, e3
_at_Declarative boolean wellTyped() return
syntaxOk() (kind TRUE gt true
) (kind
FALSE gt true )
(kind ZERO gt true
) (kind SUCC gt
e1.wellTyped() e1.type() INT)
(kind PRED gt e1.wellTyped() e1.type()
INT) (kind ISZERO gt
e1.wellTyped() e1.type() INT)
(kind IF gt e1.wellTyped() e1.type()
BOOL e2.wellTyped()
e3.wellTyped() e2.type()
e3.type()) Expression smallStep()
throws StuckException if (isValue())
return this if (!e1.isValue()) e1
e1.smallStep() return this if
(kind PRED e1.kind ZERO ) return e1
if (kind PRED e1.kind SUCC )
return e1.e1 if (kind ISZERO e1.kind
ZERO ) return True() if (kind ISZERO
e1.kind SUCC ) return False() if
(kind IF e1.kind TRUE ) return e2
if (kind IF e1.kind FALSE)
return e3 throw new StuckException()
// Helper Functions Expression
root _at_Declarative public boolean wellTyped()
return root.wellTyped() public void
smallStep() throws StuckException root
root.smallStep() public boolean
isFinalState() return root.isValue()
static class Expression int kind // TRUE,
FALSE, ZERO, SUCC, PRED, ISZERO, IF _at_Tree
Expression e1, e2, e3
17
Language Definition
class ExpressionLanguage implements Language
static class Expression int kind //
TRUE, FALSE, ZERO, SUCC, PRED, ISZERO, IF
_at_Tree Expression e1, e2, e3
_at_Declarative boolean wellTyped() return
syntaxOk() (kind TRUE gt true
) (kind
FALSE gt true )
(kind ZERO gt true
) (kind SUCC gt
e1.wellTyped() e1.type() INT)
(kind PRED gt e1.wellTyped() e1.type()
INT) (kind ISZERO gt
e1.wellTyped() e1.type() INT)
(kind IF gt e1.wellTyped() e1.type()
BOOL e2.wellTyped()
e3.wellTyped() e2.type()
e3.type()) Expression smallStep()
throws StuckException if (isValue())
return this if (!e1.isValue()) e1
e1.smallStep() return this if
(kind PRED e1.kind ZERO ) return e1
if (kind PRED e1.kind SUCC )
return e1.e1 if (kind ISZERO e1.kind
ZERO ) return True() if (kind ISZERO
e1.kind SUCC ) return False() if
(kind IF e1.kind TRUE ) return e2
if (kind IF e1.kind FALSE)
return e3 throw new StuckException()
// Helper Functions Expression
root _at_Declarative public boolean wellTyped()
return root.wellTyped() public void
smallStep() throws StuckException root
root.smallStep() public boolean
isFinalState() return root.isValue()
Expression smallStep() throws StuckException
if (isValue()) return this if (!e1.isValue())
e1 e1.smallStep() return this if
(kind PRED e1.kind ZERO ) return e1
if (kind PRED e1.kind SUCC ) return
e1.e1 if (kind ISZERO e1.kind ZERO )
return True() if (kind ISZERO e1.kind
SUCC ) return False() if (kind IF
e1.kind TRUE ) return e2 if (kind IF
e1.kind FALSE) return e3 throw new
StuckException()
18
Language Definition
class ExpressionLanguage implements Language
static class Expression int kind //
TRUE, FALSE, ZERO, SUCC, PRED, ISZERO, IF
_at_Tree Expression e1, e2, e3
_at_Declarative boolean wellTyped() return
syntaxOk() (kind TRUE gt true
) (kind
FALSE gt true )
(kind ZERO gt true
) (kind SUCC gt
e1.wellTyped() e1.type() INT)
(kind PRED gt e1.wellTyped() e1.type()
INT) (kind ISZERO gt
e1.wellTyped() e1.type() INT)
(kind IF gt e1.wellTyped() e1.type()
BOOL e2.wellTyped()
e3.wellTyped() e2.type()
e3.type()) Expression smallStep()
throws StuckException if (isValue())
return this if (!e1.isValue()) e1
e1.smallStep() return this if
(kind PRED e1.kind ZERO ) return e1
if (kind PRED e1.kind SUCC )
return e1.e1 if (kind ISZERO e1.kind
ZERO ) return True() if (kind ISZERO
e1.kind SUCC ) return False() if
(kind IF e1.kind TRUE ) return e2
if (kind IF e1.kind FALSE)
return e3 throw new StuckException()
// Helper Functions Expression
root _at_Declarative public boolean wellTyped()
return root.wellTyped() public void
smallStep() throws StuckException root
root.smallStep() public boolean
isFinalState() return root.isValue()
_at_Declarative boolean wellTyped() return
syntaxOk() (kind TRUE gt true
) (kind FALSE
gt true )
(kind ZERO gt true
) (kind SUCC gt e1.wellTyped()
e1.type() INT) (kind PRED gt
e1.wellTyped() e1.type() INT) (kind
ISZERO gt e1.wellTyped() e1.type() INT)
(kind IF gt e1.wellTyped()
e1.type() BOOL
e2.wellTyped() e3.wellTyped()
e2.type() e3.type())
19
Approach

• Defining the language
• Searching the state space
• Representing the state space
• Reducing the state space

20
Searching

• Select an unchecked well typed state

if
iszero
if
false
0
false
false
true
21
Searching

• Check progress and preservation
• Progress Run one small step

if
iszero
if
false
void smallStep() throws StuckException
0
false
false
true
if
true
if
false
false
false
true
22
Searching

• Check progress and preservation
• Preservation Check if the result is well typed

if
iszero
if
false
0
false
false
true
if
_at_Declarative boolean wellTyped()
true
if
false
false
false
true
23
Searching

• Identify and prune similar states

PRUNED
24
Search Algorithm
Let W be the well typed state space While W is
not empty Choose a state s in W Run one step of
evaluation on s If Progress or Preservation
fail Print out s as a bug trace Let P be the
set of states similar to s W W P
25
Approach

• Defining the language
• Searching the state space
• Representing the state space
• Reducing the state space

26
Representing the State Space

• Represent a set as a boolean formula
• Encode each field as bits (b0, b1, )
• Constrain the bits using boolean operations

n1
n1.kind IF n1.e1.kind TRUE
(b0 ? b1 ? b2) ? b3 ? (b6 ? b7 ? b8)
n2
At AST height 5, a formula representing
2,504,790,381 well typed states is only 24,117
terms long.
27
Representing the State Space

• Initialize to the set of well typed states
• Construct a formula describing the type system

_at_Declarative boolean wellTyped() return
syntaxOk() ( kind TRUE gt true
) ( kind FALSE
gt true ) (
kind ZERO gt true
) ( kind SUCC gt e1.wellTyped()
e1.type() INT) ( kind PRED gt
e1.wellTyped() e1.type() INT) ( kind
ISZERO gt e1.wellTyped() e1.type() INT)
( kind IF gt e1.wellTyped()
e1.type() BOOL e2.wellTyped()
e3.wellTyped() e2.type() e3.type()
)
boolean formula
28
Representing the State Space

• Initialize to the set of well typed states
• Construct a formula describing the type system
• Declarative methods
• May not use object creations, assignments, or
  loops
• May use gt, forall, exists operators
• Declarative methods translate to small formulas

_at_Declarative boolean wellTyped() return
syntaxOk() ( kind TRUE gt true
) ( kind FALSE
gt true ) (
kind ZERO gt true
) ( kind SUCC gt e1.wellTyped()
e1.type() INT) ( kind PRED gt
e1.wellTyped() e1.type() INT) ( kind
ISZERO gt e1.wellTyped() e1.type() INT)
( kind IF gt e1.wellTyped()
e1.type() BOOL e2.wellTyped()
e3.wellTyped() e2.type() e3.type()
)
29
Search Algorithm
Let W be the well typed state space While W is
not empty Choose a state s in W Run one step of
evaluation on s If Progress or Preservation
fail Print out s as a bug trace Let P be the
set of states similar to s W W P
Use a SAT solver
Add P to the SAT solver
30
Approach

• Defining the language
• Searching the state space
• Representing the state space
• Reducing the state space
• Dynamic analysis
• Static analysis
• Other optimizations

31
Dynamic Analysis

• Discover and prune states that are similar
• Touched analysis
• Track which fields are accessed
• Symbolic execution
• Track which fields are accessed and how they are
  used

root.kind IF root.e1.kind ISZERO
root.e1.e1.kind ZERO
(root.kind IF root.kind ISZERO
root.kind SUCC root.kind PRED)
root.e1.kind ISZERO root.e1.e1.kind ZERO
32
Static Analysis
33
Static Analysis
class WhyStaticAnalysis private boolean a,
b public void smallStep() a
!a _at_Declarative public boolean wellTyped()
return a b
34
Static Analysis
class WhyStaticAnalysis private boolean a,
b public void smallStep() a
!a _at_Declarative public boolean wellTyped()
return a b
Choose a well typed state
35
Static Analysis
class WhyStaticAnalysis private boolean a,
b public void smallStep() a
!a _at_Declarative public boolean wellTyped()
return a b
Choose a well typed state
Check the transition
36
Static Analysis
class WhyStaticAnalysis private boolean a,
b public void smallStep() a
!a _at_Declarative public boolean wellTyped()
return a b
Choose a well typed state
Check the transition
Prune similar transitions
well typed
PRUNED
37
Static Analysis

• Dynamic analysis alone is not always correct!
• Need to perform additional checks
• Could check each pruned transition separately
• Negates the benefits of pruning
• We use static analysis to ensure correctness
• All pruned transitions checked together
• Any error within finite bounds is caught

38
Static Analysis
class WhyStaticAnalysis private boolean a,
b public void smallStep() a
!a _at_Declarative public boolean wellTyped()
return a b
wellTyped (a b)
Prestate of a pruned transition
wellTypedPre (a b)atrue true
wellTypedPost (a b)afalse b
wellTypedPre ? wellTypedPost
b
Poststate of the pruned transition
Not valid when b false!
39
Static Analysis

• Partially evaluate wellTyped formula
• Use SAT solver to check the above implication
• If the implication holds then pruning is sound
• If not, we have a counterexample
• Often, wellTypedPre and wellTypedPost are
  identical
• Checking the implication is trivial then

wellTypedPre
wellTypedPost
?
40
Other Optimizations

• Isomorphism pruning
• Prune more when checking heaps
• Handling term cloning
• Perform symbolic copy
• Handling term substitution
• Perform incremental substitution

41
Outline

• Motivation
• Example
• Approach
• Experimental Results
• Related Work
• Conclusions

42
Experimental Results

• We verified type soundness of five languages
• Expression Language
• Imperative Language
• Featherweight Java
• Mini Java
• Mini Java with Ownership Types

43
Experimental Results

• Imperative Language
• Simple language from Winskel93, Chapter 2
• Features variables, assignment, conditionals,
  loops
• Features arithmetic, relational, logical operators

int a a 5 while (a lt 7) a a1
44
Experimental Results

• Featherweight Java Igarashi99
• Subset of Java
• Features classes, inheritance, methods, fields

class A extends Object A() super()
class B extends Object B() super()
class Pair extends Object Object fst
Object snd Pair(Object fst, Object snd)
super() this.fstfst this.sndsnd Pair
setfst(Object newfst) return new
Pair(newfst, this.snd)
45
Experimental Results

• Mini Java
• Extends Featherweight Java
• Allows mutation of objects
• Features an explicit heap
• Features integers and booleans

class Node extends Object int value
Node(int value) super() this.value
value Node mutate() return
this.value this.value 1
46
Experimental Results

• Mini Java with Ownership Types
• Clarke98Boyapati03Aldrich02
• Types are parameterized with owners
• Enforces object encapsulation

class NodeltA,Bgt extends ObjectltAgt ObjectltBgt
data NodeltA,Bgt next Node(ObjectltBgt data,
NodeltA,Bgt next) super() this.data data
this.next next class StackltA,Bgt extends
ObjectltAgt Nodeltthis,Bgt root
Stack(Nodeltthis,Bgt root) super() this.root
root StackltA,Bgt push(ObjectltBgt data)
return this.root new Nodeltthis,Bgt(data,
this.root)
47
Experimental Results
Benchmark Max Expression Size well typed States (approx.) States Checked Time (s)
Expression Language 1 2 3 4 13 40 121 364 1093 3280 21 22 22 24 210 231 293 2280 2841 22523 1 3 3 5 11 17 23 29 35 41 0.068 0.093 0.105 0.122 0.246 0.551 1.376 3.633 10.833 38.543
IMP 1 2 3 4 5 6 7 15 31 63 127 255 511 23 25 28 29 212 214 217 236 275 2157 2327 2679 21402 1 7 11 19 34 34 34 61 96 147 230 377 652 0.102 0.185 0.256 0.408 0.710 0.739 0.816 2.158 5.107 10.066 21.013 52.208 331.138
Checks 2157 well typed IMP programs in about 10
seconds
48
Experimental Results
Benchmark Max Expression Size States Checked Time (s)
Featherweight Java 1 2 3 4 5 21 85 341 3 7 9 9 13 70 298 1210 1.148 1.594 1.650 1.899 2.151 6.905 43.756 475.022
Mini Java 1 2 3 4 5 21 85 341 5 21 40 53 59 275 1133 4565 2.721 3.117 3.897 5.750 6.191 37.354 342.435 5981.114
Mini Java with Ownership Types 1 2 3 4 5 21 25 29 33 13 73 110 135 157 733 877 1021 1165 50.818 77.135 103.230 231.328 247.954 2760.734 3963.836 5271.509 6255.260
49
Experimental Results

• We inserted 25 errors into Mini Java with
  Ownership Types

Max Expression Size Percentage of Errors Caught
1 2 3 4 5 6 7 8 0 8 40 68 76 80 84 100
50
Experimental Results

• We inserted 25 errors into Mini Java with
  Ownership Types

Max Expression Size Percentage of Errors Caught Time to Check the Entire Search Space (s)
1 2 3 4 5 6 7 8 21 25 29 33 0 8 40 68 76 80 84 100 100 100 100 100 50.818 77.135 103.230 231.328 247.954 339.632 403.249 836.568 2760.734 3963.836 5271.509 6255.260
51
Related Work

• State space reduction techniques
• Abstraction refinement SLAM Blast Magic
• Partial order reduction Godefroid97 Flanagan05
• Heap canonicalization Musuvathi05 Iosif02
• Symmetry reduction Ip93

52
Related Work

• Software model checkers
• Verisoft Godefroid97
• Java PathFinder Visser00
• CMC Musuvathi02
• Bandera Corbett00
• Bogor Dwyer05
• SLAM Ball01
• Blast Henzinger02
• Magic Chaki03
• Jalloy Vaziri03
• Miniatur Dolby07

53
Conclusions

• Fully automatic technique to check type soundness
• Counterexamples provided if soundness fails
• Useful to check small to medium sized languages
• Research languages
• Core subsets of complex languages