Data Structure Repair Using Goal-Directed Reasoning

1
Data Structure Repair Using Goal-Directed
Reasoning

• Brian Demsky
• Martin Rinard
• Computer Science and
• Artificial Intelligence Laboratory
• Massachusetts Institute of Technology

2
Problem
Broken Data Structure

• Errors
• Missing elements
• Inappropriate sharing
• Dangling references
• Out of bounds array indices
• Inconsistent values

F 20 G 5
F 20 G 10
I 5
J 2
3
Solution
Broken Data Structure
Consistent Data Structure
F 10 G 5
F 2 G 1
F 20 G 10
F 20 G 5
F 20 G 10
Repair Algorithm
I 3
I 5
J 2
J 2
4
OOPSLA 2003
Broken Abstract Model
Repaired Abstract Model
Abstract Repair
Model Definition Rules
External Consistency Constraints
10111001011 10101011101 10101110110
00011001011 10101011101 10101110110
Broken Bits
Repaired Bits
5
Current Version
Broken Abstract Model
Repaired Abstract Model
Abstract Repair
Model Definition Rules
External Consistency Constraints
10111001011 10101011101 10101110110
00011001011 10101011101 10101110110
Broken Bits
Repaired Bits
6
Why Eliminate External Consistency Constraints?

• Development overhead of external consistency
  constraints
• Possibility of errors in the external consistency
  constraints
• Difficulties ensuring that repaired model
  corresponds to a concrete data structure

7
How Did We Do It?

• Goal-directed reasoning replaces external
  consistency constraints
• Start with repair that makes model consistent
• Goal implement this repair in data structures
• Reasoning analyze model definition rules
• Result data structure updates that implement
  abstract repairs

8
Result
Broken Abstract Model
Repaired Abstract Model
Abstract Repair
Model Definition Rules
External Consistency Constraints
10111001011 10101011101 10101110110
00011001011 10101011101 10101110110
Broken Bits
Repaired Bits
9
Result
Broken Abstract Model
Repaired Abstract Model
Abstract Repair
Model Definition Rules
External Consistency Constraints
10111001011 10101011101 10101110110
00011001011 10101011101 10101110110
Automatically Generated Concrete Data Structure
Updates
Broken Bits
Repaired Bits
10
File System Example
intro

-5
2
-1

-1

3

-1
Directory Entries
Disk Blocks
struct block int nextblock byte
datablocksize struct blockbitmap subtype
block int nextblock bit bitmapnumblocks

• struct disk
• int blockbitmap
• entry dirnumentries
• block blocknumblocks
• struct entry
• byte nameLength
• int firstblock

11
File System Model
Block
BlockStatus
Next
boolean
Used
Free
Bitmap

• Sets of objects
• set Block of block Used Free
• set Used of block Bitmap
• Relations between objects
• relation Next Used, Used
• relation BlockStatus Block, boolean

12
Model Translation

• Bits translated to sets and relations in abstract
  model using statements of the form
• Quantifiers, Condition gt Inclusion Constraint
• ?i ? 0..numentries-1, 0 ? d.diri.firstblock ?
  
• d.blockd.diri.firstblock ? Used
• ?b ? Used, 0 ? b.nextblock ? ?b,d.blockb.nextbloc
  k? ? Next
• ?b ? Used, 0 ? b.nextblock ? d.blockb.nextblock
  ? Used
• ?b in 0..numblocks-1, d.blockb ? Used ?
  d.blockb ? Free
• true ? d.blockd.blockbitmap ? Bitmap
• ?j ? 0..numblocks-1, ?b ? Bitmap, true gt
  ltd.blockj,b.bitmapjgt ? BlockStatus

13
Model for File System Example
intro

-5
2
-1

-1

3

-1
Directory Entries
Disk Blocks
Blocks
Used
2
3
Free
Next
1
0
Bitmap
14
Consistency Constraints in Example

• Bitmap1
• ?u ? Used, u.BlockStatustrue
• ?f ? Free, f.BlockStatusfalse
• ?b ? Used, Next.b ? 1

Blocks
Used
2
3
Free
Next
1
0
Bitmap
15
Detecting Inconsistencies

• Evaluate consistency properties, find violations
• Bitmap1 is violated - Bitmap set is empty

Blocks
Used
2
3
Free
Next
1
0
Bitmap
16
Repairing Violations of Model Consistency
Properties

• Violation provides binding for quantified
  variables
• Convert Body to disjunctive normal form
• (p1 ? ? pn ) ? ? (q1 ? ? qm )
• p1 pn , q1 qm are basic propositions
• Choose a conjunction to satisfy
• Repair violated basic propositions in conjunction

17
Repairing Violations of Basic Propositions

• Inequality constraints on values of numeric
  fields
• V.R E, V.R lt E, V.R ? E, V.R ? E, V.R gt E
• Compute value of expression, assign relation
• Presence of required number of objects
• S C, S ? C, S ? C
• Remove or insert objects from/to set
• Topology of region surrounding each object
• V.R C, V.R ? C, V.R ? C
• R.V C, R.V ? C, R.V ? C
• Remove or insert tuples from/to relation
• Inclusion constraints V in S, V1 in V2.R,
  ?V1,V2? in R
• Remove or add the object or tuple from/to set or
• relation

18
Repairing Inconsistencies

• Repair the violation of Bitmap1 by adding a
  block
• to the Bitmap set

Blocks
Used
2
3
Free
Next
1
0
Bitmap
19
Goal-Directed Reasoning Translates Abstract
Repairs Into Concrete Repairs

• Abstract repairs add or remove objects (or
  tuples) to sets (or relations)
• Goal find concrete data structure updates with
  same effect
• Find model definition rules that construct the
  relevant set or relation
• Basic strategy
• For removals, appropriately falsify the guards
  of all these model definition rules.
• For additions, appropriately satisfy the guard
  of one of these model definition rules.

20
Goal-Directed Reasoning in Example

• Abstract Repair add block 0 to the Bitmap set

21
Goal-Directed Reasoning in Example

• Abstract Repair add block 0 to the Bitmap set
• Model Definition Rules
• ?i ? 0..numentries-1, 0 ? d.diri.firstblock ?
  
• d.blockd.diri.firstblock ? Used
• ?b ? Used, 0 ? b.nextblock ? ?b,d.blockb.nextbloc
  k? ? Next
• ?b ? Used, 0 ? b.nextblock ? d.blockb.nextblock
  ? Used
• ?b in 0..numblocks-1, d.blockb ? Used ?
  d.blockb ? Free
• true ? d.blockd.blockbitmap ? Bitmap
• ?j ? 0..numblocks-1, ?b ? Bitmap, true gt
  ltd.blockj,b.bitmapjgt ? BlockStatus

22
Goal-Directed Reasoning in Example

• Abstract Repair add block 0 to the Bitmap set
• Model Definition Rules
• ?i ? 0..numentries-1, 0 ? d.diri.firstblock ?
  
• d.blockd.diri.firstblock ? Used
• ?b ? Used, 0 ? b.nextblock ? ?b,d.blockb.nextbloc
  k? ? Next
• ?b ? Used, 0 ? b.nextblock ? d.blockb.nextblock
  ? Used
• ?b in 0..numblocks-1, d.blockb ? Used ?
  d.blockb ? Free
• true ? d.blockd.blockbitmap ? Bitmap
• ?j ? 0..numblocks-1, ?b ? Bitmap, true gt
  ltd.blockj,b.bitmapjgt ? BlockStatus

23
Goal-Directed Reasoning in Example

• Abstract Repair add block 0 to the Bitmap set
• Relevant Model Definition Rule
• true ? d.blockd.blockbitmap ? Bitmap
• d.blockd.blockbitmapblock 0

24
Goal-Directed Reasoning in Example

• Abstract Repair add block 0 to the Bitmap set
• Relevant Model Definition Rule
• true ? d.blockd.blockbitmap ? Bitmap
• d.blockd.blockbitmapblock 0
• Data Structure Update d.blockbitmap
  index of block 0 in d.block array

25
Repair in Example
Original File System
intro

-5
2
-1

-1

3

-1
Directory Entries
Disk Blocks
Updated File System
intro

0
2
-1

-1

3

-1
block bitmap
Directory Entries
Disk Blocks
26
Reasoning at Compile Time

• Compile specifications into repair algorithms
• Goal-directed reasoning takes place at compile
  time
• Consider possibility that Bitmap 0
• Abstract repair
• Choose a block in Free set
• Add block to Bitmap set
• Concrete repair
• Find relevant model definition rule
• true ? d.blockd.blockbitmap ? Bitmap
• Goal-directed reasoning finds following update
• d.blockbitmap index of block in d.block array
• Check that block is an element of d.block array
  ?b in 0..numblocks-1, d.blockb ? Used ?
  d.blockb ? Free

27
Multiple Repairs

• Some broken data structures may require multiple
  repairs
• Reconstruct model
• Reevaluate consistency constraints
• Perform any required additional repairs

28
Architecture
Broken Abstract Model
Repaired Abstract Model
Abstract Repair
. . . .
Model Translation
10111001011 10101011101 10101110110
01011001011 10101011101 10101110110
00011001011 10101011101 10101110110
. . . .
Automatically Generated Concrete Repair
Broken Bits
Repaired Bits
29
Model Recomputation
BlockStatus
Blocks
true
Used
2
3
Free
Next
false
1
Bitmap
0
30
Model Recomputation

• Re-evaluate constraints, find violations of
• ?u ? Used, u.BlockStatustrue and
• ?f ? Free, f.BlockStatusfalse

BlockStatus
Blocks
true
Used
2
3
Free
Next
false
1
Bitmap
0
31
Model Recomputation

• Repair violations of
• ?u ? Used, u.BlockStatustrue and
• ?f ? Free, f.BlockStatusfalse
• by modifying the BlockStatus relation

BlockStatus
Blocks
true
Used
2
3
Free
Next
false
1
Bitmap
0
32
Repaired File System
Repaired File System
intro
1011
0
2
-1

-1

3

-1
block bitmap
Directory Entries
Disk Blocks
33
Acyclic Repair Dependences

• Questions
• Isnt it possible for the repair of one
  constraint to invalidate another constraint?
• What about infinite repair loops?
• What about unsatisfiable specifications?
• Answer
• We require specifications to have no cyclic
  repair dependences between constraints
• So all repair sequences terminate
• Repair can fail only because of resource
  limitations

34
Repair Dependence Graph
4. Satisfy Rule 6 (BlockStatus)
1. Bitmap1
2. Add block to Bitmap
5. f.BlockStatusfalse
3. d.blockbitmapindexof(bfree)
6. Replace ltf,truegt with ltf,falsegt in BlockStatus
8. Remove ltf,truegt from BlockStatus by removing
Bitmap
7. b.bitmapjfalse for jindexof(f)
35
Repair Dependence Graph
4. Satisfy Rule 6 (BlockStatus)
1. Bitmap1
2. Add block to Bitmap
5. f.BlockStatusfalse
3. d.blockbitmapindexof(bfree)
6. Replace ltf,truegt with ltf,falsegt in BlockStatus
8. Remove ltf,truegt from BlockStatus by removing
Bitmap
7. b.bitmapjfalse for jindexof(f)
36
Repair Dependence Graph
4. Satisfy Rule 6 (BlockStatus)
1. Bitmap1
2. Add block to Bitmap
5. f.BlockStatusfalse
3. d.blockbitmapindexof(bfree)
6. Replace ltf,truegt with ltf,falsegt in BlockStatus
7. b.bitmapjfalse for jindexof(f)
37
When to Test for Consistency and Repair

• Persistent data structures
• Repair can be independent activity, or
• Repair when data written out or read in
• Volatile data structures in running program
• Under programmer control
• Transaction-based approach
• Identify transaction start and end
• Repair at start, end, or both
• Failure-based approach
• Wait until program fails
• Repair and restart from latest safe point

38
Experience

• We acquired five benchmarks (written in C/C)
• AbiWord
• x86 emulator
• CTAS (air-traffic control tool)
• Simplified Linux file system
• Freeciv interactive game
• We developed specifications for all five
• Little development time (days, not weeks)
• Most of time spent figuring out Freeciv and CTAS
• Each benchmark has
• Workload
• Bug or fault insertion methodology
• Ran benchmarks with and without repair

39
AbiWord

• Open-source word processing program
• Approximately 360,000 lines of C code
• Abiword represents documents using a Piece table
• Consistency properties
• Piece table has a section fragment
• Piece table has a paragraph fragment
• Doubly-linked list of fragments is well formed

40
AbiWord Screen Shot
41
Results

• Workload import (valid) Microsoft Word document
  that crashes AbiWord
• Bug that creates inconsistent documents with a
  text fragment before the section fragment
• Without repair
• AbiWord crashes when loading the document
• With repair
• AbiWord is able to open and successfully process
  the document

42
Parallel x86 emulator

• Parallel x86 emulator for the RAW machine
• Multi-tile architecture
• Emulator runs x86 binaries on RAW
• Contains L2 cache of translated x86 assembly
  instructions
• Maintains a constant L2 cache size
• Consistency property
• Computed size of the L2 cache is consistent with
  its actual size

43
Results

• Workload gzip benchmark on x86 emulator
• Bug that (sometimes) adds the size of a cache
  item twice when it is inserted
• Without repair
• Actual cache size goes to zero
• x86 emulator crashes
• With repair
• Actual cache size is the same as computed size
• Program runs correctly

44
CTAS

• Set of air-traffic control tools
• Traffic management
• Arrival planning
• Flow visualization
• Shortcut planning
• Deployed in centers around country (Dallas/Ft.
  Worth, Los Angeles, Denver, Miami,
  Minneapolis/St. Paul, Atlanta, Oakland)
• Approximately 1 million lines of C/C code

45
CTAS Screen Shot
46
Results

• Workload recorded radar feed from DFW
• Fault insertion
• Simulate error in flight plan processing
• Bad airport index in flight plan data structure
• Without repair
• System crashes segmentation fault
• With repair
• Aircraft has different origin or destination
• System continues to execute
• Anomaly eventually flushed from system

47
Aspects of CTAS

• Lots of independent subcomputations
• System processes hundreds of aircraft problem
  with one should not affect others
• Multipurpose system
  (visualization, arrival planning, shortcuts, )
  problem in one purpose should not affect others
• Sliding time window anomalies eventually flushed
• Rebooting ineffective system will crash again
  as soon as it sees the problematic flight plan

48
Simplified Linux File System


intro
110



0

1011













directory block
inode bitmap block
block bitmap block
inode
inode

super block
group block
disk blocks
inode block

• Some Consistency Properties
• inode bitmap consistent with inode usage
• block bitmap consistent with block usage
• directory entries refer to valid inodes
• files contain valid blocks only
• files do not share blocks

49
Results

• Workload write and verify several files
• Simulated power failure
• Inode and block bitmap errors
• Partially initialized directory and inode entries
• Without repair
• Incorrect file contents because of inode and disk
  block sharing
• With repair
• Bitmaps repaired preventing illegal sharing,
  correct file contents

50
Freeciv
Terrain Grid

• Consistency Properties
• Tiles have valid terrain values
• Cities are not in the ocean
• Each city has exactly one reference from the grid

O Ocean
P
O
M
M
P Plain
O
O
M
P
M Mountain
P
O
M
M
City Structures
P
P
M
P
51
Freeciv Screen Shot
52
Results

• Workload Freeciv software plays against itself
• Fault insertion randomly corrupt terrain values
• Without repair program crashes (seg fault)
• With repair
• Game runs just fine
• But game plays out differently because of the
  different terrain values

53
Benefits of Eliminating External Consistency
Constraints

• Simplifies AbiWord specification
• Without goal-directed reasoning, need additional
  model constraints
• Shortens specifications
• Linux file system and FreeCiv specifications are
  14 shorter
• Removes possibility of errors in external
  consistency constraints
• Removes possibility of repaired model with no
  corresponding data structure

54
Repair in Conjunction With Type Checking

• Suppose we have a type system to check a property
• Data structure repair is still useful!!!

55
Hardware Errors

• Hardware errors can corrupt memory and violate
  type safety
• Corrupted data structures may not have the
  properties that the type system guarantees
• Corrupted data structures can be used by
  malicious programs to perform arbitrary memory
  operations (Appel)
• Can we use dynamic checking to fix these
  problems?

56
Dynamic Checking

• Obvious approach dynamically verify that data
  structures are well typed
• Too restrictive for well-behaved programs
• No opportunity for the program to shutdown or
  otherwise tolerate the fault

57
Data Structure Repair for Hardware Errors

• Repairs data structure to maintain type safety
• Prevents malicious programs from exploiting type
  errors
• Safely allow program to continue to execute
• Allow programs to take actions to safely shutdown
  or tolerate fault

58
Multiple Languages

• Often build software systems using multiple
  languages
• Part of the system is type checked, part of the
  systems is not checked
• Unchecked part of the system can damage the data
  structure
• Have no guarantees about even the type checked
  portion of the system

59
Multiple Languages

• Write consistency specifications at boundaries of
  checked and unchecked code
• Repair ensure that state created or modified by
  the unchecked code is type consistent
• Protects type guarantees in the checked portion
  of the code

60
Related Work

• Hand-coded repair
• Lucent 5ESS switch
• IBM MVS operating system
• Integrity Maintenance in Databases
• Deriving Production Rules for Constraint
  Maintenance (Ceri, Widom)
• Automatic Generation of Production Rules for
  Integrity Maintenance (Ceri et al)
• Constraint analysis A design process for
  specifying operations on objects (Urban et al)
• Consistency management with repair actions
  (Nentwich et al)

61
Related Work

• Constraint mechanisms in programming languages
• Kaleidoscope (Lopez)
• Alphonse (Hoover)
• Self-stabilizing algorithms (Dijkstra)
• Log-based recovery for database systems
• Recovery-oriented computing
• Microrecovery Microreboot (Candea,Fox)
• Undo framework (Brown,Patterson)
• Specification Languages
• Alloy (Jackson)
• UML

62
Conclusion

• Data structure repair exciting way to
  (potentially) improve reliability
• Specification-based approach promises to make
  technique more widely applicable
• Moving towards more robust, probabilistic,
  continuous concept of system behavior

63
Implementation

• Size of system 26,200 lines
• Compiler
• 20,400 lines of Java code
• 2,500 lines of parser definitions
• Runtime - 3,200 lines of C code

64
Time to Check Consistency Perform Repairs
Application Time to Check Consistency(ms) Time to Check and Repair (ms)
AbiWord 0.06 0.55
CTAS 0.07 0.15
FreeCiv 3.62 15.66
File system 4.22 263.14
65
Lines of Code
Application Lines of Code
AbiWord 360,000
x86 emulator 65,000
CTAS gt1 million
FreeCiv 73,000
File system 700
66
ICSE 2005 (this paper)

• Use goal directed reasoning to eliminate external
  consistency constraints
• Benefits
• Eliminate need for model constraints to ensure
  the repaired model corresponds to a data
  structure
• Eliminate the possibility of errors in the
  external consistency constraints
• Eliminate developer overhead of writing external
  consistency constraints