InProcDiskSim Testing Database Recovery on Commodity Disk Drives

1
InProcDiskSim Testing Database Recovery on
Commodity Disk Drives

• Robin Dhamankar, Hanuma Kodavalla
• Microsoft SQL Server

2
Problem

• DBMS deployments increasingly use commodity disks
• E.g. Datacenters for Internet Services
• Commodity disks dont support write-through
• Disk controller caches and reorders writes
• Database recovery requires write-ahead logging
  (WAL)
• WAL is enforced using write-through
• gt Recovery on commodity disks maybe compromised

3
Solution Force Flush
Dhamankar et al. MSR-TR-2008-36

• Enforce write-ordering using FLUSH_CACHE
• FLUSH_CACHE command forces the contents of the
  controller cache to be written to the platter
• Issue FLUSH_CACHE between I/Os that need ordering
• FLUSH_CACHE between log and data to enforce WAL
• But use FLUSH_CACHE selectively
• FLUSH_CACHE is expensive
• One FLUSH_CACHE benefits several order dependent
  I/Os

4
Testing Challenge

• Typical recovery testing involves
• Induce server crash at interesting points during
  runtime
• Verify recoverability and consistency after
  re-start
• Crashing the server process is not sufficient
• Data in disk cache lost only in a machine crash
• Test will need programmatic control on machine
  crash
• Programmatic machine crash is not sufficient
  either
• Need control over cache contents
• Disk maybe aggressive in flushing contents

5
Alternatives

• Use programmable power strips
• Tests can control when the machine crashes
• Expensive
• No control on what is in cache and what is on
  disk
• Simulate in the driver stack
• Can control cache contents and flush policy
• No need for specialized hardware
• May affect other applications
• Difficult to perform database context-specific
  re-ordering

6
Our Solution InProcDiskSim Framework

• Simulate the disk cache in the same process as
  the database engine
• Advantages
• Inexpensive No need for specialized hardware
• Predictable Can control cache contents and
  flush policy
• Efficient Machine crash can be simulated by
  process crash
• Flexible Context-specific re-ordering is easy
  in-proc
• Isolated Does not affect any other process on
  the machine

7
Architecture
Log Manager
Buffer Manager

• InProcDiskSim lies between
• DBMS (Database file management subsystem)
• and
• OS (File system APIs)
• Consists of three modules
• Control Module
• I/O Interception Module
• Disk Cache Simulator

Database File Management
Database Engine
InProcDiskSim
I/O Interception Module
Test Hooks
Control Module
Disk Cache Simulator
OS Libraries
File system APIs for File I/O
I/O Completion Notification
OS Kernel
File system
8
Architecture (Contd.)
Log Manager
Buffer Manager

• Control Module
• External interface of the framework
• Provides programming surface for the test program

Database File Management
Database Engine
InProcDiskSim
I/O Interception Module
Test Hooks
Control Module
Disk Cache Simulator

• Provides knobs to selectively cache writes
• Data pages vs. log content
• Special-case pages such as boot page or file
  headers etc.

OS Libraries
File system APIs for File I/O
I/O Completion Notification
OS Kernel
File system
9
Architecture (Contd.)
Log Manager
Buffer Manager

• I/O Interception Module
• Intercepts I/Os from the engine
• Asks the control module if I/O should be cached
• If cached sends it to the disk cache simulator
• Else, send directly to
• the disk
• 5. When read I/O completes, it is intercepted
• 6. Disk cache simulator may contain the more
  recent data

Database File Management
Database Engine
InProcDiskSim
1
I/O Interception Module
Test Hooks
3
2
Control Module
6
Disk Cache Simulator
5
4
OS Libraries
File system APIs for File I/O
I/O Completion Notification
OS Kernel
File system
10
Architecture (Contd.)
Log Manager
Buffer Manager

• Disk Cache Simulator
• 8-16 MB in-memory buffer with fixed size cache
  lines.
• Write operations Data is written to the cache
  and I/O completion is signaled.
• Read operations Fully or partially serviced
  from the cache
• May have to evict contents to when cache is full
• FLUSH_CACHE flushes the entire cache

Database File Management
Database Engine
InProcDiskSim
Read and Write
I/O Interception Module
Test Hooks
Control Module
Disk Cache Simulator
Evict and Flush
OS Libraries
File system APIs for File I/O
I/O Completion Notification
OS Kernel
File system
11
Test DevelopmentStatic Tests
Sample Static test TestWAL () EnableCaching
(LOG) BeginTransaction() InsertData()
//Leave Transaction Open ForceDataPagesToDisk
() CrashServer() // Verification
RestartServer() VerifyDBRecovery()
VerifyDBConsistency()

• Test specific algorithms and scenarios
• Use control options to construct scenarios with
  re-ordering and crashes

• Static Tests not sufficient
• Cover what is known a priori. May miss test cases
• Will not cover newly added code

12
Test DevelopmentDynamic Tests

• Methodology
• Run existing functional/stress tests with good
  coverage
• Enable write caching using InProcDiskSim
• Periodically crash the process to induce failure
• Verify that the database recovers and is
  consistent (using database consistency checks)
• Provide comprehensive coverage
• Ensure on-going coverage through code churn

13
Observations

• InProcDiskSim is suitable for high concurrency
• Low overhead thus preserves workload timing
  characteristics
• Easy to debug
• Traces context specific actions performed when
  test executes
• No false failures
• Test failure or a detected corruption was a
  product bug
• Efficacy of dynamic Tests
• Effective where it is difficult to enumerate all
  the ordering requirements a priori
• E.g. Database File Management

14
Extending beyond Disk Cache

• Key Ideas
• Intercept I/Os in-proc
• Simulate context-specific storage characteristics
• Just like disk cache, we can simulate
• Bit Flips
• Torn writes
• I/O latency

InProcDiskSim
I/O Interception Module
Test Hooks
Feature Specific Module
Control Module
Disk Cache Simulator
15
Summary

• Motivate the complexity of test task at hand
• Examine other alternatives and discuss their
  limitations
• Describe the proposed InProcDiskSim framework
• Illustrate how InProcDiskSim enables
  comprehensive testing
• Discuss how InProcDiskSim can be applied to
  similar problems

16
Thanks!

• Questions?