Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

1
Log Based Dynamic Binary Analysis for Detecting
Device Driver Defects

• Olatunji Ruwase
• Thesis Proposal

Thesis Committee Todd C. Mowry (Chair) David
Andersen Onur Mutlu Brad Chen (Google) Michael
Swift (U. Wisconsin)
2
Device Drivers The Good, The Bad, The Ugly

• Good Enable use of hardware devices
• Kernel module in commodity OS
• Distributed in binary form

• Bad Poor code quality Chou01, Murphy04
• Written by non kernel experts
• Poorly tested

• Ugly Major cause of system failures
• System crashes
• OS corruption
• Application corruption
• Device damage

3
Program Monitoring Using Lifeguards

• Lifeguards dynamic correctness checking tools
• Dynamic binary analysis to work on unmodified
  binaries
• Instruction grained analysis to catch subtle bugs
• Versatility to catch broad range of bugs
• Memory Nethercote07
• Security Newsome05, Castro05
• Concurrency Savage97, Yu05, Flanagan09
• Multilingual program interface Lee10

Can Lifeguards be used to catch Driver Bugs ?
4
Why Drivers Are Difficult To Write Correctly
Ryzhyk09_Dingo

• Concurrency issues
• Reentrant interrupt handling

User space
SYSTEM CALL BOUNDARY

• Interface issues
• Network stack
• Kernel resources
• Hardware device

Kernel space

• Generic C language issues
• Memory management

Synchronous main memory CPU registers
Lifeguards effectively detect similar spectrum of
issues in applications
Asynchronous I/O memory interrupts
5
Potential Uses of Driver Lifeguards

• Diagnosing system failures
• Test sites
• Customer sites
• Detecting silent faults
• Test sites
• Customer sites

6
Outline

• Motivation
• Overview of Lifeguard Deployment
• Thesis Question
• Related work
• Research Challenges
• Preliminary work
• Current and Future work
• Timeline

7
Lifeguard Deployment Approaches

• Dynamic Binary Instrumentation PIN, VALGRIND
• Fault isolation
• Imprecise checking of parallel execution

Monitored program
8
Lifeguard Deployment Approaches

• Dynamic Binary Instrumentation PIN, VALGRIND
• Fault isolation
• Imprecise checking of parallel execution

check_store (p)
p
p NULL
Multithreaded program
Monitored program

• Logging AFTERSIGHT, LBA, SPECK
• Monitor parallel execution
  Pokam09,Vlachos10
• Accelerate lifeguard executionChen08,Nightingale0
  8,Ruwase08,Ruwase10

• Require fault containment
• Protect Lifeguard
• Restrict damages to faulting program

Log Based Lifeguards are more promising for
monitoring kernel mode drivers
9
Thesis Questions

• Can Log Based Lifeguards precisely detect faults
  in the executions of device drivers ?
• Can Log Based monitoring be adapted for drivers ?
• Will the Lifeguards be efficient enough for
  production systems (Mobile, Desktop, Cloud) ?

10
Outline

• Motivation
• Overview of Lifeguard Deployment
• Thesis Question
• Related work
• Research Challenges
• Preliminary work
• Current and Future work
• Timeline

11
Eliminating Driver Faults During Development

• Avoid overheads of runtime fault detection or
  isolation
• Cannot find all faults in production drivers

• Static analysis Metal, RacerX, SLAM
• Drivers are too complex

SYSCALL BOUNDARY

• Testing DDT
• Drivers have too many execution paths

Upper layers of network stack

• Synthesize driver code Termite
• Cannot synthesize complex features e.g.
  multithreading

Driver

• Lifeguards to detect other faults
• Customer sites
• Testing sites

12
Using Existing Hardware to Isolate Driver Faults

• Prevent system failures due to driver faults
• Little information on driver faults

SYSCALL BOUNDARY
Upper layers of network stack

• Page table permissions Nooks
• User space drivers Microdrivers, SUD

Driver

• Lifeguards on customer systems
• Pinpoint fault location to aid debugging
• Detect silent driver faults

13
Checking Driver Execution to Isolate Faults

• Pinpoint fault location
• Detect silent faults

• Instrumented software checks SafeDrive,XFI,BGI
  
• Imprecise on parallel execution
• Only memory faults studied
• Logging works for parallel execution
• Lifeguards for high level faults

SYSCALL BOUNDARY
Upper layers of network stack

• Hardware breakpoints DataCollider
• Sampling approach misses real faults
• Lifeguard finds all faults in execution

Driver
14
Related Work Summary

• Eliminating Driver faults during development
• Static analysis Metal, RacerX, SLAM
• Testing DDT
• Synthesizing driver code Termite
• Using existing hardware to isolate Driver faults
• Page table permissions Nooks
• User space drivers Microdrivers, SUD
• Checking Driver execution to isolate faults
• Instrumented software checks SafeDrive, XFI,
  BGI
• Hardware breakpoints DataCollider

15
Outline

• Motivation
• Overview of Lifeguard Deployment
• Thesis Question
• Related work
• Research Challenges
• Preliminary work
• Current and Future work
• Timeline

16
Research Challenges

• Preliminary work
• Adapting Log Based Monitoring for Drivers
• Understanding Device Drivers
• Current and Future work
• Detecting Common Driver Faults (Driver
  Lifeguards)
• Efficiency of Driver Lifeguards

17
Log Based Architectures (LBA) Chen 08
Program

Operating System
Hardware Log
Simulated LBA Design

• Execution logging
• Toggle when monitored thread (de)scheduled
• Fault containment
• Lifeguard as separate process
• Block program at system calls until Lifeguard
  catches up

18
Adapting Execution Logging for Driver Monitoring

• Toggle point
• Difficulty
• Complete information for precise fault detection
• Efficient
• Modest storage and bandwidth costs
• No lifeguard filtering costs

SYSTEM CALL BOUNDARY
Network stack
Upper layers of network stack
Driver
19
Adapting Execution Logging for Driver Monitoring
AFTERSIGHT
DIIFICULTY
Option Toggle Complete Efficient
Kernel Ring change ? ?
I/O stack
Driver
SYSTEM CALL BOUNDARY
Network stack
Upper layers of network stack
Driver
20
Adapting Execution Logging for Driver Monitoring
DIIFICULTY
Option Toggle Complete Efficient
Kernel Ring change ? ?
I/O stack I/O syscall ? ?
Driver
SYSTEM CALL BOUNDARY
Network stack
Upper layers of network stack
Driver
21
Adapting Execution Logging for Driver Monitoring
DIIFICULTY
Option Toggle Complete Efficient
Kernel Ring change ? ?
I/O stack I/O syscall ? ?
Driver Code region ? ?
SYSTEM CALL BOUNDARY
Network stack
Upper layers of network stack
Identify driver entry points at load time
Driver
22
Adapting Fault Containment for Driver Monitoring
Operating System
Hardware Log

• Execution logging
• Toggle when monitored thread (de)scheduled
• Fault containment
• Lifeguard as separate process
• Block program at system calls until Lifeguard
  catches up

23
Adapting Fault Containment for Driver Monitoring
OS
OS
Hardware Log

• Virtual Machine (VM) separation to protect
  Lifeguard AFTERSIGHT
• Rest of system remain vulnerable to driver faults
• Overhead of VM is high

24
Understanding Device Drivers
hard_start_xmit()
irq_handler()
SYSCALL BOUNDARY
open() stop() get_stats() ...
Upper layers of network stack
PCI Bus Functions
probe() remove()
Driver
PCI
Required Functions
module_init() module_cleanup()
25
Adapting Data Race Lifeguard for Network Drivers
Thread 1 Write (X)
Thread 2 Read (X)
Fork (Thread2)

• Data race on X
• Two access on X where at least one access is a
  write
• No explicit synchronization between the accesses
• Lockset algorithm for detecting races in
  applications Eraser
• Shared data protected with consistent set of
  locks
• Happens-before relation for non-lock synch. (e.g
  fork) RaceTrack

Lockset kernel synch (interrupts, spinlocks)
KernelEraser
26
Network Driver Races Reported by KernelEraser
Classification of Races
Driver Serious Benign False Alarm False Alarm Total
Net stack synch. Device synch.
tg3 2 15 13 1533 1563
tulip 0 0 472 451 923
Fixed in versions 2.6.18 2.6.21

• Simulated LBA environment
• Kernel version Linux 2.6.17.1
• Drivers tg3 tulip
• Driver class Network
• Bus PCI
• Driver VM 2 CPU
• Lifeguard VM 1 CPU

• Workload
• Load driver
• Enable Ethernet
• Transfer file over network
• Disable Ethernet
• Unload driver

27
False Alarms due to Unobserved Invariants
SYSCALL BOUNDARY

• Synchronizations in upper layers of I/O stack
• Synchronizations due to device states

Upper layers of network stack
stop () while(tp?tg3_flags )
tg3
PCI
28
False Alarms due to Unobserved Invariants
SYSCALL BOUNDARY

• Synchronizations in upper layers of I/O stack
• Synchronizations due to device states

Upper layers of network stack
probe() tp?tg3_flags
tg3
PCI
29
Preliminary Work Summary

• Adapted Log Based Monitoring for Drivers
• Identify driver code region to log only driver
  execution
• VM separation to protect Lifeguard
• Adapted Lockset (KernelEraser) to detect races in
  network drivers
• Found 2 known but serious data races in tg3
• False alarms due to external synchronizations

30
Outline

• Motivation
• Overview of Lifeguard Deployment
• Thesis Question
• Related work
• Research Challenges
• Preliminary work
• Current and Future work
• Timeline

31
Eliminating False Alarms in KernelEraser

• External synchronizations
• Network stack
• Log network stack
• Emulate interface invariants

SYSTEM CALL BOUNDARY
Network stack
Upper layers of network stack
Driver
32
Eliminating False Alarms in KernelEraser

• External synchronizations
• Network stack
• Log network stack
• Emulate interface invariants
• Device
• Model finite state machine

SYSTEM CALL BOUNDARY
Network stack
Upper layers of network stack
Driver Serious Benign False Alarm False Alarm Total
Net stack synch. Device synch.
tg3 2 15 0 0 17
tulip 0 0 0 0 0
Driver
33
Eliminating False Alarms in KernelEraser

• External synchronizations
• Network stack
• Log network stack
• Emulate interface invariants
• Device
• Model finite state machine
• Other driver classes
• SCSI disk
• SOUND
• USB
• GRAPHICS

SYSTEM CALL BOUNDARY
Network stack
Upper layers of network stack
Driver
34
Lifeguards for Common Driver Faults
Ryzhyk09_Dingo
User space

• Concurrency faults
• Data Races

SYSTEM CALL BOUNDARY

• Memory faults
• Illegal memory access
• Memory leaks
• Uninitialized memory use

Kernel space

• Interface violations
• Device protocol
• Kernel protocol
• I/O stack protocol

Scalability ?
35
Efficiency of Driver Lifeguards

• Accelerating Lifeguard analysis
• Static analysis
• Dynamic optimizations
• Parallel Lifeguards
• Hardware accelerators
• Reduce overhead of VM fault containment
• Hardware enforced fault isolation in same VM

36
Accelerating Driver Lifeguards

• Reduce analysis workload
• Static analysis XFI

OS
OS
Hardware Log
37
Accelerating Driver Lifeguards

• Reduce analysis workload
• Static analysis XFI

OS
OS
Hardware Log

• Run analysis faster
• Dynamic compiler optimizations Qin06,Ruwase10
• Parallel Lifeguards Nightingale08,Ruwase08
• Hardware accelerators Vlachos10

38
Avoid Overhead of VM Fault Containment
User space
SYSTEM CALL BOUNDARY

• Hardware enforced fault isolation Nooks, SUD
• Issues to consider
• Protection quality
• Lifeguard using Driver (e.g. disk)

Kernel space
39
Current and Future Work Summary

• Detecting common driver faults
• Data races
• Memory
• Interface violations
• Efficiency of Driver Lifeguards
• Accelerating Lifeguard analysis
• More efficient fault containment

40
Outline

• Motivation
• Overview of Lifeguard Deployment
• Thesis Question
• Related work
• Research Challenges
• Preliminary work
• Current and Future work
• Timeline

41
Timeline
42
Questions ?

• Thanks to members of the LBA Group for their
  contributions
• Shimin Chen
• Babak Falsafi
• Phillip Gibbons
• Michelle Goodstein
• Michael Kozuch
• Onur Mutlu
• Todd Mowry
• Gennady Pekhimenko
• Vivek Seshadri
• Theodoros Strigkos
• Evangelos Vlachos