GILK: A Dynamic Instrumentation Tool for the Linux Kernel

1
GILK A Dynamic Instrumentation Tool for the
Linux Kernel

• David J. Pearce, Paul H.J. Kelly, Tony Field and
  Uli Harder
• d.pearce_at_doc.ic.ac.uk
• www.doc.ic.ac.uk/djp1/gilk.html

2
Introduction

• Do programmers really know what is going on?
• Need deeper understanding of program behaviour
• Debug/model/validate performance issues
• Program instrumentation can provide this
• GILK is about doing this for the Linux Kernel
• Adds instrumentation on the fly
• No need to restart
• No need to recompile
• No need to interrupt the web server!

3
Introduction

• Do programmers really know what is going on?
• Need deeper understanding of program behaviour
• Debug/model/validate performance issues
• Program instrumentation can provide this
• GILK is about doing this for the Linux Kernel
• Adds instrumentation on the fly
• No need to restart
• No need to recompile
• No need to interrupt the web server!

4
GILK An Introduction
5
GILK An Introduction
6
GILK An Introduction
7
Motivating Example

• Research into Self Similarity _at_ Imperial
• Using TCPDUMP to obtain network timing
  information
• Produced fair amount of data
• But occasional zero or negative inter-arrival
  time was spotted
• Are results so far wasted?
• Is TCPDUMP useable or not?
• GILK deployed for comparison with TCPDUMP
• GILK provides high resolution packet arrival
  times
• Previous results valid statistical properties
  unchanged
• TCPDUMP does generate erroneous results
• What We Did
• Next slide

8
Motivating Example

• Research into Self Similarity _at_ Imperial
• Using TCPDUMP to obtain network timing
  information
• Produced fair amount of data
• But occasional zero or negative inter-arrival
  time was spotted
• Are results so far wasted?
• Is TCPDUMP useable or not?
• GILK deployed for comparison with TCPDUMP
• GILK provides high resolution packet arrival
  times
• Previous results valid statistical properties
  unchanged
• TCPDUMP does generate erroneous results
• What We Did
• Next slide

9
Motivating Example

• Research into Self Similarity _at_ Imperial
• Using TCPDUMP to obtain network timing
  information
• Produced fair amount of data
• But occasional zero or negative inter-arrival
  time was spotted
• Are results so far wasted?
• Is TCPDUMP useable or not?
• GILK deployed for comparison with TCPDUMP
• GILK provides high resolution packet arrival
  times
• Previous results valid statistical properties
  unchanged
• TCPDUMP does generate erroneous results
• What We Did
• Next slide

10
What We Did
Linux Network Stack

BSD Socket Layer
sock_recvmsg

Ethernet Driver
speedo_rx
11
What We Did
Linux Network Stack

BSD Socket Layer
sock_recvmsg
RDTSC

Ethernet Driver
RDTSC
speedo_rx
12
Where was the Success?

• Were not Kernel Programmers
• Dont know beforehand what was needed
• Time wasted trying different instrumentation
• GILK is fast
• Turn around time only a few minutes
• Allows us to try many different instrumentations
• Fits exploratory nature of debugging

13
Where was the Success?

• Were not Kernel Programmers
• Dont know beforehand what was needed
• Time wasted trying different instrumentation
• GILK is fast
• Turn around time only a few minutes
• Allows us to try many different instrumentations
• Fits exploratory nature of debugging

14
Where was the Success?

• Were not Kernel Programmers
• Dont know beforehand what was needed
• Time wasted trying different instrumentation
• GILK is fast
• Turn around time only a few minutes
• Allows us to try many different instrumentations
• Fits exploratory nature of debugging

15
How it Works

• GILK uses code splicing technology
• Allows instrumentation of active kernel
• Instrumentation performed at machine code level
• Works on unmodified kernel image
• No patching, restarting, recompiling etc
• Instruments written in C
• Implemented as Kernel Modules
• Access to full language features
• Access to kernel functionality
• System Requirements
• Only Intel x86 Architecture supported
• Kernel versions 2.0 2.2 supported
• Extension to 2.4 is trivial

16
How it Works

• GILK uses code splicing technology
• Allows instrumentation of active kernel
• Instrumentation performed at machine code level
• Works on unmodified kernel image
• No patching, restarting, recompiling etc
• Instruments written in C
• Implemented as Kernel Modules
• Access to full language features
• Access to kernel functionality
• System Requirements
• Only Intel x86 Architecture supported
• Kernel versions 2.0 2.2 supported
• Extension to 2.4 is trivial

17
How it Works

• GILK uses code splicing technology
• Allows instrumentation of active kernel
• Instrumentation performed at machine code level
• Works on unmodified kernel image
• No patching, restarting, recompiling etc
• Instruments written in C
• Implemented as Kernel Modules
• Access to full language features
• Access to kernel functionality
• System Requirements
• Only Intel x86 Architecture supported
• Kernel versions 2.0 2.2 supported
• Extension to 2.4 is trivial

18
Instruction Boundaries
Before
After
splice
unused
instrumentation
Relocated sequence
19
Conclusion

• GILK quickly instruments Linux Kernel
• Runtime code splicing useful for instrumentation
• Much left unsaid, however
• Future work
• Simplifying instrumentation process
• GUI, Instrumentation Strategies, Visualization
• Programmatic instrumentation is possible
• Paradyn (WISC) attempts this
• Non-instrumentation based options
• Modifying Program Behaviour
• Non-Intel platforms?
• KernInst/Paradyn (WISC) on UltraSparc/Solaris
• User-Space programs?
• User-Space Threading is an issue
• Our Research group is now doing this for Java

20
Conclusion

• GILK quickly instruments Linux Kernel
• Runtime code splicing useful for instrumentation
• Much left unsaid, however
• Future work
• Simplifying instrumentation process
• GUI, Instrumentation Strategies, Visualization
• Programmatic instrumentation is possible
• Paradyn (WISC) attempts this
• Non-instrumentation based options
• Modifying Program Behaviour
• Non-Intel platforms?
• KernInst/Paradyn (WISC) on UltraSparc/Solaris
• User-Space programs?
• User-Space Threading is an issue
• Our Research group is now doing this for Java

21
Conclusion
d.pearce_at_doc.ic.ac.ukwww.doc.ic.ac.uk/djp1/gilk.
html

• GILK quickly instruments Linux Kernel
• Runtime code splicing useful for instrumentation
• Much left unsaid, however
• Future work
• Simplifying instrumentation process
• GUI, Instrumentation Strategies, Visualization
• Programmatic instrumentation is possible
• Paradyn (WISC) attempts this
• Non-instrumentation based options
• Modifying Program Behaviour
• Non-Intel platforms?
• KernInst/Paradyn (WISC) on UltraSparc/Solaris
• User-Space programs?
• User-Space Threading is an issue
• Our Research group is now doing this for Java