Improve Software Quality with Fault Injection

1
Improve Software Quality with Fault Injection

• Rob Grzywinski
• rgrzywinski_at_yahoo.com

2
Speaker Introduction

• Started writing software interfaces for
  scientific equipment
• Managed a successful consulting company
• Exposure to projects and products running from
  content management / publishing to security
• Forced a hard look at effective software
  maintenance
• Technology due diligence for Silicon Valley VCs
• Vastly increased exposure to software
  technologies
• Currently working as a startup manager
• A Business-Centric Approach to Information
  Security
• Focuses on risk management

3
Motivation

• Writing and testing an IO-intensive application
• Robust, asynchronous event (message) collector
• NIO (early 1.4 days)
• How to best simulate and test various failure
  modes?
• NIO buggy as all get-out
• Networks are inherently flaky and problematic
• Didnt want to have to obfuscate or drastically
  change code in order to test it

4
Agenda

• Quick Introduction to Fault Injection
• Why Use Fault Injection
• Fault Injection Examples
• Beyond Fault Injection
• Quick Introduction to AOP
• Fault Injection in Java
• AspectJ
• Javassist
• Wrap Up

5
Fault Injection 101

• A fault is an abnormal condition or defect which
  may lead to a failure 10
• Fault injection involves the deliberate insertion
  of faults or errors into a computer system in
  order to determine its response 9
• The goal isnt to recreate the conditions that
  created the fault

6
Fault Injection 101 (cont)

• There are two primary steps to Fault Injection
• To identify the set of faults that can occur
  within an application, module, class, method,
  etc.
• e.g. if the application does not use the network
  then theres no point in injecting network faults
• In practice this isnt as easy as it sounds
• To exercise those faults to evaluate how the
  application responds
• Does the application detect the fault, is it
  isolated and does the application recover from
  it? 8

7
Example

• void readFile() throws IOException
• ...
• final InputStream is new FileInputStream()
• ...
• while((offset lt bytes.length)
• (numRead is.read(bytes, offset,
  (bytes.length - offset))) gt 0)
• offset numRead
• ...
• is.close()
• (from http//javaalmanac.com/egs/java.io/File2Byte
  Array.html)

• What could go wrong with this code?
• new FileInputStream() can throw
  FileNotFoundException
• InputStream.read() can throw IOException and
  IndexOutOfBoundsException and can return -1 for
  end of file
• is.close() can throw IOException

How do we test how the application responds to
one of these situations? Specifically, how do we
inject these faults so that we can test how the
application responds?
8
Possible Techniques

• Force the situations at the OS level
• Quite hairy to reproduce reliably when theyre
  needed
• Refactor the snot out of it
• Replace the call to InputStream.read() to some
  local instrumented method
• Create our own instrumented InputStream subclass
  possibly using mock objects
• Inject the subclass via IoC (requires some
  framework such as PicoContainer or Spring)
• Just comment out the code and replace with throw
  new IOException()
• Egad!
• We need a way to inject a fault without changing
  the code!

9
Why Use Fault Injection

• Fault occurred in a production application and it
  is determined that that is a valid fault (i.e. it
  is expected to occur)
• Inject the fault in a testing environment without
  having to reproduce the conditions that actually
  caused the fault
• Any fault that is difficult or inconvenient to
  reproduce in testing
• Out of memory
• Disk full
• Database crash
• 3rd party library / Java bug
• Deadlock
• Resource contention
• Dumb users
• Invalid user
• Corner cases

?
10
Why Use Fault Injection (cont)

• You publish an API and you want to be tolerant of
  user input or response
• It is inconvenient or undesirable to refactor
  code to facilitate testing
• e.g. 3rd party library
• You believe that testing code coverage is a good
  metric to measure for code quality 11
• Injecting faults forces exception paths to be
  followed
• More to come

11
Reminder

• You dont have to recreate the conditions that
  caused the fault
• you only have to recreate the fault
• e.g. if an OhNoException crashed your application
  because some little old lady drove her 1966 Buick
  Skylark into your datacenter -- you dont need
  the Skylark or the little old lady to fix the
  application

12
Quick Aside

• Traditionally, Fault Injection is centers around
  faults derived from hardware
• Disk faults such as file missing and corrupt file
• Network faults such as host not found, host
  inaccessible, and high latency
• Memory faults such as corrupt memory
• Operating system faults such as unexpected
  hardware interrupts
• Traditionally used in mission-critical fault
  tolerant environments

13
Quick Aside (cont)

• Fault Injection comes in two favors
• Hardware-based
• Typically requires specialized hardware
• Software-based
• Traditionally attempts to mimic hardware-based
  fault injection and typically involves direct
  interaction with the operating system
• More recently expanded to include any fault that
  can occur in software such as
• Mutated bytecode
• Exceptions
• Invalid or mutated input / output values
• Deadlock / resource contention
• Application Security
• etc.

14
Example

• Creating a directory and temporary file

final File directory new File() final
boolean success directory.mkdirs() if(success)
final File file File.createTempFile(, ,
directory) ...

• Faults
• Making the parent directory(s) fails (returns
  false)
• Temp file creation failure (throws IOException)
• Permissions
• Parent directory(s) deleted
• Injection of a specific temp file

15
Example (cont)

• Cant easily show in examples
• Deeply nested exceptions
• Effects of unchecked exceptions
• Consequences of using catch(Exception e)
• i.e. NullPointerExceptions that go unnoticed
• Complex / real-world cases

16
Intelligent Fault Injection

• Fault injection doesnt have to be all on or all
  off
• Logic can be coded around injected faults
• e.g. InputStream.read()
• Throw IOException after n bytes are read
• Return -1 (EOF) one byte before the actual EOF
• Sporadically mutate the read bytes

17
Beyond Fault Injection

• The techniques that Im going to demonstrate
  arent specific to injecting faults (i.e.
  conditions that may lead to a failure)
• What if analysis
• Test difficult corner cases
• e.g. some optimization routine runs when certain
  conditions are met
• Initialize application state to some desired
  value
• etc.
• The techniques are similar to mock objects or IoC
  but much more granular

18
Summary

• Intercept operations and inject code to
• Perform a custom function
• Return a custom value
• Modify a input parameter
• Throw an exception
• Set the state of an object to a specific value

19
Checkpoint

• Done
• Quick Introduction to Fault Injection
• Why Use Fault Injection
• Fault Injection Examples
• Beyond Fault Injection
• Next
• Quick Introduction to AOP
• Fault Injection in Java
• AspectJ
• Javassist
• Wrap Up

20
Checkpoint (cont)

• Questions?

21
Aspect Oriented Programming 101

• Join Point
• a point in the flow of a running program 14
• It is the where in AOP
• Not all AO environments support all join points
• e.g. Spring AOP only supports method-based join
  points

22
Aspect Oriented Programming 101 (cont)

• Join point examples
• reading or writing a field
• calling or executing a method or constructor
• catching or throwing an exception
• A join point shadow is the location of a join
  point in the source code or bytecode of the
  program 5
• We will use join point and join point shadow
  interchangeably

23
Aspect Oriented Programming 101 (cont)

• Pointcut
• a set of join points 6
• Effectively a query where the join points are the
  data that is being queried the which

24
Aspect Oriented Programming 101 (cont)

• Advice
• code that executes at each join point picked
  out by a pointcut 7
• Advice is the code that youre injecting -- the
  what

25
Aspect Oriented Programming 101 (cont)

• Aspect
• a logical collection of advice
• An aspect is to AO what an object is to OO

26
Aspect Oriented Programming 101 (cont)

• Introduction (aka Mixin or inter-type
  declaration)
• advice that provides additional functionality to
  a class
• e.g. methods or member variables can be added via
  an Introduction to the interface of a class

27
Aspect Oriented Programming 101 (cont)

• Weaving
• the process by which specific join points are
  identified by pointcuts and advice is injected
  into bytecode

28
How AOP Works

• Compile-time
• a specialized compiler is used to weave the
  source code and advice together to produce
  bytecode
• Post-compile (aka binary weaving)
• weaves advice into existing class files modifying
  them as necessary
• Load-time
• a classloader performs binary weaving as the
  class files are loaded into the JVM or a Java
  agent is used via JVMTI (JVM Tool Interface)
• Run-time
• classes that are already loaded by the JVM are
  woven with aspects via JPDA (Java Platform
  Debugger Architecture) to replace method bodies.
  Dynamic proxies can also be used.

29
Load-Time Weaving

• Were going to focus on load-time weaving (LTW)
• Advantange (over compile-time or post-compile)
  does not modify classes on disk
• Disadvantage increases application startup time

30
Java AOP

• AspectJ (http//www.eclipse.org/aspectj/)
  Excellent all-around solution.
• dynaop (https//dynaop.dev.java.net/) Bob Lees
  stripped down AOP Requires proxy creation in
  code (i.e. not 100 transparent).
• CGLIB (http//cglib.sourceforge.net/) a
  roll-your-own solution using proxies. Requires
  proxy creation in code (i.e. not 100
  transparent). For load-time weaving a custom
  class loader would be needed.
• BCEL (http//jakarta.apache.org/bcel/)
  roll-your-own solution requiring knowledge of
  Java byte code to write advice. For load-time
  weaving a custom class loader would be needed.
• Javassist (http//www.csg.is.titech.ac.jp/chiba/j
  avassist/) A large step above CGLIB / BCEL in
  that it includes a stripped down compiler and
  modifies the advised class on the fly. Load-time
  weaving is supported.
• Spring AOP (http//www.springframework.org/docs/re
  ference/aop.html) Limited to method-based
  joinpoints. If using the _at_AspectJ annotations
  then it effectively becomes AspectJ otherwise
  standard Spring-style XML configuration. JDK
  dynamic proxies or CGLIB for implementation. If
  youre already using Spring and only need to
  advise coarse-grained objects (i.e. the objects
  that youre IoCing) and method-based jointpoints
  are suitable then this is a good solution.
• JMangler (http//roots.iai.uni-bonn.de/research/jm
  angler/) A step above BCEL / CGLIB but a step
  below Javassist. Requires knowledge of Java byte
  code (it uses the BCEL API for advice).
  Load-time weaving is supported.
• JBoss-AOP (http//labs.jboss.com/portal/jbossaop/)
  Effectively on par with AspectJ. If youre
  using JBoss then this would be the best solution.
  Even has support in their IDE.

31
AspectJ

• AspectJ
• http//www.eclipse.org/aspectj/
• Documentation
• http//www.eclipse.org/aspectj/doc/released/
• AspectJ FAQ
• http//www.eclipse.org/aspectj/doc/next/faq.html
• AspectJ Development Environment Guide
• http//www.eclipse.org/aspectj/doc/next/devguide/
  index.html
• AspectJ Programming Guide
• http//www.eclipse.org/aspectj/doc/released/proggu
  ide/index.html
• AspectJ Quick Reference
• http//www.eclipse.org/aspectj/doc/released/quick5
  .pdf

32
Load-Time Weaving AspectJ

• Load-time Weaving can be accomplished via 13
• -javaagentpathto/aspectjweaver.jar
• aj.bat
• Custom classloader (WeavingURLClassLoader)
• Both require a META-INF/aop.xml to be located in
  the classpath

33
AspectJ Examples
34
AspectJ Pros

• Quite mature
• Integration with IDEs (e.g. AJDT)
• Responsive forums
• Does 95 of what you need

35
AspectJ Cons

• Expect to burn a lot of time figuring out the
  difference between call and execute, within and
  withincode, etc.
• Can add quite a bit of time to incremental
  compiles (Also see bugs 146071 and 145438)
• Its very easy to confuse the IDE
• No access to local variables
• You have to be careful about other errors in your
  application. A compile error in some completely
  unrelated file may cause errors in aspects.
• Brittle with respect to refactoring especially in
  the case of load-time weaving. If using AJDT
  with compile-time weaving then refactorings apply
  across aspects.

36
Javassist

• Javassist
• http//www.csg.is.titech.ac.jp/chiba/javassist/
• Javadoc
• http//www.csg.is.titech.ac.jp/chiba/javassist/ht
  ml/index.html
• Tutorial
• http//www.csg.is.titech.ac.jp/chiba/javassist/tu
  torial/tutorial.html

37
Javassist Example
38
Javassist Pros

• Provides a relatively simple interface to do
  fault injection at the level of the bytecode
• Provides a limited compiler to take away the need
  to work directly in bytecode (e.g. BCEL)

39
Javassist Cons

• Limited compiler
• Theres much that you need to roll in order to
  use it
• Confusing code fragment language

40
Questions?
41
References

• 1 Harbulot, B. Gurd, J. (2005). A join
  point for loops in AspectJ.
• http//www.cs.iastate.edu/leavens/FOAL/papers-2
  005/harbulot-gurd.pdf
• 2 http//www.eclipse.org/aspectj/doc/released/f
  aq.htmlqjoinpoints
• 3 http//www.aspectprogrammer.org/blogs/adrian/
  2004/06/go_from_stateme.html
• 4 Ossher, H. Tarr, P. Operation-Level
  Composition A Case in (Join) Point
• http//www.ccs.neu.edu/research/demeter/SOP/Tarr
  .pdf
• 5 Hilsdale, E. Hugunin, J. Advice Weaving
  in AspectJ
• http//hugunin.net/papers/aosd-2004-cameraReady.
  pdf
• 6 http//en.wikipedia.org/wiki/Pointcut
• 7 http//www.eclipse.org/aspectj/doc/released/f
  aq.htmlqadvice
• 8 Hsueh, M, Tasi, T, Iyer, R. Fault
  InjectionTechniques and Tools.
• IEEE CS Press, 1997, pp. 75-82
• 9 Clark, J. Pradhan, D. Fault Injection.
  IEEE CS Press, 1995, pp. 47-56
• 10 http//en.wikipedia.org/wiki/Fault_28techno
  logy29
• 11 http//blogs.msdn.com/bwill/archive/2003/09/
  23/51169.aspx
• 12 Espak, M. Improving Efficiency by Weaving
  at Run-time
• http//www.old.netobjectdays.org/pdf/03/papers
  /ws-yrw/415.pdf
• 13 http//www.eclipse.org/aspectj/doc/released/
  devguide/ltw-configuration.html
• 14 http//en.wikipedia.org/wiki/Join_point

42
Thank You

• Rob Grzywinski
• http//www.realityinteractive.com/rgrzywinski/
• rgrzywinski_at_yahoo.com