Using Runtime Testing to Detect Defects in Applications without Test Oracles

1
Using Runtime Testing to Detect Defects in
Applications without Test Oracles

• Chris Murphy
• Columbia University
• November 10, 2008

2
About Me

• 3rd-year PhD student at Columbia University
• Advised by Prof. Gail Kaiser
• Research interests
• Software Testing
• Computer Science Education
• Computer-Supported Cooperative Work

3
Introduction

• This thesis addresses the problem of testing
  complex, highly configurable systems,
  particularly those without test oracles that
  indicate what the correct output should be for
  arbitrary input
• We adapt a technique that leverages built-in
  pseudo-oracles and perform testing in the
  deployment environment in order to address
  limitations regarding defects that reveal
  themselves only in certain states or with certain
  input data

4
Overview

• Problem Statement Requirements
• Approach Hypotheses
• Model Architecture
• Feasibility Preliminary Results
• Related Work
• Expected Contributions

5
Problem Statement

• Some applications, such as in Machine Learning,
  do not have test oracles for the general case
• Even if certain defects may be detectable, others
  can only be revealed as a result of particular
  input values, configurations, application states,
  or runtime environments that may not have been
  tested prior to deploying a software product

6
Observation

• Even when there is no oracle in the general case,
  there can still be a limited subset of inputs
  such that
• they can at least reveal certain types of
  defects, e.g. catastrophic failures (crashes),
  and/or
• the expected output can actually be known
• These inputs may be generated based on past
  inputs and their respective outputs, as in
  Metamorphic Testing Chen 98

7
Metamorphic Testing

• Originally designed as an approach for creating
  follow-up test cases based on those that have not
  revealed any defects
• If input x produces output f(x), then the
  functions metamorphic properties are used to
  guide a transformation function t, which is
  applied to produce t(x)
• We can then predict the expected value of f(t(x))
  based on the known value of f(x)
• If f(t(x)) is not as expected, then a defect
  exists

8
Metamorphic Testing Example

• Anomaly-based network intrusion detection systems
  build a model of normal behavior
• In some cases, the model may consider the byte
  distribution of data in the incoming network
  packet anything deemed anomalous causes an alert
• We cannot know a priori whether a particular
  packet should cause an alert
• However, if we permute the order of the bytes,
  the result (anomalous or not) should be the same
  as for the original packet, since the system only
  considers the distribution of the bytes

9
Proposed Approach

• To address the problem of functions and/or
  applications that have no test oracle, we use
  Metamorphic Testing
• In such applications, to reveal defects that are
  dependent on input data, configuration,
  application state, or the runtime environment, we
  continue Metamorphic Testing in the field, after
  deployment and during actual usage

10
Approach Details

• Initial input/output pairs are taken from actual
  executions
• We cannot know whether the output is correct but
  we at least know that the input is something that
  comes up in practice, and is useful as a valid
  test case
• We then apply metamorphic properties to get
  test input, so that we should be able to predict
  the test output
• Although we cannot know whether the test output
  is correct either, if it is not as predicted then
  there is a defect
• Since this runs in the field, we have to ensure
  that users don't notice this testing, e.g. see
  the test output, experience a sudden performance
  lag, etc.

11
Hypotheses

• For programs that do not have a test oracle,
  conducting Metamorphic Testing within the context
  of the application running in the field can
  reveal defects that would not ordinarily
  otherwise be found
• This can be done without affecting the
  application state from the users perspective,
  and with minimal performance overhead

12
Proposed Model

• Automated Metamorphic System Testing Conducts
  system-level Metamorphic Testing in the
  deployment environment
• Metamorphic Runtime Checking A separate testing
  technique that, for individual units (functions),
  supports the execution of Metamorphic Tests that
  are executed from within the context of the
  running application

13
Automated Metamorphic System Testing

• Checks that the metamorphic properties of the
  entire system hold after execution
• Treats the application as a black box
• Multiple invocations run in parallel, and results
  are compared upon completion
• User only sees output from the original
  invocation

14
(No Transcript)
15
Amsterdam Automated Metamorphic System Testing
Framework

• Metamorphic properties are specified in XML
• Input transformation
• Runtime options
• Output comparison
• Framework provides out-of-box support for
  numerous transformation and comparison functions
  but is extendable to support custom operations
• Additional invocations are executed in parallel
  in separate sandboxes that have their own virtual
  execution environment

16
Metamorphic Runtime Checking

• For individual units (functions), we check
  whether the metamorphic properties hold as the
  application is running, using actual input from
  real executions and the applications current
  state
• Function arguments are modified according to the
  metamorphic properties
• Function is called again (in an isolated sandbox)
  with the new input
• Outputs are compared

17
Create a sandbox for the test
18
Foundation In Vivo Testing

• To facilitate testing from within a running
  program, we will extend In Vivo Testing Chu
  ICST08
• In Vivo Tests are analogous to unit tests but
  they test from within the context of the running
  application as it executes in the deployment
  environment, as opposed to a clean slate
• They test that sequences of actions produce the
  expected results, no matter what the
  configuration, state, or runtime environment

19
Complementary Testing Approaches

• Metamorphic Testing addresses a limitation of In
  Vivo Testing
• The need for a test oracle
• In Vivo Testing addresses some limitations of
  Metamorphic Testing
• Availability of initial test data
• Detecting defects that only appear in certain
  states, configurations, or environments or occur
  only intermittently

20
Columbus Metamorphic Runtime Checking Framework

• Tests (specifications of metamorphic properties)
  are written by developers, then select components
  of the application are instrumented with tests at
  compile-time
• Configuration includes probability of running a
  test for each function maximum number of
  concurrent tests action to take when test fails
  whether to assign tests to separate
  processor/core
• Test sandbox can be created by simple fork or
  by creating a virtual execution environment

21
Preliminary Results

• Identified categories of metamorphic properties
  in the domain of Machine Learning
• Detected defects with Metamorphic System Testing
• Detected defects with In Vivo Testing
• Detected defects with Metamorphic Runtime Checking

22
Categories of Metamorphic Properties Murphy
SEKE08

• Additive Increase (or decrease) numerical values
  by a constant
• Multiplicative Multiply numerical values by a
  constant
• Permutative Randomly permute the order of
  elements in a set
• Invertive Reverse the order of elements in a set
• Inclusive Add a new element to a set
• Exclusive Remove an element from a set
• ML apps such as ranking, classification, and
  anomaly detection exhibit these properties

23
Feasibility Metamorphic System Testing

• We performed system-level metamorphic testing on
  various types of Machine Learning applications
  Murphy SEKE08
• Detected previously-unknown defects in a
  real-world network intrusion detection system
• However, this testing was not automated inputs
  were modified with one-off scripts and outputs
  were compared manually

24
Feasibility In Vivo Testing

• We have previously developed an implementation of
  the In Vivo Testing framework for Java
  applications called Invite Chu ICST08
• Targeted towards applications in which defects
  were not obvious to the user (not necessarily
  those without test oracles)
• Detected known defects in OSCache that were not
  found by traditional unit tests
• Uses fork to create new processes for sandbox

25
Feasibility Metamorphic Runtime Checking

• We have developed a system by which functions
  metamorphic properties are specified using an
  extension to JML
• Specifications converted into metamorphic unit
  tests by a tool called Corduroy Murphy 08
• tests run using JML Runtime Assertion Checking
• Detected defects in WEKA and RapidMiner machine
  learning toolkits

26
Related Work Absence of Oracles

• Pseudo-oracles Davis ACM81
• Testing non-testable programs Weyuker TCJ82
• Overview of approaches Baresi 01
• Embedded assertion languages
• Extrinsic interface contracts
• Pure specification languages
• Trace checking log file analysis
• Using metamorphic testing Chen JIST02

27
Related Work Testing in the Field

• Perpetual Testing Osterweil QW96
• Gamma Orso ISSTA02
• Skoll Memon ICSE04
• Cooperative Bug Isolation Liblit RAMSS04
• Failure-Oblivious Computing Rinard OSDI04
• Security systems that monitor for errors

28
Methodology (1)

• To further demonstrate feasibility, we will
  conduct Automated Metamorphic System Testing and
  Metamorphic Runtime Checking on real-world
  Machine Learning applications as they run under
  normal operation in the field
• We will also show that certain defects would not
  ordinarily have been detected by using
  Metamorphic Testing (or other techniques) prior
  to deployment

29
Methodology (2)

• To show that our testing approach advances the
  state of the art in testing applications that
  have no test oracle, we will compare it to other
  techniques that could be used to address this
  same problem
• Symbolic execution
• Model checking
• Program invariants
• Formal specification languages

30
Expected Contributions

• Automated Metamorphic System Testing and a
  testing framework called Amsterdam
• Metamorphic Runtime Checking and a testing
  framework called Columbus
• A set of guidelines for assisting the formulation
  and specification of metamorphic properties

31
Using Runtime Testing to Detect Defects in
Applications without Test Oracles Chris
Murphy cmurphy_at_cs.columbia.edu