Distributed In Vivo Testing of Software Applications - PowerPoint PPT Presentation

Loading...

PPT – Distributed In Vivo Testing of Software Applications PowerPoint presentation | free to view - id: 78d8e3-ZDgzM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Distributed In Vivo Testing of Software Applications

Description:

Distributed In Vivo Testing of Software Applications Matt Chu, Christian Murphy, Gail Kaiser Columbia University Introduction It is infeasible to fully test a large ... – PowerPoint PPT presentation

Number of Views:46
Avg rating:3.0/5.0
Slides: 20
Provided by: Gail140
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Distributed In Vivo Testing of Software Applications


1
Distributed In Vivo Testing of Software
Applications
  • Matt Chu, Christian Murphy, Gail Kaiser
  • Columbia University

2
Introduction
  • It is infeasible to fully test a large system
    prior to deployment considering
  • different runtime environments
  • different configuration options
  • different patterns of usage
  • Extending testing into the deployment environment
    can help reveal faults

3
In Vivo Testing
  • We have developed an approach called in vivo
    testing
  • Unit tests from the development environment (in
    vitro) are executed at run-time in the deployment
    environment (in vivo)
  • Done without affecting the state of the
    application
  • However, this approach can incur a large
    performance overhead

4
Distributed In Vivo Testing
  • In this work we present a distributed approach
    that takes advantage of networked application
    communities
  • Members of the community each run fewer tests
    (thus less overhead) and results are aggregated
    centrally

5
Benefits of Distributed In Vivo Testing
  • Amortizing the workload over many instances
    tackles the high performance cost without
    sacrificing the quantity of tests being conducted
    globally
  • Having a community of applications
    collaboratively working together increases the
    possibility that an instance will find these
    erroneous permutations of state

6
Overview
  • Related work
  • In vivo testing the Invite framework
  • The distributed Invite framework
  • Experimental results
  • Conclusions and future work

7
Related Work
  • Perpetual testing Clarke SAS00
  • Skoll Memon ICSE04
  • Gamma Orso ISSTA02
  • CBI Liblit PLDI03
  • MORABIT Merdes SEM06

8
In Vivo Testing
  • Unit tests are executed in the deployment
    environment, in the context of the running
    application, as opposed to a controlled
    development environment
  • Tests are run continuously as the application
    runs, at selected points in the program execution
  • Tests are run in a separate process (which has
    been created as an exact copy of the original),
    so as not to affect the state of the original
    process

9
In Vivo Testing Defects Targeted
  1. Unit tests make incorrect assumptions about the
    state of objects in the application
  2. Possible field configurations were not tested in
    the lab
  3. A legal user action puts the system in an
    unexpected state
  4. An sequence of unanticipated user actions breaks
    the system
  5. Defects that only appear intermittently

10
Invite The in vivo testing framework
  • Implemented in Java
  • Uses AspectJ to weave in the instrumentation at
    compile-time
  • Configuration performed by software developer
  • Frequency of test execution (per method)
  • Reaction policy
  • Software deployed as usual, with no requirements
    on end user

11
Invite Running a test
Method foo is about to be executed
Run the test in a separate process
12
Distributed in vivo testing approach
  • Amortize the cost of test execution by
    distributing the workload over a number of
    instances
  • Cover a variety of different platforms, usage
    patterns, etc.
  • Client/Server architecture
  • Server can distribute tests according to
    targeting or sampling policies

13
Distributed Invite framework
Invite Server
14
Experiment Setup
  • Goal ensure that evenly distributing testing
    load improves performance by decreasing the
    number of tests each individual member in the
    community has to run.
  • Subject Jetty WebServer
  • 5000 dynamic static web pages requested from
    each instance

15
Experimental Results
overhead
community size
16
Summary
  • Overhead declines as the number of members in the
    community grows
  • Analytical results
  • 18 members 25
  • 45 members 10
  • 90 members 5
  • Optimal minimum overhead (from instrumentation)
    was 2.4

17
Limitations Future Work
  • Ensure that the test does not affect the external
    system state
  • Improve distribution of tests
  • Adjust frequency of test execution based on
    context or resource availability
  • Include support for capture/replay, fault
    localization, etc.

18
Conclusion
  • Distributed in vivo testing addresses one of the
    biggest problems of the in vivo testing approach
    (extremely high performance penalties) by
    distributing the number of tests that need to be
    run onto the entire community of applications
  • The approach maintains the same number of tests
    globally, and also encompasses more diverse
    environments

19
Distributed In Vivo Testing of Software
Applications
  • Matt Chu, Christian Murphy, Gail Kaiser
  • Columbia University
About PowerShow.com