Software Testing Research and Roadmap

1
Software TestingResearch and Roadmap

• The Future of Software Engineering
  (http//www.softwaresystems.org/future.html)
• http//www.softwaresystems.org/testing.html

2

• Testing dynamic verification
• Advantages (over static verification)
• Automatic generation of test case requirements
• Instrument the software to report execution
  results
• Actual execution in expected environment
• Automated test execution and result analysis
• Limitations
• Cannot show the absence of faults
• Cannot test for certain qualities
• Cannot generalize specific test case results

3
Observations

• gt 50 of software development cost is testing
• more for critical systems
• Pervasive software requires higher quality
• Software quality will become the dominant success
  criterion in the software industry
• Systematic testing techniques are not widely used

4
Roadmap
5
Research Areas

• Component-based and architecture-based testing
• Testing based on pre-code artifacts
• Testing evolving software
• Demonstrating effectiveness of test techniques
• Establishing effective processes for testing
• Using testing artifacts
• Other testing techniques
• Methods and tools
• Empirical studies

6
Component-based testing

• Perspectives
• Component provider
• Context independent
• All possible configurations
• White box
• Component user
• Context specific
• Typically black box

7
Techniques for Component Providers

• Formal component syntactic and semantic interface
• Algebraic specifications
• Cluster and class testing
• Object state-based techniques
• Symbolic evaluation to generate sequences of
  method calls and definition-use pairs

8
Techniques for Component Users

• Axioms for a theory of test adequacy
• Test the component from each subdomain of use
• Cryptographic techniques to verify coverage
  without disclosure of intellectual property

9
Needs

• Testing quality aspects security, dependability,
  safety
• Specification and test of component/application
  couplings
• Component meta-specifications
• Represent testing information
• Deliver efficient testing techniques with the
  component
• And reuse them in regression test, integration
  test, and system test
• Query interfaces for test coverage or integration
  coupling

10
Architecture-based testing

• Formal architecture description languages
• Formal analysis of testability (controllability,
  reachability, observability, etc.), coverage,
  etc.
• Testability as an explicit architectural quality
• Testing of architecture framework early in
  development
• Architecture slicing
• Style/Quality-based test approaches
• Driver of test-case planning and development
• Testing architecture
• Systematic development, organization and
  management of complex tests and regression tests
• Product-line architecture test suites

11
Testing evolving software

• Observation regression testing can account for
  1/3 of the total cost of a software system
• Regression-test selection methods
• Based on test suite and prior test results
• Based on requirements and architecture coupled
  with change traceability
• Optimization of size in regression test suite
• Attribute trade-off coverage vs. cost vs.
  running time
• Assessment of regression testability and test
  size
• Analysis of test coupling

12
Demonstrating effectiveness of test techniques

• Evaluation criteria for determining adequacy of
  test suites
• (Goodenough and Gerhart)
• Behavior-based and code-based selection
• Need fault classification and alignment with test
  selection criteria
• e.g., focus on data flow or boolean logic
• Analytical, statistical and empirical
  effectiveness
• Criteria interaction
• User (vs. tester) testing of form-based visual
  programming languages
• Efficient approximations to optimal test suites

13
Establishing effective processes for testing

• Waterfall and V-Model testing vs. incremental
  synchronize and stabilize
• Perpetual testing
• Optimal testing increment to balance coverage
  and cost
• Coupling of dynamic verification with other VV
  tasks
• Quality development processes produce quality
  products
• How?

14
Using testing artifacts

• Artifacts test cases, execution traces, path
  coverage, intermediate and final test results,
  test pass/fail, etc.
• Artifact configuration management and reuse
• Using execution traces and dynamic program slices
• Heuristics to consider intersections and unions
  of program slices driven by test subset failure
  (to localize faults)
• Analysis of invariants
• Using coverage and traceability to
  select/prioritize regression tests and to predict
  magnitude of regression and new test
• Path or branch spectra collection and comparison
  to drive profiling
• Visualization
• Data mining

15
Other testing techniques

• Efficient generation of test inputs and analysis
  of outputs
• Especially for integration and system test
• Testing distributed systems
• Cost of instrumentation in real-time systems

16
Methods and tools

• Transfer of technology to practice
• 18 years?!
• Scalability to large, complex systems
• Usable by practitioners
• Computational efficiency trade-offs

17
Empirical studies

• Demonstrate practical utility of techniques
• Empirical Software Engineering
• Infrastructures to support case studies and
  experimental systems
• Availability of test subjects
• Sensitivity of test results in marketplace

18
Thank You!
a b