PropertyBased Test Generation Li Tan, Oleg Sokolsky, and Insup Lee University of Pennsylvania

1
Property-Based Test GenerationLi Tan, Oleg
Sokolsky, and Insup LeeUniversity of
Pennsylvania
2
The Overview of Our Approach
Quasi linear (Lasso-shape) proof structure
LTL formulae F1, , Fn
Test trace generator
Model Checker
Feature conflict detection
Behavior specification
LTL formulae F1, , Fn
Temporal Property Translator
Traces r1,,rn
Criteria not being covered
Specification Model (CHARON)
Simulation-based randomized test generator
Traces
Coverage Criteria
Testing result
Environ. Modeling
System Modeling
Interface Definition
Informal System Specification
Hardware Specification/limitation
Test Harness
Implementation
Test result
3
Model-checking based Test generator
Temporal (feature) Spec. Model (optional)
Test suite (Finite set of finite traces)

• Goal Using model-checking technique to make test
  generation more efficient, flexible, and centered
  on the system-specific properties (features).
• Step I. Preparing specifications
• Properties (feature specification) as linear
  temporal logic formula
• (optional) System specification system as CHARON
  (for hybrid systems) and EFSM (for discrete
  systems)
• Step II. Test generation using model checkers.
• (For hybrid systems) Simulation-based test
  generation with the assistance of predicate
  abstraction reachability analysis.
• (For Discrete system)
• (Option A) Using the proof structures of
  evidence-ready model checkers.
• (Option B) Reducing the test generation for LTL
  formula to safety check
• (For temporal specification only) Functional
  test.
• Generating non-trivial test traces for temporal
  specification (feature specification)
• Detecting conflicting in temporal specification.
  

4
From Property and Model to Test suite
Property-based test generation

• I. From infinite length to finite Synthesizing
  test suites for 9LTL property

5

• A infinite Lasso-shaped test suite can be checked
  adequately by finite steps if the implementation
  is bounded.

Turn1, c1, c2
Turn1, c1, c2

Turn2, c1, c2
Turn2, c1, c2
Turn1, c1, c2
Estimating the number of relevant implementation
states using slicing
Turn1, c1, c2
A quasi-linear proof skeleton
A finite test suite
6

• Test Generation using Model Checkers
• Option A Modifying model checkers and retaining
  proofs.
• Option B Using the idea of reducing LTL model
  checking to reachability analysis A. Biere etc,
  but enhancing the observer to retain proof

SMV model
SMV model Extended Observer Model

Repetition information
Linear Temporal Logic Specification
Extracted Proof
Generated test trace
7

• II. From infinite numbers of traces to finite
  selecting interesting traces
• System properties are required to be held on all
  the paths, we will select only nontrivial paths,
  whose characteristics are caught by ELTL formula
  systematically deriving from the properties.
• LTL f gt ELTL formulae a2e(f)f(f ! ð(f))f
  Á f

F G(req -gt F(cancelÇresponse))
F Æ ( G(req ! F(cancelÇ false))) F Æ F(req Æ
G( cancel)) Test the case that no cancel follows
a request (hence a reponse must be placed)
F Æ ( G(req ! F(false Ç response))) F Æ F(req
Æ G( response)) Test the case that no response
follows a request (hence a cancel must be placed)
F Æ ( G(true ! F(cancel Ç response))) F Æ FG(
(cancel Ç response)) Test the case that no cancel
or reponses occurs after time t, (hence should
not a request occur).
F Æ ( G(req ! false)) F Æ F(req) Test the case
that there is request
8
From only Property to Test suite Functional test
generation

• So, what if only behavior (feature) specification
  is available

LTL formulae F
Nontrivial ELTL formulae Derived from F Ya2e(F)
f0 2 F Æ 2 Y
f1 2 F Æ 2 Y
fn 2 F Æ 2 Y
.
Buchi automaton B0
Buchi automaton B1
Buchi automaton Bn
Check nonemptiness
Check nonemptiness
A trace satisfies f0
A trace satisfies f1
A trace satisfies fn
9
Testing Hybrid system simulation-based test
generator with predicate-abstract reachability
analysis
System Modeling
Coverage Criteria
No
Bad set
Reachability Checker
CHARON (Model)
Flatten hybrid model
Predicate set
Simulation /refinement
NO w/ more predicates
Yes w/ Trace
YES
Test Suite
Concretize
Implementation
10

• An implementation of simulation-based test
  generator

a. CHARON simulator with test generator
b. Progress report of test generator
c. Visual display of generated test traces.
11

Realizing Test Harness
Test trace Variable back_EMF Value Time 60.0
0.001 70.0 0.002
Simulation-based test generator
Coverage criteriae
Test Result
12

• Conclusion
• Applying model-checking technique to traditional
  domain of test generation is appealing.
• Test generation is centralized on system-specific
  properties
• State-of-art model checkers may be adapted as
  general purpose test generator (and think
  properties as programs ).
• Techniques in model checking may help find
  interesting test traces and provide new angle to
  view and think test generation.
• Property-based test generation requires
  integrated efforts.
• Test generation ¹ witness generation.
• Proof is necessary to generate partial test suite
  and perform optimization.
• Proof is also needed to extend the notion of
  testable properties.
• Model-based code generation may help build test
  harness.