Formal Methods of Systems Specification Logical Specification of Hard and Software

1
Formal Methods of Systems SpecificationLogical
Specification of Hard- and Software

• Prof. Dr. Holger Schlingloff
• Institut für Informatik der Humboldt Universität
• and
• Fraunhofer Institut für Rechnerarchitektur und
  Softwaretechnik

2
Test Specification

• Spec model program serves as
• executable specification
• for simulation/animation of intended behaviour
• test generator
• test models are obtained from model program by
  abstraction
• test oracle
• assertion of safety properties, pre /
  postconditions
• Extra effort to derive!!!
• pays off

3
Spec Explorer Tool
http//staff.washington.edu/jon/icfem/specs-icfem.
html

• Specification based testing
• Development of Spec model programs
• "literate programming" editor, debugger
• Validation of Spec model programs
• simulation (Main execution)
• exploration (FSM generation)
• visualization
• safety (reachability), liveness analysis
• static analyses (BoogiePL)
• Test case generation
• scenarios from explored FSMs
• complete coverage of spec or stochastic testing
• Test execution
• offline test case generation for conformance
  testing
• on-the-fly testing with spec as test oracle

4
Spec Explorer Artefacts
\Spec Explorer\doc\SpecExplorerReference.doc
5
Test Spec vs. System Spec

• System spec used to derive the implementation
• transformational development
• correctness of derivation steps
• assertion checks can be switched on or off
• Test spec aimed at testing and validating the
  SUT
• investigate properties of model programs wrt SUT
• generate and execute test suites
• assertions as test oracle
• Different intentions, different levels of
  abstraction!

6
Modeling Abstraction

• Minimum code needed to generate scenarios of
  interest no need to be comprehensive
• Adequate level of abstraction (state variables,
  actions in SUT to test)
• Global point of view, each agent can see each
  other agents' state
• All state information (files, messages, ) in
  model variables
• Each action (at chosen level of abstraction) is
  coded as a method in the model (need not
  correspond 11 to methods in IUT)
• Model program has a single thread, interleaving
  actions can represent concurrency
• Actions are atomic, no interleaving within action
  bodies
• Multiple assignments within an action can
  represent parallelism
• Model introduces new state space, reconcile with
  SUT

7
Modeling Coding

• To code each action
• When is it enabled? (requires ...)
• multiple actions enabled in the same state models
  nondeterminism
• allows for interleaving concurrency
• What (if anything) does it return? (return ...)
• What is next state? Is it different?
  (assignments, ... ... )
• Distinguish top level Action methods from
  helper methods
• Possibly write Main method(s) to simulate
  scenario(s)

8
Example(s)

• Stack
• Counting Problem (Prisoners Dilemma)

9
Test design

• Generating test suites from Spec model
• Rationale find equivalence classes of
  behaviourno need for exhaustive testing
• Different fault types call for different kinds of
  tests
• Wrong logic/wrong expression
• Complete but minimal coverage over small domains
• Problem scaling up data structures (like hash
  table resize, editor buffer gap)
• Vary a few properties over large ranges
• Unreliable infrastucture, hidden state leaks out
• Long test cases, revisit the same (model) states

10
Exploration

• Exploration generates a finite state machine
  (FSM) from the model program for
• validation (visualization, check safety and
  liveness), and
• offline test case generation
• Exploration executes the model program in a
  special environment, building the FSM as it goes.
  
• each invocation (method call including args) is a
  transition in the FSM
• execute all enabled invocations from a state
  (backtracking, in effect)
• execute each method with all combinations of
  arguments from given finite domain (can simulate
  internal nondeterminism with additional
  arguments).
• Generated FSM is an underapproximation of the
  model program
• can be nondeterministic

11
Exploration algorithm

• Exploration treats model program state as first
  class.
• Spec compiler generates code with storage
  management hooks
• explorer creates set of hyperstates as
  approximation of sets of states
• executes the actions of the given spec on
  concrete states of that spec and building up the
  hyperstates
• end state of a new transition is added to the
  frontier if the transition is relevant (is an
  improvement towards goal)
• Abstraction vs. Exploration
• abstraction (e.g., hiding of variables) yields
  over-approximation (more transitions than
  really)
• model exploration yields under-approximation

12
Exploration Parameters
13
Test case generation

• Offline test case generation traverse FSM
  generated by exploration
• Different traversal algorithms achieve different
  coverage
• Postman tour gives minimal transition coverage
  (not path coverage)
• Identify "accepting states" where test run may
  terminate
• Identify "cleanup actions" that make progress
  toward accepting state
• Tool ensures each test case reaches accepting
  state (via cleanup actions)
• Tool can store test suite internally for
  subsequent conformance testOR tool can write out
  test suite as C program

14
Conformance testing

• Tool can act as test harness for conformance
  testing
• Tool can reference and execute IUT (binary, DLL)
• Model and IUT can be at different levels of
  abstraction, must reconcile model state space
  with IUT state space
• Write wrapper or test driver around IUT
• Wrapper can translate IUT values to model values
• Probe actions can return (translated) IUT state
  variables
• Action bindings, type bindings defined in
  configuration
• Object bindings made dynamically
• Lockstep execution, model with IUT, check
• actions are enabled
• correct return values