Formal%20Conformance%20Testing%20of%20Systems%20with%20Refused%20Inputs%20and%20Forbidden%20Actions

1
Formal Conformance Testing of Systemswith
Refused Inputs and Forbidden Actions

• Igor Burdonov,
• Alexander Kossatchev,
• Victor Kuliamin
• ISP RAS, Moscow

2
Outline

• Introduction
• Formal Testing with LTSes and ioco
• Proposed conformance relation
• LTS Completion
• Conclusion

3
Formal Conformance Testing
Real World
Model World
Requirements
Specification
conforms
???
conforms formally
!!!
modeling
?
?
System under Test
Implementation
derivation
passes formally
passes
translation
Test Suite
Model Test Suite
4
Labeled Transition Systems

• Model world LTSes
• States
• Transitions
• Labels
• Inputs ?a, ?b
• Outputs !x, !y
• Internal transitions t
• Initial state

?a
!x
!y
?b
t
5
ioco Conformance Relation

• Testing abilities
• Ability to provide inputs
• Ability to observe outputs
• Ability to observe quiescence
• I ioco S ?
• for each d-trace (trace with quiescence) s
  of S
• an output is possible in I after s only if its is
  possible in S
• quiescence is possible in I after s only if it is
  possible in S

?a
!x
d
?b
6
Examples
S
I1
I2
I3
d
d
d
!y
!y
!y
?a
?a
?a
?b
?a
?b
d
!x
!y
!x
!x
t
?b
d
d
?a
d
?a
?a
?b
?b
?b
?b
t
d
d
?a
?a
?a
d
I1 ioco S
I2 ioco S
I3 ioco S
7
More Subtle Abilities
I

• Testing abilities
• Ability to provide inputs
• Ability to observe outputs
• Ability to observe quiescence
• Constraints
• Implementation should be input-enabled
• Tester has full control over action firing of
  implementation - synchronous testing( parallel
  composition semantics )

?b
!y
?
!b
?a
?b
T
?a
t
!x
?a
?b
?b
?a
8
Practical Considerations

• Some implementations are notinput-enabled
• Launch button
• Memory allocation
• Testing in almost cases isasynchronous
• Especially for distributed systems

free(void ptr) ptr should be earlier returned
by malloc(), calloc(), realloc() (POSIX)
9
Asynchronous Testing
Real World
Model World
Environment
System under Test
Implementation
I
I C
Context
C
Tester
Model Tester
T
10
Compositional Conformance

• We may try to check that(I C) ioco (S C)
• But ioco interacts with in a bad way!
• Unexpected conformance (I C) ioco (S C)
  while I ioco S
• Unexpected non-conformance (I C) ioco (S
  C) while I ioco S

11
Bad Examples
C input and output queues
S
I1
I2
?a
!x
?a
!x
?a
!x
!y
?b
!y
!x
?b
?b
!z
I1 ioco S (I1 C) ioco (S C)
I2 ioco S (I2 C) ioco (S C)
12
Possible Ways out

• ioco is asymmetric I should be input-enabled,
  while S should not
• ioco is preserved when S is input-enabled
• Consider input-enabled specs only
• Too narrow
• Replace S with S such that?I I ioco S ? I ioco
  S ( S ioco S)and S is input-enabled
• Consider not input-enabled implementations and
  ioco analogue for them
• Consider context-specific conformance relations
• Queues already considered, but others are also
  needed

13
Outline

• Introduction
• Proposed conformance relation
• Extension of ioco for non-input-enabled
  implementations
• LTS Completion
• Conclusion

14
Meaning of Undefined Inputs

• ForbiddenShould not be provided
• RefusedCan be provided and its refusal can be
  observed
• IgnoredCan be provided, does nothing

? should be specified
15
Proposed Model

• LTS with forbidden actions and refused inputs
• Additional elements
• Forbidden action ?
• Refused inputs ?a, ?b
• ß?d-traces
• Contain inputs, outputs, d, input refusals, ?
• ? can only be the last symbol

?b
?b
?a
!x
?a,?b
?a
?b
!y
t
d
?a
?a
?
?b
?a,?b
?a,?b, d
16
Safe Traces

• ß?d-traces that cannot cause forbidden action to
  occur

?b
!x
Safe ß?d-traces
?ad?b?b?a
?a
d
?b
?a
!x?ad?a
t
t
?b
?
!y
?a
?b
!x
d, ?a
t
?b
?
!x
17
Safety of Testing

• Test should not destroy implementation
• Safety hypothesis(weakening of input-enabledness
  hypothesis)Each safe ß?d-trace of S, which is
  also ß?d-trace of I, can be safely extended in I
  by each symbol safe after this trace in S
• Such I is called safe for S

18
iocoß?d Conformance Relation

• Testing abilities
• Ability to provide inputs
• Ability to observe outputs
• Ability to observe quiescence
• Ability to observe input refusal
• Constraints
• Implementation is safe for specification
• Tester has full control over action firing of
  implementation (synchronous testing)

19
iocoß?d Conformance Relation

• I iocoß?d S ?
• I is safe for S and
• for each safe ß?d-trace s of S
• an output safe in S after s is possible in I
  after s only if its is possible in S
• quiescence safe in S after s is possible in I
  after s only if it is possible in S
• an input safe in S after s is possible in ? after
  s only if it is possible in S
• an input refusal safe in S after s is possible in
  I after s only if it is possible in S

20
Examples
S
I2
I1
I3
I4
I5
I6
?b
?b
?b
?b
d
?a
?a
!y
?a
!x
!x
!x
?b
?a
!x
?a
?a
?a
,?b
!y
?b
?b
!y
?a
?b
!y
!y
t
?a
?a
?b
?b
d
?a
?a
d
d
d
?a
?a
?
?
?b
?b
?b
?b
!y
?b
!y
?
?
?a,?b, d
?a,?b, d
?a,?b, d
?a,?b, d
I2 is not safe for S
I1 iocoß?d S
I3 iocoß?d S
I4 iocoß?d S
I5 iocoß?d S
I6 iocoß?d S
21
Test Case Derivation

• Very similar to ioco
• Differences
• We should escape forbidden action
• Use safe traces only
• We should test input refusals
• Try to provide an input and observe refusal
  as a deadlock

22
Examples
S
T1
T2
?y,?
!b
?b
?
!x
?x,?y
?
!a
?
?a
!x
!a
?
?a
?
!b
?x,?
?x,?y
?y
?b
?
!a
?x,?y
!y
t
?
?
!b
?
d
?a
?
!b
?
?x,?y
?b
?
!a
?
?a,?b, d
!b
?
23
Outline

• Introduction
• Proposed conformance relation
• LTS Completion
• Trying to force composition to preserve
  conformance
• Conclusion

24
Completion Operations

• ioco is preserved when S is input-enabled
• Replace S with S such that?I I ioco S ? I ioco
  S ( S ioco S)and S is input-enabled
• Then,I ioco S ? I ioco S ? ?C (I C) ioco (S
  C)
• S is more correct form of S
• it can be used in any context

25
Demonic completion ?

• Tretmans et al.

S
I
?(S)
Undefined inputs
All inputs
t
LTS
!y
!x
t
?a
All outputs
?a
?a
!y
?a
?a
I ioco S
!x
!y
?a
t
?a
I ioco ?(S)
May add more conforming implementations
26
Basic Completion

• States Bc(S) d-traces of S
• For each d-trace s of S add the following
  transitions
• R(s) means all d-traces obtained from s by
  deleting some d-s
• Add s ? slt?agt marked with ?a if ?µ ? R(s) µlt?agt
  d-trace of S
• Add s ? slt!xgt marked with !x if ?µ ? R(s) µlt!xgt
  d-trace of S
• Add internal s ? sltdgt if ?µ ? R(s) µltdgt
  d-trace of S and s does not end with d
• Add s ? slt!errorgt marked with !error if ?µ ?
  R(s) µ cannot be extended with any output, nor
  with d

!x
!y
t
?a
?a
27
Proposed Completions
Undefined inputs
S
?(S)
G(S)

• ?
• G
• For each I and S in ioco domainI ioco S ? I
  iocoß?d ?(S) ? I iocoß?d G(S)

All inputs
t
Bc(LTS)
All outputs
!y
!x
t
?a
Undefined inputs
?a
?a
?a
Bc(LTS)
?
!x
!y
?
t
?a
28
Outline

• Introduction
• Proposed conformance relation
• LTS Completion
• Conclusion

29
Summary
iocoß?d
?
Extension for non-input-enabled implementations
?, G
ioco
Completion construction of equivalent
input-enabled spec
30
Announcement

• Notation
• U non-empty subsets of reachable states of S
• A?U and z is safe in A ? Az states reachable
  from A by z and t
• A?U and z is unsafe in A ? Az ?
• A?U and z is refusal set ? Az maximal stable
  subset of A that for each refusal in z it exists
  in each element of the subset
• States of F(S) U, Uoutputs, d, ?
• Transitions
• ? ? ? marked with ?
• If for symbol z Az is nonempty ? A ? Az and (A,
  d) ? Az marked with z
• If !y is safe and exists in Az where z is refusal
  set ? internal A ? (Az,!y) and (Az,!y) ? (Az)!y
  marked with !y
• If each !y is unsafe or not exists in nonempty Az
  where z is refusal set ? internal A ? (Az, d)
• If for ?a A?a is nonempty ? (A, !y) ? A?a marked
  with ?a

31
Practical Implications

• Not any specs are written correctly for use in
  component-based systems
• The transformation rules can serve for
  specification adjustment
• The rules can be rephrased into characteristics
  of correctly written specs

32
Contacts

• Igor Burdonovigor_at_ispras.ru
• Alexander Kossatchevkos_at_ispras.ru
• Victor Kuliaminkuliamin_at_ispras.ru

33
Thank you!