Integration of Functional and Timed Testing of Real Time and Concurrent Systems

1
Integration of Functional and Timed Testing
ofReal Time and Concurrent Systems

• V. Kuliamin, A. Petrenko, N.!Pakoulin,
  I.!Bourdonov, A.!Kossatchev
• kuliamin_at_ispras.ru
• ISP RAS, Russia

2
Classic testing scheme
stimulus
s1
Test System
Target System
r1
s2
r2
reaction
s3
r3
Test Test sequence
s4
r4
s5
r5
3
Concurrency and real-time issues
Target System
stimulus1
stimulus2
stimulus
reaction3
reaction1
reaction
reaction2
reaction

• Multiple access points
• Simultaneous inputs
• Multiple reactions
• Spontaneous reactions
• Temporal constraints

stimulus
?t lt T0
reaction
4
Outline

• Theoretical background
• Practical considerations
• Case studies

5
Proposed model

• IOSM Input/Output State Machine
• States S
• Stimuli X
• Reactions R
• Transitions T
• f T ? S(X ? R ? i) S

?b
!z
?a
Internal transition
!y
Spontaneous reaction
!x
Multiple reactions
6
Empty stimulus

• X I ? e
• Empty stimulus e models the knowledge of system
  that it has no inputs
• Asynchronous State Machine IOSM with empty
  stimulus

!x
!y
?a
?e
7
Behavior function
!x
!y
!x
!y
?a
?e
?a

• B X ? P (R)
• ai ? xi

• B X ? P (R)
• ai ? yjxkxlyn kl i

I.Burdonov, A.Kossatchev, V.Kuliamin.
Asynchronous State Machines Classification and
Applications to Testing. Proceedings of ISP RAS,
Moscow, 2003. In Russian
8
Modeling parallel inputs
?a
?b
How to model responses to all possible
multistimuli?
?a,a
?b,b
?a,b
?a,a,b
?a,b,b

• Plain concurrency multistimulus is
• equivalent to some sequence of the stimulus
• contained

9
Outline

• Theoretical background
• Practical considerations
• UniTesK test development method
• Modifications for testing concurrent systems

• Case studies

10
UniTesK test construction
Behavior Model
Testing Model
?
Target System
?
?
Test Input Construction
Behavior Correctness Checking
11
UniTesK test architecture
Test sequence construction
Test Engine
Test sequence construction
Test Action Iterator
Test Scenario
Specification
Target system
12
Example of specifications

• specification int deposit ( int x )
• pre
• return x gt 0
• post
• return balance _at_balance x

13
Example of scenario

• scenario class AccountTest
• AcountSpecification model
• AbstractState getState()
• return new IntState( model.balance )
• scenario deposit()
• iterate(int i 0 i lt 5 i)
  model.deposit(i)
• scenario withdraw()
• iterate(int i 0 i lt 5 i)
  model.withdraw(i)

14
Test sequence construction
Test Engine
Test Action Iterator
15
Modifications for testing concurrency

• Specifications
• Asynchronous reactions
• Temporal constraints
• Testing
• Providing parallel inputs
• Collecting reactions
• Checking the behavior as a whole

16
Specifying asynchronous reactions

• specification reaction UDPPacket UDPResponse ( )
• pre return !ModelUDPPackets.isEmpty ( )
• post
• return
• _at_ModelUDPPackets.clone().contains(UDPRespons
  e)
• !ModelUDPPackets.contains(UDPResponse)

17
Specifying temporal constraints

• specification reaction UDPPacket UDPResponse ( )
• pre return !ModelUDPPackets.isEmpty ( )
• post
• return
• _at_ModelUDPPackets.clone().contains(UDPRespons
  e)
• !ModelUDPPackets.contains(UDPResponse)
• time
• _at_ModelUDPPackets.clone().time(UDPResponse
  )
• lt 50numberOfNodes

18
Providing parallel inputs
s11
s12
s13
s14
Target System
s21
s22
s23
s24
s31
s32
s33

• Multisequence is used instead of sequence of
  stimuli

19
Collecting asynchronous reactions
11
21
31
r12
r11
Target System
12
22
32
r23
r22
r21
23
r33
r32
r31
33
Time

• Reactions form a partially ordered set

20
Constructing model action series
Stimuli
Reactions
Plain concurrency axiom
?
21
Modified test architecture
Test sequence construction
Test Engine
?
?
Test Action Iterator
Synchronization Manager
Serializer
Oracle
Mediator
Target system
Reaction Collector
22
Outline

• Theoretical background
• Practical considerations
• Case studies

23
Case studies

• Nortel Networks SOS 1994-1996
• about 30 serious bugs found
• MSR IPv6 2001-2002
• several serious bugs found
• MS Mobile IPv6 2002-2003

24
References

• http//www.ispras.ru/groups/rv/rv.html
• I. Burdonov, A. Kossatchev, V. Kuliamin.
  Asynchronous State Machines Classification and
  Applications to Testing. Proceedings of ISP RAS,
  Moscow, 2003. In Russian
• V. Kuliamin, A. Petrenko, I. Bourdonov, and A.
  Kossatchev. UniTesK Test Suite Architecture.
  Proceedings of FME 2002. LNCS 2391, pp.!77-88,
  Springer-Verlag, 2002.
• A. Petrenko. Specification-based Testing Towards
  Practice. Proceedings of PSI01, LNCS,
  Springer-Verlag. v. 2244, pp. 287-300
• V. Kuliamin, A. Petrenko, I. Bourdonov, A.
  Demakov, A. Jarov, A. Kossatchev, S.!Zelenov.
  Java Specification Extension for Automated Test
  Development. Proceedings of PSI'01. LNCS,
  Springer-Verlag. v. 2244, pp. 301-307.
• A. Petrenko, I. Bourdonov, A. Kossatchev, and V.
  Kuliamin. Experiences in using testing tools and
  technology in real-life applications. Proceedings
  of SETT'01, India, Pune, 2001.

25
Thank you!