STRESS: Systematic Testing of Protocol Robustness by Evaluation of Synthesized Scenarios

1
STRESS Systematic Testing of Protocol Robustness
by Evaluation of Synthesized Scenarios

• Sandeep Gupta and Ahmed Helmy
• University of Southern California
• http//catarina.usc.edu/stress

2
Overview

• Objective To develop a methodology to generate
  scenarios that cause a protocol to exhibit error
  or worst-case performance, and suggest protocol
  improvements.
• Simulations can be no better than the scenarios
• Approach Systematic evaluation of protocols via
• Semi-formal modeling of multicast protocols and
  ad hoc wireless protocols
• Thorough examination of the protocol state space
  using efficient forward/backward search
  algorithms
• Automatic synthesis of scenarios topology,
  protocol events, and network faults

3
Key Concepts

• Most of protocol complexity is due to the need to
  deal with non-ideal behavior (faults), such as
  packet loss
• Consequently, analysis of protocol
  correctness/performance in presence of faults
  called robustness analysis constitutes a
  significant goal of protocol verification/evaluati
  on
• In all our case studies, this approach has been
  very successful in protocol falsification (vs.
  verification), i.e., in finding errors

4
Key Concepts

• Robustness analysis is fault-oriented, where a
  scenario is generated for a given protocol and a
  target fault, so as to lead to error/worst-case
  performance
• Fault-oriented scenario generation borrows
  modeling and search techniques from approaches
  for test generation for digital VLSI
• Fault underlying network anomaly (e.g., packet
  loss, crash)
• Error undesirable protocol behavior (e.g.,
  collision, packet duplication)

5
Key Contributions

• Developed a fault-oriented approach for scenario
  generation that generates network configuration
  (topology) as part of the scenario
• Developed an error-oriented approach that
  generates scenarios starting from a target
  protocol error
• We have considered different types of topologies
  including LAN, range sets for wireless networks,
  and virtual LANs to capture end-to-end
  characteristics of multi-hop networks
• No parallel exists in test generation for VLSI,
  where tests are generated for a given circuit
  configuration

6
Previous Results

• Robustness study for multicast routing (PIM)
• Used scenario generation techniques to uncover
  looping, blackholes, overhead, join latency
  problems. Some encountered in practice two years
  after our results
• End-to-end multicast
• Synthesized worst/best case performance scenarios
  for the timer suppression mechanism employed by
  numerous reliable multicast protocols
• Wireless ad-hoc routing
• Generated scenarios for network partition for
  DSR,AODV
• Multicast over ATM network MARS
• Generated scenarios with blackholes and packet
  loss
• Mobile IP
• Generated blackholes, unrecoverable crash
  scenarios

7
Recent Research Emphasis

• Focus on ad-hoc wireless protocols
• High level of interest in wireless protocols
• Many ad-hoc protocol variations are
  application-specific and do not receive the same
  level of scrutiny as general protocols. Hence,
  these protocols can most benefit from automatic
  scenario generation
• These protocols provide new and unique challenges
  to automatic generation of scenarios topology
  representation, mobility, power

8
Recent Research Emphasis

• Analysis and comparison of ad-hoc MAC level
  protocols (e.g., 802.11, MACAW)
• Development of new error-oriented scenario
  generation for wireless protocols, that targets
  errors (including primary and secondary
  collisions)
• Systematic evaluation of multicast congestion
  control protocols (e.g., pgmcc)

9
Recent Results

• Ad hoc MAC protocols (MACAW,802.11)
• Developed tools for scenario generation using
• Forward search
• Error-oriented backward search with topology
  synthesis
• Synthesized scenarios for
• Channel under-utilization
• Primary and secondary collisions
• Unfairness and starvation
• Developed models for wireless and multi-hop
  topologies

10
Ad Hoc Mac Protocols (802.11)
n
1
Idle
Tx
k
WFCTS
o
p
i
j
Rx
Datao-p
Defer
l
m

• Unnecessary defer by all nodes in the range of
  transmitter i as the destination j ignores RTS
  while deferring gt channel underutilization
• Duration of o-p transmission is updated with
  every Datao-p and Ackp-o j updates its defer
  period
• i comes out of wait-for-CTS period and
  retransmits RTSi-j
• l updates defer period and backsoff as l has data
  for q
• This effect may cascade throughout the network
• Adding RRTS in 802.11 or MACAW handles this
  scenario but similar effects still exist with
  other scenarios

q
11
Identification of Target Collisions Data-Data
Collision (Primary Collision)
A
D
C
B
Messages which are heard, but no action is taken
12
Modeling Ad Hoc MAC Protocols MACAW Transition
Table
13
Topology and scenario generation MACAW
Possible receivers UK1 1, UK2 1 UK1 1, UK2
3 UK1 1, UK2 4 UK1 3, UK2 1 UK1 3,
UK2 2 UK1 3, UK2 4
2
1
RTS2-UK1
CTS3-UK2
3
3 could be located either in the range of 1 alone
(location x), or in the range of both 1 and 2
(location y)
14
Topology and scenario generation MACAW

• 4 should be present within 3s range, i.e. at
  any of the positions a, b, c or d

1
2
RTS2-3

• 4 can not be present at a and b, because it
  cannot be present within the range of 2

3
CTS3-4
4
15
Case Study (pgmcc)

• Single rate, scalability, responsiveness, TCP
  friendliness

Acker sends ACK
Sender sends packet
16
Case Study (pgmcc)

• Single rate, scalability, responsiveness, TCP
  friendliness

Sender
Receiver
Acker
Data
NACK
ACK
If the NACK is from a receiver worse than Acker,
then it is designated as the new Acker
Sender sends next packet
Packet lost by one receiver
Receiver sends NACK and Acker sends ACK
17
Case Study (pgmcc)

• Single rate, scalability, responsiveness, TCP
  friendliness

Sender
Receiver
Acker
Data
NACK
ACK
Sender sends next Packet
New Acker sends ACK
18
Recent Results (contd.)

• Multicast congestion control (pgmcc)
• Developed tools for scenario generation using
  forward search
• Synthesized scenarios for
• Wrong Acker selection
• Unnecessary starvation
• Wrong congestion notification

19
Scenarios for pgmcc

• The effect of receivers joining and leaving on
  the feedback measurements
• Wrong loss ratio estimate by a receiver due to
  leaving and joining causing wrong acker selection
• The effect of having special receivers as the
  acker which can leave or change
• Acker switch causing out-of-order ACKs and wrong
  congestion notification
• Acker leaving or crashing causing unnecessary
  starvation
• The effect of NACK suppression on feedback
• Wrong acker selection

20
Upcoming Milestones

• Completion of study of ad-hoc wireless protocols,
  including
• Scenario generation
• Scenario classification
• Protocol enhancement and re-verification
• Modeling of mobility and power
• Completion of study of pgmcc
• Release of unified tool for scenario generation
• Release of updated NS-2 models and utilities
• Report describing scenario generation, suggested
  protocol enhancements and demonstration via
  simulation

21
(No Transcript)