New Directions in Traffic Measurement and Accounting

1
New Directions in Traffic Measurement and
Accounting

• Focusing on the Elephants, Ignoring the Mice

Cristian Estan and George Varghese University of
California, San Diego
2
Talk outline

• Problem definition
• Sample and hold
• Multistage filters
• Validation, measurements
• Conclusions

3
Traffic analysis today
Workstation
Router
Concise analysis results
Collection and analysis software
Large raw data
Measurement module
Sampled packets
Offline analysis
Fast link
4
Our research agenda
Router
Concise analysis results
Real-time analysis
Measurement module

• Is it doable?
• Is it better?

Fast link
5
What is traffic analysis used for?

• Network planning need to know traffic between
  pairs of networks (traffic matrix)
• Accounting usage based billing
• Detecting DoS attacks flood attacks
• Application characterization breaking up the
  traffic based on port numbers

6
Common abstractions

• Packets are grouped together into streams based
  on header fields
• Traffic matrix by source and destination AS
• DoS attacks by destination IP address
• Measuring large streams (this paper)
• Estimating the number of active streams (poster)

7
Why is measuring streams hard?

• Cheap memories (DRAM) are too slow to count all
  packets
• Fast memories (SRAM) are too small to keep
  counters for all streams
• Opportunity elephants matter, mice dont
• Problem usually we dont know in advance which
  streams are large

8
Problem definition

• Given a fixed definition for streams, measure
  large streams accurately
• Large above 1 of link capacity over a 1 minute
  interval
• Assumptions
• Mice dont matter
• Accuracy of results important

9
Talk outline

• Problem definition
• Sample and hold
• Multistage filters
• Validation, measurements
• Conclusions

10
How does sample and hold work?
stream memory
Sample
Insert
stream1 1
11
How does sample and hold work?
stream memory
Update
stream1 1
stream1 2
12
How does sample and hold work?
stream memory
Sample
stream1 2
Insert
stream2 1
13
Why is sample hold better?
Sample and hold
Ordinary sampling
14
How much better is it?

• Comparing the relative error of the estimate for
  a stream at 1/F of the link bandwidth
• Memory limited to M entries

15
Talk outline

• Problem definition
• Sample and hold
• Multistage filters
• Validation, measurements
• Conclusions

16
Multistage filters

• Characteristics
• No large stream is ever omitted
• Very few entries are used by small streams
• Better performance but implementation and tuning
  is more complex

17
How do multistage filters work?
stream memory
Array of counters
Hash(Pink)
18
How do multistage filters work?
stream memory
Array of counters
Hash(Green)
19
How do multistage filters work?
stream memory
Array of counters
Hash(Green)
20
How do multistage filters work?
stream memory
21
How do multistage filters work?
stream memory
Collisions are OK
22
How do multistage filters work?
Reached threshold
stream memory
stream1 1
Insert
23
How do multistage filters work?
stream memory
stream1 1
24
How do multistage filters work?
stream memory
stream1 1
stream2 1
25
How do multistage filters work?
stream memory
Stage 1
stream1 1
26
Conservative update
Gray all prior packets
27
Conservative update
28
Conservative update
29
Talk outline

• Problem definition
• Sample and hold
• Multistage filters
• Validation, measurements
• Conclusions

30
Validation

• Analytical evaluation
• Comparison of analytical results to measured
  performance
• Comparison of full measurement devices using
  different algorithms

31
On traces, algorithms much better than analysis
predicts
Percentage of small streams passing filter (log
scale)
Theory Zipf Actual
Conservative update
Number of stages
32
Measurement results

• Setup OC48 trace, 100,000 TCP flows, 5 second
  intervals, ordinary sampling - unlimited memory,
  sampling 1 in 16 our algorithms - 1Mbit, adapting
  parameters to keep it around 90 full
• Large streams (above 0.1) ordinary sampling has
  an error of 9 sample and hold 0.075, multistage
  filter 0.037

33
Talk outline

• Problem definition
• Sample and hold
• Multistage filters
• Validation, measurements
• Conclusions

34
Our contributions

• Abstraction
• Real-time packet analysis abstractions can help
  systematize router implementations.
• While the notion of elephants and mice is
  inherent in earlier work, we abstracted
  measurement of large streams - it can be used by
  many applications.

35
Our contributions (2)

• Algorithms
• Sample and hold is a simple and efficient
  algorithm for identifying and measuring large
  streams.
• Multistage filters with conservative update
  perform better but are more complex.
• Both can be used for real-time as well as offline
  analysis.

36
Our contributions (3)

• Validation
• Theoretical results that make no assumptions on
  traffic distribution
• Simulations on traces are orders of magnitude
  better
• Preliminary hardware design (John Huber)
  indicates feasibility at OC192 speeds

37
Thank you!