Title: Counter Braids: A Novel Counter Architecture for PerFlow Measurement
1Counter Braids A Novel Counter Architecture for
Per-Flow Measurement
- Yi Lu
- Stanford University
- Joint work with Andrea Montanari, Balaji
Prabhakar, Sarang Dharmapurikar and Abdul Kabbani - May 21, 2008
2Network Measurement
packets
- Measurability is key to understanding networks
today and engineering future networks - Routers collect traffic statistics for
- Accounting / billing
- Traffic engineering
- Security / Forensics
counters
3Per-Flow Network Measurement
- Flow a sequence of packets satisfying a common
set of rules.
easy
hard
harder
- aggregate
- per-port
- manually defined filters
- more refined filtering
- per-prefix
- per source address
- per flow tuple (src addr, dest addr, src port,
dest port, type)
- Products
- Cisco Systems NetFlow
- Juniper Networks cflowd
- Huawei Technologys Netstream
a large number of flows (can be O(100,000) )
a large number of flows AND flows come and go
a small number of flows (
4Per-Flow Network Measurement
per-flow in usual sense
- large number of counters.
- flows being measured vary over time. Need
one-to-one relationship binding flows and
counters. (flow-to-counter association) - dynamically constructed
- randomly accessible
5In enterprise networks
Packets arrive slowly
DRAM
6In enterprise networks
hash table
Flow ID 1
Flow ID 12
Flow ID 3
Flow ID 23
7Modern Core Routers
Packets arrive Fast (every 15 ns)
SRAM
Infeasible
8Approximate counting
Previous Approaches
1. Approximate counting. (Estan and Varghese,
2004)
SRAM
9Previous Approaches
2. Hybrid SRAM-DRAM architecture (Shah et. al.
2001, Ramabhadran and Varghese 2003, Zhao et.
al. 2006)
10Previous Approaches
2. Hybrid SRAM-DRAM architecture (Shah et. al.
2001, Ramabhadran and Varghese 2003, Zhao et.
al. 2006)
SRAM
DRAM
Shallow counters
Deep counters
11Our Approach
- Counter Braids
- Small and SRAM-only
- No dedicated counters for each flow, hence no
need for flow-to-counter association - Compresses as it counts
- Simple updates, without decompression
Counter Braids
12Our Approach
Offline
- Compression ratio
- with a high-complexity decoder, asymptotically
equals Shannon entropy - a simple, fast message-passing decoder, a few
bits per flow - related to compressed sensing and LDPC codes
13Counter Braids
In anticipation
14Counter Braids
In anticipation
graphs generated on-the-fly using hash functions
15Design Intuition
1. Hash collisions can be helpful.
2. Braiding shares bits among flows.
16Hash Collisions
- n flows, m counters.
- Hash table. Single hash. No collisions.
- Count-Min. Multiple hashes. At least 1
hash has no collision. - (Cormode and
Muthukrishnan) - Counter Braids. Multiple hashes. Collisions help.
17Braiding
18Design Intuition
1. Hash collisions can be helpful.
2. Braiding shares bits among flows.
19Update in SRAM
2
0
3
0
packet
packet
3
20Message-passing Decoder
- one message on each edge
- In each iteration, messages go from
- flows to counters
- then from
- counters to flows
21Message-passing Decoder
- one message on each edge
- In each iteration, messages go from
- flows to counters
- then from
- counters to flows
22Message from Counters
Anti-monotonicity
23Message from Flows
Anti-monotonicity
even iteration underestimate odd iteration
overestimate
24Decoding example
25Analytical Result
n flows, m counters.
Fix m/n and let n go to infinity, we can compute
a threshold above which all flows are decoded
correctly.
26Simulation
27Conclusion
- Counter Braids, an SRAM-only solution for
per-flow measurement, which compresses while
counting - Simple, analyzable message-passing decoding
algorithm, whose compression rate is close to the
entropy of flow sizes - Current work FPGA implementation of Counter
Braids - Further work Lossy compression
28Trace Simulation
- Two OC-48 (2.5 Gbps) one-hour contiguous traces
collected by CAIDA at a San Jose router. - Each measurement epoch (5 minutes) contains 0.9
million flows
29Computation Cost of Decoder
- Decoding 1 million flows on a two-layer Counter
Braids takes - 15 seconds on a 2.6 GHz machine.
30Compressed Sensing
Candes and Tao, Donoho, Indyk, Muthukrishnan,
Wainwright, and many others
- Differences
- Counting is incremental, hence sparse
transformation is necessary. - Braiding minimizes space in addition to
minimizing number of samples.
31LDPC codes
- Caire, Shamai, Verdú Noiseless data compression
with LDPC codes - Sparse matrix and belief propagation (BP)
decoder. - For q-ary source symbols, the complexity of BP
decoder increases with q2, and the prior
distribution of flow size is needed - Our message-passing algorithm does not use the
prior distribution of flow sizes. It only takes
advantage of the sparsity. ( P (f min) )
32Density Evolution
- We use the following measure of error
- As n becomes large, the evolution of Pe
concentrates around the iterative path
- There is a sharp threshold for the number of
counters such that above the threshold, Pe goes
to 0 and below the threshold, Pe 0 always.