NetQuest: A Flexible Framework for LargeScale Network Measurement

1
NetQuest A Flexible Framework for Large-Scale
Network Measurement

Lili Qiu University of Texas at
Austin lili_at_cs.utexas.edu Joint work with Han
Hee Song and Yin Zhang ACM SIGMETRICS 2006 June
27, 2006
2
Motivating Scenario INetwork Diagnosis
AOL
CW
Sprint
ATT
UUNet
Qwest
Earthlink
3
Motivating Scenario IIPerformance Monitoring in
ISP Networks
ISP
4
Key Requirements

• Scalable work for large networks (e.g.,
  thousands of nodes)
• Flexible accommodate different applications
• Differentiated design
• Different quantities have different importance,
  e.g., a subset of paths belong to a major
  customer
• Augmented design
• Conduct additional experiments given existing
  observations, e.g., after measurement failures
• Multi-user design
• Multiple users interested in different parts of
  network or have different objective functions

Q Which measurement to conduct to estimate the
quantities of interest?
5
What We Want

• A function f(x) of link performance x
• We use a linear function f(x)Fx in this talk

Ex. 1 average link delay f(x)
(x1x11)/11 Ex. 2 end-to-end delay
Apply to any additive metric, eg. Log (1 loss
rate)
x2
3
2
x4
x1
x3
x5
x6
5
4
1
x10
x7
x8
x11
7
6
x9
6
Problem Formulation

• What we can measure e2e performance
• Network performance estimation
• Goal e2e performance on some paths ? f(x)
• Input yS (end-to-end performance on a subset of
  paths S), AS, and ySASx
• Output f(x)
• Design of measurement experiments
• Select which subset of paths S for active probing
  
• Network inference
• Infer x based on partial indirect observations

7
Design of Experiments

• State of the art
• Probe every path (e.g., RON)
• Not scalable since paths grow quadratically
  with nodes
• Rank-based approach sigcomm04
• Let A denote routing matrix
• Monitor rank(A) paths that are linearly
  independent to exactly reconstruct end-to-end
  path properties
• Still very expensive
• Our work
• If we can tolerate some error, we can
  significantly reduce measurement cost.
• How to select a given paths to probe to
  estimate f(x) as accurately as possible?
• Need a metric to quantify goodness of a given set
  of paths

8
Bayesian Experimental Design

• Many practical applications
• Car crash test, medicine design, software testing
• A good design should maximize the expected
  utility under the optimal inference algorithm
• Different utility functions yield different
  design criteria
• We use Bayesian A-optimal and Bayesian D-optimal
  design criteria

9
Bayesian A-Optimal Design

• Goal
• Minimize the squared error
• Maximize the following expected utility
• Let , where
  is covariance matrix of x
• Assuming a normal linear system, the Bayesian
  procedure yields
• This is equivalent to minimize

10
Bayesian D-Optimal Design

• Goal
• Maximize the expected gain in Shannon information
• Let , where
  is covariance matrix of x
• Assuming a normal linear system, the Bayesian
  procedure yields
  
• This is equivalent to minimize

11
Search Algorithm

• Given a design criterion , next step is
  to find s rows of A to minimize
• This problem is NP-hard
• We use a sequential search algorithm
• Start with an empty initial design
• Sequentially add rows to the design that results
  in the largest reduction in

12
Flexibility

• Differentiated design
• Give higher weights to the important rowsof
  matrix F
• Augmented design
• Identify the additional paths to probe such that
  in conjunction with previously monitored paths
  maximize the utility
• Multi-user design
• New design criteria a linear combination of
  different users design criteria

13
Network Inference

• Goal infer x s.t. ySASx
• Main challenge under-constrained problem
• L2-norm minimization
• L1-norm minimization
• Maximum entropy estimation

14
Evaluation Methodology

• Data sets
• Accuracy metric
• Only show the RTT results from PlanetLab
• Refer to our paper for more extensive results,
  e.g., loss estimation and comparison of inference
  algorithms

15
Comparison of DOE Algorithms Estimating
Network-Wide Mean RTT
A-optimal yields the lowest error.
16
Comparison of DOE Algorithms Estimating Per-Path
RTT
A-optimal yields the lowest error.
17
Differentiated Design Inference Error on
Preferred Paths
Lower error on the paths with higher weights.
18
Differentiated Design Inference Error on the
Remaining Paths
Error on the remaining paths increases slightly.
19
Augmented Design
A-optimal is most effective in augmenting an
existing design.
20
Multi-user Design
A-optimal yields the lowest error.
21
Summary

• Our contributions
• Apply Bayesian experimental design to large-scale
  network performance monitoring
• Develop a flexible framework to accommodate
  different design requirements
• Develop a toolkit on PlanetLab to measure and
  estimate network performance
• Our results
• Higher or comparable accuracy
• Flexible differentiated design, augmented
  design, multi-user design
• Scalable can handle 1,000,000 paths and 50,000
  links
• Future work
• Making measurement design fault tolerant
• Extend our framework to incorporate additional
  design constraints
• Apply our technique to traffic matrix estimation

22
Thank you!
23
Toolkit Architecture
24
Comparison of DOE Algorithms Estimating Per-Path
Loss
The performance difference across different
experimental designs is smaller for loss
estimation.
25
Comparison of Inference Algorithms RTT Inference
Different inference algorithms perform similarly
for delay inference.
26
Comparison of Inference Algorithms Loss Inference
The inference algorithms that enforce
non-negativity constraints perform better.
27
Motivating Scenario IIIMonitoring Performance in
Overlay Networks
28
Multi-user Design

• Separate design and separate inference
  (sep./sep.)
• Each user individually determines the set of
  paths to monitor, and makes inference only based
  on his/her own observations.
• Separate design and joint inference (sep./joint)
• An enhancement of the previous version.
• Users still individually decide which paths to
  monitor, but they make inference based on the
  observations made from all users.
• Augmented design and joint inference (aug./joint)
• In the augmented design, we first design
  measurement experiments for user 1, and then
  apply the the augmented design to construct
  measurement experiments for user 2. We continue
  the process for all the other users.
• Union design and joint inference (union/joint)
• In the union design, we take a union of all the
  paths that are interesting to at least one user,
  and then apply the (basic) measurement design.
• Joint design and joint inference (joint/joint)
• Unlike in the union design, where all interesting
  paths are treated equally, in joint design we set
  a path's weight to be the square root of the
  number of users who are interested in the path.

29
Multi-user Design
Joint/joint gt union/joint aug/joint gt sep/joint
gt sep/sep
30
Motivation Network Benchmarking

• 1000s of virtual networks over the same physical
  network
• Wants to summarize the performance of each
  virtual net
• E.g. traffic-weighted average of individual
  virtual path performance (loss, delay, jitter, )
• Similar problem exists for monitoring
  per-application/customer performance
• Challenge Cannot afford to monitor all
  individual virtual paths
• N2 explosion times 1000s of virtual nets
• Solution monitor a subset of virtual paths and
  infer the rest
• Q which subset of virtual paths to monitor?

31
Motivation Network Diagnosis

• Clients probe each other
• Use tomography/inference to localize trouble spot
• E.g. links/regions with high loss rate, delay
  jitter, etc.
• Challenge Pair-wise probing too expensive due to
  N2 explosion
• Solution monitor a subset of paths and infer the
  link performance
• Q which subset of paths to probe?

AOL
CW
Sprint
UUNet
ATT
Qwest
EarthLink
32
More Examples

• Wireless sniffer placement
• Input
• A set of locations to place wireless sniffers
• Not all locations possible some people hate to
  be surrounded by sniffers
• Monitoring quality at each candidate location
• E.g. probabilities for capturing packets from
  different APs
• Expected workload of different APs
• Locations of existing sniffers
• Output
• K additional locations for placing sniffers
• Cross-layer diagnosis
• Infer layer-2 properties based on layer-3
  performance
• Which subset of layer-3 paths to probe?

33
Practical Network Diagnosis

• Ideal
• Every network element is self-monitoring,
  self-reporting, self-, there is no silent
  failures
• Oracle walks through the haystack of data,
  accurately pinpoints root causes, and suggests
  response actions
• Reality
• Finite resources (CPU, BW, human cycles, )
• cannot afford to instrument/monitor every
  element
• Decentralized, autonomous nature of the Internet
• infeasible to instrument/monitor every
  organization
• Protocol layering minimizes information exposure
• difficult to obtain complete information at
  every layer

Practical network diagnosis Maximize diagnosis
accuracy under given resource constraint and
information availability
34
Design of Diagnosis Experiments

• Input
• A candidate set of diagnosis experiments
• Reflects infrastructure constraints
• Information availability
• Existing information already available
• Information provided by each new experiment
• Resource constraint
• E.g., number of experiments to conduct (per
  hour), number of monitors available
• Output A diagnosis experimental plan
• A subset of experiments to conduct
• Configuration of various control parameters
• E.g., frequency, duration, sampling ratio,

35
NetQuest

• Achieves scalability and flexibility by combining
  
• Bayesian experimental design
• Statistical inference
• Developed in the context of e2e performance
  monitoring
• Can extend to other network monitoring/ diagnosis
  problems

36
Beyond Networking

• Software debugging
• Select a given number of tests to maximize the
  coverage of corner cases
• Car crash test
• Crash a given number of cars to find a maximal
  number of defects
• Medicine design
• Conducting a given number of tests to maximize
  the chance of finding an effective ingredient
• Many more

37
Motivation
Server
Sprint
Server
CW
AOL
ATT
UUNet
Qwest
Earthlink
Server
Server
38
Motivation (Cont.)

• Many applications require scalable network
  monitoring
• Server selection
• Network diagnosis
• Traffic engineering
• Overlay networks
• Peer-to-peer applications
• Scalable network measurement is important to
• ISPs
• Enterprise and university networks
• Application and protocol designers
• End users

39
Bayesian Experimental Design

• Many practical applications
• Car crash test, medicine design, software testing
• A good design maximizes the expected utility
  under the optimal inference algorithm
• Different utility functions yield different
  design criteria
• Let , where
  is covariance matrix of x
• Bayesian A-optimality
• Goal minimize the squared error
• Bayesian D-optimality
• Goal maximize the expected gain in Shannon
  information