Traffic Engineering for ISP Networks

1
Traffic Engineering for ISP Networks

• Jennifer Rexford
• Computer Science Department
• Princeton University
• http//www.cs.princeton.edu/jrex

2
Outline

• Internet routing
• Overview of the Internet routing architecture
• Shortest-path link-state routing between edge
  routers
• Optimization Tune routing to the traffic
• Optimizing routing given a topology and traffic
  matrix
• Local search to select the integer link weights
• Design for optimizability Design routing
  protocol
• Optimal traffic engineering with link-state
  routing
• Tomography Infer the traffic matrix
• Estimating traffic matrix from routing and link
  load

3
Internet Routing
4
Autonomous Systems (ASes)

• Internet is divided into Autonomous Systems
• Distinct regions of administrative control
• Routers/links managed by a single institution
• Service provider, company, university,
• Hierarchy of Autonomous Systems
• Large, tier-1 provider with a nationwide backbone
• Medium-sized regional provider with smaller
  backbone
• Small network run by a single company or
  university
• Cooperate to ensure end-to-end reachability

5
Interdomain Routing

• AS-level topology
• Destinations are IP prefixes (e.g., 12.0.0.0/8)
• Nodes are Autonomous Systems (ASes)
• Edges are links and business relationships

4
3
5
2
6
7
1
Web server
Client
6
Example Backbone Abilene Internet2 Newtork
7
Points-of-Presence (PoPs)

• Inter-PoP links
• Long distances
• High bandwidth
• Intra-PoP links
• Short cables between racks or floors
• Aggregated bandwidth
• Links to other networks
• Wide range of media and bandwidth

Inter-PoP
Intra-PoP
Other networks
8
Intradomain Routing Shortest-Path Routing

• Path-selection model
• Destination-based
• Load-insensitive (e.g., static link weights)
• Minimum hop count or sum of link weights

2
1
3
1
4
2
1
5
4
3
9
Computing Shortest Paths Link-State Routing

• Topology discovery
• Routers flood information to learn the topology
• Each router constructs a link-state database

2
1
3
1
3
2
1
5
4
3
10
Computing Shortest Paths Link-State Routing

• Shortest-path computation
• Each router runs Dijkstras shortest-path
  algorithm
• Computes the next hop to reach other routers

2
1
3
1
3
2
1
5
4
3
11
Computing Shortest Paths Link-State Routing

• Packet forwarding
• Each router maintains a forwarding table
• To forward incoming packets to the right next-hop
  link

2
1
3
1
3
2
1
5
4
3
12
Our Focus Traffic Engineering (TE)

• Adjusting routing to the flow of traffic
• How should network administrators run their
  networks?
• Specifically, how should they set the link
  weights?
• Designing protocols for better traffic
  engineering
• How should future routing protocols be designed?
• Specifically, how to make TE efficient and easy?
• Collecting measurements of the offered traffic
• How should administrators learn the traffic
  matrix?
• Specifically, how to infer the matrix from link
  loads?

13
Optimization Tuning Routing to the Traffic

• Joint work with Bernard Fortz and Mikkel Thorup
• http//www.cs.princeton.edu/jrex/papers/ieeecomm0
  2.pdf
• http//www.cs.princeton.edu/jrex/papers/opthand04
  .pdf

14
Link Weights Control the Flow of Traffic

• Routers compute paths
• Shortest paths as sum of link weights
• Operators set the link weights
• To control where the traffic goes

2
1
3
1
3
2
3
1
5
4
3
15
Heuristics for Setting the Link Weights

• Proportional to physical distance
• Cross-country links have higher weights than
  local ones
• Minimizes end-to-end propagation delay
• Inversely proportional to link capacity
• Smaller weights for higher-bandwidth links
• Attracts more traffic to links with more capacity
• Tuned based on the offered traffic
• Network-wide optimization of weights based on
  traffic
• Directly minimizes key metrics like max link
  utilization

16
Why Are the Link Weights Static?

• Strawman alternative load-sensitive routing
• Link metrics based on traffic load
• Flood dynamic metrics as they change
• Adapt automatically to changes in offered load
• Reasons why this is typically not done
• Delay-based routing unsuccessful in the early
  days
• Oscillation as routers adapt to out-of-date
  information
• Most Internet transfers are very short-lived
• Research and standards work continues
• but operators have to do what they can today

17
Big Picture Measure, Model, and Control
Network-wide what if model
Offered traffic
Topology/ Configuration
Changes to the network
measure
control
Operational network
18
Traffic Engineering in an ISP Backbone

• Topology
• Connectivity and capacity of routers and links
• Traffic matrix
• Offered load between points in the network
• Link weights
• Configurable parameters for Interior Gateway
  Protocol
• Performance objective
• Balanced load, low latency, service level
  agreements
• Question Given the topology and traffic matrix
  in an IP network, which link weights should be
  used?

19
Key Ingredients of Our Approach

• Measurement
• Topology monitoring of the routing protocols
• Traffic matrix widely deployed traffic
  measurement
• Network-wide models
• Representations of topology and traffic
• What-if models of shortest-path routing
• Network optimization
• Efficient algorithms to find good configurations
• Operational experience to identify key
  constraints

20
Formalizing the Optimization Problem

• Input graph G(R,L)
• R is the set of routers
• L is the set of unidirectional links
• cl is the capacity of link l
• Input traffic matrix
• Mi,j is traffic load from router i to j
• Output setting of the link weights
• wl is weight on unidirectional link l
• Pi,j,l is fraction of traffic from i to j
  traversing link l

21
Multiple Shortest Paths With Even Splitting
Values of Pi,j,l
22
Defining the Objective Function

• Computing the link utilization
• Link load ul Si,j Mi,j Pi,j,l
• Utilization ul/cl
• Objective functions
• min(maxl(ul/cl))
• min(Sl f(ul/cl))

23
Complexity of the Optimization Problem

• NP-hard optimization problem
• No efficient algorithm to find the link weights
• Even for the simple convex objective functions
• Why cant we just do multi-commodity flow?
• E.g., solve the multi-commodity flow problem
• and the link weights pop out as the dual
• Because IP routers cannot split arbitrarily over
  ties
• What are the implications?
• Have to resort to searching through weight
  settings

24
Optimization Based on Local Search

• Start with an initial setting of the link weights
• E.g., same integer weight on every link
• E.g., weights inversely proportional to link
  capacity
• E.g., existing weights in the operational network
• Compute the objective function
• Compute the all-pairs shortest paths to get
  Pi,j,l
• Apply the traffic matrix Mi,j to get link loads
  ul
• Evaluate the objective function from the ul/cl
• Generate a new setting of the link weights

repeat
25
Making the Search Efficient

• Avoid repeating the same weight setting
• Keep track of past values of the weight setting
• or keep a small signature (e.g., a hash) of
  past values
• Do not evaluate a weight setting if signatures
  match
• Avoid computing the shortest paths from scratch
• Explore weight settings that changes just one
  weight
• Apply fast incremental shortest-path algorithms
• Limit the number of unique values of link weights
• Do not explore all 216 possible values for each
  weight
• Stop early, before exploring the whole search
  space

26
Incorporating Operational Realities

• Minimize number of changes to the network
• Changing just 1 or 2 link weights is often enough
• Tolerate failure of network equipment
• Weights settings usually remain good after
  failure
• or can be fixed by changing one or two weights
• Limit dependence on measurement accuracy
• Good weights remain good, despite random noise
• Limit frequency of changes to the weights
• Joint optimization for day and night traffic
  matrices

27
Application to ATTs Backbone Network

• Performance of the optimized weights
• Search finds a good solution within a few minutes
• Much better than link capacity or physical
  distance
• Competitive with multi-commodity flow solution
• How ATT changes the link weights
• Maintenance done every night from midnight to 6am
• Predict effects of removing link(s) from the
  network
• Reoptimize the link weights to avoid congestion
• Configure new weights before disabling equipment

28
Example from My Visit to ATTs Operations Center

• Amtrak repairing/moving part of the train track
• Need to move some of the fiber optic cables
• Or, heightened risk of the cables being cut
• Amtrak notifies us of the time the work will be
  done
• ATT engineers model the effects
• Determine which IP links go over the affected
  fiber
• Pretend the network no longer has these links
• Evaluate the new shortest paths and traffic flow
• Identify whether link loads will be too high

29
Example Continued

• If load will be too high
• Reoptimize the weights on the remaining links
• Schedule the time for the new weights to be
  configured
• Roll back to the old weight setting after Amtrak
  is done
• Same process applied to other cases
• Assessing the networks risk to possible failures
• Planning for maintenance of existing equipment
• Adapting the link weights to installation of new
  links
• Adapting the link weights in response to traffic
  shifts

30
Conclusions on Traffic Engineering

• IP networks do not adapt on their own
• Routers compute shortest paths based on static
  weights
• Service providers need to adapt the weights
• Due to failures, congestion, or planned
  maintenance
• Leads to an interesting optimization problems
• Optimize link weights based on topology and
  traffic
• Optimization problem is computationally difficult
• Forces the use of efficient local-search
  techniques
• Results of the local search are pretty good
• Near-optimal solutions that minimize disruptions

31
Ongoing Work

• Robust link-weight assignments
• Link/node failures
• Range of traffic matrices
• More complex routing models
• Hot-potato routing
• BGP routing policies
• Interaction between ASes
• Inter-AS negotiation for joint optimization
• Grappling with scalability and trust issues

32
Design for Optimizability Optimal Link-State
Routing Protocol

• Joint work with Dahai Xu
• and Mung Chiang
• http//www.cs.princeton.edu/jrex/papers/pefti.pdf
  

33
Revisiting TE With Link-State Routing Protocols

• Advantages of link weights
• One parameter for each unidirectional link
• Hop-by-hop forwarding (no tunneling, no per-flow
  state)
• New routes computed automatically after failure
• Changing just a few weights can alleviate
  congestion
• Disadvantages of link weights
• Computationally expensive optimization
• Suboptimal distribution of traffic
• (Disruptions when changing the link weights)

34
Example of Inefficient TE

• Simple topology
• Demand of 300 units
• All on top path 300 utilization of top path
• All on bottom path 150 utilization of bottom
  path
• Even splitting 150 on top path, 75 on bottom

c1 100
t
s
c2 200
35
Stepping Back Design for Optimizability

• Two research approaches
• Bottom up do the best with what you have
• Top down design systems that are easier to
  manage
• Design for manage-ability
• If you are both the professor and the student,
  you create exam questions that are easy to
  answer. Mung Chiang
• Knowing what we know now
• How should intradomain routing protocols work
• to make TE more efficient and hopefully easier?

36
Optimal TE With Multicommodity Flow

• Problem with shortest-path routing
• Inflexible even splitting over shortest paths
• Optimal distribution of traffic
• Send traffic over any paths in any proportions
• Using tunneling to force traffic on the paths
• Realizable with MultiProtocol Label Switching
  (MPLS)
• Disadvantage of MPLS high overhead
• Large number of paths between pairs of routers
• Must adapt the splitting ratios after each failure

37
Can We Have Link-State Routing and Optimal TE?

• Link-state routing and hop-by-hop forwarding
• Single weight on each link
• Local rule to compute splitting over paths
• Each router forwards based only on the
  destination
• Link-state routing ! shortest-path routing
• Routers could use other traffic-splitting rules
• as long as they are locally computable
• only from the link weights

38
Forward Packet Based on Link Weights

• Available information at router u
• wu,v weights for all links
• dut shortest distance from u to t
• hu,vt distance gap (dvt wu,v dut)

distance gap of 1
2
1
3
1
3
2
1
5
4
3
distance gap of 0
39
Traffic-Splitting Function

• Relative flow distributed on outgoing links
• G(hu,vt) proportion sent out link v toward t
• Split traffic to t in proportion
• Even splitting
• G(hu,vt) is 1 if hu,vt 0 (all traffic on
  shortest paths)
• G(hu,vt) is 0 if hu,vt gt 0 (no traffic on longer
  paths)

G(hu,vt)
SG(hu,jt)
40
Exponential Splitting

• Exponentially diminishing traffic on longer paths
• Proportion on path i proportional to exp(-pi)
• where pi is the cost of path i

41
Optimal TE

• A surprising result
• This kind of link-state routing can achieve
  optimal TE
• Optimality
• Can realize the multicommodity flow traffic
  distribution
• Expressible in terms of settings of link weights
• Efficient algorithm
• Computationally tractable to compute optimal
  weights
• for a given traffic matrix and capacitated
  topology

42
Intuition Behind the Theory
Realizable with link-state routing
Optimal flow routing
Feasible flow routing
43
Finding Link-State Protocols That Achieve Optimal
TE

• Need an additional objective function
• To find solutions expressible in terms of link
  weights
• But we already have an objective function
• So, how can we add another one?
• First, solve the original optimization problem
• To determine the load on each link at optimality
• i.e., the necessary capacity of each link
• Then, solve a second optimization problem
• On this new topology, with our new objective

44
TE Optimization Problem Compute Necessary
Capacity

• Convex objective
• Min sum f() over all links
• Constraints
• Flow conservation must carry the traffic matrix
• Capacity constraint cannot exceed link capacity
• Variables
• Flow along each path
• Given
• Traffic matrix and link capacities

45
New Optimization Problem

• Necessary link capacity
• Flow on link u,v in the multicommdity-flow
  solution
• becomes the capacity of the link in the new
  problem
• In the new optimization problem
• Any feasible solution is optimal
• relative to the original optimization problem
• So, now we can pick a new objective
• Key intuition maximizing entropy

46
Entropy Maximization

• Assume we could enumerate all paths from s to t
• (Though in practice this wouldnt be practical)
• Entropy
• xks,t fraction of traffic from s to t put on
  path k
• z(x) - x log(x) entropy function
• New objective maximize entropy
• S Mi,j (S z(xks,t))

47
High-Level Overview of the Details

• NEM problem always has a solution
• Earlier multicommodity flow solution
• Solving directly is not efficient
• Need to avoid enumerating all the paths
• Solving with dual decomposition
• Derivation leads to the exponential function
• for splitting traffic over the multiple paths
• Derivation also leads to weight-setting algorithm
• Computationally efficient, better than local
  search

48
Conclusions

• Protocols induce optimization problems
• E.g., setting link weights to do traffic
  engineering
• Complexity of the optimization problems
• A symptom that the protocol is not quite right
• E.g., NP-hard problem and suboptimal traffic flow
• Design for optimizability
• Design the protocol to be easy to optimize
• using optimization theory as a protocol design
  tool

49
Tomography Inferring the Traffic Matrix

• Work by Yin Zhang, Matthew Roughan, Nick
  Duffield, and Albert Greenberg
• http//www.cs.utexas.edu/yzhang/papers/tomogravit
  y-sigm03.pdf

50
Computing the Traffic Matrix Mi,j

• Hard to measure the traffic matrix
• IP networks transmit data as individual packets
• Routers do not keep traffic statistics, except
  link utilization on (say) a five-minute time
  scale
• Need to infer the traffic matrix Mi,j from
• Current topology G(R,L)
• Current routing Pi,j,l
• Current link load ul
• Link capacity cl

51
Inference Network Tomography
From link counts to the traffic matrix
Sources
3Mbps
5Mbps
4Mbps
4Mbps
Destinations
52
Tomography Formalizing the Problem

• Ingress-egress pairs
• p is a ingress-egress pair of nodes (i,j)
• xp is the (unknown) traffic volume for this pair
  Mi,j
• Routing
• Plp is proportion of ps traffic that traverses l
• Links in the network
• l is a unidirectional edge
• ul is the observed traffic volume on this link
• Relationship u Px (work backwards to get x)

53
Tomography One Observation Not Enough

• Linear system of n nodes is underdetermined
• Number of links e is around O(n)
• Number of ingress-egress pairs c is O(n2)
• Dimension of solution sub-space at least c - e
• Multiple observations are needed
• k independent observations (over time)
• Stochastic model with Poisson iid counts
• Maximum likelihood estimation to infer matrix
• Doesnt work all that well in practice

54
Approach Used at ATT Tomo-gravity

• Gravitational assumption
• Ingress point a has traffic via
• Egress point b has traffic veb
• Pair (a,b) has traffic proportional to via veb

9
20
21
10
55
Approach Used at ATT Tomo-gravity

• Problem with gravity model
• Gravity model ignores the load on the inside
  links
• Gravity assumption isnt always 100 correct
• Resulting traffic matrix might not satisfy the
  link loads
• Combining the two techniques
• Gravity find a traffic matrix using the gravity
  model
• Tomography find the family of traffic matrices
  consistent with all link load statistics
• Tomo-gravity find the tomography solution that
  is closest to the output of the gravity model
• Works extremely well (and fast) in practice

56
Conclusions on Tomography

• Routers dont reveal much information about
  traffic
• Measurement provides a network-wide view
• E.g., network topology and traffic matrix
• Available data induces a tomography problem
• Input network topology, routing, and link loads
• Output inferred traffic matrix
• Design for tomography?
• Design future monitoring systems to induce
  easier-to-solve tomography problems?

57
Conclusions

• Internet routing is a rich problem space
• Designed incrementally as Internet evolved
• Not designed with network management in mind
• Network management bottom up
• Working with what you have
• Tuning link weights, and inferring traffic
  matrices
• Exciting new area design for manageability
• Protocols that are easy to tune
• Measurements that make inference easy