An Approach to IP Network Traffic Engineering

1
An Approach to IP Network Traffic Engineering

• NANOG
• Miami, FL
• Chris Liljenstolpe
• Cable Wireless
• chris_at_cw.net

2
Scope and Purpose

• Describes CWs Traffic Engineering methodology
  as well as some of the reasoning behind it.
• Not The One True Way, but a method that works
  for us.

3
Scale of the Previous Design

• Originally a flat network - one layer of routers
  interconnected over a complete PVC mesh.
• A Network event in 1998 on AS3561 educated
  the engineering staff on IGP scaling issues.
• This event lead to a week of network
  instability as it was re-engineered.
• At one time there were 380 routers in the direct
  mesh, accounting for 30k PVCs in the network
  760 direct IGP associations per router.

4
Hierarchy

• At one time there were 380 routers in the direct
  mesh, accounting for 30k PVCs IGP associations.
• Currently there are no more than 80 routers in
  any one mesh due to the addition of hierarchy.
• Due to the shrinking mesh sizes, and code
  optimization efforts, calculation times have
  dropped from 4 hours to 20 minutes.

5
Online vs. Offline

• We like to always know where our traffic is and
  where it is routed.
• Calculating optimal routing takes time on
  dedicated compute platforms

6
Layer 2 vs. Layer 3

• Utilizing IGP metrics to adjust traffic flows on
  an IP network leads to network-wide (and
  sometimes/usually, unplanned) effects in a large
  network, due to flooding.
• This can lead to the network equivalent of the
  midway game Hit the groundhog

7
IGP Use

• The IGP (in our case 2 level IS-IS) is only used
  for link state signaling in normal and most
  failure mode conditions.
• In the worst case dual failure mode condition,
  the IGP does provide real next-hop calculations.

8
IGP Metrics

• Because of the direct router-router adjacencies
  provided by the underlying network, a large set
  of IGP metrics are not needed.
• The set in use is small, and only used to select
  primary vs. secondary path, and discourage
  expensive link utilization in a multi-point
  failure that leads to multi-hop routing.

9
ATM to MPLS for TE

• ATM w/ PVCs worked quite nicely
• Except for ATM overhead
• And lack of high-speed router interfaces
• For our traffic engineering network, we are
  treating MPLS as an IP friendly ATM (actually
  more like Frame Relay, but never mind)

10
Will ?s Replace MPLS?

• Only when the bandwidth required for any
  router-router pair approaches the bandwidth
  available from a single ? on the DWDM plant AND
  the cost of a port on an OXC is significantly
  cheaper than an equivalent bandwidth port on an
  MPLS switch.
• When that occurs, the ?s will be provisioned
  just as the MPLS LSPs are statically with
  resilience.
• GMPLS may be the technology used to signal the
  path over the OXC, just as MPLS is used for the
  LSPs today.

11
Tools

• Currently the tools that compute the paths, and
  configure the layer 2 and layer 3 equipment with
  those paths are all developed and maintained
  in-house.
• Some have been in continual development and
  tweak mode for 6 years.

12
Futures

• Most link failures will be detected and handled
  at the layer 2 traffic engineering layer, instead
  of at layer 3.
• Path redundancy will grow from 2 to 4 paths per
  router-router pair.
• Developments optimization mathematics originally
  researched for circuit path layout and analog
  circuit design will be utilized in the path
  layout tools.
• Networks other than the IP backbone will utilize
  the traffic engineering core.