Routing Design in Operational Networks: A Look from the Inside

1
Routing Design in Operational NetworksA Look
from the Inside

• David A. Maltz, Geoffrey Xie, Jibin Zhan, Hui
  Zhang
• Carnegie Mellon University
• Gisli Hjalmtysson, Albert Greenberg
• ATT Labs Research

2
The Problem of Routing Design
policy
HostC
HostD
iBGP
ACLs
HostA
eBGP
RIP
OSPF
EIGRP
HostB
3
Many Routing Designs Possible
AS1
AS2
AS3
Drop Alt-gtB
Drop Alt-gtB
Multiple ASs BGP
Multiple OSPF instances
Drop Alt-gtB
Packet filters
4
Routing Design

• Selecting routing protocols
• Configuring their boundaries
• Setting the policies that control their
  interaction
• Adding packet filters, other mechanisms
• Routing design fundamentally establishes the
  networks properties
• Remains constant as network expands
• Details of protocol, path computation are
  second-order effects
• Topology doesnt say much about reachability

5
Reachability Example
C
A
Permit A-gtC
Internet
Permit B-gtC
B

• Enterprise with two remote offices
• Only AB should be able to talk to server C

6
Reachability Example
C
A
Permit A-gtC
Internet
Permit B-gtC
B

• Network designers add two links for robustness
• Configure routing protocols to use new links in
  failure

7
Reachability Example
Permit A-gtC
C
A
Permit A-gtC
Permit A-gtC
Permit B-gtC
Internet
Permit B-gtC
B

• Designers apply packet filters to new links

8
Reachability Example
Permit A-gtC
C
A
Permit A-gtC
Permit A-gtC
Permit B-gtC
Internet
B
9
Reachability Example
Permit A-gtC
C
A
Permit A-gtC
Permit A-gtC
Permit B-gtC
Internet
B

• Packet from B-gtC dropped!
• Testing under normal conditions wont find this
  error!

10
How Are Routing Designs Expressed Today?

• interface Ethernet0
• ip address 6.2.5.14 255.255.255.128
• interface Serial1/0.5 point-to-point
• ip address 6.2.2.85 255.255.255.252
• ip access-group 143 in
• frame-relay interface-dlci 28
• router ospf 64
• redistribute connected subnets
• redistribute bgp 64780 metric 1 subnets
• network 66.251.75.128 0.0.0.127 area 0
• router bgp 64780
• redistribute ospf 64 match route-map
  8aTzlvBrbaW
• neighbor 66.253.160.68 remote-as 12762
• neighbor 66.253.160.68 distribute-list 4 in

access-list 143 deny 1.1.0.0/16 access-list 143
permit any route-map 8aTzlvBrbaW deny 10 match
ip address 4 route-map 8aTzlvBrbaW permit 20
match ip address 7 ip route 10.2.2.1/16 10.2.1.7
11
Lots of Configuration Files
2000
Lines in config file
1000
0
881
0
Router ID
12
Problems with State of the Art

• No good way to visualize or describe routing
  design
• Impossible to establish linkage between
  configurations and routing design
• Only a few textbook routing designs are widely
  known

13
Approaches?

• Option 1 High-level design compiled down to
  configuration commands
• Feasible?
• What are the constructs?
• How to capture design intents?
• Our starting point Bottom-up white-box approach
• Start with router configuration files
• Reverse-engineer the routing design

14
Contributions

• Abstractions for modeling routing design
• Routing Process Graph
• Routing Instance Graph
• Reverse-engineering methodology
• Anonymization of configuration files
• Tools to convert configurations into model
• Study of 31 production networks using model
• Verified some bits of common wisdom
• Found counter examples for other bits

15
Router Model
OSPF
BGP
OSPF
Route Selection
Route Table
Router 1
16
Route Redistribution
Routing policy 1
Routing policy 2
OSPF
BGP
OSPF
Route Selection
Route Table
Router 1
17
Routing Protocol Adjacencies
Routing policy 1
Routing policy 2
OSPF
BGP
OSPF
OSPF
Route Selection
RS
Route Table
Route Table
Router 1
Router 2
18
Reverse-Engineering Overview
Configuration files
Find links
Construct Layer 3 Topology
Find adjacent routing processes
Construct Routing Process Graph
Condense adjacent routing processes
AS2
Construct Routing Instance Graph
OSPF 1
OSPF 2
BGP AS1
19
Reconstruct the Layer 3 Topology
Internet
Router 1 Config
Router 2 Config
interface Serial1/0.5 ip address 1.1.1.1
255.255.255.252 .
interface Serial2/1.5 ip address 1.1.1.2
255.255.255.252 .
20
Construct the Routing Process Graph
OSPF
OSPF
Internet
RT
RT
Policy1
Policy2
EBGP
OSPF
BGP
OSPF
OSPF
OSPF
RT
Route Table
RT
21
Abstract to a Routing Instance Graph
AS2
Policy1
Policy2
OSPF 1
OSPF 2
BGP AS1

• Pick an unassigned Routing Process
• Flood fill along process adjacencies, labeling
  processes
• Repeat until all processes assigned to an
  Instance

22
Abstract to a Routing Instance Graph
Router2
Router1
AS2
Policy1
Policy2
OSPF 1
OSPF 2
BGP AS1
Router1
Router2
23
A Study of Operational Production Networks

• Obtained anonymized configuration files for 31
  active networks (gt8,000 configuration files)
• Networks include
• 6 Tier-1 and Tier-2 Internet backbone networks
• 25 enterprise networks
• Sizes between 10 and 1,200 routers
• 4 enterprise networks significantly larger than
  the backbone networks
• Networks created by diverse set of designers and
  companies

24
Textbook Routing Design for Enterprise Networks
EBGP
EBGP

• Border routers speak eBGP to external peers
• BGP selects a few key external routes to
  redistribute into OSPF
• 7 of 25 enterprise networks follow this pattern

AS2
OSPF
BGP AS 1
AS3
25
Reality A Diversity of Unusual Routing Designs
Rest of the World
BGP AS 2
BGP AS 1
BGP AS 3
BGP AS 4
BGP AS 5

• Network broken up into compartments, each with
  only 1 to 4 routers
• Each compartment has its own AS number
• Hub and spoke logical topology
• Why? Lots of control over how spokes communicate

26
Routing Design for 900 Router Network
27
Reality A Diversity of Unusual Routing Designs
Rest of the World
BGP AS 1
BGP AS 2
EIGRP
EIGRP
EIGRP
Rest of the World
BGP AS 3
BGP AS 4

• Network broken up into many compartments, each
  running EIGRP, some with 400 routers
• BGP used to filter routes passed between
  compartments
• Compartments themselves pass information between
  BGP speakers
• Why? Little need for IBGP few routers speak
  BGP Lots of control over how packets move
  between compartments

28
Myth Policy Enforced at Edge of Network

• Conventional wisdom
• Place packet filters on the edge to defend
  infrastructure
• Routing policy applied where networks touch

29
Reality Policy Exists Throughout Networks

• Packet filters commonly used on internal links
• Protect routers from attack
• Implement reachability matrix
• Prevent some hosts from communicating with others
• Localize traffic, particularly multicast

30
Summary

• Developed abstractions to model routing design
• Routing Instance abstracts away details
• Reverse-engineer routing design from configs
• We presented our extracted design to designers
• They agreed we captured their design intent
• Focusing on individual protocols is not enough
• Understanding composition is equally important
• First step towards turning routing design from an
  art into a science

31
Applications of Routing Design Analysis

• Enables static analysis of network properties
• Reachability/security analysis
• Route leaks? Reachability violations?
• Robustness analysis
• How sensitive is the network to external events
  such as route announcements?
• Resource usage analysis
• Will a particular configuration cause the routing
  table of a router to overflow?

32
The Value of Investigating Routing DesignNext
Steps

• Found many different designs in use
• Do we need so many designs?
• Framework to ask and answer questions of
  scalability, completeness, optimality
• Do we have the right abstractions?
• Is this the right way to program routers?
• Suggest improvements to protocols and
  configuration languages
• Can the network be run using abstractions?

33
Questions?
34
Textbook Routing Design for Backbone Network
EBGP
IGBP MESH

• Border routers speak eBGP to external peers
• All routers speak iBGP with each other
• All routers participate in both BGP and OSPF
  (learning infrastructure routes from OSPF,
  external routes from BGP)

AS2
OSPF
BGP AS 1
AS3

ASn
35
Real Routing Designs for Backbone Networks
AS2
OSPF
BGP AS 1
AS3
ASn

• All 6 backbone networks used basic OSPF/BGP
  pattern

36
Real Routing Designs for Backbone Networks
AS2
OSPF
BGP AS 1
AS3
EIGRP
ASn
EIGRP
RIP
Customer

• All 6 backbone networks used basic OSPF/BGP
  pattern
• 3 of 6 include many additional routing instances
• Used to exchange routes with customers

37
BGP Used an IGP
38
What do Designers do Today?

• Network designers balance many goals
• Scalability
• Resiliency to failure
• Make it easy to expand network
• Many rules of thumb in use
• Instability results from overloaded routers
• Too much routing state is bad
• Use routing boundaries to control spread of
  change
• Routing Design is currently an art can we add
  more science?

39
Approaches?

• Need deeper understanding than network topology
• Need broader study than backbone networks
• Interviewing network designers isnt enough
• No language/visualization exists for
  communicating about routing design
• Documentation is out-of-date or non-existent
• Our approach Bottom-up white-box
• Start with router configuration files
• Reverse-engineer the routing design

40
Potential Approaches

• Top-down design problem
• How should networks be designed?

First must understand what happens in real
networks

• Bottom-up black-box approach
• Send probe traffic to explore network properties
• Very successful at recovering topology
  RocketFuel Skitter Mercator

Measured topology a result of a routing design
--- it does not expose the routing design itself

• Our approach Bottom-up white-box
• Start with router configuration files
• Reverse-engineer the routing design

41
Router Configuration Files
42
Lots of Configuration Files