Optimal External Route Selection: Tips and Techniques for ISPs

1
Optimal External Route Selection Tips and
Techniques for ISPs

• Avi Freedman
• Net Access

2
Overview

• Brief review of BGP routing concepts
• Safe routing
• Determining policy
• Using MEDs
• Setting MEDs on internal routes
• as-path padding to tune external traffic
• Using local-prefs to tune external traffic
• Setting MEDs to tune external traffic

3
BGP Concept Review
4
BGP Intro

• BGP4 is the protocol used on the Internet to
  exchange routing information between providers,
  and to propagate external routing information
  through networks.
• Each autonomous network is called an Autonomous
  System.
• ASs which inject routing information on their own
  behalf have ASNs.

5
BGP Peering

• BGP-speaking routers peer with each other over
  TCP sessions, and exchange routes through the
  peering sessions.
• Providers typically try to peer at multiple
  places. Either by peering with the same AS
  multiple times, or because some ASs are
  multi-homed, a typical network will have many
  candidate paths to a given prefix.

6
The BGP Route

• The BGP route is, conceptually, a promise to
  carry data to a section of IP space. The route
  is a bag of attributes.
• The section of IP space is called the prefix
  attribute of the route.
• As a BGP route travels from AS to AS, the ASN of
  each AS is stamped on it when it leaves that AS.
  Called the AS_PATH attribute, or as-path in
  Cisco-speak.

7
BGP Route Attributes

• In addition to the prefix, the as-path, and the
  next-hop, the BGP route has other attributes,
  affectionately known as knobs and
  twiddles -
• weight, rarely used - sledgehammer
• local-pref, sometimes used - hammer
• origin code, rarely used
• MED (metric) - a gentle nudge

8
BGP Policy

• BGP was designed to allow ASs to express a
  routing policy. This is done by filtering
  certain routes, based on prefix, as-path, or
  other attributes - or by adjusting some of the
  attributes to influence the best-route selection
  process.

9
BGP Best-Route Selection

• With all of the paths that a router may
  accumulate to a given prefix, how does the BGP
  router choose which is the best path?
• Through an RFC-specified (mostly) route selection
  algorithm.

10
BGP Best-Route Selection

• Do not consider IBGP path if not synchronized
• Do not consider path if no route to next hop
• Highest weight (local to router)
• Highest local preference (global within AS)
• Shortest AS path
• Lowest origin code IGP lt EGP lt incomplete
• Lowest MED
• Prefer EBGP path over IBGP path
• Path with shortest next-hop metric wins
• Lowest router-id

11
BGP Selection, Summary

• So, local-pref is stronger than as-path is
  stronger than MED.
• Setting local-pref without careful planning can
  cause strange things (preferring other paths to
  get to your own customers)

12
Safe Routing
13
Safe Routing

• BGP routes are promises to carry traffic to a
  certain destination. Still, not every provider
  makes good promises at all times.
• So, it is best to sanity-filter all eBGP sessions.

14
Safe Routing

• Method 1
• The Cisco maximum-prefix keyword
• neighbor ltremote-ipgt maximum-prefix percent
  warning
• Sets a maximum number of prefixes allowed for a
  peer.
• Behavior 1 - Shut down the session and log the
  fact.
• Behavior 2 - Leave the session up just log the
  warning.

15
Safe Routing - Filtering

• Another method of sanity filtering is to restrict
  your peers based on routes or as-paths.
• Usually, it is hard to filter based on routes
  (except for our friends, the fanatics at ANS).
• So, from smaller providers it is a good idea to
  prevent random route redistribution.

16
Safe Routing - Filtering

• ip as-path access-list 40 deny _701_
• ip as-path access-list 40 deny _1239_
• ip as-path access-list 40 deny _3561_
• ip as-path access-list 40 deny _1_
• ip as-path access-list 40 deny _1673_
• ip as-path access-list 40 deny _174_
• ip as-path access-list 40 permit .
• Apply this access-list inbound for sanity.

17
I am Blackholio

• In sufficiently strange circumstances, this wont
  help.
• If someone (AS 7007, perhaps) strips the as-path
  information, as-path filters do no good.

18
Determining Policy
19
Determining Policy

• What do you want to do?
• The tricky part.
• Configuring is easy
• Do you want to prefer higher-quality connections?
• Optimize for cost of the links?

20
Connection Quality

• We will assume that you want to optimize for
  connection quality.
• This generally means, in the Platonic
  zero-packet-loss Internet, minimizing latency and
  avoiding small pipes.
• Well come back to small pipes and backup paths
  when we talk about local-prefs.
• Well talk about minimizing latency when we
  explore MEDs.

21
Connection Quality

• At all times, we must minimize packet loss.
• In general, this means avoiding public exchanges
  in favor of private peering and/or transit.
• Sometimes this might not be economically
  desirable, but if you dont tune this way, stay
  vigilant about inter-connection quality.
• Best to measure it if you really care...

22
Measuring Packet Loss with MRTG
Max Max 423.0 ms (352.5) Average Max 32.0 ms
(26.7) Current Max 37.0 ms (30.8) Max Min
9.0 ms (7.5) Average Min 5.0 ms (4.2) Current
Min 6.0 ms (5.0)
23
Peering Points

• You want to prefer paths that you hear over
  uncongested pipes.
• Assuming you have non-full private interconnects,
  PIs will be better than public exchanges.
• Of course, that can depend on which Gigaswitch
  youre on whether youre at PSK, PACBell, AADS,
  or the MAEs.

24
Hot-Potato

• In general, traffic is handed off as soon as
  possible to external providers to minimize
  backbone utilization and costs.
• This is not always the best plan if you want to
  maximize connection quality (assuming your
  inter-LATA and/or cross-country links are not
  full).
• Solution - Listen to and use MEDs.

25
Asymmetry

• For this presentation, we are going to ignore the
  return path - data coming back into your network.
• Still, for best tuning you will want to explore
  this and use as-path padding and possibly
  controlled de-aggregation (to willing partners)...

26
Review Policy

• Somehow, you want to prefer better-quality links.
• In the examples that follow, well assume a small
  but national network, peering at MAE-West,
  MAE-East, and Pennsauken.
• Additionally, private interconnects with IDT,
  PSI, Digex, above.net, and Exodus.
• Transit through above.net and UUNET.

27
Goals

• Our goals will be to prefer, in this order
• Private interconnects
• Regionality of traffic
• Pennsauken over MAE-East
• Public Exchanges
• Transit pipes, above.net first

28
Using MEDs
29
Introduction to MEDs

• The MULTI_EXIT_DISCRIMINATOR, or MED, is a BGP
  attribute used to
• Describe internal network topology.
• Pass on this topology to external peers.
• A smaller knob than others, like local-pref or
  as-path padding.
• Major problem - no inter-provide consistency on
  MED semantics.
• Internally, also called metrics.

30
Setting MEDsfor Internal Route
31
Setting MEDs

• Use an internally consistent scheme.
• Usually, peoples MEDs are in the low hundreds or
  less.
• Suggestion - use average delay in ms between
  POPs.
• Set MEDs in one direction only.
• To be advanced, MEDs can be set on a per-router
  basis in a POP, but usually are not.

32
Network Diagram
CHI
PHL
SF
DC
33
Setting MEDs

• For SF, CHI, PHL, DC
• SF-DC 60
• SF-CHI 40
• CHI-PHL 30
• CHI-DC 25
• PHL-DC 10
• PHL-PSK 0
• DC-MAE-E 5
• SF-MAE-W 5

34
Network Diagram w/ MEDs
CHI
30
40
25
PHL
60
SF
10
DC
35
Route Maps in DC

• route-map from-sf
• set metric 60
• route-map from-chi
• set metric 40
• route-map from-phl
• set metric 10
• neighbor ltsf-ipgt route-map from-sf in
• etc...

36
What this Does

• A route originating in PHL will have
• metric 60 or or 70 in SF (unless there are
  multiple link failures)
• metric 10 or 60 in SF
• metric 10 or 35 in DC
• etc
• Thus, a provider honoring MEDs (not doing
  hot-potato) will send packets destined to that
  route in PSK, to PHL.

37
Slight Improvement?

• Or, change things to weight PSK vs. DC over PHL
  vs. DC.
• PSK 0
• MAE-E 20
• Thus, a provider honoring MEDs will send a
  PHL-destined packet to PSK. This is generally a
  good thing.

38
Using as-path Padding
39
as-path padding

• Some think that modifying as-paths is a nasty
  business.
• It is a good beginning way to do preferences.
• If providers have already padded to de-prefer,
  preserves that de-preference.
• Simple to do.

40
as-path padding

• First, policy?
• Private interconnects - pad no times
• Regionality of traffic - pad four times x-country
• Pennsauken over MAE-East - pad once twice
• Public Exchanges - twice at MAE-West
• Transit pipes, above.net first - pad three
• Problem - cant pad easily going cross-country.
• But we can do the rest.
• Problem - lots of route-maps and typing.
• Why? Cant prepend our own AS inside network, so
  must have separate roue-map per session.

41
route-maps

• On everyone, at above.net
• route-map prepend-once permit 10
• set as pre 6461 6461 6461
• On everyone, at UUNET
• route-map prepend-once permit 10
• set as pre 701 701 701
• On PSI, at MAE-East and MAE-West
• route-map prepend-once permit 10
• set as pre 174 174
• On PSI, at Pennsauken
• route-map prepend-once permit 10
• set as pre 174

42
Using local-prefs
43
Local-prefs

• Most common method of preferring external routes.
• Local-pref is a number, by default 100, put on
  routes and passed to all routers within a
  network.
• Never passed to an eBGP peer.

44
Implementing Policy

• Customers - local-pref 200
• Private interconnects - local-pref 150
• Pennsauken over MAE-East - 120 for Pennsauken
• Public Exchanges - 100 at MAE-East and MAE-West
• Transit pipes, above.net first - 80 from transit
  pipes
• Regionality of traffic - defer to MEDS for equal
  local-pref. May want to add PACBELL cxn and make
  it 120.

45
route-maps

• At Pennsauken
• route-map psk in
• set local-pref 120
• set community 4969800
• neighbor peer-group external-peer-psk route-map
  psk in
• or
• neighbor ltremoteipgt route-map psk in

46
Problem Prefers Bad Paths

• The problem with this approach
• Take AS 14000, who has a T1 to Sprintlink and a
  backup-backup-backup 56k to another local
  provider, say, 13000.
• Announces as
• 1239 14000 and
• 701 13000 14000 14000 14000 14000 1400
• Local-prefs can screw with this.

47
Listening to MEDs
48
Listening to MEDs Same Peer

• Nothing special is required to listen to MEDs.
• Because MEDs mean different things to different
  networks, one approach is no only set MEDs
  inbound for your own routes.
• When listening to MEDs at multiple locations from
  a peer, set to internal MEDs if you want to
  hot-potato.

49
route-map on DC, v2

• route-map from-sf permit 10
• match community 1
• set metric 60

50
MEDs from Diff. eBGP Peers

• bgp always-compare-med keyword allows Ciscos to
  use MEDs among different providers.
• Otherwise, will use them to compare iBGP routes,
  or eBGP routes from the same AS.

51
Setting MEDs on External Routes
52
Preferring External Routes w/ MEDs

• Can be done, sometimes while preserving remote
  MED info, but usually remote MED info is lost.
• Better in some cases than as-path padding or
  local-prefs (as-path padding is undesirable when
  you have to pass routes on to customers
  local-prefs might use backup links).

53
Preferring External Routes w/ MEDs

• Assuming not honoring remote MEDs
• Set metric inbound to 0 and set internal-route
  MEDs on routes, then
• Private interconnects - no change
• Regionality of traffic - no change - add normal
  MEDs
• Pennsauken over MAE-East - add 20 for MAE-East
• Public Exchanges - no change, or add 20
• Transit pipes, above.net first - add 30 or 40

54
Active Route Override
55
Overriding BGP

• Some have started to override BGP when evidence
  suggests better routing, on a per-prefix basis.
• ASAP from above.net, ?fastpath?, ?others?
• Ideally actively and autonomously, determine best
  path to frequently-used prefixes and inject
  fixer-routes.
• Soon, Cisco will have hooks for injection.