Congestion Mitigation - PowerPoint PPT Presentation

About This Presentation
Title:

Congestion Mitigation

Description:

There are frequent periods of congestion between our network and a peer's network in Europe ... [jabley_at_nautilus]$ seeasp -s 3320 ./dtag.agg | more. Facets: ... – PowerPoint PPT presentation

Number of Views:44
Avg rating:3.0/5.0
Slides: 21
Provided by: joea9
Category:

less

Transcript and Presenter's Notes

Title: Congestion Mitigation


1
Congestion Mitigation
  • Trying to maximise performance between MFNs
    network and a peers network over some busy PNIs

2
Hello, Good Evening
  • Joe Abley
  • jabley_at_mfnx.net
  • Toolmaker, Token Canadian
  • Metromedia Fiber Network

3
Background
  • There are frequent periods of congestion between
    our network and a peers network in Europe
  • The peer is a major operator in the region, and
    evil forces are preventing us from simply
    lighting up new fibre
  • We need to work with what we have in place already

4
Characteristics of Problem
  • The peer is home to a lot of broadband
    subscribers
  • Transient hot-spots in content hosted within
    MFNs network cause localised congestion in some
    PNIs, with other PNIs showing headroom
  • We get paid for shifting packets (if we dont
    carry the packets, we dont get paid)

5
State of Play Day One
Location Capacity Utilisation
Amsterdam STM-1 75M
Frankfurt STM-1 140M
Vienna STM-1 90M
6
Goals
  • Identify the important consumers of traffic in
    and beyond the peers network
  • Once we can characterise the major traffic sinks,
    we can try and balance them out across our
    various PNIs
  • Hopefully this will make the PNIs less sensitive
    to bursty traffic
  • We expect to have to keep measuring and
    rebalancing

7
Tools
  • Ixia IxTraffic
  • Gumption
  • Unix-based BGP speaker that participates in the
    IBGP mesh
  • Gives us route history
  • SeeFlow
  • Smart NetFlow collector which talks to Gumption
  • Awk
  • We always have awk

8
Infrastructure
  • NetFlow traffic to be collected in-band from
    GSR12012s
  • Single IxTraffic box
  • FreeBSD 4.5, i386, dual 700MHz P3, 2GB RAM
  • Overkill
  • Load average occasionally peaks above 0.07
  • 10GB filesystem for storing routing and flow data
    in
  • Located in Virginia
  • MRTG-like thing (duck) which also lives in VA on
    a different box gives us nice visibility of
    congestion trends

9
Exporting Flow Data
  • Flow switching needs to be turned on in a maint
    window, because it makes the routers belch
    impolitely
  • All interfaces that can contribute towards
    traffic sent towards the peer get ip route-cache
    flow sampled
  • See kiddie script! See router die!
  • Export config is trivial
  • ip flow-export source Loopback0
  • ip flow-export version 5 peer-as
  • ip flow-export destination 192.168.38.9 3871
  • ip flow-sampling-mode packet-interval 1000
  • Note low sampling rate of 11000.

10
Collecting Flow Data
  • SeeFlow is configured to populate net2net and
    aspath matrices (buckets)
  • We suspect that a lot of data is getting sunk
    within the peer network, hence net2net
  • We could be wrong, and aspath matrices are cool,
    so we collect those too
  • Buckets chew up about 50MB of disk per day (all
    routers)

11
Initial Discoveries
  • All the traffic is being sunk within the peer
    network, and not in a downstream network
  • Damn
  • All the traffic is being sunk into a single /12
    advertisement
  • Damn
  • We need better granularity if we are going to be
    able to spread the demand across our PNIs

12
ASPATH Matrices
  • jabley_at_nautilus seeasp -s 3320 ./dtag.agg
    more
  • Facets
  • TimeInterval 05/09/2002 150003.663492 -
    06/05/2002 045758.747866 PDT
  • SuperDataFacet
  • Facets
  • RouterIpv4Addr 209.249.254.142
  • RouterName pr1.fra1.de.mfnx.net
  • Facets
  • RouterIpv4Addr 209.249.254.195
  • RouterName mpr2.vie3.at.mfnx.net
  • Facets
  • RouterIpv4Addr 216.200.254.246
  • RouterName mpr1.ams1.nl.mfnx.net
  • AS P PktsThru BytesThru PktsTo
    BytesTo PktsTotal BytesTotal
  • ----- - ---------- ------------ ----------
    ------------ ---------- ------------
  • 0 - 0 0 371.203M
    143.286G 371.203M 143.286G
  • AAAA P 1.816M 333.567M 108.112M
    36.705G 109.928M 37.038G
  • BBBB - 0 0 1.516M
    191.657M 1.516M 191.657M

13
Net2Net Matrices
  • 172.184.0.0/13 -gt A.A.0.0/12
    13001989 5601247858
  • 172.184.0.0/13 -gt A.A.0.0/12
    12983375 5592070913
  • 62.4.67.0/24 -gt B.B.0.0/11
    9459634 1687041555
  • 62.4.67.0/24 -gt B.B.0.0/11
    9443861 1677536483
  • 172.176.0.0/14 -gt A.A.0.0/12
    7113026 2985029679
  • 172.176.0.0/14 -gt A.A.0.0/12
    7099648 2977787074
  • 62.80.115.0/24 -gt C.C.0.0/11
    6873518 1236318991
  • 62.4.67.0/24 -gt A.A.0.0/12
    6689319 1180741686
  • 62.4.82.0/24 -gt A.A.0.0/12
    6611879 1171430532
  • 62.4.67.0/24 -gt C.C.0.0/11
    3469776 629221553
  • 62.4.82.0/24 -gt C.C.0.0/11
    3433970 625422145
  • 62.4.67.0/24 -gt D.0.0.0/13
    2422913 442942807
  • 62.4.67.0/24 -gt D.0.0.0/13
    2407651 470778890
  • 62.4.65.96/27 -gt A.A.0.0/12
    1981446 287218317
  • 62.80.116.0/24 -gt E.E.0.0/15
    1802114 378062358
  • 62.4.67.0/24 -gt F.F.0.0/14
    1510412 315282857
  • 62.4.67.0/24 -gt F.F.0.0/14
    1421497 277014582
  • 62.4.65.96/27 -gt B.B.0.0/11
    1341063 378931389
  • 62.4.81.128/27 -gt B.B.0.0/11
    1330058 378268227

14
Destination Prefix Histogram
  • destination net megabytes proportion
  • A.A.0.0/12 21478 59
  • B.B.0.0/11 6494 18
  • C.C.0.0/11 4388 12
  • D.0.0.0/13 1365 3
  • F.F.0.0/14 1033 2
  • G.G.0.0/15 416 1
  • H.H.0.0/14 311 0
  • I.I.I.0/21 160 0
  • J.J.0.0/15 117 0
  • K.K.0.0/19 89 0

15
Drilling down into A.A.0.0/12
  • Ask peer to advertise longer prefixes within the
    /12, so we can measure the traffic per prefix
  • Wait for response
  • GOTO 10
  • Maybe we can fix this ourselves

16
/home/dlr/bin/bgpd
  • We injected 15 covered /16 prefixes into IBGP,
    with a NEXT_HOP that lay within the remaining /16
  • All tagged no-export, to avoid messing with the
    peers public route policy
  • Strictly local-use within AS6461

17
More Collection
  • The increased granularity gives us better
    visibility into the traffic sinks within the peer
    network
  • We will try to spread the traffic over the
    available PNIs so we can weather bursts of demand
    more effectively
  • We will also continue to let the peer know what
    we are doing
  • You never know, they may be listening

18
New Dest Prefix Histogram
  • destination net megabytes proportion
  • B.B.0.0/11 1912 14
  • A.76.0.0/16 1530 11
  • C.C.0.0/11 1516 11
  • A.73.0.0/16 1120 8
  • A.72.0.0/16 1024 7
  • A.64.0.0/12 874 6
  • A.74.0.0/16 683 5
  • A.70.0.0/16 696 5
  • A.68.0.0/16 601 4
  • A.66.0.0/16 437 3

19
State of Play Day N
Location Capacity Utilisation
Amsterdam STM-1 110M
Frankfurt STM-1 105M
Vienna STM-1 85M
20
Conclusions
  • Light more fibre is not always a realistic
    strategy
  • You are not always your peers number one
    priority, so its nice to be able to take matters
    into your own hands
  • Distributing the heavy traffic sinks across
    different PNIs makes bursty demand less
    unpleasant
  • Routing plus flow data Power. Or something.
Write a Comment
User Comments (0)
About PowerShow.com