Part III: Measuring Inter-domain Paths

1
Part III Measuring Inter-domain Paths
2
Packet forwarding path
Destination
Internet
IP traffic
Source
Forwarding path - the path packets traverse
through the Internet from a source to a
destination
3
An inter-domain level view
AS D
Destination
Internet
AS C
IP traffic
AS A
AS B
Source
An IP forwarding path often span across multiple
Autonomous Systems.
4
Why do we care?

• Characterize end-to-end network paths
• Diagnose routing anomalies
• Discover Internet topology

5
Why do we care?

• Characterize end-to-end network paths
• Latency
• Capacity
• Link utilization
• Loss rate.
• Diagnose routing anomalies
• Discover Internet topology

6
Varies link capacity
Destination
Internet
Source
7
Different loss rate
Destination
Internet
Source
8
Traffic engineering
Destination
Internet
Source
Customer service enhancement
9
Why do we care?

• Characterize end-to-end network paths
• Diagnose routing anomalies
• Forwarding loop, black holes, routing changes,
  unexpected paths, main component of end-to-end
  latency.
• Discover Internet topology

10
Forwarding loops
Destination
Internet
Source
11
Black holes
Destination
Internet
Source
12
Routing changes
Destination
Internet
Source
13
Unexpected routes
Destination
Internet
Source
14
Performance bottleneck
Destination
Internet
Source
15
Why do we care?

• Characterize end-to-end network paths
• Diagnose routing anomalies
• Discover Internet topology
• Server placement

16
Internet topology
Server
Internet
Client
Client
Client
17
Server placement
Server
Internet
Client
Client
Client
18
Key challenge

• Need to understand how packets flow through the
  Internet without real-time access to proprietary
  routing data from each domain.
• Identify accurate packet forwarding paths
• Characterize the performance metrics of each hop
  along the paths

19
Identify forwarding path

• Traceroute gives IP level forwarding path
• IP address of the router interfaces on a
  forwarding path
• RTT statistics for each hop along the way

20
Traceroute from UC Berkeley to www.cnn.com
Traceroute output (hop number, IP address, DNS
name)

• 1 169.229.62.1
• 2 169.229.59.225
• 3 128.32.255.169
• 4 128.32.0.249
• 5 128.32.0.66
• 6 209.247.159.109
• 7
• 8 64.159.1.46
• 9 209.247.9.170
• 10 66.185.138.33
• 11
• 12 66.185.136.17
• 13 64.236.16.52

• inr-daedalus-0.CS.Berkeley.EDU
• soda-cr-1-1-soda-br-6-2
• vlan242.inr-202-doecev.Berkeley.EDU
• gigE6-0-0.inr-666-doecev.Berkeley.EDU
• qsv-juniper--ucb-gw.calren2.net
• POS1-0.hsipaccess1.SanJose1.Level3.net
• ?
• ?
• pos8-0.hsa2.Atlanta2.Level3.net
• pop2-atm-P0-2.atdn.net
• ?
• pop1-atl-P4-0.atdn.net
• www4.cnn.com

1 169.229.62.1 2 169.229.59.225 3
128.32.255.169 4 128.32.0.249 5 128.32.0.66
6 209.247.159.109 7 8 64.159.1.46 9
209.247.9.170 10 66.185.138.33 11 12
66.185.136.17 13 64.236.16.52
inr-daedalus-0.CS.Berkeley.EDU soda-cr-1-1-soda-br
-6-2 vlan242.inr-202-doecev.Berkeley.EDU gigE6-0-
0.inr-666-doecev.Berkeley.EDU qsv-juniper--ucb-gw.
calren2.net POS1-0.hsipaccess1.SanJose1.Level3.net
? ? pos8-0.hsa2.Atlanta2.Level3.net pop2-atm-P0-2
.atdn.net ? pop1-atl-P4-0.atdn.net www4.cnn.com
21
Traceroute from ATT Research to www.cnn.com

• traceroute to cnn.com (64.236.24.12), 30 hops
  max, 40 byte packets
• 1 oden (135.207.16.1) 1 ms 1 ms 1 ms
• 2
• 3 attlr-gate (192.20.225.1) 2 ms 2 ms 2 ms
• 4 12.119.155.157 (12.119.155.157) 3 ms 4 ms
  4 ms
• 5 gbr6-p52.n54ny.ip.att.net (12.123.192.18) 4
  ms 4 ms 4 ms
• 6 tbr2-p012401.n54ny.ip.att.net (12.122.11.29)
  4 ms (ttl249!) 5 ms (ttl249!) 5 ms (ttl249!)
• 7 ggr2-p390.n54ny.ip.att.net (12.123.3.62) 4
  ms 5 ms 4 ms
• 8 att-gw.ny.aol.net (192.205.32.218) 4 ms 4
  ms 4 ms
• 9 bb2-nye-P1-0.atdn.net (66.185.151.66) 4 ms
  4 ms 4 ms
• 10 bb2-vie-P8-0.atdn.net (66.185.152.201) 13 ms
  (ttl245!) 12 ms (ttl245!) 12 ms (ttl245!)
• 11 bb1-vie-P11-0.atdn.net (66.185.152.206) 10
  ms 10 ms 10 ms
• 12 bb1-cha-P7-0.atdn.net (66.185.152.28) 20 ms
  20 ms 20 ms
• 13 bb1-atm-P6-0.atdn.net (66.185.152.182) 25 ms
  25 ms 25 ms
• 14 pop1-atl-P4-0.atdn.net (66.185.136.17) 25 ms
  (ttl243!) 24 ms (ttl243!) 24 ms (ttl243!)
• 15

• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30

Destination unreachable!
Who is responsible for the forwarding problem?
22
Need to know Inter-domain level path
AS D
www.cnn.com
Internet
AS C
AS A
AS B
ATT Research
Routing loop in AS C!
23
How to obtain AS level paths

• BGP AS path
• Traceroute AS path

24
BGP AS path
Signaling path control traffic
d pathBC
d pathC
Prefix AS path d A B C
Is BGP AS path the answer?
No!
25
BGP AS path is not the answer

• Requires timely access to BGP data
• Signaling path may differ from forwarding path
• Route aggregation and filtering
• Routing anomalies e.g., deflections, loops
  Griffin2002
• BGP misconfigurations e.g., incorrect AS
  prepending

Two paths may differ precisely when operators
most need accurate data to diagnose a problem!
26
Traceroute AS path

• Obtain IP level path using traceroute
• Map IP addresses to ASes

b
c
d
e
a
Source
Destination
Is traceroute AS path the answer?
NO!
27
Example UC Berkeley to CNN
Traceroute output (hop number, IP)
1 169.229.62.1 2 169.229.59.225 3
128.32.255.169 4 128.32.0.249 5 128.32.0.66
6 209.247.159.109 7 8 64.159.1.46 9
209.247.9.170 10 66.185.138.33 11 12
66.185.136.17 13 64.236.16.52
28
Traceroute AS path is not the answer

• Identifying ASes along forwarding path is
  surprisingly difficult!
• Internet route registry
• Origin AS in BGP routes

29
Internet route registry

• Whois database
• E.g. NANOG traceroute, prtraceroute
• Out-of-date, incomplete
• Address allocation to customers
• Acquisition, mergers, break-ups

30
Origin AS in BGP routes

• Last AS in the AS path for each prefix
• More accurate and complete than whois data

31
Limitations of BGP origin AS

• Multiple Origin AS (MOAS)
• Infrastructure addresses may not be advertised
• Addresses announced by someone else

32
Limitations of BGP origin AS

• Multiple Origin AS (MOAS)
• Multi-homing
• Misconfiguration
• Internet eXchange Points (IXPs)
• Infrastructure addresses may not be advertised
• Addresses announced by someone else

33
Limitations of BGP origin AS

• Multiple Origin AS (MOAS)
• Infrastructure addresses may not be advertised
• Does not require to be announced publicly
• Security concerns
• Addresses announced by someone else

34
Limitations of BGP origin AS

• Multiple Origin AS (MOAS)
• Infrastructure addresses may not be advertised
• Addresses announced by someone else
• Static routed customers
• Shared equipments at boundary between ASes

Need accurate IP-to-AS mapping!
35
Accurate AS-level traceroute
Combine BGP and traceroute data to find a better
answer!
36
Assumptions

• IP-to-AS mapping
• Mappings from BGP tables are mostly correct.
• Change slowly
• BGP paths and forwarding paths mostly match.
• 70 of the BGP path and traceroute path match

37
BGP path and traceroute path could differ!

• Inaccurate IP-to-AS mapping
• Traceroute problems
• Legitimate mismatches

38
BGP path and traceroute path could differ!

• Inaccurate IP-to-AS mapping
• Internet eXchange Points (IXPs)
• Sibling ASes
• Unannounced infrastructure addresses
• Traceroute problems
• Legitimate mismatches

39
Internet eXchange Points (IXPs)

• Shared infrastructure connected to multiple
  service providers
• Exchange BGP routes and data traffic
• May have its own AS number or announced by
  participating ASes
• Dedicated BGP sessions between pairs of
  participating ASes
• E.g., Mae-East, Mae-West, PAIX.

40
IXPs cause extra AS hop

• Extra AS hop in traceroute path
• Large number of fan-in and fan-out ASes
• Non-transit AS, small address block, likely MOAS

41
IXPs cause extra AS hop
A
E
A
E
B
F
B
F
D
C
G
C
G
Traceroute AS path
BGP AS path
42
Sibling ASes

• Single organization owns and manages multiple
  ASes
• May share address space
• Large fan-in and fan-out for the sibling AS
  pair

43
Sibling ASes cause extra AS hop

• Large fan-in and fan-out for the sibling AS pair

A
E
A
E
B
F
B
F
D
H
D
C
G
C
G
Traceroute AS path
BGP AS path
44
Unannounced infrastructure addresses

• ASes do not necessarily announce infrastructure
  via BGP
• Lead to unmapped addresses
• Sometimes fall into supernet announced by ASs
  provider or sibling

45
Unannounced infrastructure addresses
AS loop in traceroute path
AS A
Substitute AS hop
AS B
Missing AS hop in traceroute path
AS C
Extra AS hop in traceroute path
46
BGP path and traceroute path could differ!

• Inaccurate IP-to-AS mapping
• Traceroute problems
• Forwarding path changing during traceroute
• Interface numbering at AS boundaries
• ICMP response refers to outgoing interface
• Legitimate mismatches

47
Forwarding path changing during traceroute
AS D
AS E
Route flaps between A B C and A D E
AS A
AS B
AS C
AS A
AS C
AS D
AS hop B is substituted by AS D in the traceroute
path
48
Interface numbering at AS boundaries
AS A
AS C
AS A
AS B
AS C
Missing AS hop B in traceroute path
49
ICMP response refers to outgoing interface
AS A
AS C
ICMP message
AS B
Extra AS hop B in traceroute path
50
BGP path and traceroute path could differ!

• Inaccurate IP-to-AS mapping
• Traceroute problems
• Legitimate mismatches
• Route aggregation and filtering
• Routing anomalies, e.g., deflections

51
Route aggregation/filtering
AS B
AS C
AS A
8.0.0.0/8 B C
8.0.0.0/8 C 8.64.0.0/16 C D
Extended traceroute path due to filtering by AS B
52
Mismatch patterns and causes
53
BGP and traceroute data collection
Initial mappings from origin AS of a large set
of BGP tables
Traceroute paths from multiple locations
(Ignoring unstable paths)
For each location
Local BGP paths
Traceroute AS paths
For each location

• Compare
• Look for known causes of mismatches
• (e.g., IXP, sibling ASes)
• Edit IP-to-AS mappings
• (a single change explaining a large number of
  mismatches)

Combine all locations
54
Experimental methodology
200,000 destinations d0, d1, d2, d3, d4,
d200,000
For each di -Traceroute path -BGP path
55
Measurement setup

• Eight vantage points
• Upstream providers US-centric tier-1 ISPs
• Sweep all routable IP address space
• About 200,000 IP addresses, 160,000 prefixes,
  15,000 destination ASes

56
Eight vantage points
Many thanks to people who let us collect data!
57
Preprocessing BGP paths

• Discard prefixes with BGP paths containing
• Routing changes based on BGP updates
• Private AS numbers (64512 - 65535)
• Empty AS paths (local destinations)
• AS loops from misconfiguration
• AS SET instead of AS sequence
• Less than 1 prefixes affected

58
Preprocessing traceroute paths

• Resolving incomplete traceroute paths
• Unresolved hops within a single AS map to that AS
• Unmapped hops between ASes
• Try match to neighboring AS using DNS, Whois
• Trim unresponsive () hops at the end
• Compare with the beginning of local BGP paths
• MOAS at the end of paths
• Assume multi-homing without BGP
• Validation using ATT router configurations
• More than 98 cases validated

59
Initial IP-to-AS Mapping
60
Heuristics to improve mappings

• Overall modification to mappings
• 10 IP-to-AS mappings modified
• 25 IXPs identified
• 28 pairs of sibling ASes found
• 1150 of the /24 prefixes shared

61
Heuristics to improve mappings
62
Systematic optimization

• Dynamic-programming and iterative improvement
• Initial IP-to-AS mapping derived from BGP routing
  tables
• Identify a small number of modifications that
  significantly improve the match rate.
• 95 match ratio, less than 3 changes, very
  robust

63
Optimization results
64
Validation

• Public data
• Whois/DNS data
• pch.net for known IXPs
• Private data
• AS 7018

65
Validations IXP heuristic

• 25 inferences 19 confirmed
• Whois/DNS data confirm 18 of 25 inferences
• AS5459 -- London Internet Exchange
• 198.32.176.0/24
• part of Exchange Point Blocks
• DNS name sfba-unicast1-net.eng.paix.net
• Known list from pch.net confirm 16 of 25
• Missing 13 known IXPs due to
• Limited number of measurement locations
• Mostly tier-1 US-centric providers

66
Validations Sibling heuristic

• 28 inferences all confirmed
• Whois for organization names (15 cases)
• E.g., AS1299 and AS8233 are TeliaNet
• MOAS origin ASes for several address blocks
• (13 cases)
• E.g., 148.231.0.0/16 has MOAS
• AS5677 and AS7132
• (Pacific Bell Internet Services and SBC Internet
  Services)

67
Summary

• Identify accurate AS level forwarding path
• improve infrastructure IP to AS mappings
• Heuristics and Dynamic programming optimization
• Match/mismatch ratio improvement 8-12 to 25-35
• Reduction of incomplete paths 18-22 to 6-7

68
Summary

• Dependence on operational realities
• Most BGP routes are relatively stable
• Few private ASes, AS_SETs
• Public, routable infrastructure addresses
• Routers respond with ICMP replies

http//www.research.att.com/jiawang/as_traceroute