Nanog24 GEANT

1
Inter-domain Multicast in European Research
NetworkingTEN-155 Operational Experience and
Deployment on GÉANT
NANOG24 Miami 12th February 2002
Agnes Pouélé, DANTE Ltd. Network Engineer Jan
Novak, Cisco Systems Inc. Network Consulting
Engineer
1
2
Agenda

• DANTE
• TEN-155 Operational Experience
• Evolution of the TEN-155 multicast topology from
  1998 to 2000
• Operational Experience
• GÉANT deployment
• Overview of GÉANT Network
• GÉANTs Multicast design
• GÉANT Multicast and Unicast Coverage
• GÉANT Multicast Service and Monitoring
• Conclusion

2
3
DANTE, TEN-155, GÉANT

• DANTE
• DANTE is a not-for-profit company set up in 1993
  by European National Research Network
  organizations.
• TEN-155
• was an ATM based network built initially on OC3
  links and then upgraded in 2000
• GÉANT
• successor of TEN-155
• launched in December 2001
• 10 Gbps Pan-European Network

3
4
Part I TEN-155 Operational Experience
4
5
Starting point MBONE TUNNELED TOPOLOGY
Typical mess of DVMRP tunnels on Sun WSs with
usual tunnel routing problems.
5
6
1999 PIM-SM and DVMRP set-up
PIM-SM domain
OSLO IETF transmitted over both native STM-1 ATM
based network and DVMRP tunnelled infrastructure
6
7
Multicast deployment in TEN-155

• Based on this first set-up and successful
  operation, we concluded to enable multicast on
  all production routers in TEN-155.
• Multicast code stable, CPU usage ok, M-BGP ok
• PIM-SM only for production service
• PIM-SM to DVMRP border works, but not possible to
  operate routinely (NOC)
• From 1999 to 2000 each country was migrated from
  the DVMRP cloud to a interconnection with TEN-155
  using PIM-SM/MBGP/MSDP

7
8
2000 TEN-155 final topology
e x t e r n a l
NL
SE
DE
UK
e x t e r n a l
e x t e r n a l
US
AT
FR
CH
tunnel
IT
dedicated mcast
B G P
unicast/mcast line
8
9
TEN-155 Operational ExperiencePerformance Impact

• Parameters to be considered for the exploitation
  of Multicast
• CPU - parameters
• about 1600 forwarding (mroute) entries
• max 8 outgoing interfaces (average 2-3)
• max 20 Mbit/s of data forwarded by one router
• resulting in 5-10 of CPU increase (mainly PIM)

9
10
TEN-155 Operational ExperiencePerformance Impact

• Parameters to be considered for the exploitation
  of Multicast
• Memory
• about 3000 SA messages in the cache
• about 1600 forwarding entries (mroute)
• about 10 000 routes in the MBGP table
• These parameters didnt represent a significant
• memory usage

10
11
Usage of MSDP new component new storms
MSDP peers
default peer

• Default peer in redundant topology

• Redundant mesh-groups

Impact of a MSDP storm on the CPU load
11
12
Usage of MSDP early implementation problems

• Cisco IOS 12.0.6S and lower
• ghost SA entries in the MSDP cache
• SA messages recreated by the incoming (S,G) joins
  
• FIXED
• Origination of SA messages only when source
  registered
• to the originating RP
• ip msdp redistribute command without
  arguments
• re-originated all known SAs
• caused huge increase of the SA counts worldwide
• FIXED

12
13
TEN-155 MSDP monitoring

• Monitoring set-up

MSDP peering

• Software adapted C MSDP
• implementation of Steve Rubin

• MSDP usage monitoring
• number of RPs, average 90 (40 EU)
• number of groups
• number of pairs source, group (S,G)
• number of SA messages per minute

13
14
TEN-155 MSDP monitoring
number of RPs
Number of RPs announced to TEN-155 before and
after the loss of US connectivity
14
15
TEN-155 MBGP monitoring

• Monitoring set-up

MBGP peering

• Software - Merits MRTD -
• modified SAFI definition for MBGP

• MBGP monitoring
• stability/updates
• number routes, max about 9000 (760 EU)
• number of ASNs, average 240 (80 EU)

15
16
TEN-155 MBGP monitoring
stability/updates

• In blue,
• the number of routes originated from one AS
• In green,
• the number of updates originated by the same AS

16
17
TEN-155 Multicast Monitoring tools

• http//www.dante.net/pubs/dip/40/40.html
• http//www.dante.net/pubs/dip/41/41.html
• http//www.dante.net/mbone/
• http//sigma.dante.org.uk/stats/mrtg/msdp/data/
• http//www.dante.net/mbone/mbgp
• The graph values from mbgp and msdp monitoring
  are historical values.

17
18
TEN-155 Operational ExperienceConclusion

• Move to native and SM multicast in national
  networks (NRENs)
• Unicast and multicast non congruent
• MSDP peer doesnt need to be RP.
• Concept of two BGP tables and multi-protocol
  RPF check often still misunderstood.
• Inter-domain Multicast debugging
• gt Almost impossible to fix problems in just one
  week

18
19
PART IINEXT GENERATIONGÉANT DEPLOYMENT
19
20
www.dante.net/nep/GÉANT-MULTICAST/map.html
20
21
GÉANT Services

• GÉANT Standard IP Service
• IP traffic from NREN to NREN and Research
  peerings.
• Multicast Service (rolling out now)
• Replacement of TEN-155 Managed Bandwidth Service
• GÉANT Premium IP Service
• Layer-2 VPNs (forthcoming)
• Upcoming
• Security and Dos attack detection, IPV6

21
22
GÉANTs Customers and Other Peerings

• GÉANT Unicast Customers
• 27 countries in Europe
• GÉANT Multicast Customers
• 24 countries in Europe
• GÉANT Unicast and Multicast research and
  commercial peerings
• Abilene, Canarie and ESnet via GTREN
• Infonet, UUNET

22
23
GÉANT Multicast Customers
www.dante.net/nep/GÉANT-MULTICAST/map.html
23
24
GÉANT Multicast Implementation

• Current design built on
• The experience gained from TEN-155
• Guidelines from multicast experts
• Juniper Laboratory tests
• GÉANT Multicast services
• Multicast transit domain NRENs to NRENs
• Multicast transit domain NRENs to other PEERS
• Beacon monitored backbone
• GÉANT Multicast routing policy at
• http//www.dante.net/nep/GEANT-MULTICAST/routing-
  policy.html

24
25
TEST-BED LAB 26/27th Nov 2001

• Junos 5.0R3.3

HostX
Group 233.1.10.1
Network 10.1/16 -AS 101
Source 10.1.10.2
Sulfur

• mesh group

Iridium
Mangan
ese
M-BGP peering
Network 10.0/16 -AS 100
Helium
Gallium
MSDP peering
HostZ
Xenon
Tin
HostY
Network 10.2/16 AS 102
Network 10.3/16 AS 103
Group 233.2.20.1
Group 233.3.30.1
Source 10.2.20.2
Source 10.3.30.2
25
26
PIM SMv2 GÉANT domain

• GÉANT single PIM-SMv2
• Version 2 , enabled on all interfaces.
• Three Rendez-vous Points with private anycast
  address.
• backup for internal sources and receivers.
• Private Anycast address (filtered out)
• closest RP based on the OSPF cost
• All other interconnected administrative domain
  have to be PIM-SM v2 enabled with their dedicated
  RP.

26
27
PHYSICAL TOPOLOGY RP
Eenet
Litnet
Janet
Nordunet
NY4-1
Latnet
IUCC
Posnan
Heanet
Surfnet
Belnet
PoP
DFN
NY4-2
Cesnet
Restena
Infonet
Renater
Rediris
Sanet
FCCN
Hungarnet
RoEduNet
INFN
Arnes
SwitchCern
STM64/OC192
STM16/OC48
Carnet
STM4/OC12
Aconet
Grnet
STM1/OC3
Unicom-b
Cynet
www.dante.net/nep/GÉANT-MULTICAST/map.html
27
28
GÉANT Design MBGP and MSDP

• MBGP
• Separate multicast routing table (inet.2)
• Currently congruent BGP and MBGP topology in
  Europe
• iMSDP
• MSDP is fully meshed between 19 PoPs
• Use of mesh group
• i-MSDP Peering with loopback addresses (Not the
  anycast address !!)
• eMSDP
• NREN lt---gt GÉANT Access Router

28
29
MSDP peering Logical view
Janet
Ny4-1
Iucc
GTREN
GRnet
Eenet
Cynet
Nordunet
Unicom-b
Litnet
External MSDP peering

uk
Renater
Latnet
se
gr
fr
iMSDP Mesh Group iMBGP Full Mesh
SwitchCern
Abilene
de2
ch
NRN ----- GÉANT router
Infonet
de1
es
Rediris
DFN
be
NY4-2
Belnet
nl
RP
it
Surfnet
INFN
pt
si
Arnes
at
FCCN
lu
hu
Carnet
ie
Restena
Aconet
sk
pl
cz
RoEdunet
Heanet
Hungarnet
Posnan
Sanet
Cesnet
www.dante.net/nep/GEANT-MULTICAST/map.html
29
30
MSDP SA Filtering

• A list of filtered SA is defined at
• http//www.dante.net/nep/GEANT-MULTICAST/deploymen
  t-msdp.html
• We filter the recommended list.
• We authorise 239.194.0.0/16 from the IPV4
  Organisation Local Scope through GÉANT.

30
31
GÉANT Multicast and Unicast Coverage
CAnet
Esnet
Abilene
GTREN
RESEARCH Peerings
www.dante.net/nep/gtren.html
GÉANT AS20965
DFN
JANET
UK
DE
NREN4
NREN2
...
...
Third party provider European Distributed Access
STM-16
Commodity Internet Access
STM-4
31
32
GÉANT Multicast Service

• Access to the service
• Via the primary access to GÉANT
• Via a GRE tunnel (currently nobody)
• Support of PIM-SM v2 only
• Operational procedures (rolling out now)
• Goal same level of service as Unicast.
• Troubleshooting
• Extension of the trouble ticket systems to
  multicast incidents

32
33
GÉANT multicast monitoring

• Beacon
• Tool initially developed by Kai Chen from NLANR
• dast.nlanr.net/projects/Beacon/
• Relies on a number of Agents spread over the
  network which simultaneously send and receive
  multicast packets carrying a packet sequence
  number and a timestamp.
• Communicates with a central server which displays
  matrices of Agents via web pages.

33
34
GÉANT multicast monitoring

• Recommended by TF-NGN group
• (www.dante.net/tf-ngn)
• To use from day 1
• monitoring of multicast inside and outside of
  GÉANT.
• Server code enhanced with historical
  functionality
• http//noc.man.poznan.pl/noc/index/strony (Menu
  item Applications)
• romradz_at_man.poznan.pl
• Multicast Beacon Agent written in C
• http//www.cesnet.cz/tf-ngn/multicast/

34
35
GÉANT multicast monitoring

• Beacons matrices
• One for the internal sources of GÉANT
• Each GÉANT POP has a beacon agent installed
• One for the external sources of GÉANT
• http//beaconserver.geant.net19999/
• We have assigned two multicast groups from GLOP
  range RFC2770 for each matrix
• Parameters monitored
• Loss
• Delay
• Jitter

35
36
Beacon internal/external matrix
www.dante.net/nep/GÉANT-MULTICAST/deployment-beaco
n.html
36
37
Other Monitoring Tools

• Per group monitoring from TEN-155
• Based on the IETF IP-MROUTE MIB
• Shows traffic per multicast group per interface
• Under installation on ws1.se.geant.net
• MSDP and MBGP monitoring tools
• Not yet available

37
38
CONCLUSION

• Conclusion
• From TEN-155 to GÉANT Unicast and multicast moves
  to a congruent topologyacross Europe and towards
  research peerings.
• Deployment status
• Links
• http//www.dante.net/nep/GEANT-MULTICAST/
• http//beaconserver.geant.net19999/

38
39
THANKS
We would like to thank and acknowledge the help
of the people who worked and are working with us
on these projects, mainly from all EU and US
research networks
39
40
Questions ?
40
41
SA Filter list
41