Long Haul Native 10 Gigabit Ethernet

1
Long Haul Native 10 Gigabit Ethernet

• Catalin Meirosu
• CERN and PolitehnicaUniversity of Bucuresti
• On behalf of a large collaboration
• CGW03, October 27th 2003

2
Many people have contributed
3
Contents

• Whats in it for the ATLAS experiment ?
• Our 1 Gbps long haul experiments
• 10GE Ethernet goes global
• Long Haul Native Ethernet
• Over dark fibre
• Over a DWDM path
• Over a SONET circuit

4
Long haul networks in ATLAS ?

• Allow access to data from the ATLAS experiment,
  from off-site computing facilities, in real time
• Monitoring
• Calibration
• Event farms for data filtering
• Estimated sustained traffic rate for remote data
  filtering minimum 20 Gbps

5
Our first WAN experiments at 1 Gbps
NBI - Copenhagen
CERN
850 Mbps sustained TCP throughput
6
The IEEE 802.3ae 10 Gigabit Ethernet

• 10GE standard adopted in July 2002
• Increases the speed of the LAN to 10 Gbps
• Supports only fibre and full duplex transmission
  mode
• Maximum reach limited by the optical components
  40km using a 1550nm laser on single mode fibre

7
10 GE over dark fibre - ESTA
250km

• 250 km of DARENETs dark fibre in July 2003
• 4 optical amplifiers in each direction
• error-free transmission for 66 hours

8
The LAN goes global

• 10GE introduces a gateway from LAN to the WAN by
  means of the WAN PHY
• Compatible with existing WAN infrastructure
• Transmission rate
• Encapsulation
• Different optical characteristics
• Minor differences in the use of management bits
  of the SONET/SDH frame
• Note Todays WAN PHY modules in fact use SONET
  optics

9
Connecting the LAN to the WAN
Router
LTE
OC192
Router
WAN
LTE
3R
OC192
3R
3R
3R
3R
10GE Switch
LTE
WAN PHY

WAN PHY
OC192
10GE Switch
Router
10
Lab tests of the 10 GE WAN PHY
Force10 E-600
Force10 E-600
Cisco ONS 15454
Cisco ONS 15454
IXIA 400T
WAN PHY
OC192
OC192
WAN PHY
GE
OC192c

• Performed at CANARIEs lab at Ottawa
• Establishes interoperability between a certain
  WAN PHY and a specific SONET OC-192 equipment
• Gate opener for the rest of our WAN PHY
  experiences

11
Out in the W(ild )AN

• Between Geneva and Amsterdam, using an OC-192
  circuit provided by SURFnet over a wavelength
  delivered by GlobalCrossing
• started on August 15th by CERN and SURFnet and
  later joined by the University of Amsterdam
• Aims
• Validate the WAN PHY in the field
• over a DWDM path
• over a SONET circuit
• How are higher-layer protocols handling such a
  long fat pipe ?

12
WAN PHY over DWDM
DWDM
DWDM
IXIA
IXIA
10 GE
10 GE
CERN
SURFnet
Geneva
Amsterdam

• No packet loss for 91 hours and 365 TB of
  continuous streaming (equivalent to a BER better
  then of 0.310-15)

13
WAN PHY over SONET
ONS 15454
ONS 15454
IXIA
IXIA
DWDM
DWDM
10 GE
OC192
10 GE
OC192
SURFnet
CERN
Geneva
Amsterdam

• Error-free transmission
• Field validation of our previous lab experiments

14
Server to server traffic
Force10 E600
HP Itanium-2
DWDM
Ixia 400T
10 GE WAN
10GE LAN
Force10 E600
Intel Xeon
HP Itanium-2
Ixia 400T
10 GE WAN
10GE LAN
DWDM
Amsterdam
Geneva
HP Itanium-2

• Maximum numbers for single stream, 8000 bytes
  frames
• from Itanium to Itanium Raw Ethernet 5.50
  Gbps UDP 5.45 Gbps
• from Xeon to Itanium Raw Ethernet 4.16
  Gbps UDP4.08 Gbps
• 2 concurrent streams, 2 pairs of PCs A total of
  9.07 Gbps throughput

15
Sustained TCP streaming

• CERN UvA Average rate of 5448 Mbps for almost
  15 hours

16
The First Transatlantic Native 10 GE
Oct. 6th, 2003
HP Itanium-2
10GE WAN PHY
10GE LAN PHY
OC192c
Intel Itanium-2
HP Itanium-2
Intel Xeon
Ixia 400T
Cisco ONS 15454
Cisco ONS 15454
Cisco ONS 15454
Cisco ONS 15454
Cisco ONS 15454
Force10E 600
Force10 E 600
Ixia 400T
Geneva
Ottawa
Toronto
Chicago
Amsterdam
WAN PHY over an OC-192c circuit using lightpaths
provided by SURFnet and CANARIE
9.24 Gbps using traffic generators
5.24 Gbps using TCP on PCs
6 Gbps using UDP on PCs
17
Long Haul Ethernet

• End-to-end Ethernet frames over legacy long haul
  networks
• Encapsulation (OSI layer 2.5)
• Several pre-standard methods (IETF drafts for
  Ethernet frames over MPLS (EoMPLS) martini, vpls
  )
• Implemented by most of the major vendors
• Native (OSI layer 2)
• 10 GE WAN PHY
• Layer 2 or Layer 3, same performance to the end
  user
• The newest TCP single stream Internet 2 Land
  Speed Record (Layer 3 transmission) 5.66 Gbps
  between Geneva and Los Angeles (Caltech/CERN over
  the DataTAG/LHCgrid testbed)
• 5.66 Gbps for a single stream TCP transfer
  between Geneva and Ottawa on our Layer 2
  connection (after we have finalized our system
  tuning with help from Caltech)

18
Summary

• The reach of the 10 GE LAN PHY can be extended
  using optical amplifiers to at least 250 km
• 10 Gigabit Ethernet Alliance goals related to the
  WAN PHY
• Direct attachment of the 10GE WAN PHY to todays
  SONET/SDH transponders, providing access to the
  installed base of DWDM checked
  (Geneva-Amsterdam)
• Direct attachment of the 10GE WAN PHY to an
  OC-192 tributary interface checked (Canarie lab,
  Geneva Amsterdam, Geneva - Ottawa)
• Direct attachment of the 10GE WAN PHY to emerging
  ITU OTN transponders, providing access to the
  next era of wide area infrastructure looking
  forward to doing it
• Current PC architecture cannot keep up with 10 GE

19
Acknowledgments for our industrial partners
20
Questions ?