Atlas Canada Lightpath Data Transfer Trial

1
Atlas Canada Lightpath Data Transfer Trial

• Corrie Kost, Steve McDonald (TRIUMF)
• Bryan Caron (Alberta), Wade Hong (Carleton)

2
The iGrid2002 Network
3
TRIUMF Local Loop
Only one SM fiber pair available between TRIUMF
and the CAnet 4 ONS at the BCnet POP CWDM
gear arrived at TRIUMF on Sept 4th Extreme
10GbE LR blades arrived on the same day
4
CERN Local Loop

• Extreme BD with LR Blade arrives on Sept 17
  within
• 2 days of shipping
• ONS finally installed and operational on Sept
  20th

5
We are live continent to continent!

• e2e lightpath up and running 2145 CET

traceroute to cern-10g (192.168.2.2), 30 hops
max, 38 byte packets 1 cern-10g (192.168.2.2)
161.780 ms 161.760 ms 161.754 ms
6
The Network Hardware

• At CERN and TRIUMF
• Intel 10GbE Server NICs (alpha)
• Extreme Networks Black Diamond 6808
• At TRIUMF
• MRV Nbase Xyplex Fiber Driver WDM44s to multiplex
  2GbEs to CAnet 4 ONS at BCnet POP

7
Intel 10GbE Cards

• Intel kindly loaned us 2 of their Pro/10GbE LR
  server adapters cards despite the end of their
  Alpha program
• based on Intel 82597EX 10 Gigabit Ethernet
  Controller

8
Extreme Networks

• Extreme Networks generously loaned and shipped to
  us 2 Black Diamond 6808s with their new10GbE LRi
  blades

9
Extreme Networks
TRIUMF
CERN
10
The Endpoint Hosts

• Linux-based PC Servers
• SuperMicro P4DL6 (Dual Xeon 2GHz)
• ServerWorks GC-LE Chipset
• 400 MHz front side bus
• 1 GB DDR2100 RAM
• Dual Channel Ultra 160 onboard SCSI
• SysKonnect 9843 SX GbE
• 3 independent PCI buses
• PCI-X 64 bit/133 Mhz capable

11
IDE Disk Arrays
CERN Receive Host
TRIUMF Send Host
12
Operating System

• Redhat 7.3 based Linux kernel 2.4.18-3
• Needed to support filesystems gt 1TB
• Upgrades and patches
• Patched to 2.4.18-10
• Intel Pro 10GbE Linux driver (early stable)
• SysKonnect 9843 SX Linux driver (latest)
• Ported Sylvain Ravots tcp tune patches

13
BBFTP Transfer

• Vancouver ONS
• ons-van01(enet_15/1)
• Vancouver ONS
• ons-van01(enet_15/2)

14
BBFTP Transfer

• Chicago ONS
• GigE Port 1
• Chicago ONS
• GigE Port 2

15
Tsunami Transfer

• Vancouver ONS
• ons-van01(enet_15/1)
• Vancouver ONS
• ons-van01(enet_15/2)

16
Tsunami Transfer

• Chicago ONS
• GigE Port 1
• Chicago ONS
• GigE Port 2

17
Sunday Nite Summaries
18
Tuesday Nite Summaries

• Vancouver ONS
• ons-van01(enet_15/1)
• Vancouver ONS
• ons-van01(enet_15/2)

19
Comparative Results
20
Lessons Learned

• Linux software RAID faster than most conventional
  SCSI based systems
• More disk spindles the better
• Channel bonding of two GbEs seems to work very
  well (on an unshared link)
• The larger files the better for thruput

21
Acknowledgements

• Canarie
• Bill St. Arnaud, Rene Hatem, Damir Pobric, Thomas
  Tam, Jun Jian
• Atlas Canada
• Mike Vetterli, Randall Sobie, Jim Pinfold, Pekka
  Sinervo, Gerald Oakham, Bob Orr, Michel Lefebrve,
  Richard Keeler
• HEPnet Canada
• Dean Karlen
• TRIUMF
• Peter Gumplinger, Fred Jones, Mike Losty, Jack
  Chakhalian, Renee Poutissou
• BCNET
• Mike Hrybyk, Marilyn Hay, Dennis OReilly, Don
  McWilliams

22
Acknowledgements

• Extreme Networks
• Amyn Pirmohamed, Steven Flowers, John Casselman,
  Darrell Clarke, Rob Bazinet, Damaris Soellner
• Intel Corporation
• Hugues Morin, Caroline Larson, Peter Molnar,
  Harrison Li, Layne Flake, Jesse Brandeburg

23
Acknowledgements

• Indiana University
• Mark Meiss, Stephen Wallace
• Caltech
• Sylvain Ravot, Harvey Neuman
• CERN
• Olivier Martin, Paolo Moroni, Martin Fluckiger,
  Stanley Cannon, J.P Martin-Flatin
• SURFnet/Universiteit van Amsterdam
• Pieter de Boer, Dennis Paus, Erik.Radius,
  Erik-Jan.Bos, Leon Gommans, Bert Andree, Cees de
  Laat

24
Acknowledgements

• Yotta Yotta
• Geoff Hayward, Reg Joseph, Ying Xie, E. Siu
• BCIT
• Bill Rutherford
• Jalaam
• Loki Jorgensen
• Netera
• Gary Finley