The Performance of High Throughput Data Flows for eVLBI in Europe Multiple vlbi_udp Flows, Constant

1
The Performance of High Throughput Data Flows for
e-VLBI in EuropeMultiple vlbi_udp
Flows,Constant Bit-Rate over TCPMulti-Gigabit
over GÉANT2
Richard Hughes-Jones The University of
Manchester www.hep.man.ac.uk/rich/ then
Talks
2
What is VLBI ?

• VLBI signal wave front

• Data wave front sent over the network to the
  Correlator

3
European e-VLBI Test Topology
Metsähovi Finland
Gbit link
Chalmers University of Technology, Gothenburg
Jodrell BankUK
OnsalaSweden
Gbit link
TorunPoland
2 1 Gbit links
DedicatedDWDM link
Dwingeloo Netherlands
MedicinaItaly
4
vlbi_udp UDP on the WAN

• iGrid2002 monolithic code
• Convert to use pthreads
• control
• Data input
• Data output
• Work done on vlbi_recv
• Output thread polled for data in the ring buffer
  burned CPU
• Input thread signals output thread when there is
  work to do else wait on semaphore had packet
  loss at high rate, ? variable throughput
• Output thread uses sched_yield() when no work to
  do
• Multi-flow Network performance set up in Dec06
• 3 Sites to JIVE Manc UKLight Manc production
  Bologna GEANT PoP
• Measure throughput, packet loss, re-ordering,
  1-way delay

5
vlbi_udp Some of the Problems

• JIVE made Huygens, mark524 (.54) and mark620
  (.59) available
• Within minutes of Arpad leaving, the Alteon NIC
  of mark524 lost the data network!
• OK used mark623 (.62) faster CPU
• Firewalls needed to allow vlbi_udp ports
• Aarrgg (!!!) Huygens is SUSE Linux
• Routing well this ALWAYS needs to be fixed !!!
• AMD Opteron did not like sched_getaffinity()
  sched_setaffinity()
• Comment out this bit
• udpmon flows Onsala to JIVE look good
• udpmon flows JIVE mark623 to Onsala Manc UKL
  dont work
• Firewall down stops after 77 udpmon loops
• Firewall up udpmon cant communicate with Onsala
• CPU load issues on the MarkV systems
• Dont seem to be able to keep up with receiving
  UDP flow AND emptying the ring buffer
• Torun PC / Link lost as the test started

6
Multiple vlbi_udp Flows

• Gig7 ? Huygens UKLight 15 us spacing
• 816 Mbit/s sigma lt1Mbit/sstep 1 Mbit/s
• Zero packet loss
• Zero re-ordering
• Gig8 ? mark623 Academic Internet 20 us spacing
• 612 Mbit/s
• 0.6 falling to 0.05 packet loss
• 0.02 re-ordering
• Bologna ? mark620 Academic Internet 30 us
  spacing
• 396 Mbit/s
• 0.02 packet loss
• 0 re-ordering

7
The Impact of Multiple vlbi_udp Flows

• Gig7 ? Huygens UKLight 15 us spacing 800
  Mbit/s
• Gig8 ? mark623 Academic Internet 20 us
  spacing 600 Mbit/s
• Bologna ? mark620 Academic Internet 30 us spacing
  400 Mbit/s

SJ5 Access link
SURFnet Access link
GARR Access link
8
e-VLBI Driven by Science
Microquasar GRS1915105 (11 kpc) on 21 April
2006 at 5 Ghz using 6 EVN telescopes, during a
weak flare (11 mJy), just resolved in jet
direction (PA140 deg). (Rushton et al.)

• 128 Mbit/s from each telescope
• 4 TBytes raw samples data over 12 hours
• 2.8 GBytes of correlated data

Microquasar Cygnus X-3 (10 kpc) on 20 April (a)
and 18 May 2006 (b). The source as in a
semi-quiescent state in (a) and in a flaring
state in (b), The core of the source is probably
20 mas to the N of knot A. (Tudose et al.)
b
a
9
RR001 The First Rapid Response Experiment
(Rushton Spencer)

• The experiment was planned as follows
• Operate EVN 6 telescope in real time on 29th Jan
  2007
• Correlate and Analyse results in double quick
  time
• Select sources for follow up observations
• Observe selected sources 1 Feb 2007

The experiment worked we successfully observed
and analysed 16 sources (weak microquasars),
ready for the follow up run but we found that
none of the sources were suitably active at that
time. a perverse universe!
10
Constant Bit-Rate Data over TCP/IP
11
CBR Test Setup
12
Moving CBR over TCP
Effect of loss rate on message arrival time. TCP
buffer 1.8 MB (BDP) RTT 27 ms
When there is packet loss TCP decreases the
rate. TCP buffer 0.9 MB (BDP) RTT 15.2 ms
Timely arrivalof data
Can TCP deliver the data on time?
13
Resynchronisation
Arrival time
Message number / Time
14
CBR over TCP Large TCP Buffer

• Message size 1448 Bytes
• Data Rate 525 Mbit/s
• RouteManchester - JIVE
• RTT 15.2 ms
• TCP buffer 160 MB
• Drop 1 in 1.12 million packets
• Throughput increases
• Peak throughput 734 Mbit/s
• Min. throughput 252 Mbit/s

15
CBR over TCP Message Delay

• Message size 1448 Bytes
• Data Rate 525 Mbit/s
• RouteManchester - JIVE
• RTT 15.2 ms
• TCP buffer 160 MB
• Drop 1 in 1.12 million packets
• OK you can recover BUT
• Peak Delay 2.5s
• TCP buffer ? RTT4

16
Multi-gigabit tests over GÉANT But will 10
Gigabit Ethernet work on a PC?
17
High-end Server PCs for 10 Gigabit

• Boston/Supermicro X7DBE
• Two Dual Core Intel Xeon Woodcrest 5130
• 2 GHz
• Independent 1.33GHz FSBuses
• 530 MHz FD Memory (serial)
• Parallel access to 4 banks
• Chipsets Intel 5000P MCH PCIe MemoryESB2
  PCI-X GE etc.
• PCI
• 3 8 lane PCIe buses
• 3 133 MHz PCI-X
• 2 Gigabit Ethernet
• SATA

18
10 GigE Back2Back UDP Latency

• Motherboard Supermicro X7DBE
• Chipset Intel 5000P MCH
• CPU 2 Dual Intel Xeon 5130 2 GHz with 4096k L2
  cache
• Mem bus 2 independent 1.33 GHz
• PCI-e 8 lane
• Linux Kernel 2.6.20-web100_pktd-plus
• Myricom NIC 10G-PCIE-8A-R Fibre
• myri10ge v1.2.0 firmware v1.4.10
• rx-usecs0 Coalescence OFF
• MSI1
• Checksums ON
• tx_boundary4096
• MTU 9000 bytes
• Latency 22 µs very well behaved
• Latency Slope 0.0028 µs/byte
• B2B Expect 0.00268 µs/byte
• Mem 0.0004
• PCI-e 0.00054

19
10 GigE Back2Back UDP Throughput

• Kernel 2.6.20-web100_pktd-plus
• Myricom 10G-PCIE-8A-R Fibre
• rx-usecs25 Coalescence ON
• MTU 9000 bytes
• Max throughput 9.4 Gbit/s
• Notice rate for 8972 byte packet
• 0.002 packet loss in 10M packetsin receiving
  host
• Sending host, 3 CPUs idle
• For lt8 µs packets, 1 CPU is gt90 in kernel
  modeinc 10 soft int
• Receiving host 3 CPUs idle
• For lt8 µs packets, 1 CPU is 70-80 in kernel
  modeinc 15 soft int

20
10 GigE UDP Throughput vs packet size

• Motherboard Supermicro X7DBE
• Linux Kernel 2.6.20-web100_pktd-plus
• Myricom NIC 10G-PCIE-8A-R Fibre
• myri10ge v1.2.0 firmware v1.4.10
• rx-usecs0 Coalescence ON
• MSI1
• Checksums ON
• tx_boundary4096
• Steps at 4060 and 8160 byteswithin 36 bytes of
  2n boundaries
• Model data transfer time as t C mBytes
• C includes the time to set up transfers
• Fit reasonable C 1.67 µs m 5.4 e4 µs/byte
• Steps consistent with C increasing by 0.6 µs
• The Myricom driver segments the transfers,
  limiting the DMA to 4096 bytes PCI-e chipset
  dependent!

21
10 GigE X7DBE?X7DBE TCP iperf

• No packet loss
• MTU 9000
• TCP buffer 256k BDP330k
• Cwnd
• SlowStart then slow growth
• Limited by sender !
• Duplicate ACKs
• One event of 3 DupACKs
• Packets Re-Transmitted
• Iperf TCP throughput 7.77 Gbit/s

Web100 plots of TCP parameters
22
OK so it works !!!
23
ESLEA-FABRIC4 Gbit flows over GÉANT2

• Set up 4 Gigabit Lightpath Between GÉANT2 PoPs
• Collaboration with DANTE
• GÉANT2 Testbed London Prague London
• PCs in the DANTE London PoP with 10 Gigabit NICs
• VLBI Tests
• UDP Performance
• Throughput, jitter, packet loss, 1-way delay,
  stability
• Continuous (days) Data Flows VLBI_UDP and
  udpmon
• Multi-Gigabit TCP performance with current
  kernels
• Multi-Gigabit CBR over TCP/IP
• Experience for FPGA Ethernet packet systems
• DANTE Interests
• Multi-Gigabit TCP performance
• The effect of (Alcatel 1678 MCC 10GE port) buffer
  size on bursty TCP using BW limited Lightpaths

24
The GÉANT2 Testbed

• 10 Gigabit SDH backbone
• Alcatel 1678 MCCs
• GE and 10GE client interfaces
• Node location
• London
• Amsterdam
• Paris
• Prague
• Frankfurt
• Can do lightpath routingso make paths of
  different RTT
• Locate the PCs in London

25
Provisioning the lightpath on ALCATEL MCCs

• Some jiggery-pokery needed with the NMS to force
  a looped back lightpath London-Prague-London
• Manual XCs (using element manager) possible but
  hard work
• 196 needed other operations!
• Instead used RM to create two parallel VC-4-28v
  (single-ended) Ethernet private line (EPL) paths
• Constrained to transit DE
• Then manually joined paths in CZ
• Only 28 manually created XCs required

26
Provisioning the lightpath on ALCATEL MCCs

• Paths come up
• (Transient) alarms clear
• Result provisioned a path of 28 virtually
  concatenated VC-4sUK-NL-DE-NL-UK
• Optical path 4150 km
• With dispersion compensation4900 km
• RTT 46.7 ms

27
Photos at The PoP
Test-bed SDH
Production SDH
10 GE
ProductionRouter
Optical Transport
28
4 Gig Flows on GÉANT UDP Throughput

• Kernel 2.6.20-web100_pktd-plus
• Myricom 10G-PCIE-8A-R Fibre
• rx-usecs25 Coalescence ON
• MTU 9000 bytes
• Max throughput 4.199 Gbit/s
• Sending host, 3 CPUs idle
• For lt8 µs packets, 1 CPU is gt90 in kernel
  modeinc 10 soft int
• Receiving host 3 CPUs idle
• For lt8 µs packets, 1 CPU is 37 in kernel
  modeinc 9 soft int

29
4 Gig Flows on GÉANT 1-way delay

• Kernel 2.6.20-web100_pktd-plus
• Myricom 10G-PCIE-8A-R Fibre
• Coalescence OFF
• 1-way delay stable at 23.435 µs
• Peak separation 86 µs
• 40 µs extra delay

30
4 Gig Flows on GÉANT Jitter hist

• Kernel 2.6.20-web100_pktd-plus
• Myricom 10G-PCIE-8A-R Fibre
• Coalescence OFF
• Peak separation 36 µs
• Factor 100 smaller

31
4 Gig Flows on GÉANT UDP Flow Stability

• Kernel 2.6.20-web100_pktd-plus
• Myricom 10G-PCIE-8A-R Fibre
• Coalescence OFF
• MTU 9000 bytes
• Packet spacing 18 us
• Trials send 10 M packets
• Ran for 26 Hours
• Throughput very stable3.9795 Gbit/s
• Occasional trials have packet loss 40 in 10M -
  investigating

• Our thanks go to all our collaborators
• DANTE really provided Bandwidth on Demand
• A record 6 hours ! including
• Driving to the PoP
• Installing the PCs
• Provisioning the Light-path

32

• Any Questions?

33
Provisioning the lightpath on ALCATEL MCCs

• Create a virtual network element to a planned
  port (non-existing)in Prague VNE2
• Define end points
• Out port 3 in UK VNE2 CZ
• In port 4 in UK VNE2 CZ
• Add Constraint to go via DE
• Or does OSPF
• Set capacity ( 28 VC-4s )
• Alcatel Resource Manager allocates routing of
  EXPReS_outVC-4 trails
• Repeat for EXPReS_ret
• Same time slots used in CZ for EXPReS_out
  EXPReS_ret paths