High Rate Internet Data Transfer for eVLBI

1
High Rate Internet Data Transfer for eVLBI

• Ralph Spencer, Richard Hughes-Jones and Simon
  Casey
• The University of Manchester
• All Hands Sept 2005

2
Radio Astronomy

• The study of celestial objects at lt1 mm to gt1m
  wavelength.
• Sensitivity for continuum sources
• Bbandwidth, tintegration time.
• High resolution achieved by interferometers.
• Some radio emitting X-ray binary stars in our own
  galaxy

GRS 1915105 MERLIN
Cygnus X-1 VLBA
SS433 MERLIN and European VLBI
3
GRS 1915105 15 solar mass BH in an X-ray
binary MERLIN observations
receding
600 mas 6000 A.U. at 10 kpc
4
Very Long Baseline Interferometry
5
How interferometry works

• Interferometer signals from each telescope
  brought together coherently for correlation
  (multiplication)
• Resolution l/D ltlt that possible from a single
  telescope, e.g the pimples on a golf ball at 3000
  miles at the highest resolution
• Output of the correlator is one component of the
  Fourier transform of the sky brightness

6
Earth-Rotation Synthesis
The baseline between each pair of telescopes
forms an ellipse as the Earth rotates
.
7
The Aperture Plane (u,v)
EVN

• Need 12 hours for full synthesis, not
  necessarily collecting data for all that time.
• Telescope data correlated in pairs N(N-1)/2
  baselines
• NBTrade-off between B and t for sensitivity
  sqrt(Bt)
• Data are calibrated, Fourier inverted and then
  deconvolved to give the images.

MERLIN
8
The European VLBI NetworkEVN

• Detailed radio imaging uses antenna networks over
  100s-1000s km
• At faintest levels, sky teems with galaxies being
  formed
• Currently use disk recording at 512Mb/s (MkV)
• real-time connection allows greater
• response
• reliability
• sensitivity
• Need Internet eVLBI

9
EVN-NREN
Gbit link
Chalmers University of Technology, Gothenburg
OnsalaSweden
Gbit link
TorunPoland
Jodrell BankUK
WesterborkNetherlands
DedicatedGbit link
MERLIN
Dwingeloo DWDM link
CambridgeUK
MedicinaItaly
10
eVLBI Milestones

• January 2004 Disk buffered eVLBI session
• Three telescopes at 128Mb/s for first eVLBI image
• On Wb fringes at 256Mb/s
• April 2004 Three-telescope, real-time eVLBI
  session.
• Fringes at 64Mb/s
• First real-time EVN image - 32Mb/s.
• September 2004 Four telescope real-time eVLBI
• Fringes to Torun and Arecibo
• First EVN, eVLBI Science session
• January 2005 First dedicated light-path eVLBI
• Australia to JIVE via Canarie data from Huygens
  descent transferred at 450Mb/s

11

• 20 December 20 2004
• connection of JBO to Manchester by 2 x 1 GE
• eVLBI tests between Poland Sweden UK and
  Netherlands at 256 Mb/s

• February 2005
• TCP and UDP memory memory tests at rates up to
  450 Mb/s (TCP) and 650 Mb/s (UDP)
• Tests showed inconsistencies between Red Hat
  kernels, rates of 128 Mb/s only obtained on 10
  Feb
• Haystack (US) Onsala (Sweden) runs at 256 Mb/s

• Regular tests with VLBI data every 6 weeks
• 128 Mpbs OK, 256 Mpbs often,
• But not 512 Mbps WHY NOT?
• Correlator can cope with large error rates
• up to 2
• but need high throughput for sensitivity
• implications for protocols

12
UDP Throughput Oct-Nov 2003 Manchester-Dwingeloo
Production

• Throughput vs packet spacing
• Manchester 2.0G Hz Xeon
• Dwingeloo 1.2 GHz PIII
• Near wire rate, 950 Mbps
• UDPmon
• Packet loss
• CPU Kernel Load sender
• CPU Kernel Load receiver
• 4th Year project
• Adam Mathews
• Steve OToole

13
January 2005 repeated UDP over 2 days
14
ESLEA

• Packet loss will cause low throughput in TCP/IP
• Congestion will result in routers drooping
  packets use Switched Light Paths!
• Tests with MB-NG network Jan-
• JBO connected to JIVE via UKLight in June (thanks
  to John Graham, UKERNA)
• Comparison tests between UKLight connections
  JBO-JIVE and production (SJ4-Geant)

15
UKLight Switched light path
16
Tests on the UKLight switched light-path
Manchester Dwingeloo

• Throughput as a function of inter-packet spacing
• Packet loss
• Maximum size packets can reach full line rates
  with no loss, and there was no re-ordering (plot
  not shown).

17
Tests on the production network Manchester
Dwingeloo.

• Throughput
• Small (0.2) packet loss was seen
• Re-ordering of packets was significant

18
UKLight using MkV terminals
19
The GÉANT2 Launch June 2005
20
e-VLBI at the GÉANT2 Launch Jun 2005
21
UDP Performance 3 Flows on GÉANT

• Throughput 5 Hour run
• 1500 byte MTU
• Jodrell JIVE2.0 GHz dual Xeon 2.4 GHz dual
  Xeon670-840 Mbit/s
• Medicina (Bologna) JIVE 800 MHz PIII Mk5
  (623) 1.2 GHz PIII 330 Mbit/s limited by sending
  PC
• Torun JIVE 2.4 GHz dual Xeon Mk5 (575) 1.2
  GHz PIII 245-325 Mbit/s limited by security
  policing (gt600Mbit/s ? 20 Mbit/s) ?
• Throughput 50 min period?
• Period is 17 min

22
18 Hour Flows on UKLightJodrell JIVE, 26 June
2005

• Throughput
• Jodrell JIVE2.4 GHz dual Xeon 2.4 GHz
  dual Xeon960-980 Mbit/s
• Traffic through SURFnet
• Packet Loss
• Only 3 groups with 10-150 lost packets each
• No packets lost the rest of the time
• Packet re-ordering
• None

23
Conclusion and Future directions

• eVLBI needs to get to rates of 512 Mpbs not
  there yet.
• Rates of 256 Mpbs on production networks possible
  now using TCP/IP in good conditions, but not
  higher rates.
• G2 launch showed continuous long term flows as in
  VLBI have different characteristics to those in
  short term tests (Iperf, UDPmon) more
  investigation needed collaborating with DANTE
  and SURFnet
• Packet loss is a killer for TCP new protocols
  needed (ESLEA) - VSI-E standard
• The End Host Problem
• The performance of Motherboards, NICs, RAID
  controllers and Disks matter Mk V units
  (Conduant)
• MkV-B 2 Gbps data recording in 2 yrs?
• Next generation correlator 10 Gbps in 5-10 yrs

24
Thanks to

• Paul Burgess JBO
• Steve Parsley, Arpad Szomoru, Cormac Reynolds
  JIVE
• Nicola Pezzi, John Graham, Colin Greenwood, Peter
  Clarke ESLEA
• EVN observatories staff