High Performance Grid Data Transfer

1
High Performance Grid Data Transfer

• Bill Allcock, ANL
• BNL Technology Meeting
• 26 April 2004

2
Outline

• Transport and Related Services
• GridFTP and XIO
• Alternate Transport Protocols
• Optical Networking
• Firewalls

3
Transport and Related Services
4
Services Whats the Big Deal?

• Grids (Globus anyway) has been service oriented
  from the start.
• However, every service had its own, unique,
  non-standard, and not necessarily well documented
  protocol
• Web services began to gain traction and via SOAP
  and WSDL provided a nice interface, good tooling,
  etc..
• But, did not deal with some issues critical to
  the Grid, such as state, lifetime management, etc.

5
Open Grid Services Infrastructure (OGSI)

• The first crack at melding the Grid with Web
  Services.
• Largely driven by Globus and IBM, but OGSI
  working group in GGF was very active in its
  creation.
• Worked well, and gained lots of support
• Except from the Web Service Stack Vendors

6
Grid and Web ServicesConvergence?
Grid
Web
However, despite enthusiasm for OGSI, adoption
within Web community turned out to be problematic
7
Three Major Web Services Concerns about OGSI

• Too much stuff in one specification
• Does not work well with existing Web services
  tooling
• Too object oriented
• Read It didnt fit their model

8
Concerns Addressed

• Too much stuff in one specification
• WSRF partitions OGSI v1.0 functionality into a
  family of composable specifications
• Does not work well with existing Web services
  tooling
• WSRF tones down the usage of XML Schema
• Too object oriented
• WSRF makes an explicit distinction between the
  service and the stateful resources acted upon
  by that service

9
Grid and Web ServicesConvergence Yes!
Grid
Web
The definition of WSRF means that Grid and Web
communities can move forward on a common base
10
From OGSI to WSRFRefactoring and Evolution
Draft document at www.globus.org/wsrf this week
11
GridFTP and XIO
12
What is GridFTP?

• A secure, robust, fast, efficient, standards
  based, widely accepted data transfer protocol
• A Protocol
• Multiple Independent implementation can
  interoperate
• This works. Both the Condor Project at Uwis and
  Fermi Lab have home grown servers that work with
  ours.
• Lots of people have developed clients independent
  of the Globus Project.
• We also supply a reference implementation
• Server
• Client tools (globus-url-copy)
• Development Libraries

13
wuftpd based GridFTP

• Existing Functionality
• Security
• Reliability / Restart
• Parallel Streams
• Third Party Transfers
• Manual TCP Buffer Size
• Server Side Processing
• Partial File Transfer
• Large File Support
• Data Channel Caching
• Integrated Instrumentation
• De facto standard on the Grid

• New Functionality in 3.2
• Server Improvements
• Structured File Info
• MLST, MLSD
• checksum support
• chmod support
• globus-url-copy changes
• File globbing support
• Recursive dir moves
• RFC 1738 support
• Control of restart
• Control of DC security

14
New GridFTP Implementation

• 100 Globus code. No licensing issues.
• GT3.2 Alpha had a very minimal implementation
• GT3.2 final does NOT have the new server. It
  will be released in 4.0.
• wuftpd specific functionality, such as virtual
  domains, will NOT be present
• Will have IPV6 support included (EPRT, EPSV)
• Based on XIO
• Extremely modular to allow integration with a
  variety of data sources (files, mass stores,
  etc.)
• Striping will also be present in 4.0

15
GridFTP at SC2000 Long-Running Dallas-Chicago
Transfer
SciNet Power Failure
Other demos starting up (Congestion)
Parallelism Increases (Demos)
DNS Problems
Backbone problems on the SC Floor
Transition between files (not zero due to
averaging)
16
GridFTP Standards Based

• FTP protocol is defined by several IETF RFCs
• Start with most commonly used subset
• Standard FTP get/put etc., 3rd-party transfer
• Implement standard but often unused features
• GSS binding, extended directory listing, simple
  restart
• Extend in various ways, while preserving
  interoperability with existing servers
• Striped/parallel data channels, partial file,
  automatic manual TCP buffer setting, progress
  monitoring, extended restart

17
GridFTP Standards Based (cont)

• Existing standards
• RFC 959 File Transfer Protocol
• RFC 2228 FTP Security Extensions
• RFC 2389 Feature Negotiation for the File
  Transfer Protocol
• Draft FTP Extensions
• New drafts
• GridFTP Protocol Extensions to FTP for the Grid
• Grid Forum Recommendation
• GFD.20
• http//www.ggf.org/documents/GWD-R/GFD-R.020.pdf

18
GridFTP Caveats

• Protocol requires that the sending side do the
  TCP connect (possible Firewall issues)
• Client / Server
• Currently, no server library, therefore Peer to
  Peer type apps VERY difficult
• Generally needs a pre-installed server
• Looking at a dynamically installable server
• Striped Server is a prototype, though a heavily
  used and well tested prototype.
• alpha of new server with striping by mid summer
• Should be released by Fall 2004

19
Striped Server

• Multiple nodes work together and act as a single
  GridFTP server
• An underlying parallel file system stores blocks
  of the file, usually in round robin fashion,
  across all of the nodes.
• Each node then moves only the pieces of the file
  that it is responsible for.
• The other side then writes the file in the same
  way, block round robin on a parallel file system.
• This allows multiple levels of parallelism, CPU,
  bus, NIC, disk, etc.

20
(No Transcript)
21
Extensible IO (XIO) system

• Provides a framework that implements a
  Read/Write/Open/Close Abstraction
• Drivers are written that implement the
  functionality (file, TCP, UDP, GSI, etc.)
• Different functionality is achieved by building
  protocol stacks
• GridFTP drivers will allow 3rd party applications
  to easily access files stored under a GridFTP
  server
• Other drivers could be written to allow access
  to other data stores.
• Changing drivers requires minimal change to the
  application code.

22
Optical Networking
23
Optical Networking

• Will have (is having) a profound effect on the
  way we do computing.
• Networking speeds increassing gtgt Moores Law
• The network is no longer necessarily the weak
  link.
• Network bandwidth is gtgt than the bus on your
  computer.
• Explode the computer out over the network
• Current economics make it reasonable to have your
  own fiber.

24
Optical Networking

• Whats the big deal?
• Real QoS available
• Can carve out an OC-12 that is all yours
• Its all yours, use any protocol you want
• i.e. ditch TCP
• Optical switches are cheap
• Bringing about a change in the business model.
• Users can request reconfiguration and get it
  within minutes or even seconds.

25
Optical Networking

• What does it mean to you?
• User
• It should be nothing more than a service to
  request a reservation for a path.
• Should be able to get guaranteed, or at least
  reasonable and consistent network performance.
• Developer
• For middleware or network developers some people
  think the level of exposure will go down to every
  optical switch (mems, or individual path switch,
  not switch like the entire box)
• Applications can (and should) become network (or
  if you luck buzz words, photonic) aware. They
  can alter their behavior based on what network
  resources are available.

26
Optical Networking Caveat

• The following is all IMHO ?
• We will never go to a fully circuit switched
  network
• Just does not make sense for most web traffic
• Will be treated as a premium service
• Most people will not have the luxury of true
  end-to-end optical switching
• Will often do packet switching on-site to the
  Optical gear.
• Often will be virtual circuits over MPLS or GMPLS
• In those cases you still need proper traffic
  engineering
• This may seem obvious, but this will only work if
  it is profitable.

27
Optical Networking Picture
28
Protocols
29
Protocols

• Current TCP variants are great for web traffic,
  but not for bulk data transfer.
• Attempts to improve this fall into a few
  categories
• Evolution of AIMD
• Rate based
• Reliable UDP based
• XCP

30
Whats wrong with TCP?

• Uses packet loss to signal congestion
• Not necessarily the case
• Feedback is too coarse
• You only know when it is too late
• Additive Increase, Multiplicative Decrease (AIMD)
  for its control mechanism
• Need to have a buffer equal to the Bandwidth
  Delay Product
• The Interaction between AIMD and BWDP

31
AIMD

• To the first order this algorithm
• Begin in slow start by putting one packet on
  the network and then doubling (exponentially
  increasing) the number of unacknowledged packets
  (congestion window or CWND) each time an ACK is
  received, until it gets a congestion event
• When there is a congestion event, enter
  congestion avoidance
• Cut the CWND in half
• Linearly increase one or a few packets per ACK
• Note that CWND size is equivalent to Max BW

32
BWDP

• TCP is reliable and must keep a copy of every
  packet until the receiver acknowledges it has
  been received.
• Use a tank as an analogy
• I can keep putting water in until it is full.
• After that, I can only put in one gallon for each
  gallon removed.
• The volume of the tank is the cross sectional
  area times the height
• Think of the BW as the area and the RTT as the
  length of the network pipe.

33
Recovery Time
34
Recovery Time for a Single Congestion Event

• T1 (1.544 Mbs) with 50ms RTT ? 10 KB
• Recovery Time (1500 MTU) 0.16 Sec
• GigE with 50ms RTT ? 6250 KB
• Recovery Time (1500 MTU) 104 Seconds
• GigE to Amsterdam (100ms) ? 1250 KB
• Recovery Time (1500 MTU) 416 Seconds
• GigE to CERN (160ms) ? 2000 KB
• Recovery Time (1500 MTU) 1066 Sec (17.8 min)

35
Evolution of AIMD

• Still do Additive Increase, Multiplicative
  Decrease, but change the rate at which you
  increase and decrease
• High Speed TCP (RFC ?) Sally Floyd
• After a set point (typically 38 packets) a table
  is used and more aggressive AIMD parameters are
  set
• Scalable TCP
• Increase is exponential and decrease is 0.125
• H-TCP (Leith)
• Uses time between congestion events to
  differentiate high speed network from low speed.
• Good Paper that compares these and others
• http//dsd.lbl.gov/DIDC/PFLDnet2004/papers/Bullot.
  pdf

36
Rate Based Control

• Flow control is based on changes in RTT
• Assume an increase in RTT means buffers in the
  router are filling up
• This was tried back in the 70s, but TCP won out.
• FAST TCP out of Cal Tech (Stephen Low) uses this
  approach.
• So far, results look very promising
• Good performance
• Fairly TCP friendly
• Still TCP, so socket libraries work

37
Reliable UDP

• Send the data UDP and use some form of retransmit
  for lost data
• Reliable UDP (RFC ?)
• Retransmit and control on TCP channel
• Reliable Blast UDP (UIC/EVL/Leigh)
• Retransmit and control on TCP channel
• File based (must know of bytes being
  transmitted)
• Tsunami (IU/ANML/Wallace)
• Retransmit and control on TCP channel
• File based (must know of bytes being
  transmitted)
• SABUL/UDT (UIC/Grossman)
• SABUL (UDP/TCP), UDT (UDP ONLY)
• General transport, not file based

38
Reliable UDP

• Issues
• TCP friendliness
• Only SABUL/UDT attempt to back off in the face of
  congestion.
• This means all others will shut out TCP flows
• This is why UDT went to UDP control, others
  strangle their own control / retransmit flows.
• Standardization
• Some routers wont let UDP through
• Great choice over an optical circuit though
• Cant use the socket libraries
• Can use XIO

39
XCP

• XCP (Katabi / MIT)
• Generalization of Explicit Congestion
  Notification (ECN)
• Decouples utilization control from fairness
  control
• Based on control systems theory
• Requires router modifications ?
• Again, IMHO
• This is the way to go
• You need a real feedback and a proper control
  system

40
Firewalls
41
What Security People Would like ?
GridFTPServer
GridFTPServer
The NETWORK
Client
42
What application People Want ?
GridFTPServer
GridFTPServer
Big, Fat, Pipe
The NETWORK
Client
43
Where are we at today?

• Applications often cant run, and if they are
  high bandwidth apps, the firewall often limits
  performance.
• Today, it means manual configuration to open
  holes, but then they stay open too long, and
  anyone can exploit them.
• We need something better
• We need to open holes in the firewall
• Only while absolutely necessary
• For a specific party
• With confidence that the use of that port falls
  within authorized behavior

44
How can we do that?

• We envision two services
• A service proxy
• Intercepts incoming service requests (SOAP in Web
  Services)
• Validates / Authorizes the request
• Pluggable framework so it can be easily extended
• Once authorized forwards it to the service
• A secure, dynamic firewall automatic garage door
  opener
• Temporarily opens holes through the firewall
• Uses lifetime management to ensure the holes
  close
• Ideally can specify exact host and service that
  will contact
• Possibly have no monitoring of packets after
  that?
• But could work with IDS, if it were fast enough.
• Could, in theory, use these separately

45
A picture
service
Authorized request
Open for n minutes
Proxy
Opener
request
client
46
Issues

• You might notice the picture does not show who
  talks to the opener, this has significant
  security and effort impacts
• The Proxy?
• The Service?
• The Client?
• Stability / Security of plug-ins
• What about non-SOAP requests?
• Would this be secure enough to let the fat pipe
  run un-monitored?

47
Fusion Collaboratory
48
Even with Optical Paths