The Kangaroo Approach to Data Movement on the Grid

1
The Kangaroo Approach to Data Movementon the
Grid

• Douglas Thain, Jim Basney,
• Se-Chang Son, and Miron Livny
• http//www.cs.wisc.edu/condor/kangaroo

2
Half-Interactive Jobson the Grid

• The Grid is a vision of an international,
  highly available, robust, shared computation
  service.
• Grid applications generally
• ..are half-interactive.
• ..are brittle wrt errors.
• ..have weak consistency requirements.
• Can we perform on-line data access without
  subjecting apps to unexpected errors?

App
Unreliable Network
3
The Kangaroo Approach To Data Movement

• Make a third party responsible for executing each
  applications I/O operations.
• Use all available resources to hide latencies.
• Never return an error to the application.
• (Maybe tell the user or scheduler.)
• Benefits Fault tolerance, higher throughput.
• Cost Weaker consistency.

4
The Vision A Data Grid
K
K
K
Data Movement System
K
K
K
K
Disk
5
Kangaroo Prototype
An application may contact any node in the system
and perform partial-file reads and writes.
The node may then execute or buffer operations as
conditions warrant.
K
K
K
App
Buffer
Buffer
Disk
6
User Interface

• An interposition agent converts POSIX operations
  into the Kangaroo protocol.
• vi /kangaroo/coral.cs.wisc.edu/etc/hosts
• gcc -c /gsiftp/ftp.cs.wisc.edu/test.c -o
• /kangaroo/coral/tmp/test.o

K
Agent
App
get, put
read, write
Kangaroo
POSIX
7
MacrobenchmarkImage Processing

• Post-processing of satellite image data Need to
  compute various enhancements and produce output
  for each.
• Read input image
• For I1 to N
• Compute transformation of image
• Write output image
• Example
• Image size about 5 MB
• Compute time about 6 sec
• IO-cpu ratio about 0.9 MB/s

8
I/O Models Compared
CPU Released
Task Done
Offline I/O
OUTPUT
OUTPUT
CPU
OUTPUT
INPUT
OUTPUT
CPU
CPU
CPU
CPU Released
Online I/O
OUTPUT
OUTPUT
CPU
OUTPUT
INPUT
OUTPUT
CPU
CPU
CPU
CPU Released
Task Done
Task Done
Kangaroo
CPU
INPUT
CPU
CPU
CPU
PUSH
OUTPUT
OUTPUT
OUTPUT
OUTPUT
9