Efficient Access to Many Small Files in a Grid Filesystem - PowerPoint PPT Presentation

About This Presentation
Title:

Efficient Access to Many Small Files in a Grid Filesystem

Description:

Chirp and Parrot: A Grid Filesystem. Requirements for a Grid Filesystem ... parrot tcsh ls /chirp. alpha.nd.edu. beta.nd.edu cd /chirp/alpha.nd.edu/mydir ... – PowerPoint PPT presentation

Number of Views:48
Avg rating:3.0/5.0
Slides: 45
Provided by: dougla9
Learn more at: http://www.cse.nd.edu
Category:

less

Transcript and Presenter's Notes

Title: Efficient Access to Many Small Files in a Grid Filesystem


1
Efficient Access toMany Small Files in a Grid
Filesystem
  • Douglas Thain and Christopher Moretti
  • University of Notre Dame

2
Efficient Access to ManySmall (and Big) Files
in a Grid Filesystem
  • Douglas Thain and Christopher Moretti
  • University of Notre Dame

3
Abstract
  • Many grid data tools focus on transferring,
    storing, and managing large (GB-TB) files.
  • But, many users need to manage, transfer, and
    process lots (1000s) of small (KB-MB) files.
  • We describe protocols and interfaces for
    manipulating many small files over wide area
    networks. (Doesnt hurt large files, either.)
  • Implemented in the Chirp file system.
  • Performance
  • Best case order of magnitude improvement.
  • Worst case no slower than before.

4
The Small File Problem
5
Who has lots of small files?
  • Anyone using a batch system.
  • One file for submit, input, output, error, log...
  • Anyone using a large software package.
  • Executables, libraries, config files...
  • Anyone using a filesystem like a database.
  • Genomics, astronomy, physics...
  • Anyone who likes to write shell scripts.
  • foreach host in list ssh host gt host.output

6
Why is this a problem?
  • Users do the sensible thing
  • foreach file in (list) do transfer done
  • The sensible thing performs miserably
  • New TCP Connection
  • SSL Authentication
  • Configuration Operations
  • Slow Start Again
  • Result is KB/s on a GB/s link.

7
Why not just use tar?
  • If you can, you should!
  • Sometimes you cannot
  • The system semantics demand multiple files.
  • Packing and unpacking can be very slow.
  • Not enough disk space to unpack.
  • Different apps select different data subsets.
  • Using an existing script or program.
  • Users dont know or care that its a dist system,
    why should they change?

8
The ChallengeHow to design interfacesso that
users get the expectedperformance and behavior?
9
Chirp and ParrotA Grid Filesystem
10
Requirements for a Grid Filesystem
  • Transparent access to files in the same manner as
    a local Unix filesystem.
  • Non privileged deployment at both client and
    server. (root not possible on the grid.)
  • User control over policies for naming, caching,
    consistency, and fault tolerance.
  • Flexible access controls for sharing.
  • Good performance on both small and large files.

11
Chirp/Parrot A Grid Filesystem
Ordinary Unix Program
Authentication Kerberos /
Globus / Hostname / Unix
No Privs Needed!
Automatic Recovery
unix system calls
ptrace trap
Single TCP Stream
Chirp
Parrot
Protocol open / pread / pwrite / close stat /
mkdir / rmdir / unlink getfile / putfile /
movefile
No Privs Needed!
Ordinary Unix Filesystem
Authorization kerberosjoe_at_nd.edu
RWLDA globus/OND/CNJoe
RWLDA hostname.nd.edu
RL groupserver.nd.edu/team RWL
12
Ordinary Unix Commands
  • gt parrot tcsh

gt ls /chirp alpha.nd.edu beta.nd.edu ... gt cd
/chirp/alpha.nd.edu/mydir gt cp /tmp/bigdata . gt
emacs mydata.txt
13
Parrot Specific Commands
  • gt parrot tcsh

gt parrot_whoami globus/OND/CNJoe gt
parrot_getacl /chirp/alpha.nd.edu/ kerberosjoe_at_nd
.edu RWLDA globus/OND/CNJoe
RWL hostname.nd.edu RL
14
Chirp as Remote Filesystem
Grid Site A
Grid Site B
Secured by GSI
Chirp Server
Grid Middleware
Unix Filesystem
15
Chirp as Cluster Filesystem
Grid Site A
Grid Site B
Chirp Server
Chirp Server
Chirp Server
Chirp Server
Unix Filesystem
Unix Filesystem
Unix Filesystem
Unix Filesystem
16
http//www.cse.nd.edu/ccl/viz
17
Sample Applications
  • Image Processing for Biometrics
  • Moretti et al, PCGRID 2007
  • Bioinformatics on EGEE
  • Blanchet et al, Grid 2006
  • High Energy Physics on LCG
  • Sfiligoi et al, CHEP 2005,
  • Molecular Dynamics Repository
  • Wozniak et al, HPDC 2005
  • Remote DB Access on EDG
  • Klous et al, CCPE 2005

18
Protocols for Small Files
19
What About FTP?
  • FTP is a great data transfer system, but it was
    never designed to be a file system
  • New TCP stream per data transfer.
  • New TCP stream for each directory list.
  • Lots of connections can overwhelm net devices.
  • Coarse errors 550 for all file system errors.
  • Semantic problems e.g. empty directory.
  • Unix access controls, (But, see SecPAL)
  • Wildly varying implementations and support.

20
FTP Protocol Reminder
Control Connection
AUTH GSSAPI MIC MIC PORT RETR
FTP Client
FTP Server
Data Connection
Minimum of four round trips (plus auth overhead)
to fetch a file loss of TCP window.
AUTH GSSAPI MIC MIC Data Transfer
Common practice is new control connection for
every data transfer!
21
What About NFS?
  • NFS was designed for a local area network among
    (relatively) trusted hosts.
  • Fine-grained file access very slow on WAN.
  • Kernel support and root assistance needed to
    start server, mount client, change target.
  • Unix UID for ownership, access control.
  • Need to bind to privileged port, often filtered.
  • Use of file handles to refer to files makes it
    very difficult to build a user-level server.
  • lots of lookup operations over the WAN.

22
NFS Protocol Reminder
lookup(00,a) lookup(10,b) lookup(20,c) ...
NFS Client
NFS Server
read 4KB read 4KB read 4KB ...
On a WAN, throughput limited to 4KB/latency. 10ms
400 KB/s 100ms 40 KB/s
23
Chirp Hybrid Protocol Overview
auth globus (8 RTT) open read write close ... getf
ile(mydata) putfile(otherdata,size)
Chirp Client
Chirp Server
size and data
data
24
Protocol Comparison
  • FTP - Stream per File
  • Latency 4 RTT for each file
  • Throughput TCP limit after slow start
  • NFS Remote Procedure Call
  • Latency 1 RTT for each file
  • Throughput block size / latency
  • Chirp - Hybrid
  • Latency 1 RTT for each file
  • Throughput TCP limit in steady state

25
Local Area Performance
26
Wide Area Performance
27
Real WAN Performance
28
Interfaces for Small Files
29
Standard Unix Copy
cp /tmp/source /chirp/B/target
cp
open(source)
open(target)
loop read/write
Parrot
open(source)
read
open
write
Local
Chirp
open(source)
read
open
write
Chirp Server
Local Disk
30
ProblemThe system does not know the context of
the operation!SolutionIntroduce a
higher-level operationcopyfile that exploits the
context.
31
Improved Copy with Copyfile
cp /tmp/source /chirp/B/target
new cp
Parrot
Local
Chirp
Chirp Server
Local Disk
32
Is it reasonable to modify cp?
  • Installation
  • Cannot modify /bin/cp.
  • Install new parrot_cp
  • Alias cp or link named cp in PATH.
  • Backwards compatibility
  • parrot_cp without Parrot falls back to normal.
  • Ordinary cp on Parrot behaves as before.
  • Parrot_cp on a different filesystem falls back.

33
Improved Copy with Copyfile
cp /chirp/A/source /chirp/B/target
new cp
Parrot
Chirp
Chirp Server B
Chirp Server A
34
Directory Copy
cp r /chirp/A/mydir /chirp/B/mydir
cp
Parrot
Chirp Server B
Chirp Server A
mydir
ACL
X
Y
Z
35
Improved Directory Copy
cp r /chirp/A/mydir /chirp/B/mydir
cp
Parrot
mkdir putfile3 setacl
Chirp Server B
Chirp Server A
mydir
ACL
X
Y
Z
36
Third Party Performance
37
You get the idea...
  • ls la D
  • Original getdir D Nstat
  • Improved getlongdir D
  • rm rf D
  • Original getdir D Nunlink (recursive)
  • Improved rmall D
  • md5sum F
  • Original open F Nread close
  • Improved md5 F

38
Final Example
  • ls la /chirp/alpha/data
  • md5sum /chirp/alpha/data/
  • cp -r /chirp/alpha/data
  • /chirp/beta/data
  • md5sum /chirp/beta/data/
  • rm rf /chirp/alpha/data

39
Original Implementation
app
parrot
chirp server A
chirp server B
40
Improved Implementation
app
parrot
chirp server A
chirp server B
41
Performance on Script
42
The ChallengeHow to design interfacesso that
users get the expectedperformance and behavior?
43
Summary
  • Good small file performance requires attention to
    low level network protocols.
  • getfile, putfile, thirdput, rmall, checksum
  • Exploiting protocols requires minor changes to
    the Unix I/O interface.
  • copyfile, rmall, checksum, others?
  • Easy to apply those changes in a user transparent
    way.
  • cp, rm, md5sum all operate as normal
  • Usable performance in a wide-area FS.

44
For more information...
  • Douglas Thain
  • dthain_at_nd.edu
  • Chris Moretti
  • cmoretti_at_nd.edu
  • Parrot and Chirp
  • http//www.cctools.org
Write a Comment
User Comments (0)
About PowerShow.com