Gfarm Grid File System for Distributed and Parallel Data Computing - PowerPoint PPT Presentation

Loading...

PPT – Gfarm Grid File System for Distributed and Parallel Data Computing PowerPoint presentation | free to download - id: 7b6b4d-YmQ3O



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Gfarm Grid File System for Distributed and Parallel Data Computing

Description:

Title: How to build Grid Datafarm Author: Osamu Tatebe Last modified by: ASCC Created Date: 9/13/2004 11:41:43 AM Document presentation format: On-screen Show – PowerPoint PPT presentation

Number of Views:32
Avg rating:3.0/5.0
Slides: 25
Provided by: Osam45
Learn more at: http://www.apan.net
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Gfarm Grid File System for Distributed and Parallel Data Computing


1
Gfarm Grid File System for Distributed and
Parallel Data Computing
APAN Workshop on Exploring eScience Aug 26,
2005 Taipei, Taiwan
  • Osamu Tatebe
  • o.tatebe_at_aist.go.jp
  • Grid Technology Research Center, AIST

2
Background Petascale Data Intensive Computing
  • High Energy Physics
  • CERN LHC, KEK-B Belle
  • MB/collision, 100 collisions/sec
  • PB/year
  • 2000 physicists, 35 countries

Detector for LHCb experiment
Detector for ALICE experiment
  • Astronomical Data Analysis
  • data analysis of the whole data
  • TBPB/year/telescope
  • Subaru telescope
  • 10 GB/night, 3 TB/year

3
Petascale Data-intensive Computing Requirements
  • Peta/Exabyte scale files, millions of millions of
    files
  • Scalable computational power
  • gt 1TFLOPS, hopefully gt 10TFLOPS
  • Scalable parallel I/O throughput
  • gt 100GB/s, hopefully gt 1TB/s within a system and
    between systems
  • Efficiently global sharing with group-oriented
    authentication and access control
  • Fault Tolerance / Dynamic re-configuration
  • Resource Management and Scheduling
  • System monitoring and administration
  • Global Computing Environment

4
Goal and feature of Grid Datafarm
  • Goal
  • Dependable data sharing among multiple
    organizations
  • High-speed data access, High-performance data
    computing
  • Grid Datafarm
  • Gfarm Grid File System Global dependable
    virtual file system
  • Federates scratch disks in PCs
  • Parallel distributed data computing
  • Associates Computational Grid with Data Grid
  • Features
  • Secured based on Grid Security Infrastructure
  • Scalable depending on data size and usage
    scenarios
  • Data location transparent data access
  • Automatic and transparent replica selection for
    fault tolerance
  • High-performance data access and computing by
    accessing multiple dispersed storages in parallel
    (file affinity scheduling)

5
Gfarm file system (1)
  • Virtual file system that federates local disks of
    cluster nodes or Grid nodes
  • Enables transparent access using Global namespace
    to dispersed file data in a Grid
  • Supports fault tolerance and avoid access
    concentration by automatic and transparent
    replica selection
  • It can be shared among all cluster nodes and
    clients

Global namespace
mapping
File replica creation
Gfarm File System
6
Gfarm file system (2)
  • A file can be shared among all nodes and clients
  • Physically, it may be replicated and stored on
    any file system node
  • Applications can access it regardless of its
    location
  • In cluster environment, shared secret key is used
    for authentication

GridFTP, samba, NFS server
Gfarm metadata server
Compute node
Compute node
Compute node
Compute node
Compute node
Client PC
/gfarm
Gfarm file system
metadata
File A
File A
Note PC
File B
File C
File C
File A
File B
File B

File C
7
Grid-wide configuration
  • Grid-wide file system by integrating local disks
    in several areas
  • GSI authentication
  • It can be shared among all cluster nodes and
    clients
  • GridFTP and samba servers in each site

Gfarm Grid file system
GridFTP server, samba server, (NFS server)
Metaserver node
Compute fs node
Compute fs node

Compute fsnode
GridFTP server, samba server, (NFS server)
Compute fs node
Compute fs node

Compute fs node
GridFTP server, samba server, (NFS server)
Compute fs node
Compute fs node

Compute fs node
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
/gfarm
Japan
Singapore
US
8
Feature of Gfarm file system
  • A file can be stored on any file system (compute)
    node (Distributed file system)
  • A file can be replicated and stored on different
    nodes (Fault tolerant, access concentration
    tolerant)
  • When there is a file replica on a compute node,
    it can be accessed without overhead (High
    performance, scalable I/O)

9
More Scalable I/O Performance
Users view
Physical execution view in Gfarm (file-affinity
scheduling)
File A
User A submits that accesses
is executed on a node that has
File A
Job A
Job A
User B submits that accesses
is executed on a node that has
File B
Job B
File B
Job B
network
Cluster, Grid
CPU
CPU
CPU
CPU
Gfarm file system
File system nodes compute nodes
Shared network file system
Do not separate storage and CPU (SAN not
necessary)
Move and execute program instead of moving
large-scale data
Scalable file I/O by exploiting local I/O
10
GfarmTM Data Grid middleware
  • Open source development
  • GfarmTM version 1.1.1 released on May 17th, 2005
    (http//datafarm.apgrid.org/)
  • Read-write mode support, more support for
    existing binary applications, metadata cache
    server
  • A shared file system in a cluster or a grid
  • Accessibility from legacy applications without
    any modification
  • Standard protocol support by scp, GridFTP server,
    samba server, . . .

Metadata server
  • Existing applications can access Gfarm file
    system without any modification using
    LD_PRELOAD of syscall hooking library or
    GfarmFS-FUSE

application
gfmd
slapd
Gfarm client library
CPU
CPU
CPU
CPU
gfsd
gfsd
gfsd
gfsd
. . .
Compute and file system nodes
11
GfarmTM Data Grid middleware (2)
  • libgfarm Gfarm client library
  • Gfarm API
  • gfmd, slapd Metadata server
  • Namespace, replica catalog, host information,
    process information
  • gfsd I/O server
  • Remote file access

Metadata server
application
File, host information
gfmd
slapd
Gfarm client library
Remote file access
CPU
CPU
CPU
CPU
gfsd
gfsd
gfsd
gfsd
. . .
Compute and file system nodes
12
Access from legacy applications
  • libgfs_hook.so system call hooking library
  • It emulates to mount Gfarm file system at /gfarm
    hooking open(2), read(2), write(2),
  • When it accesses under /gfarm, call appropriate
    Gfarm API
  • Otherwise, call ordinal system call
  • Re-link not necessary by specifying LD_PRELOAD
  • Linux, FreeBSD, NetBSD,
  • Higher portability than developing kernel module
  • Mounting Gfarm file system
  • GfarmFS-FUSE enables to mount Gfarm file system
    using FUSE mechanism in Linux
  • released on Jul 12, 2005
  • Need to develop a kernel module for other OSs
  • Need volunteers

13
Gfarm Application and performance result
  • http//datafarm.apgrid.org/

14
Scientific Application (1)
  • ATLAS Data Production
  • Distribution kit (binary)
  • Atlfast fast simulation
  • Input data stored in Gfarm file system not NFS
  • G4sim full simulation
  • (Collaboration with ICEPP, KEK)
  • Belle Monte-Carlo/Data Production
  • Online data processing
  • Distributed data processing
  • Realtime histgram display
  • 10 M events generated in a few days using a
    50-node PC cluster
  • (Collaboration with KEK, U-Tokyo)

15
Scientific Application (2)
  • Astronomical Object Survey
  • Data analysis on the whole archive
  • 652 GBytes data observed by SUBARU telescope
  • Large configuration data from Lattice QCD
  • Three sets of hundreds of gluon field
    configurations on a 24348 4-D space-time
    lattice (3 sets x 364.5 MB x 800 854.3 GB)
  • Generated by the CP-PACS parallel computer
    at Center for Computational Physics, Univ. of
    Tsukuba (300Gflops x years of CPU time)

16
Performance result of parallel grep
  • 25 GBytes text file
  • Xeon 2.8GHz/512KB, 2GB memory
  • NFS 340 sec (sequential grep)
  • Gfarm 15 sec (16 fs nodes, 16 parallel processes)
  • 22.6 times superlinear speed up

Compute node
Compute node
Compute node
Compute node
.
.
.
Gfarm file system
NFS
Gfarm file system consists of local disks of
compute nodes
17
GridFTP data transfer performance
Client
Client
Client
Client
Client
Client
Client
Client
ftpd
Local disk vs Gfarm (12 nodes)
Two GridFTP servers can provide almost peak
performance (1 Gbps)
18
Gaussian 03 in Gfarm
  • Ab initio quantum chemistry Package
  • Install once and run everywhere
  • No modification required to access Gfarm
  • Test415 (IO intensive test input)
  • 1h 54min 33sec (NFS)
  • 1h 0min 51sec (Gfarm)
  • Parallel analysis of all 666 test inputs using 47
    nodes
  • Write error! (NFS)
  • Due to heavy IO load
  • 17h 31m 02s (Gfarm)
  • Quite good scalability of IO performance
  • Elapsed time can be reduced by re-ordering test
    inputs

Compute node
NFS vs Gfarm
Compute node
Compute node
Compute node
.
.
.
NFS vs Gfarm
Gfarm consists of local disks of compute nodes
19
Bioinformatics in Gfarm
  • iGAP (Integrative Genome Annotation Pipeline)
  • A suite of bioinformatics software for protein
    structural and functional annotation
  • More than 140 complete or partial proteomes
    analyzed
  • iGAP on Gfarm
  • Install once and run everywhere using Gfarms
    high performance file replication and transfer
  • no modifications required to use distributed
    compute and storage resource

Burkholderia mallei (Bacteria)
Gfarm makes it possible to use iGAP to
analyze the complete proteome (available
9/28/04) of the bacteria Burkholderia mallei, a
known biothreat agent, on distributed
resources. This is a collaboration under PRAGMA
and the data is available through
http//eol.sdsc.edu.
Participating sites SDSC/UCSD (US), BII
(Singapore), Osaka Univ, AIST (Japan), Konkuk
Univ, Kookmin Univ, KISTI (Korea)
20
Protein sequences
structure info
sequence info
Prediction of signal peptides (SignalP,
PSORT) transmembrane (TMHMM, PSORT) coiled
coils (COILS) low complexity regions (SEG)
NR, PFAM
SCOP, PDB
Step 1
Building FOLDLIB PDB chains SCOP domains PDP
domains CE matches PDB vs. SCOP 90 sequence
non-identical minimum size 25 aa coverage (90,
gaps lt30, endslt30)

Structural assignment of domains by WU-BLAST
Step 2
Structural assignment of domains by PSI-BLAST
profiles on FOLDLIB
Step 3
Structural assignment of domains by 123D on
FOLDLIB
Step 4
Functional assignment by PFAM, NR assignments
FOLDLIB
Step 5
Domain location prediction by sequence
Step 6
Data Warehouse
21
Cluster configuration of Worldwide iGAP/Gfarm
data analysis
N P Type Proc Speed (GHz) Memory (GB) Hard Disk (GB) OS
Kookmin 8 1 Athlon 1 0.256 40 RH 7.3
Konkuk 8 1 Intel P4 3 1 40 RH 7.3
KISTI 8 1 Intel P4 2.4 1 80 RH 7.3
UCSD 5 2 Intel Xeon 3.06 2 40 Rocks 3.2
BII 4 1 Intel P3 1.3 1 30 RH 8.0
Osaka 10 1 Intel P3 1.4 1 70 RH 7.2
Total 43 48 202 52.048 2500
22
Preliminary performance result
  • Multiple cluster data analysis

NFS
4-node cluster A 30.07 min
Gfarm
4-node cluster A 4-node cluster B 17.39 min
23
(No Transcript)
24
Development Status and Future Plan
  • Gfarm Grid file system
  • Global virtual file system
  • A dependable network shared file system in a
    cluster or a grid
  • High performance data computing support
  • Associates Computational Grid with Data Grid
  • Gfarm Grid software
  • Version 1.1.1 released on May 17, 2005
    (http//datafarm.apgrid.org/)
  • Version 1.2 available real soon now
  • Existing programs can access Gfarm file system
    using syscall hooking library or GfarmFS-FUSE
  • Distribute analysis shows scalable I/O
    performance
  • iGAP/Gfarm bioinformatics package
  • Gaussian 03 Ab initio quantum chemistry package
  • Standardization effort with GGF Grid File System
    WG (GFS-WG)

https//datafarm.apgrid.org/
About PowerShow.com