Separating Abstractions from Resources in a Tactical Storage System

1
Separating Abstractions from Resources in a
Tactical Storage System

• Douglas Thain
• University of Notre Dame
• http//www.nd.edu/ccl

2
Abstract

• Users of distributed systems encounter many
  practical barriers between their jobs and the
  data they wish to access.
• Problem Users have access to many resources
  (disks), but are stuck with the abstractions
  (cluster NFS) provided by administrators.
• Solution Tactical Storage Systems allow any user
  to create, reconfigure, and tear down
  abstractions without bugging the administrator.

3
The Standard Model
4
The Standard Model
5
Problems with the Standard Model

• Users encounter partitions in the WAN.
• Easy to access data inside cluster, hard outside.
• Must use different mechanisms on diff links.
• Difficult to combine resources together.
• Resources go unused.
• Disks on each node of a cluster.
• Unorganized resources in a department/lab.
• Unnecessary cross-talk between users.
• User A demands async NFS for performance.
• User B demands sync NFS for consistency.
• A global file system is not possible!

6
What if...

• Users could easily access any storage?
• I could borrow an unused disk for NFS?
• An entire cluster can be used as storage?
• Multiple clusters could be combined?
• I could reconfigure structures without root?
• (Or bugging the administrator daily.)
• Solution Tactical Storage System (TSS)

7
Outline

• Problems with the Standard Model
• Tactical Storage Systems
• File Servers, Catalogs, Abstractions, Adapters
• Applications
• Remote Dynamic Linking in HEP Simulation
• Remote Database Access in HEP Simulation
• Expandable Filesystem for Experimental Data
• Expandable Database for Bioinformatics Simulation
• Ongoing Work
• Malloc, Dynamic Views, DACLs, PINS
• Final Thought

8
Tactical Storage Systems (TSS)

• A TSS allows any node to serve as a file server
  or as a file system client.
• All components can be deployed without special
  privileges but with security.
• Users can build up complex structures.
• Filesystems, databases, caches, ...
• Two Independent Concepts
• Resources The raw storage to be used.
• Abstractions The organization of storage.

9
App
Adapter
???
file system
file system
file system
file system
file system
file system
file system
10
Components of a TSS

• 1 File Servers
• 2 Catalogs
• 3 Abstractions
• 4 Adapters

11
1 File Servers

• Unix-Like Interface
• open/close/read/write
• getfile/putfile to stream whole files
• opendir/stat/rename/unlink
• Complete Independence
• choose friends
• limit bandwidth/space
• evict users?
• Trivial to Deploy
• run server setacl
• no privilege required
• can be thrown into a grid system
• Flexible Access Control

Chirp Protocol
file server A
file server B
file system
owner of server A
owner of server B
12
Access Control in File Servers

• Unix Security is not Sufficient
• No global user database possible/desirable.
• Mapping external credentials to Unix gets messy.
• Instead, Make External Names First-Class
• Perform access control on remote, not local,
  names.
• Types Globus, Kerberos, Unix, Hostname, Address
• Each directory has an ACL
• globus/ONotreDame/CNDThain RWLA
• kerberosdthain_at_nd.edu RWL
• hostname.cs.nd.edu
  RL
• address192.168.1.
  RWLA

13
Problem Shared Namespace
file server
globus/ONotreDame/ RWLAX
14
Solution Reservation (V) Right
file server
mkdir only!
ONotreDame/CN V(RWLA)
15
2 - Catalogs
HTTP XML, TXT, ClassAds
catalog server
catalog server
periodic UDP updates
16
3 - Abstractions

• An abstraction is an organizational layer built
  on top of one or more file servers.
• End Users choose what abstractions to employ.
• Working Examples
• CFS Central File System
• DSFS Distributed Shared File System
• DSDB Distributed Shared Database
• Others Possible?
• Distributed Backup System
• Striped File System (RAID/Zebra)

17
CFS Central File System
appl
appl
appl
adapter
adapter
adapter
CFS
CFS
CFS
file server
file
file
file
18
DSFS Dist. Shared File System
appl
appl
adapter
adapter
DSFS
DSFS
file server
file server
file server
file
file
file
file
file
ptr
file
file
file
file
file
ptr
ptr
19
DSDB Dist. Shared Database
appl
appl
adapter
adapter
DSDB
DSDB
insert
query
direct access
file server
file server
database server
create
file
file
file index
file
file
file
file
file
file
file
file
file
20
4 - Adapter

• Like an OS Kernel
• Tracks procs, files, etc.
• Adds new capabilities.
• Enforces owners policies.
• Delegated Syscalls
• Trapped via ptrace interface.
• Action taken by Parrot.
• Resources chrgd to Parrot.
• User Chooses Abstr.
• Appears as a filesystem.
• Option Timeout tolerance.
• Option Cons. semantics.
• Option Servers to use.
• Option Auth mechanisms.

system calls trapped via ptrace
Adapter - Parrot
process table
file table
Abstractions CFS DSFS - DSDB
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
App
Adapter
???
file system
file system
file system
file system
file system
file system
file system
28
Performance Summary

• Nothing comes for free!
• System calls order of magnitude slower.
• Memory bandwidth overhead extra copies.
• TSS can drive network/switch to limits.
• Compared to NFS Protocol
• TSS slightly better on small operations. (no
  lookup)
• TSS much better in network bandwidth. (TCP)
• NFS caches, TSS doesnt (today), mixed blessing.
• On real applications
• Measurable slowdown
• Benefit far more flexible and scalable.

29
Outline

• Problems with the Standard Model
• Tactical Storage Systems
• File Servers, Catalogs, Abstractions, Adapters
• Applications
• Remote Dynamic Linking in HEP Simulation
• Remote Database Access in HEP Simulation
• Expandable Filesystem for Astrophysics Data
• Expandable Database for Mol. Dynamics Simulation
• Ongoing Work
• Malloc, Dynamic Views, DACLs, PINS
• Final Thoughts

30
Remote Dynamic Linking
Credit Igor Sfiligoi _at_ Fermi National Lab

• Modular Simulation Needs Many Libraries
• Devel. on workstations, then ported to grid.
• Selection of library depends on analysis tech.
• Solution Dynamic Link with TSS and FTP
• LD_LIBRARY_PATH/ftp/server.name/libs
• Send adapter along with job.

appl
select several MB from 60 GB of libraries
liba.so
FTP server
file system
ld.so
libb.so
Anon. Login.
adapter
WAN
libc.so
FTP driver
31
Related Work

• Lots of file services for the Grid
• GridFTP, Freeldr, NeST, IBP, SRB, RFIO,...
• Adapter interfaces with many of these!
• Why have another file server?
• Reason 1 Must have precise Unix semantics!
• Apps distinguish ENOENT vs EACCES vs EISDIR.
• FTP always returns error 550, regardless of
  error.
• Reason 2 TSS focused on easy deployment.
• No privilege required, no config files, no
  rebuilding, flexible access control, ...

32
Remote Database Access
Credit Sander Klous _at_ NIKHEF

• HEP Simulation Needs Direct DB Access
• App linked against Objectivity DB.
• Objectivity accesses filesystem directly.
• How to distribute application securely?
• Solution Remote Root Mount via TSS
• parrot M //chirp/fileserver/rootdir
• DB code can read/write/lock files
  directly.

GSI
script
DB data
TSS file server
file system
adapter
libdb.so
WAN
GSI Auth
CFS
sim.exe
33
Performance on EDG Testbed
Setup Time to Init Time/Event
Unix 446 /- 46 64s
LAN/NFS 4464 /- 172 113s
LAN/TSS 4505 /- 155 113s
WAN/TSS 6275 /- 330 88s
34
Expandable Filesystemfor Experimental Data
Credit John Poirer _at_ Notre Dame Astrophysics
Dept.
Project GRAND http//www.nd.edu/grand
35
Expandable Filesystemfor Experimental Data
Credit John Poirer _at_ Notre Dame Astrophysics
Dept.
Project GRAND http//www.nd.edu/grand
file server
36
Appl Distributed MD Database

• State of Molecular Dynamics Research
• Easy to run lots of simulations!
• Difficult to understand the big picture
• Hard to systematically share results and ask
  questions.
• Desired Questions and Activities
• What parameters have I explored?
• How can I share results with friends?
• Replicate these items five times for safety.
• Recompute everything that relied on this
  machine.
• GEMS Grid Enabled Molecular Sims
• Distributed database for MD siml at Notre Dame.
• XML database for indexing, TSS for storage/policy.

37
GEMS Distributed Database
Credit Jesus Izaguirre and Aaron Striegel, Notre
Dame CSE Dept.
database server
catalog server
catalog server
38
Active Recovery in GEMS
39
GEMS and Tactical Storage

• Dynamic System Configuration
• Add/remove servers, discovered via catalog
• Policy Control in File Servers
• Groups can Collaborate within Constraints
• Security Implemented within File Servers
• Direct Access via Adapters
• Unmodified Simulations can use Database
• Alternate Web/Viz Interfaces for Users.

40
Outline

• Problems with the Standard Model
• Tactical Storage Systems
• File Servers, Catalogs, Abstractions, Adapters
• Applications
• Remote Dynamic Linking in HEP Simulation
• Remote Database Access in HEP Simulation
• Expandable Filesystem for Astrophysics Data
• Expandable Database for Mol. Dynamics Simulation
• Ongoing Work
• Malloc, Dynamic Views, DACLs, PINS
• Final Thoughts

41
Ongoing Work

• Malloc() for the Filesystem
• Resource owners want to limit users. (quota)
• End users need space assurance. (alloc)
• Need per-user allocations, not just global
  limits.
• Dynamic Data Views
• Convert from DB to FS and back again.
• Distributed Access Control
• ACLs refer to group definitions elsewhere.
• Whats new? Fault tolerance / policy management.
• Processing in Storage (PINS)
• Move computation to data.
• Needs new programming (scripting) model.

42
Malloc in the Filesystem

• Paper Grid3 Principles and Practice
• 90 of jobs would fail, most due to disk!
• Users need to alloc disk like anything else.
• (Not accessible to user quotas, loopback)
• Allocation integrated with directory tree

scratch 100 GB
43
Dynamic Data Views

• The same data can be perceived as either a file
  system or a database.
• Example
• DB get files s.t. (Tgt300K) (MolCH4)
• FS then process using scripts and shell
• DB associate derived files with original
• FS export and tar files for others.

44
Dynamic Data Views
database server
A
C
B
Y
Z
X
45
Distributed Access Control Lists

• Users are very comfortable with the ACL and group
  model.
• Can it be adapted to a grid environment?
• Yes, can let an ACL refer to remote server.
• Challenges failures, caching, sharing policy.

TSS client
46
PINS Processing in Storage

• Observation
• Traditional clusters separate CPU and storage
  into two distinct systems/problems.
• Distributed computing is always some direct
  combination of CPU and I/O needs.
• Idea PINS
• Cluster HW is already a tighly integrated complex
  of CPU and I/O. Make the SW reflect the HW.
• Key Always compute in the same place that the
  data is located. Leave newly created data in
  place.

47
Processing in Storage
database server
XML index of data files
file server
file server
file server
file server
(X 200)
A
B
A
C
X
D
C
S1
S2
S3
S4
48
Outline

• Problems with the Standard Model
• Tactical Storage Systems
• File Servers, Catalogs, Abstractions, Adapters
• Applications
• Remote Dynamic Linking in HEP Simulation
• Remote Database Access in HEP Simulation
• Expandable Filesystem for Astrophysics Data
• Expandable Database for Mol. Dynamics Simulation
• Ongoing Work
• Malloc, Dynamic Views, DACLs, PINS
• Final Thoughts

49
Tactical Storage Systems

• Separate Abstractions from Resources
• Components
• Servers, catalogs, abstractions, adapters.
• Completely user level.
• Performance acceptable for real applications.
• Independent but Cooperating Components
• Owners of file servers set policy.
• Users must work within policies.
• Within policies, users are free to build.

50
(No Transcript)
51
Acknowledgments

• Science Collaborators
• Jesus Izaguirre
• Sander Klous
• Peter Kunzst
• Erwin Laure
• John Poirer
• Igor Sfiligoi
• Aaron Striegel

• CSE Graduate Students
• Paul Brenner
• James Fitzgerald
• Jeff Hemmes
• Paul Madrid
• Chris Moretti
• Phil Snowberger
• Justin Wozniak

52
For more information...

• Cooperative Computing Lab
• http//www.cse.nd.edu/ccl
• Cooperative Computing Tools
• http//www.cctools.org
• Douglas Thain
• dthain_at_cse.nd.edu
• http//www.cse.nd.edu/dthain

53
Extra Slides
54
Performance System Calls
55
Performance - Applications
parrot only
56
Performance I/O Calls
57
Performance Bandwidth
58
Performance DSFS