Flexibility, Manageability and Performance in a Grid Storage Appliance

1
Flexibility, Manageability and Performance in a
Grid Storage Appliance

• John Bent, Venkateshwaran Venkataramani, Nick
  Leroy, Alain Roy, Joseph Stanley, Andrea
  Arpaci-Dusseau, Remzi Arpaci-Dusseau, and Miron
  Livny
• University of Wisconsin

2
Two trends

• Data sets
• Performance

• Storage appliances address both trends.

3
Storage Appliances and -

• Storage Appliances Great for basic file service
• Easy to manage Plug in and it works
• Good performance Specialized just for I/O
• Reliable and available too
• Storage Appliances for the Grid Mismatch?
• Inflexible Few, specific protocols (e.g., NFS)
• Costly 10x the cost of PC a few disks
• Difficult to integrate Just one piece of the
  puzzle

4
A Solution NeST

• NeST A Storage Appliance for the Grid
• Flexible Multiple simultaneous protocols
• Virtual protocol layer
• Low-cost Use commodity machines
• Dynamic adaptation
• Grid-aware Integrate w/ higher-level systems
• Design specifically for the Grid

5
Outline

• Introduction
• General architecture
• Design goals
• Flexibility
• Low-cost
• Grid-aware features
• NeST in the Grid example
• Conclusions

6
NeST Protocol layer

• Virtualizes different protocols
• Mediates access to network

Physical network layer
Dispatcher
Storage Mgr
Transfer Mgr
Physical storage layer
7
NeST Dispatcher

• Mediates interaction between other components
• Gathers information, advertises

Physical network layer
Dispatcher
Storage Mgr
Transfer Mgr
Physical storage layer
8
NeST Storage manager

• Space management
• Access control
• Virtualizes physical storage

Physical network layer
Dispatcher
Storage Mgr
Transfer Mgr
Physical storage layer
9
NeST Transfer manager

• Implementsscheduling policies
• Chooses concurrency model

Physical network layer
Dispatcher
Storage Mgr
Transfer Mgr
Physical storage layer
10
Outline

• Introduction
• General architecture
• Design goals
• Flexibility
• Low-cost
• Grid-aware features
• NeST in the Grid example
• Conclusions

11
Flexibility Multiple protocols

• Problem How to support multiple protocols?
• One approach Just a Bunch of Servers (JBOS)
• Problems with JBOS
• Lack of control (scheduling)
• Painful administration
• No shared code
• Larger memory footprint

wu-ftpd
nfsd
httpd
JBOS Server
12
NeST Flexibility By Design

• NeST Integrate protocols and gain advantage
• Implementation like VFS
• Integration introduces new challenges
• Different protocols allow different auth models
• More expensive to add a new protocol
• Less fault isolation

13
NeST vs JBOS
Linux cluster - Dual PIII - 1 GB Ram - linux
2.2.19 Each protocol - 4 clients - 10 MB files
35
35
30
30
25
25
20
20
Server bandwidth (MB/s)
15
15
10
10
linux nfsd
Apache
wu-ftpd
5
5
0
0
NFS
Chirp
HTTP
Total
GridFTP

• For each protocol, NeST is comparable to JBOS
  server.

14
Exerting scheduling control

• Different scheduling policies
• FCFS
• Cache-aware USENIX 02
• Proportional share
• Proportional share scheduling
• Allows administrators to set protocol proportions
• e.g. favor NFS
• Very difficult in JBOS

15
Proportional share
35
30
25
Linux cluster - Dual PIII - 1 GB Ram - linux
2.2.19 Each protocol - 4 clients - 10 MB files
20
Server bandwidth (MB/s)
15
10
5
0
FCFS
1111
1211
1114
Scheduling configuration

• In most cases, achieves Jains metric of
  fairness gt 0.98 (1 is fair).

16
Outline

• Introduction
• General architecture
• Design goals
• Flexibility
• Low-cost
• Grid-aware features
• NeST in the Grid example
• Conclusions

17
Low-Cost New challenges

• Desire Run on arbitrary OS on arbitrary PC
• Software-only, user-level storage appliance
• Currently on Linux (release 0.9) and Solaris
  (beta)
• Problem Portable performance
• Performance under load is platform / workload
  dependent
• Threads or processes on some systems, events on
  others
• May also be workload dependent (e.g. whether in
  cache)
• NeST approach Dynamic adaptivity
• Simultaneously support multiple concurrency
  models
• Monitor performance using each model
• Bias towards better model over time

18
Adaptive Concurrency
Linux 10 MB files
Solaris 1K files
1.5
3
1
2
Ave time per request (ms)
Ave time per request (sec)
0.5
1
0
0
Events
Events
Threads
Threads
19
Outline

• Introduction
• General architecture
• Design goals
• Flexibility
• Low-cost
• Grid-aware features
• NeST in the Grid example
• Conclusions

20
Grid-Aware Mechanisms

• Basic functionality
• Users and groups Dynamic creation/deletion
• does not need administrative intervention
• Access control Generic AFS-style ACLs
• Advanced functionality
• QoS Preferential scheduling
• Advertises into global scheduling systems
• Flexible protocol and authentication mechanisms
• Self-cleaning storage guarantees Lots

21
Storage guarantees Lots

• Characteristics of Lots
• Capacity Total amount of data lot can store
• Duration Time for which data is guaranteed to
  exist
• Set of files Multiple files may co-exist within
  lot
• Self-cleaning
• Expired lots become best-effort lots
• Lot management
• Either default set created by administrator,
  ORuse resource management protocol to create
  before usage
• Implementation File system quotas
• Advantage Integrates cleanly with local access
  methods
• Disadvantage Performance hit for large writes

22
Outline

• Introduction
• General architecture
• Design goals
• Flexibility
• Low-cost
• Grid-aware features
• NeST in the Grid example
• Conclusions

23
NeST in the Grid
Global Execution Manager
Linux NeST
Solaris NeST
compute node
compute node
compute node
compute node
Home
Remote
24
NeST in the Grid
25
Conclusions

• NeST A storage appliance for the Grid
• Gain manageability
• Without sacrificing performance
• Design goals
• Flexibility Virtual protocol architecture
• Low-cost Adaptation mechanisms
• Grid-aware Space management
• Current status release 0.9 available
• Future work
• Hot deployable NeSTs, lot management extensions

26
Questions?
http//www.cs.wisc.edu/condor/nest
27
NeST Architecture
The Network
GridFTP
NFS
HTTP
Chirp
FTP
Common Protocol Layer
Dispatcher
Storage Manager
Transfer Manager
Physical Storage
28
Example client interaction

• Client sends mkdir
• Dispatcher
• asks protocol handler to parse
• asks storage mgr to mkdir
• checks permissions
• sends ack through handler
• Client puts file in new dir
• Dispatcher
• asks protocol handler to parse
• asks storage mgr for permission
• hands off to transfer mgr
• Transfer manager
• monitors transfer
• sends final ack

The Network
GFTP
NFS
HTTP
Chirp
FTP
Common Protocol Layer
Dispatcher
Storage Manager
Transfer Manager
Physical Storage