Distributed File System: Data Storage for Networks Large and Small

1
Distributed File System Data Storage for
Networks Large and Small

• Pei Cao
• Cisco Systems, Inc.

2
Review DFS Design Considerations

1. Name space construction
2. AAA
3. Operator batching
4. Client caching
5. Data consistency
6. Locking

3
Summing it Up CIFS as an Example

• Network transport in CIFS
• Use SMB (Server Message block) messages over a
  reliable connection-oriented transport
• TCP
• NetBIOS over TCP
• Use persistent connections called sessions
• If a session is broken, client does the recovery

4
Design Choices in CIFS

• Name space construction
• per-client linkage, multiple methods for server
  resolution
• file//fs.xyz.com/users/alice/stuff.doc
• \\cifsserver\users\alice\stuff.doc
• E\stuff.doc
• CIFS also offers redirection method
• A share can be replicated in multiple servers or
  moved
• Client open ? server reply STATUS_DFS_PATH_NOT_C
  OVERED ? client issues TRANS2_DFS_GET_REFERRAL
  ? server reply with new server

5
Design Choices in CIFS

• AAA Kerberos
• Older systems use NTLM
• Operator batching supported
• These methods have AndX variations
  TREE_CONNECT, OPEN, CREATE, READ, WRITE, LOCK
• Server implicitly takes results of preceding
  operations as input for subsequent operations
• First command that encounters an error stops all
  subsequent processing in the batch

6
Design Choices in CIFS

• Client caching
• Cache both file data and file metadata,
  write-back cache, can read-ahead
• Offers strong cache consistency using an
  invalidation-based approach
• Data access consistency
• Oplocks similar to tokens in AFS v3
• level II oplock read-only data locks
• exclusive oplock exclusive read/write data
  lock
• batch oplock exclusive read/write open lock
  and data lock and metadata lock
• Transition among the oplocks
• Observation can have a hierarchy of lock
  managers

7
Design Choices in CIFS

• File and data record locking
• Offer shared (read-only) and exclusive
  (read/write) locks
• Part of the file system Mandatory
• Can lock either a whole file or byte-range in the
  file
• Lock request can specify a timeout for waiting
• Enables atomic writes with the ANDX batching
  with Writes
• Lock/write/unlock as a batched command sequence
• Additional capability directory change
  notification

8
DFS for Mobile Networks

• What properties of DFS are desirable
• Handle frequent connection and disconnection
• Enable clients to operate in disconnected state
  for an extended period of time
• Ways to resolve/merge conflicts

9
Design Issues for DFS in Mobile Networks

• What should be kept in client cache?
• How to update the client cache copies with
  changes made on the server?
• How to upload changes made by the client to the
  server?
• How to resolve conflicts when more than one
  clients change a file during disconnected state?

10
Example System Coda

• Client cache content
• User can specify which directories should always
  be cached on the client
• Also cache recently used files
• Cache replacement walk over the cached items
  every 10 min to reevaluate their priorities
• Updates from server to client
• The server keeps a log of callbacks that couldnt
  be delivered and deliver them upon client
  connection

11
Coda File System

• Upload the changes from client to server
• The client has to keep a replay log
• Contents of the replay log
• Ways to reduce the replay log size
• Handling conflicts
• Detecting conflicts
• Resolving conflicts

12
Performance Issues in File Servers

• Components of server load
• Network protocol handling
• File system implementation
• Disk accesses
• Read operations
• Metadata
• Data
• Write operations
• Metadata
• Data
• Workload characterization

13
DFS for High-Speed Networks DAFS

• Proposal from Network Appliance and companies
• Goal eliminate memory copies and protocol
  processing
• Standard implementation network buffers ? file
  system buffer cache ? user-level application
  buffers
• Designed to take advantage of RDMA (Remote DMA)
  network protocols
• Network transport provides direct memory ? memory
  transfer
• Protocol processing is provided in hardware
• Suitable for high-bandwidth, low-error-rate,
  low-latency network

14
DAFS Protocol

• Data read from the client
• RDMA request from the server to copy file data
  directly into application buffer
• Data write from the client
• RDMA request from the server to copy application
  buffer into server memory
• Implementation
• as a library linked to user application interface
  with RDMA network library directly
• Eliminate two data copies
• as a new file system implementation in the kernel
• Eliminate one data copy
• Performance advantage
• Example 90 usec/op in NFS vs. 25 usec/op in DAFS

15
DAFS Features

• Session-based
• Offer authentication of client machines
• Flow control by server
• Stateful lock implementation with leases
• Offers atomic writes
• Offers operator batching

16
Clustered File Servers

• Goal scalability in file service
• Build a high-performance file service using a
  collection of cheap file servers
• Methods for Partitioning the Workload
• Each server can support one subtree
• Advantages
• Disadvantages
• Each server can support a group of clients
• Advantages
• Disadvantages
• Client requests are sent to server in round-robin
  or load-balanced fashion
• Advantages
• Disadvantages

17
Non-Subtree-Partition Clustered File Servers

• Design issues
• On which disks should the data be stored?
• Management of memory cache in file servers
• Data consistency management
• Metadata operation consistency
• Data operation consistency
• Server failure management
• Single server failure fault tolerance
• Disk failure fault tolerance

18
Mapping Between Disks and Servers

• Direct-attached disks
• Network-attached disks
• Fiber-channel attached disks
• iSCSI attached disks
• Managing the network-attached disks volume
  manager

19
Functionalities of a Volume Manager

• Group multiple disk partitions into a logical
  disk volume
• Volume can expand or shrink in size without
  affecting existing data
• Volume can be RAID-0/1/5, tolerating disk
  failures
• Volume can offer snapshot functionalities for
  easy backup
• Volumes are self-evident

20
Implementations of Volume Manager

• In-kernel implementation
• Example Linux volume manager, Veritas volume
  manager, etc.
• Disk server implementation
• Example EMC storage systems

21
Serverless File Systems

• Serverless file systems in WAN
• Motivation peer-to-peer storage never lose the
  file
• Serverless file system in LAN
• Motivation client powerful enough to be like
  servers use all clients memory to cache file
  data