RAMS A RDMAenabled IO Cache Architecture for Clustered network Servers

1
RAMS--A RDMA-enabled I/O Cache Architecture for
Clustered networkServers

• Peng Gu and Jun Wang
• Univeristy of Nebraska-Lincoln
• Lincoln NE 68588-0115

2
Introduction

• What is RAMS? RDMA-based cAching Middleware for
  cluster-based Servers
• What can RAMS do for us?To improve the
  intra-communication performance between cluster
  nodes

3
Motivation

• The drawback of traditional RPC transfer.
• Multi-copy of the data (space consuming)
• Data was copied many times (time consuming)
• High Interrupt Density (CPU demanding)

4
Motivation (continued)

• The advantage of RDMA transfer.
• Only one copy of data (save space)
• Reduce the data coping time (save time)
• RDMA transfer was controlled by the NIC instead
  of CPU (offload CPU)
• User-level communication (easy to customize)

5
Current Applications

• Direct Access File System (Specification 1.0
  released in 2001)
• RDMA over IP (Cisco System and Sandburst Corp.)
• User-level communication in cluster based servers
  (Rutgers University)

6
Limitation

• Only use server initiated RDMA write
• Do not used client initiated RDMA read in spite
  of potential performance gain.

7
Reasons of the Limitation

• Current commercially-available RDMA capable NICs
  lack of the capabilities necessary to enable the
  use of client initiated RDMA read. These
  capabilities includes
• Wire/Unwire memory paged on demand
• Remote access exception notification mechanism
  not implemented.
• No integration between NICs and mainstream OS
  necessary to deal with virtual memory locking.
• No support for exception handling on behalf of
  NIC.
• Weak safety if stray memory reference used.

8
Current solution

• Optimistic Direct Access File System
• Make use of the client initiated RDMA read by
  following approach
• Sending the data information together with the
  virtual memory address.
• Using client initiated RDMA read when possible.
• Do statistic on the RDMA stale hit rate, if the
  penalty of handling RDMA read failure, then
  switch back to normal RPC transfer.

9
Our solution

• RDMA-based caching middleware for cluster-based
  servers (RAMS).
• Our goal is to reduce the RDMA read failure rate
  in a cluster.
• Our method Introducing Caching Middleware for
  RDMA operation.

10
RAMS Architecture
11
System behavior

• The three ways of serving a request
• Local
• Local File system
• Remote
• Normal RPC
• RDMA transfer

12
Working process
13
System Components

• The outgoing directory
• The incoming directory
• The outgoing data cache
• The incoming data cache

14
The Key problem

• Updating the directories
• Why?
• When?
• How?

15
Lazy update

• Advantages
• Easy to implement
• Do not occupy extra network bandwidth
• Disadvantages
• No control on the risk of RDMA read failure rate,
  which will potentially introduce more exception
  handling time.

16
Aggressive update

• Advantages
• Easy to implement
• No risk of RDMA read failure rate, which is a
  ideal environment for RDMA read operation.
• Disadvantages
• The updating process itself will consume a lot of
  network bandwidth because of its high frequency.

17
Threshold update

• Advantages
• Combine the advantages of the two update policies
  mentioned before.
• The process itself do not consume too much
  network bandwidth.
• Provide a reasonable RDMA read failure rate.
• Easy to adjust for all kind of applications by
  revise the threshold value.
• Disadvantages
• A little bit more complicated to implement.

18
Statistic strategies

• Source side statistic
• Destination side statistic
• Both side statistic

19
Remaining work

• Collect data for all the update policies and the
  all the statistic strategies.
• Design a good cache replacement algorithm
  especially for our RDMA cache directories and
  cache data.

20
Thanks

• Q A