xrootd

1
xrootd

• Andrew Hanushevsky
• Stanford Linear Accelerator Center
• 30-May-03

2
Goals

• High Performance File-Based Access
• Scalable, extensible, usable
• Fault tolerance
• Server failures handled in a natural way
• Servers may be dynamically added and removed
• Flexible Security
• Allowing use of almost any protocol
• Rootd Compatibility

3
Achieving High Performance

• Scalable request/response protocol
• Multi-threaded multi-process architecture
• Architecture sensitive polling
• MRU scheduling
• Sticky sockets
• Adaptive reconfiguration
• Versatile sfs layer (based on proven oofs)

4
Scalable Protocol I

• Connection multiplexing
• One connection per client/host
• Multiple logically independent streams
• Request redirection supported
• Similar to http redirection
• Supports dynamic load balancing and fail-over
• Uses an intentional request header
• Can better optimize request processing

5
Scalable Protocol II

• Asynchronous mode allowed
• Multiple processing-order-independent requests
• Optional application-directed pre-read
• I/O segmenting
• Able to naturally deal with very large transfers
• Better use of server resources
• Request deferral
• Client waits for resources without using server
  resources

6
Scalable Protocol III

• Unsolicited Reverse Request Mode
• Allows server to manage client for recovery
• Asynchronous redirect, deferral, and messages
• Protocol may be compatibly extended
• Mechanism to send opaque information
• Accommodate things that were forgotten
• Messaging interface
• Cache group
• Request priority
• And so on.

7
MT/MP Architecture

• Normally one multi-threaded server per host
• Should be able to utilize available resources
• Easy to administer
• Optionally, multiple servers per host
• Fully utilize large machines

8
Architecture Sensitive Polling

• All POSIX systems support poll()
• Used by default
• Not always an efficient I/O interrupt mechanism
• Alternate polling mechanisms allowed
• /dev/poll
• Available on Solaris and patched RH Linux
• Up to an order of magnitude reduction in CPU
• Essential to reduce latency

9
MRU Scheduling

• Connections processed in most recently used order
• Gives priority to active connections
• Reduces polling overhead
• Essentially a fair scheduling algorithm
• Starvation cannot occur
• Longer running tasks tend to get started first
• Assuming all other things being equal

10
Sticky Sockets

• Connection temporarily binds to a thread
• Avoids polling and scheduling overhead
• Significantly reduces latency
• Connection automatically unbinds
• Client is not sufficiently active
• Number of other requests approaches available
  threads

11
Adaptive Reconfiguration

• Server dynamically adjusts configuration
• Number of threads
• Kept proportionate to number of active requests
• Pre-allocated buffers
• Sizes track actual usage profile
• Recomputed periodically
• Pre-allocated objects
• Number tracks recent needs
• High latency connections rescheduled

12
Versatile sfs Layer I

• Integrates multiple performance features
• Dynamic load balancing
• Client redirected to best server of the moment
• File descriptor partitioning
• Reduces socket polling overhead
• File system interface reuse
• Prevents open file proliferation and attendant
  overhead
• Same file opened in same mode shared by multiple
  clients
• File system interface timeout
• Reduces overhead caused by idle opened files

13
Dynamic Load Balancing
xrootd
Dynamic Selection
xrootd
client
mss
xrootd
14
DLB Implementation
I do
subscribe
who has the file?
open again
open
Client
wait
try hostport
15
Versatile sfs Layer II

• Dynamic disk cache integration
• Allows unlimited file system size
• Provides superior internal load balancing
• Mass Storage Integration
• HPSS, Castor, Enstore, etc
• RFIO Integration
• Scalable authorization
• From file sub-trees to single files

16
Cache File System
Index Area Optional data cache Default data
area
/databases/mydbfile
symlink
Naming convention allows for audit and index
recovery
Multiple Independent Filesystems
/cache1/databasesmydbfile
Data Area Any number Any Size Chosen based on
free space in LRU order
/cache2
/cache3
17
Fault Tolerance I

• Servers may come and go
• Uses load balancing to effect recovery
• New servers can be added at any time
• Servers may be brought down for maintenance
• Files can be moved around in real-time
• Client simply adjust to the new configuration
• XTNetFile object handles recovery protocol

18
Fault Tolerance II

• Whenever client looses r/o connection
• Back to distinguished xrootd(s) for reselection
• Whenever client looses r/w connection
• Limited wait/retry loop on the same server
• We will be working to improve this next year!
• All handled in the XTNetFile class
• Disruptions merely delay the client

19
Flexible Security

• Negotiated Security Protocol
• Allows client/server to agree on protocol
• E.g., Kerberos, GSI, AFS Kerberos, etc.
• Can be easily extended
• Multi-protocol authentication support

20
Security Architecture
xrootd
client
login
Client-Specific Security Configuration
authenticate
Multiple handshakes allowed during authentication
phase (required by some PKI protocols)
Protocol Selection
Self Configuration
Security Shared Library
Security Shared Library
libooseccl.so
libooseccl.so
21
Heterogeneous Security Support
xrootd 1
Security Contexts
Servers have one or more protocol objects
Server protocol objects created at server
initialization time Client selects which
protocol to use when security context created
Protocol object created based on configuration
returned by xrootd One security context object
per physical xrootd connection Protocol
objects may be shared by one or more contexts
Each pass through a security context object
may generate credentials to be passed to xrootd
xrootd 2
client
xrootd 3
protocols
Protocol Objects
22
Simple Effective Interface

• For each login that requires authentication
• XrdSecCreateSecurityContext(ipaddr, config)
• Returns security protocol object
• XrdSecClientSecurity
• Based on server ipaddr and server-supplied config
• XrdSecClientSecuritygetCredentials()
• Returns credentials to be sent to the server
• Done via authenticate request and possible
  authmore response
• Based on well tested and documented oofs security

23
Optional Scalable Authorization
libooacc.so
xrootd
Authentication
Authorization Shared Library
Authorization
XrdSfs
Replaceable Database Interface
database
u abh rw /slac/rootfiles/usr/abh
r /cern/rootfiles
24
Security Summary

• Multi-protocol Authentication
• Supports distributed heterogeneous environments
• Scalable Authorization
• Open-ended capability based model
• Integrated Auditing
• To keep the security hard hats happy
• Well defined, proven interfaces
• Trivially replaceable for a plug play
  architecture

25
rootd Compatibility

• Bilateral compatibility
• XTNetfile reverts to TNetFile for rootd servers
• XRootd reverts to rootd protocol for TNetFile
• Allows for transparent introduction
• Can run mixed mode
• Binary is multi-environment compatible

26
Compatibility Modes
Client-Side Compatibility
Application
xrootd
XTNetFile
rootd
TNetFile
Server-Side Compatibility
Application
xrootd rootd compability
rootd
TNetFile
27
xrootd Architecture
application
Protocol Manager
xrd
Protocol Layer
xroot
Authentication
xrootd
Filesystem Logical Layer
sfs
Authorization
Filesystem Physical Layer
oss
Filesystem Implementation
mss
_fs
28
xrootd Internals
network manager
config manager
socket manager
thread manager
buffer manager
protocol manager
xroot
rootd
security
Dynamically loaded (can also be static)
oofs
logging
29
Conclusion

• xrootd provides high performance file access
• Improves over afs, ams, nfs, etc.
• Unique performance, usability, scalability,
  security, compatibility, and recoverability
  characteristics
• xrootd can provide a firm server foundation for
  native file system implementations
• E.g. alienfs, gridfs, slashgrid, etc
• For now, aim is to support BaBar