Scallaxrootd Introduction

1
Scalla/xrootd Introduction

• Andrew Hanushevsky, SLAC
• SLAC National Accelerator Laboratory
• Stanford University
• 6-April-09
• ATLAS Western Tier 2 Users Forum

2
Outline

• File servers
• NFS xrootd
• How xrootd manages file data
• Multiple file servers (i.e., clustering)
• Considerations and pitfalls
• Getting to xrootd hosted file data
• Available programs and interfaces

3
File Server Types
Application
Alternatively
Data Files
Linux
Linux
Client Machine
Server Machine
xrootd is nothing more than an application level
file server client using another protocol
4
Why Not Just Use NFS?

• NFS V2 V3 inadequate
• Scaling problems with large batch farms
• Unwieldy when more than one server needed
• NFS V4?
• Relatively new
• Multiple server support still being vetted
• Still has a single point of failure problems

5
NFS Multiple File Servers
cp /foo /tmp
open(/foo)
Which Server?
NFS cant naturally deal with this
problem. Typical ad hoc solutions are
cumbersome, restrictive and error prone!
6
xrootd Multiple File Servers I
Data Files
xrdcp root//R//foo /tmp
Application
xroot Client
Redirector
open(/foo)
Linux
Client Machine
/foo
Data Files
xroot Server
The xroot client does all of these steps
automatically without application
(user) intervention!
Linux
Server Machine B
7
File Discovery Considerations I

• The redirector does not have a catalog of files
• It always asks each server, and
• Caches the answers in memory for a while
• So, it wont ask again when asked about a past
  lookup
• Allows real-time configuration changes
• Clients never see the disruption
• Does have some side-effects
• The lookup takes less than a microsecond when
  files exist
• Much longer when a requested file does not exist!

8
xrootd Multiple File Servers II
Data Files
xrdcp root//R//foo /tmp
Application
xroot Client
Redirector
open(/foo)
Linux
Client Machine
/foo
5
Data Files
xroot Server
File deemed not to exist if there is no
response after 5 seconds!
Linux
Server Machine B
9
File Discovery Considerations II

• System optimized for file exists case!
• Penalty for going after missing files
• Arent new files, by definition, missing?
• Yes, but that involves writing data!
• The system is optimized for reading data
• So, creating a new file will suffer a 5 second
  delay
• Can minimize the delay by using the xprep command
• Primes the redirectors file memory cache ahead
  of time
• Can files appear to be missing any other way?

10
Missing File vs. Missing Server

• In xroot files exist to the extent servers exist
• The redirector cushions this effect for 10
  minutes
• Afterwards, the redirector cannot tell the
  difference
• This allows partially dead server clusters to
  continue
• Jobs hunting for missing files will eventually
  die
• But jobs cannot rely on files actually being
  missing
• xroot cannot provide a definitive answer to " s
  Øx
• This requires manual safety for file creation

11
Safe File Creation

• Avoiding the basic problem....
• Todays new file may be on yesterdays dead
  server
• Generally, do not re-use output file names
• Otherwise, serialize file creation
• Use temporary file names when creating new files
• E.g., path/....root.temp
• Remove temporary to clean-up any previous
  failures
• E.g., -f xrdcp option or truncate option on open
• Upon success, rename the temporary to its
  permanent name

12
Getting to xrootd hosted data

• Use the root framework
• Automatically, when files named root//....
• Manually, use TXNetFile() object
• Note identical TFile() object will not work with
  xrootd!
• xrdcp
• The copy command
• xprep
• The redirector seeder command
• Via fuse on atlint01.slac.stanford.edu
• POSIX preload library

13
Copying xrootd hosted data

• xrdcp options source dest
• Copies data to/from xrootd servers
• Some handy options
• -f erase dest before copying source
• -s stealth mode (i.e., produce no status
  messages)
• -S n use n parallel streams (use only across WAN)

14
Preparing xrootd hosted data

• xprep options hostport path ...
• Prepares xrootd access via redirector hostport
• Minimizes wait time if you are creating many
  files
• Some handy options
• -w file will be created or written
• -f fn file fn holds a list of paths, one per line

15
Interactive xrootd hosted data

• Atlas xroot redirector mounted as a file system
• /xrootd on atlint01.slac.stanford.edu
• Use this for typical operations
• dq2-get
• dq2-put
• dq2-ls
• rm

16
For Everything Else

• POSIX preload library (libXrdPosixPreload.so)
• Works with any POSIX I/O compliant program
• Provides direct access to xrootd hosted data
• Does not need any changes to the application
• Just run the binary as is
• Talk to Wei or Andy if you want to use it

17
Conclusion

• We hope that this is an effective environment
• Production
• Analysis
• But, we need your feedback
• What is unclear
• What is missing
• What is not working
• What can work even better

18
Future Directions

• More simplicity!
• Integrating the cnsd into cmsd
• Reduces configuration issues
• Pre-linking the extended open file system (ofs)
• Less configuration options
• Tutorial-like guides!
• Apparent need as we deploy at smaller sites

19
Acknowledgements

• Software Contributors
• Alice Derek Feichtinger
• CERN Fabrizio Furano , Andreas Peters
• Fermi Tony Johnson (Java)
• Root Gerri Ganis, Beterand Bellenet, Fons
  Rademakers
• STAR/BNL Pavel Jackl
• SLAC Jacek Becla, Tofigh Azemoon, Wilko Kroeger
• LBNL Alex Sim, Junmin Gu, Vijaya Natarajan
  (BeStMan team)
• Operational Collaborators
• BNL, FZK, IN2P3, RAL, UVIC, UTA
• Partial Funding
• US Department of Energy
• Contract DE-AC02-76SF00515 with Stanford
  University