Mass Storage Management improvised report for LHC Computing Review Software Panel

1
Mass Storage Managementimprovised report for
LHC Computing Review Software Panel

• Manuel Delfino
• CERN IT Division
• Disclaimer I wrote this mostly from memory and
  had no time to check with experts. I am sure
  general ideas are right, some details may be
  slightly misrepresented

2
What is the problem ?

• Copying a file between disk?tape is easy
• Except if the tape breaks
• Except if the drive is being used by someone
• Except if you have forgotten the tape number
• Except if you have forgotten the file number
• (maybe) except if you need high performance
• (maybe) except if hunting needle in haystack
• Another major issue is caring for the media
• Tapes should be wound at least once a year
• Migrate between media to follow technology
• Replicate media between regional centers

3
What is the solution ?

• Managed Storage
• (of a kind we can performance tune)
• If fully deployed, it is a bit more than
  Hierarchical Storage Management
• Inventory
• Error statistics and failure prediction
• Automatic migration and replication
• One question How much functionality do we need
  vs. how much we can easily afford ?

4
Why multiple solutions ?

• History dating back many years
• Each major lab has 3 FTEs on this
• Most of their effort goes into running the
  production systems
• Enough left to do something
• Probably not enough to productize
• Historically software very tied to the hardware
  (not so true anymore)
• Needs pushed emphasis on performance rather than
  functionality (n.s.t.a)
• Lack of coordination of next step

5
Eurostore (I and II)

• Beginning of HERA era, a glimer of hopeLachman
  OSM commercial HSM
• Alas, it went bankrupt
• Released the code but
• Unmaintainable
• Missing functionality
• DESY leads EU project to continue OSM
• CERN involved in testing
• Eurostore did not manage to deliver a production
  quality system
• Hence Eurostore II
• CERNs commitment is in testing (again)
• Main test will be ALICE Data Challenge 2001

6
HPSS from IBM Government Sys

• Another possible commercial solution, but
• Works best on IBM computers/tapes
• Dominated by supercomputer crowd
• Is it really a mainstream IBM product ?
• CERN involvement
• Evaluate product
• Port some components to Digital Unix (project
  funded by Digital/Compaq)
• Real tests with production
• NA45 data
• Current backend support for hsm command
• Objectivity (interface developed orig. by SLAC)

7
CASTOR

• Modernize and cleanup CERN Stage
• Move from Tape oriented to Managed Storage
  concepts
• Complete backward compatibility
• Phase I functionality now being deployed is
  enough for LEP, fixed target
• Not a black box anymore, but a set of well
  defined components that interact
• Preparation for ALICE Data Challenge 2000
  achieved gt100 MB/s sustained
• Surprise ! See performance limitations due to
  memory-bandwidth in the machines

8
Others

• interSTAGE Attempt by CERN and FNAL
• Introduce requirement of WAN transfers
• Try to entice all labs to work together on next
  step
• Mostly fizzled out
• ENSTORE FNAL based on OSM design
• No name Vestige of interSTAGE, agreement to
  produce a set of APIs for HEP mass storage
  management

9
Which way in future ?

• Lots of experience being gained
• ALICE data challenges
• COMPASS and HARP
• RUN II, BaBar, HERA, etc
• Relatively relaxed about it because
• The big problem is disk?network?program
• Nevertheless, future plans should include
• Demonstrate simple sure shot for CDR
• Decide needed functionality and have one system
  used by all LHC experiments if poss