UK Atlas Petabyte DataStore

1
UK Atlas Petabyte DataStore Data Curation

• MSSN Remote Access Automation Workshop
• March 29th 2005
• David Corney d.corney_at_rl.ac.uk
• Group Leader, Petabyte Storage Group, e-Science,
• CCLRC Rutherford Appleton Laboratory

2

• Context Overview of CCLRC
• Atlas Petabyte Data Store
• System and Service
• Grid technologies incl Storage Resource Broker
  (SRB)
• Usage User Communities
• Costs and charging
• Looking ahead
• Data Curation The Digital Curation Centre

3
The Context Overview of CCLRC
4

• Rutherford Appleton Laboratory
• Daresbury Laboratory
• Chilbolton Observatory

The Council for the Central Laboratory of the
Research Councils
..enabling technology for science and discovery...
5
CCLRC Sites
Daresbury Laboratory, Cheshire
Rutherford Appleton Laboratory, Oxfordshire
Chilbolton Observatory, Hampshire
6
(No Transcript)
7
National responsibilities

• The focus for large-scale facilities
• neutron scattering
• synchrotron radiation
• lasers

• Technological resource centres
• e-science
• accelerator science and technology
• instrumentation
• space science technology
• high performance computing

8
CCLRC in the UK scene
9
Sources of income

10
Research communities
Around 7000 researchers world-wide use the
facilities and services each year.
11
Large scale facilities
ISIS
The worlds most powerful pulsed neutron and muon
source. Provides beams of neutrons and muons to
probe the structure and dynamics of condensed
matter on a microscopic scale.

• Scientific highlight
• Structure determination of C60 and High Tc
  Superconductors using the worlds leading pulsed
  neutron and muon source, ISIS.

12
Large scale facilities
Diamond
A new world-class synchrotron light source,
diamond, will be operational at the Rutherford
Appleton Laboratory in 2007. Diamond will enable
scientists and engineers to probe deep into the
basic structure of matter and materials
13
The Atlas Petabyte Data Store

• System and Service
• Grid technologies incl. Storage Resource Broker
  (SRB)
• Usage User Communities
• Costs and charging
• Looking ahead

14
System and Service
STK 9310
15
(No Transcript)
16
Atlas Datastore Architecture
Robot Server (buxton)
Catalogue Server (brian)
Copy C
Copy A
Copy B
flfsys tape commands (sysreq)
CSI
recycling (libflf)
read
ACSLS
ACSLS API
flfqryoff (copy of flfsys code)
Backup catalogue
flfsys import/export commands (sysreq)
flfdoback (libflf)
read
Tape Robot
control info (mount/ dismount)
read
stats
flfdoexp (libflf)
LMU
cellmgr
backend
Pathtape Server (rusty)
IBM tape drive
STK tape drive
flfsys (libflf)
pathtape
data
short name (sysreq)
data
servesys
flfsys farm commands (sysreq)
frontend
long name (sysreq)
flfsys admin commands (sysreq)
flfstk
SSI
?
flfsys user commands (sysreq)
(sysreq)
(sysreq)
flfaio
flfscan
cache disk
flfaio
Farm Server
datastore (script)
user program
flfaio
vtp
tapeserv
data transfer (libvtp)
I/E Server (dylan)
vtp
tape
28 Feb 03 - 2 B Strong
importexport
libvtp
User Node
17
Test system
Production system
dylan AIX Import/export
8 x 9940 tape drives
STK 9310
buxton SunOS ACSLS
Tape devices
4 drives to each switch
basil AIX test dataserver
Brocade FC switches
SRB pathtape commands
ADS_switch_1
ADS_Switch_2
ADS0CNTR Redhat counter
ADS0PT01 Redhat pathtape
ADS0SB01 Redhat SRB interface
cache
User pathtape commands
Logging
cache
mchenry1 AIX Test flfsys
ermintrude AIX dataserver
florence AIX dataserver
zebedee AIX dataserver
dougal AIX dataserver
brian AIX flfsys
admin commands create query
catalogue
array3
array4
array1
array2
catalogue
All sysreq, vtp and ACSLS connections to dougal
also apply tothe other dataserver machines, but
are left out for clarity
User
SRB Inq S commands MySRB
ADS tape
ADS sysreq
Thursday, 04 November 2004
18
Users

• Need to know their data is
• Safe (esp in case of disaster)
• Easily accessible (from anywhere)
• Always there 24/7 - this year, the next and the
  next..
• Dont need to know how
• Need to be freed from IT concerns to focus on
  their science

19
Strategy

• De-couple user and application from storage
  media.
• Upgrades and media migration occur behind the
  scenes
• High resilience - very few Single Point Failures
• High reliability high, availability (24/7)
• Constant environmental monitoring linked to
  alarm/call out
• Lifetime data integrity checks hardware and
  software
• Fire safe and off-site backups Tested disaster
  recovery procedures media migration, recycling
• Easy to exploit (endless) new technology
• Technology watch to monitor future technology path

20
Interfaces

• Light weight interfaces
• Client server configuration
• tape command for many platforms
• Virtual Tape Protocol (VTP)
• Fortran and C callable library
• Heavy weight grid interfaces
• SRB interface Storage Resource Broker
• SE interface developed for EDG/GRIDPP/GRIDPP2

21
Grid technologies and StorageResource Broker
(SRB)

• Key component of UK Tier1 for LHC
• Grid access is essential
• SRM - Storage Resource Manager the protocol
  defined from collaboration between Lawrence
  Berkeley, FermiLab, Jefferson Lab, CERN,
• SE Storage Element the interface - a
  component of European Data Grid (EDG) middleware
  developed by CCLRCs e-Science PPD departments
• Provides a uniform Grid Interface to diverse Mass
  Storage Systems (MSS)

22
Storage Resource Broker (SRB)

• Storage Resource Broker (SRB) is a software
  product developed by the San Diego Supercomputing
  Centre (SDSC).
• Allows users to access files and database objects
  across a distributed environment.
• Actual physical location and way the data is
  stored is abstracted from the user
• Allows the user to add user defined metadata
  describing the scientific content of the
  information

23
How SRB Works

• 4 major components
• The Metadata Catalogue (MCAT)
• The MCAT-EnabledSRB Server
• The SRB Storage Server
• The SRB Client

24
The MCAT Database

• The MCAT database is a metadata repository that
  provides a mechanism for storing information used
  by the SRB system.
• Includes both
• Internal system data required for running the
  system
• Application (user) metadata regarding data sets
  being brokered by SRB.

25
The MCAT Server

• At least one SRB Server must be installed on the
  node that can access the MCAT database. This is
  known as the MCAT-Enabled SRB Server.
• MCAT SRB Server works directly against the MCAT
  database to provide SRB Services
• All other SRB Servers interact through the MCAT
  Server

26
The SRB Server

• The SRB Server is a middleware application that
  accepts requests from clients and
  obtains/queries/manages the necessary data sets.
• It queries the MCAT SRB Server to gather
  information on datasets and supplies this back to
  the SRB client.

27
SRB Client Tools

• Provide a user interface to send requests to the
  SRB server.
• 4 main interfaces
• Command line (S-Commands)
• MS Windows (InQ)
• Web based (MySRB).
• Java (JARGON)
• Web Services (MATRIX)

28
SRB Example CMS

• Largest project using CCLRC SRB services to date
  is the CERN CMS experiment.
• SRB chosen for Pre-Challenge Production,
  producing data for Data Challenge 2003/2004
  (DC03/DC04)
• Need to prove data can be transferred, replicated
  and stored at LHC rates
• DC04 provided key input to SRB Version 3.2

29
MCAT CCLRC Database Service

• MCAT requires professionally run database
• Two IBM x440 clusters, one based at Daresbury
  Laboratory and the other at Rutherford Appleton
  Laboratory.
• The clusters connect to their own 1TB RAID 5
  storage arrays via a independent fibre channel
  Storage Area Networks (SAN).
• Run Oracle Real Application Clusters software on
  Redhat Advanced Server for high
  availability/scalability RDBMS
• CMS MCAT hosted by 2 nodes
• Can load balance

30
ADS Driver for SRB

• Implemented Storage System Driver
• Implement (most) of the 16 standard calls that
  implement the driver layer such as copy, move,
  delete and create.
• Some functions have no equivalent in ADS

31
SRB-ADS architecture
SRB ADS Server
Port 5600
SRB-ISIS server instance
Port 5601
SRB-BADC server instance
Port 5602
SRB-CCLRC server instance
32
User Communities

• Particle Physics Community (LHC CMS, Atlas,
  LHcb,.)
• ISIS, British Atmospheric Data Centre
• EISCAT (Radar research)
• National Earth Observation Data Centre
• World Data Centre, BITD
• Central Laser Facility
• Diamond Light Source
• National Crystallography Service, Southampton
  University
• WASP, VIRGO Consortium
• BBSRC (BITS)
• Arts and Humanities Data Service
• Integrative Biology

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
Costs and charging

• H/w Costs
• Total 1m every 4-5 years, equiv to 250K/yr
• H/W upgrades are costly installation,
  configuration, test and associated data
  migration - many months
• Example component costs
• Robot (6000 slots) 300K
• Media 420K (_at_ 70 per unit)
• Disk 1.5K/TB? 50K for 75TB commodity?
• Tape drives 20K each. (est. T1s and T2s) Total
  200K for 10
• Data Servers
• Linux 3K each. Total 30K for 10
• AIX 14K each. Total 140K for 10
• Network/switches 50K
• Numbers are the Key to flexible performance
  esp. data servers and tape drives.
• S/w Costs Currently limited to staff
  development costs
• Staff 2.5 FTE system manager system developer
  0.5 operations staff

38
Service Charging

• CCLRC facilities and users
• Previously internal charging, now moving to
  nominal charging supported by OST funding to
  e-science
• Particle Physics users
• Provide financial resources of 120K/year for
  h/w upgrades, media and staff
• External users
• Nominal 1K/TB/yr Media (60/200GB)
• Reviewing modelling to take account of
• Total volume stored
• Access patterns

39
Looking ahead
40
Expected future demand
41
Capacity performance - Hardware

• Hardware
• Defines both performance and capacity
• Changing fast but well understood (buy as late
  as possible)
• Tied into technology futures of manufacturers and
  HEP community
• Currently hardware is effectively infinitely
  scalable
• Future estimated storage capacity bandwidth for
  a 6000 slot robot

42
Capacity performance Software requirements
for future scaleable solutions

• Low cost solution
• GRID Interfaces current (SRB/SE/SRM) and
  future??
• Allow input to development direction, esp. wrt
  interfaces
• Scaleable to 10s of Pb and millions or billions
  of files
• Performance Must allow a flexible simple
  response to indeterminate and changing patterns
  of access
• Access to an existing pool of user and sysadmin
  experience
• Strong future development path
• Allow future chargeable service to be offered

43
Storage capacity performanceCASTOR a
long(er) term solution?

• Low cost solution
• free to collaborators Linux compliant
• GRID Interfaces current (SRB/SE/SRM) and
  future??
• Well placed to provide these as our contribution.
  Much negotiation required.
• Allow input to development direction, esp. wrt
  interfaces
• Developed by HEPIX collaboration - 70 of the ADS
  user community
• Scaleable to 10s of Pb and millions or billions
  of files
• Is or soon will be. If CASTOR fails CERN fails
• Performance Must allow a flexible simple
  response to indeterminate and changing patterns
  of access
• Design allows scaleable use of multiple low cost
  data servers
• Access to an existing pool of user and sysadmin
  experience
• Already exists in CERN and developing in other
  Tier 1 sites
• Strong future development path
• Next release due out shortly. Tied to success and
  development of LHC and HEPIX community.
• Allow future chargeable service to be offered
• TBD. CASTOR currently runs under GPL Gnu Public
  Licence. Details under investigation

44
Digital Curation Centre

• Joint collaboration between CCLRC, UKOLN, and
  Edinburgh and Glasgow Universities.
• Provide advice, support, research and Development
  into aspects of Digital Curation for the UK HE
  community
• Funded jointly by JISC and EPSRC - 1m/year for
  three years initially. Feb 2004- 2007
• Establish collaboration with industrial partners

45
What is Digital Curation?

• the actions needed to maintain digital research
  data and other digital materials over their
  entire life-cycle and over time for current and
  future generations of users
• processes of digital archiving and preservation
• Processes for good data creation and management
• capacity to add value to data to generate new
  sources of information and knowledge.

46
Why is it needed?

• problem of the moment
• fragility of digital information recognised
• data curation data deluge in e-science/research
• longevity of digital heritage research
  investment
• re-examining Communication in ICT
• Internet and GRID communication across
  space with utmost accuracy
• Digital Curation communication across time,
  with utmost accuracy
• ensure Content travels despite turbulence of IT
• agree strategies methods for digital
  preservation

47
(How?) Organisation to Engage Collaborate
curation organisations eg DPC
communities of practice users
UKOLN
Collaborative Associates Network of
Data Organisations
U of Edinburgh
U of Edinburgh
U of Glasgow
research collaborators
CCLRC
testbeds tools
Industry
standards bodies
48
Organisation to Engage Collaborate
curation organisations eg DPC
communities of practice users
community support outreach
service definition delivery
Collaborative Associates Network of
Data Organisations
management admin support
research collaborators
research
development co-ordination
testbeds tools
Industry
standards bodies
49
Who?

• Phase One leadership over first eight months of
  funding
• Community Support Outreach
• Led by Dr Liz Lyon (UKOLN, University of Bath)
• Service Definition Delivery
• Led by Professor Seamus Ross (HATII ERPANET,
  University of Glasgow)
• Development
• Led by Dr David Giaretta (Astronomical Software
  Services, CCLRC)
• Research
• Led by Professor Peter Buneman (Informatics,
  University of Edinburgh)
• Management Co-ordination
• Director Chris Rusbridge
• Peter Burnhill had been Phase One Director

50
Outreach

• User interviews and focus groups
• Internet Journal
• Web presence (http//www.dcc.ac.uk) and Portal
• DPC membership and collaboration
• Associates Network
• DCC Conference (Sept 29-30)
• PV2005 Conference (Nov 21-23)

51
Services

• Advisory service and Help desk
• Site visits and case studies
• Curation Manual and Briefings
• Tools and testbeds
• Standards watch
• Certification
• Training

52
Development

• OAIS fundamentals
• Registries/Repositories for Representation
  Information
• offering a repository of tools and technical
  information, a focal point for digital curators
• metadata standards
• Testbeds
• for testing and evaluating tools, methods,
  standards and policies in realistic settings
• Certification
• Standards (Trusted digital repositories RLG)

53
OAIS Reference Model Functional Model
54
Current research priorities

• Data integration and publication
• Performance and optimisation
• Annotation
• Appraisal and long-term preservation
• Socio-economic and legal context rights,
  responsibilities and viability
• Cost-benefit analysis of the data curation
  process
• Security safe and effective data analysis
  environments
• Automation of metadata extraction
• Visitors Programme and Seminar Series

55
Working with Others

• Digital Library Federation
• The National Archives
• Global Grid Forum
• NARA
• Library of Congress
• Research Library Group
• Digital Preservation Coalition
• JISC community
• E-Science Community
• Associates Network
• and many many more

Development info see http//dev.dcc.rl.ac.uk for
details of Wiki and email list open to all
56
Credits Links

• Credits
• Data Storage (APS)
• Tim Folkes System manager
• Bonny Strong System developer
• Jens Jensen SE development
• Data Management (SRB)
• Kerstin Kleese Van Dam
• Peter Berisford SRB implementation
• Gordon Brown Database admin
• Digital Curation Centre
• David Giaretta Development Director
• Links
• CCLRC home http//www.cclrc.ac.uk/
• CCLRC e-Science http//www.e-science.clrc.ac.uk/we
  b
• CCLRC Atlas Data Store http//www.e-science.clrc.a
  c.uk/web/services/datastore
• SDSC SRB http//www.npaci.edu/DICE/SRB/