Developing a Data Management System for the ATLAS Experiment

1
Developing a Data Management System for the ATLAS
Experiment

• September 20, 2005
• Miguel Branco
• miguel.branco_at_cern.ch

2
Outline

• Data Challenges 2 and Rome Production
• Lessons Learned
• DQ2
• Design
• Implementation
• Data model
• Services
• Conclusion

3
DC2 and Rome Production

• Production started Spring 2004 and finished
  recently
• ProdSys
• Data Management (DQ) high-level service that
  interacted with all ATLAS Grid catalogs and
  storages
• File-based relied on backend RLS (Globus RLS,
  EDG RLS)
• Also implemented a simple reliable file transfer
  (FIFO queue)

Supervisors collect jobs from production
database dispatch to executors Executors (per
Grid) translate physics definition to a Grid
job and launch it DQ All components interacted
with data management
4
(a few of the) Lessons learned

• Catalogs were provided by Grid providers and used
  as-is
• Granularity file-level. No datasets, no file
  collections
• No scoping of queries (difficult to find data,
  slow)
• No bulk operations for most operations
• No managed and transparent data access from grid
  m/w
• GridFTP, SRM unreliable (error reporting)
• Problems with mass storage. Situation has since
  improved
• Difficult to handle multiple conditions from Grid
  abstraction
• Dynamically mapping users to mass storage
  stagers?
• Metadata support not usable too slow
• Logical Collection Name as metadata string
  /datafiles/rome/
• Catalogs not always geographically distributed
• Single point of failure (middleware,
  people/timezones)
• No ATLAS resources information system (with
  known/negotiated QoS)
• and unreliable information systems from Grid
  providers, when available
• Operational problems
• Timezones, lack of people, experience,
  communication

5
DQ2 Design rationale

• Evolve from past experience
• Interoperability
• Grid m/w components Start with
• Replica Catalog, Storage Management, Reliable
  File Transfer
• Global ! Site ! Worker Node ! Client Tools
• Handle Production and User Analysis
• Well-defined data flows and chaotic data movement
• Security
• Datasets, not files
• Bulk requests native to the design and interfaces
• Scalability
• Administrative, Geographical, Load

6
DQ2

• Moves from a file based system to one based on
  datasets
• Hides file level granularity from users
• A hierarchical structure makes cataloging more
  manageable
• However file level access is still possible
• Scalable global data discovery and access via a
  catalog hierarchy
• No global physical file replica catalog
• but global dataset replica catalog and global
  dataset location catalog

Files
Files
Datasets
Sites
Files
Files
Files
Dataset
7
Catalog architecture and interactions
8
Global catalogs
Dataset Repository
Holds all dataset names and unique IDs ( system
metadata)
Dataset Hierarchy
Maintains versioning information and information
on container datasets, datasets consisting of
other datasets
Dataset Content Catalog
Maps each dataset to its constituent files This
one holds info on every logical file so must be
highly scalable, however it can be highly
partitioned using metadata etc..
Dataset Location Catalog
Stores locations of each dataset
All logically global but may be distributed
physically
9
Local Catalogs
Local Replica Catalog
Per grid/site/tier providing logical to physical
file name mapping. Implementations of this
catalog are Grid specific but must use a standard
interface.
Claims Catalog
Per site storage, keeping user claims on
datasets. Claims are used to manage stage
lifetime, resources and provide accounting.
Currently all Local catalogs are deployed per
ATLAS site
10
Datablocks

• Datablocks are defined as immutable and
  unbreakable collections of files
• They are a special case of datasets
• A site cannot hold partial datablocks
• There are no versions for datablocks
• Used to aggregate files for convenient
  distribution
• Files grouped together by physics properties, run
  number etc..
• Much more scalable than file level distribution
• Useful for provenance immutable sets of data
• The principal means of data distribution and data
  discovery
• immutability avoids consistency problems when
  distributing data
• moving data in blocks improves data distribution
  (eg. datablock can map to bulk SRM copy or stage
  request)

11
Subscriptions

• A site can subscribe to data
• When a new version is available, this latest
  version of the dataset is automatically made
  available through site-local specific services
  carrying out the required replication - Automated
  movement
• Subscriptions can be made to datasets (for file
  distribution) or container datasets (for
  datablock distribution)
• Use cases
• Automatic distribution of datasets holding a
  variable collection of datablocks (container
  datasets)
• Automatic replication of files by subscribing to
  a mutable dataset (eg file-based calibration data
  distribution)

Site X
Dataset A
Subscriptions
File1
File2
Dataset A Site X Dataset B Site Y
(Container) Dataset B
Site Y
Data block1
Data block2
12
Subscriptions

• Various data movement use cases
• Datasets
• latest version of a dataset (triggers automatic
  updates whenever a new version appears)
• Container Datasets
• which in turn contain datablocks or datasets
• supports subscriptions to the latest version of a
  container dataset (automatically triggers updates
  whenever e.g. the set of datablocks making up the
  container dataset changes)
• Datablocks (single copy of immutable set of
  files)
• Databuckets (diagram next slide)
• replication of a set of files using notification
  model (whenever new content appears on the
  databucket, the replication is triggered)

Subscribes to DS1
Dataset Location Catalog updated
13
Data buckets

• Data must be replicated (quickly) not by the
  appearance of a new version but by new content
• alternative would be constantly defining new
  versions of datasets!
• Will use notification model
• Whenever new content appears on a data bucket,
  sites subscribing to it are notified and data is
  moved accordingly
• Data buckets can contain files
• Data buckets can contain datablocks

14
Claims

• Claims catalog manages the usage of datasets
• User requests have a lifetime
• Claim is assigned
• User may add claims on existing datasets
• Claim owner may (should) release claim when done
• Claim owner may extend lifetime of claim
• Automatically handled by user client tools
• Behavior
• Each claim has an expiration time (now plus
  lifetime)
• Claim is active until released or expired
• Datasets may have multiple active claims for
  different users
• Claims is a generic approach to handle usage of
  data
• Other tools use it for cache-turnover (expired
  claims), accounting, policy enforcement and even
  for dealing with mass storage (claim triggers SRM
  stage request)

15
Implementation

• Architectural Style
• REST-style (not entirely RESTful)
• Communication intend to migrate non
  performance-critical payload (monitoring,
  real-time status reporting) to XML soon
• vocabularies will emerge from experience of
  running system
• Development
• First usable prototype deployed 47 days after
  project started
• Technology choices
• HTTP text/plain HTTP text/xml
• Python servers hosted on Apache (mod_python,
  mod_gridsite) clients using PyCurl
• POOL File Catalog interface gives us choice of
  back-end for catalogs
• Globus RLS, LCG FC, MySQL, Oracle, XML
• File movement SRM (v1.1), GridFTP, gLite FTS,
  HTTP, dccp, cp
• Security
• Use HTTPS (with Globus proxy certs) for
  POST/PUT/DELETE and HTTP for GETs, ie
  world-readable data, best performance (can be
  made secure to ATLAS VO if required)

16
API (HTTP)

• GET http//atl02dq.cern.ch/dq2/repository/dataset?
  lfnsome_atlas_dataset
• GET http//atl02dq.cern.ch/dq2/content/dataset?lfn
  some_atlas_dataset
• POST http//atl02dq.cern.ch/dq2/repository/dataset
  PAYLOAD dsnDSN
• POST http//atl02dq.cern.ch/dq2/content/dataset
  PAYLOAD guids files

17
Detail on Subscriptions
Function
Agents
State Machine
Fetcher
Finds incomplete datasets
unknownSURL
ReplicaResolver
Finds remote SURL
knownSURL
MoverPartitioner
Assigns Mover agents
assigned
Mover
Moves file
toValidate
ReplicaVerifier
Verifies local replica
(usable standalone)
validated
BlockVerifier
Verifies whole dataset complete
done
List of software required to handle
subscriptions. Requires minimal deployment effort
(laptop support!)
18
Conclusion

• Evolve the model based on past experience
• based on proven technologies
• Appears to scale so far
• although it remains to be proven on mass-scale
  production
• load datasets subscriptions running per site
  multiple catalogs
• geographic local site information stays local
  (eg. SURLs) site services
• very important administrative scalability ATLAS
  users can manage their site installation
• It is running now across some US ATLAS and LCG
  sites
• Ramping up (starting now!) to the full set of LCG
  and US ATLAS resources.
• http//uimon.cern.ch/twiki/bin/view/Atlas/DDM

19
Summary of Services

• Global services
• Dataset catalogs
• Requirements grid environment, database, Apache
  services
• Site services
• Subscriptions, Databuckets, Claims (plus daemons
  running per storage) and minimal information
  system (monitoring, real-time reporting)
• Requirements grid environment, database, Apache
  services, DQ2 agents and daemons for moving data
  claims validation, grid-specific data
  movement clients, Python, PyCURL, grid (host)
  certificate
• Local worker node client
• Contact local LRC, get and put data to local
  Storage
• Requirements grid environment, (grid-specific
  data movement clients?)
• Clients
• Define datasets and datablocks, subscribe them to
  sites
• Associate files with new dataset versions
• Query dataset definition, contents, location
• Requirements Python, PyCURL, grid certificate
  (for writing)