27th LHCb Software Week Data taking: Use Of DIRAC Data Management Tools

1
27th LHCb Software Week Data taking Use Of
DIRAC Data Management Tools
2
Overview

• Overview of DIRAC Data Management System
• Components
• Storage Element, File Catalogs, ReplicaManager
• Operation of Interest
• Possible solution for Pit-gtCastor transfers

3
DIRAC DM Components

• The main components are
• Storage Element and Storage access plug-ins
• Replica Manager
• File Catalogs

Data Management Clients
UserInterface
WMS
TransferAgent
FileCatalogC
ReplicaManager
FileCatalogB
FileCatalogA
DIRAC Data Management Components
StorageElement
GridFTPStorage
HTTPStorage
SRMStorage
SE Service
Physical storage
4
Storage Element

• DIRAC StorageElement is an abstraction of a
  Storage facility
• Access to storage is provided by plug-in modules
  for each available access protocol.
• Pluggable transport modules srm, gridftp, bbftp,
  sftp, http,
• Storage Element is used mostly to get access to
  the files
• Grid SE (also Storage Element) is the underlying
  resource used

5
File Catalogs

• DIRAC Data Management was designed to work with
  multiple File Catalogs
• All available catalogs have identical APIs
• Can be used interchangeably
• Available catalogs
• LCG File Catalog LFC
• Current baseline choice
• Processing Database File Catalog
• Exposing Processing DB Datafiles and Replicas
  table as a File Catalog
• (more later)
• BK database replica tables
• Soon to be phased out
• others.

6
Replica Manager

• Replica Manager provides logic for all data
  management operations
• File upload/download to/from Grid
• File replication across SEs
• Registration in catalogs
• etc.
• Keeps a list of active File Catalogs
• All registrations applied to all catalogs

7
Transfer Agent RequestDB

• Data Management requests stored in RequestDB
• XML containing params. required for operation
• e.g. Operation, LFN, SourceSE, TargetSE, etc
• Transfer Agent
• Picks up requests from RequestDB and executes
  them
• Operations performed through Replica Manager
• Replica Manager returns full log of operations
• Transfer Agent performs retries based on logs
• Retries attempted till success

8
Processing DB

• ProcessingDB and accompanying Replication Agent
  provide tool to perform automatic data management
  tasks
• ProcessingDB contains
• Replica information for files
• Transformation definitions containing
  properties of particular tasks to be perfomed
• Developed for automatic .digi distribution for
  DC06

DIRAC WMS
Request DB
ReplicationAgent
9
Useful Operations

• Files from pit copied to castor and registered in
  catalogs
• 2GB file per 30 seconds
• Perform single file operations
• No need for heavy bulk machinery
• Upload file to Castor, remove when on tape
• Would replication to second SE be sufficient?
• More later
• So
• copyAndRegister operation
• used extensively already in DIRAC (easy)
• Removal operation already exists
• Small development required for this use-case

10
PIT-gtCastor Solution?

• DIRAC installed on a secondary server at pit
• copyAndRegister request placed in RequestDB
• DataMover places request when file ready
• Transfer Agent executes replication to Castor
• Registers file in LFC and ProcessingDB
• This file is then replicated to additional SE
  (Tier1)
• This machinery to be utilised during DC06
• Implement method to discover when file migrated
• Set Remove request for file at pit
• Send a message to DataMover to perform removal

11
PIT-gtCastor Solution?

• Possible file could be replicated to T1 before
  migrated to tape
• If this is sufficient protection?
• If so
• File registered in ProcDB on arrival at T1
• Transformation defined to check number of
  replicas
• If more than one flag file for removal from pit
• If not
• Method which regularly checks migration of file
• If complete flag file for removal from pit
• Or both in parallel!

12
Summary

• DIRAC DM System offers tested method of reliable
  file upload to the grid.
• DIRAC installed on secondary server at pit
• copyAndRegister request set by data mover to
  request DB
• Transfer Agent performs upload and registration
• New methods required
• Check status of migration (also perhaps Tier1
  replica)
• Send message to DataMover to remove file

13
Questions?