Status and plans for bookkeeping system and production tools

1
Status and plans for bookkeeping system and
production tools

• Eric van Herwijnen
• Thursday, 19 september 2002

2
Contents

• Status remote centres datachallenge plans
• DataGrid status and plans
• Data management and production tools
• Job submission
• Bookkeeping
• Configuration
• Data production and monitoring
• Roadmap and conclusions

3
Status remote centres
4
Physics Data Challenge plans

• MC production Physics Data Challenge
• Available capacity seems to match requirements
• Planning
• Preproduction mid Dec 2002 mid Jan 2003
• Production Feb May 2003

5
DataGrid status and plans

• Installation 1.2.2 operational
• Long job problem fixed
• Long file transfer problem ( 1 Gb)
• New production tools being installed
• Test
• Run 500 event MC generation
• Store on SE
• Recover logs and histograms to CERN
• Run reconstruction. Output to SE. Recover log
  files and histos.
• Write recon output to mass store (Castor)
• Read Castor data with an analysis job outside
  Grid

6
Data Management
7
Job submission

• New tools designed and written by A.
  Tsaregorodtsev
• Site dependencies concentrated in 3 scripts
• Simple installation procedure
• http//lhcb-wdqa.web.cern.ch/lhcb-wdqa/distributio
  n/
• AFS independent
• Standard directory structure
• No (or very little) java
• More flexible updating of bookkeeping database
• Remote centers should migrate to new system

8
(No Transcript)
9
Job submission, future
Data Production DB
Job.opts
Bookkeeping DB
Productiondone
Modify
Create job(s)
script
Prod.Mgr
Configuration DB

• Build newconfiguration
• Selection ofDefaults

Information Flow
10
Job submission, future

• Workflow to be implemented (template scripts for
  each step, parameters added via web page)
• Prototype exists (M. Frank)

11
Bookkeeping
12
Bookkeeping

• Most components ready (S. Ponce, F. Loverre)
• Structure of database independent of tools, easy
  to add new datatypes, identify productions,
  replicas of datasets
• Database (Oracle) filled by a Java server via XML
  files
• API implemented in Java and Python
• Migration to new database transparent for user
• Final tests under way before production

13
Configuration

• Prototype of configuration database exists
• Need to integrate with job submission tools (5
  line python script?)
• Prototype of GUI to view the database developed
  by G. Klamke (summer student)
• Need to create tool to add configurations to
  database, and integrate with Ganga

14
(No Transcript)
15
Data Production and Monitoring
16
Data production and monitoring

• Current PVSS system needs to be brought beyond
  the prototype stage
• Cosmetic changes, cleanup of dead entries, speed
• Adapt to new job submission tools
• Alarms should really be alarms, add corrective
  action
• Migrate to new version of PVSS
• Web interface
• Production database interface need to be
  created
• Data quality check tools need to be reviewed and
  created
• Ongoing activity

17
Roadmap and conclusions

• Installation of new job submission tools oct-nov
  2002
• New bookkeeping db dec 2002
• Integration of job configuration db with job
  submission tools feb 2003
• Integration of job configuration db with Ganga
  summer 2003
• Creation of data production db, integration with
  monitoring system summer-fall 2003