Production Manager Tools and Ganga (New Architecture)

1
Production Manager Tools and Ganga (New
Architecture)
Ganga/Dirac Data Management meeting 16-17 October
2003
Gennady Kuznetsov
2
Job Control Monitoring

• Requirements for the Job Control and Monitoring
  script
• Interpreter.
• Platform independent.
• Well developed language with a rich API.
• Object-oriented (we need separate variable
  scopes).
• Candidates Python, Java, Rugby, shells
• We chose Python. (Java required compilation)

Gennady Kuznetsov
3
Current Design
Job Configuration, Data Production, Bookkeeping -
Design Document, M. Frank (Editor), 04/12/2001
Gennady Kuznetsov
4
Components
We add Module as additional component of the
architecture

• Levels of usage
• Module Programmer
• Step Production manager
• Workflow User

This module structure allow us to construct any
algorithm as a combination of modules.
Each step generates job as a Python program.
Gennady Kuznetsov
5
Module

• Each variable has
• Name
• Type (for information only)
• Value
• Stated as input output
• Name of the instance of linked module
• Name of the variable of linked module
• Description

Variable
n
1
VariableCollection
Module
ModuleDefinition
ModuleInstance
n
n
1
1
StepDefinition
ModuleInstances can be connected with each other
and forms a Step.
Example of the generated code Module
Instanceiterration3 TypePrintTestiterration3
PrintTest()iterration3.value1
iterration1.result typeintiterration3.value2
iterration2.result typedoubleiterration3.exe
cute()
Each Step can be instantiated as a single Job
Gennady Kuznetsov
6
Module Editor
Python code of single module. Can be many classes.
Module Name
Stored as XML file
Description
Module variables.
Gennady Kuznetsov
7
Step Editor
Graphical representation of Module relationship
(not ready)
Stored as XML file, where all modules are embedded
Step Name
Description
Variables of currently selected instance
Definitions of Modules
Instances of Modules
Selected instance
Gennady Kuznetsov
8
Workflow Editor
Not ready yet.

• It will link all jobs in a workflow instance.
• The simplest workflow is a single step.
• It will combine workflow and Vector and
  generate as many instances of workflow as many
  members in a vector. For example, it can be list
  of numbers (job number or name) or it can be a
  list of files for reconstruction or analysis.

Gennady Kuznetsov
9
Layered Architecture
Module Library
Editors
Programmer
Module
Module Editor
Programmer Production Manager
Step
Step Editor
Production Manager User
Workflow
Workflow Editor
User
GangaWorkflow Editor with limited
functionality (configuration only)
Module library is a collection of XML files on
shared file system (like AFS). Can be updated by
anyone. It give us a lot of flexibility and
freedom.
Gennady Kuznetsov
10
Combining all together
Permanent part of the code
Ganga
IntegratedEnvironment
Used as it is now
Ganga
Job Execution Code
Repacked as Modules
Dirac Editors
Creates
Flexible part of the code
Module library
User Interfaceto Workflow
Workflow
Use an interface specification from Workflow
Gennady Kuznetsov
11
Complete Structure
Submit Python job
Grid, Batch or Submission Service
Ganga UI Permanent
Use
Module LibraryFlexible Experiment dependent
Create and use
Submit Python job
Dirac EditorsPermanent
Gennady Kuznetsov
12
What To Do?

• If this proposal would be accepted
• For the Data challenge (MC Simulation)
• Extract execution code from Ganga and reorganize
  it as a modular structure and pack it as a Module
  Library using Dirac Editors. (Ganga, Dirac)
• To make appropriate modification in the Job
  Handler class. (Ganga)
• For the User Analysis
• Design formal specification for the User
  Interface (XML?)
• Implement User Interface Rendering Engine (Ganga)
• Reserve field in Workflow (XML) and design
  editing tool as part of the Workflow Editor
  (Dirac).

Gennady Kuznetsov
13
Collaboration with Ganga

• Layered architecture.
• No functional limitations specified at the 1st
  level.
• Modular architecture of the executables.

• Combine development power of two groups

Flexibility Save initial project development
time. Good base for the code evolution. Wide and
robust testing. Efficient code maintenance. Satisf
y users with wide requirements.

• Modular structure
• Reusable code

• The same modules could be tested in different
  experiments or by different groups.

• Service standardisation
• Code sharing

• 3 Levels of GUI (Programmer, Production manager,
  End user).

Gennady Kuznetsov