The Ganga User Interface for Physics Analysis on Distributed Resources

1
The Ganga User Interface for Physics Analysis on
Distributed Resources
Alexander Soroko (Oxford University)

• Motivation and Background
• Components
• Submission of Analysis Jobs
• Plans and Conclusions

2
The Project

• Ganga is being developed as a joint project
  between ATLAS and LHCb
• Began in the UK supported by GridPP, important
  collaborations with US colleagues

• Current contributors are
• Code Developers A.Soroko (Oxford), K.Harrison
  (Cambridge), C.L.Tan (Birmingham), J.Moscicki
  (CERN), J.Martyniak (Imperial), A.Maier (CERN)
• Technical direction D.Adams (BNL), U.Egede
  (Imperial), P.Charpentier (CERN), P.Mato (CERN)
• GridPP coordination G.N.Patrick (RAL),
  R.W.L.Jones (Lancaster)

Ganga-related information regularly updated on
web site http//ganga.web.cern.ch/ganga
3
Motivation and Background

• ATLAS and LHCb develop applications within a
  common framework Gaudi/Athena
• Both collaborations aim to exploit potential of
  Grid for large-scale, data-intensive distributed
  computing

• Simplify management of analysis and production
  jobs for end-user physicists by developing tools
  for accessing Grid and other services
• Built-in knowledge of how Gaudi/Athena works
  Gaudi/Athena and Grid Alliance (Ganga)

4
Motivation and Background
5
General Outlook
6
Software Bus Design
Specialised and External components work like
plug-ins and not necessary present in the
installation
For more info about Software Bus see poster of
Wim LAVRIJSEN (session 24, track t3)
7
Core Components
Core Components
UI Abstract Definition
Job Interface
Application Interface
Data Interface
Job Registry

• External
• Components

LHCb Bookkeeping DB
PyRoot
Gaudi Python
PyCMT
PyAMI
AJDL
8
Core Components

• Core Components define classes to support basic
  functionality of Ganga
• They provide generic Interfaces for the
• Job (what is submitted)
• Application (the executable run by the Job)
• Data (input and output data to the application)
  
• Job Registry component provides the means to
  store and recover job information
• UI Abstract Definition component enables dynamic
  GUI construction
• Core components are not bound to any specific
  type of application (e.g. Gaudi/Athena)
• Core components are self consistent and can be
  used outside of Ganga to build other applications

9
Specialised Components
Specialised Components
Core Components
Job Handlers
UI Abstract Definition
Application Handlers
Job Interface
Software Bus
File Handlers
Application Interface
Job Options Editor
Data Interface
Job Splitters
Job Registry

• External
• Components

LHCb Bookkeeping DB
PyRoot
Gaudi Python
PyCMT
PyAMI
AJDL
10
Specialised Components

• Specialised components like Job, Application and
  File Handlers implement interfaces provided by
  Core components for job, application and data
  correspondingly
• Job-handling components
• Manage job submission to different batch systems
  (EDG/LCG, LSF, PBS, DIRAC WMS) and local PCs
• Perform job configuration e.g., create JDL files
• Generate work flow scripts to be executed on the
  target batch system
• Application-handling components
• Provide configuration templates covering specific
  tasks e.g., setting up environment, discovery of
  user dlls etc
• File-handling components
• Transfer input output files between worker
  nodes and different storage elements

11
Job Options Editor (JOE)

• Difficulties in creating and manipulating
  application configuration files within the
  Gaudi/Athena frameworks (typically known as job
  option files)
• Knowledge of available options/values
• Knowledge of job options syntax
• Spelling errors
• JOE makes these difficulties easy with convenient
  GUI
• Collapsible tree representation allows options,
  attributes or included files of interest to be
  viewed on demand.
• Multiple job option files may be open for
  simultaneously editing using window tabs.

12
JOE GUI
View of job options file
Main window of JOE
13
Job Splitters
14
External Components

• External
• Components

LHCb Bookkeeping DB
PyRoot
Gaudi Python
PyCMT
PyAMI
AJDL
15
LHCb Bookkeeping DB
16
ATLAS AJDL Components

• AJDL is Abstract Job Description Language, used
  by DIAL (ATLAS analysis framework)
• The building blocks of an AJDL job are
• Application (described by its name and version,
  taken from central repository)
• Task (collection of files required by the
  application)
• Dataset (the user issues an SQL query to a
  dataset catalog)
• Preferences (provide the means to fine-tune the
  AJDL job)

17
Submission of LHCb Analysis Job
Prepare/compile/select packages
Mainly outside of Ganga
Edit algorithm workflow and parameters of
algoritms
JOE
Select input datasets
Bookkeeping DB interface
Split job
Job splitters
Press Submit button
Watch job status
See job output in the job folder
18
How Ganga Works
Grid/Batch System Gatekeeper
Worker Nodes
DIRAC WMS
DIAL
Files Transfer
Storage Element
Get Monitoring Info
Get job output
Send
Ganga
Job object
Job object
Job object
Job object
Job object
Job Factory (Job Registry Class)
Job Options Editor
Strategy Selection (e.g., Splitting Scripts)
Data Selection
Job Requirements (Resources)
Job Options DB
Bookkeeping DB
19
GUI Job Submission
Job was submitted to DIRAC WMS
For more information about DIRAC see talk of
Andrei TSAREGORODTSEV (session 7 track t4)
20
Plans and Conclusions

• Ganga has been developed as a joint project
  between ATLAS and LHCb
• Ganga modular design
• Provides uniform user interface to different
  distributed resources
• Enables experiment customisation
• Plans
• Convenient CLI (scripting)
• Writing interfaces to other analysis services
  (gLite)
• Building remote Ganga services running under
  control of a local client
• GUI interface for the above client
• Services to provide centralised persistency of
  user jobs
• Ganga session persistency (history, GUI
  preferences)
• Roaming access to user profiles
• The intention is to make life easier for users