Software Review panel

1
Software Review panel

• Non experiment-specific Software

ALICE
R.Brun, F.Carminati, A.Morsch For the ALICE
collaboration
2
Upon which commercial, public domain, or HEP
community software is the experiment relying?

• Code checker, software engineering tools
• Successful industrial (IRST Trento) collaboration
  project
• Collaborate with the provider to develop the
  product satisfying our requirements
• Possible development of a geometry database with
  external experts
• Involved in EuroStore II MSS/HSM
• Will be tested in the ALICE Data Challenges
• Possible use of ORACLE RDBMS for Run catalogue
  and condition databases

3
Upon which commercial, public domain, or HEP
community software is the experiment relying?

• A model for industrial collaboration
• Joint development of product
• CERN provides a low inertia community and
  challenging applications
• Reduced costs due to the collaboration aspects
• CERN and associate institutes own the source of
  the development
• The company owns and can commercialise the result
• Could this be a better model than the one used in
  LHC/Objectivity acquisition?
• Same direction is taken by the GRID project

4
Upon which commercial, public domain, or HEP
community software is the experiment relying?

• Linux GNU
• Open source products
• Considering evaluation of GSL
• Gnu Scientific Library
• ROOT
• Main Off-line framework
• Official support asked to CERN
• Possible use of MySQL RDBMS for Run catalogue and
  condition databases
• Will be tested in ALICE Data Challenges

5
Upon which commercial, public domain, or HEP
community software is the experiment relying?

• FLUKA
• Instrumental for radiation studies and when it is
  important to understand the physics of a detector
• CERN support should materialise urgently to allow
  a source release of the code and a better
  integration in our environment
• Important also for cross-validation of the
  physics of GEANT4
• GEANT4
• Pending acceptance as explained before
• Still many problems to use it as a toolkit
• More like SUMX than HBOOK (senior physicists
  only)

6
Upon which commercial, public domain, or HEP
community software is the experiment relying?

• GEANT 3
• Maintained by ALICE
• Still used by all the LHC experiments as the main
  MC
• Event Generators
• Heavy ion physics mainly experiment-driven
• Large use of parametrised background signal
• Less frequent use of physics generators
• MSS/HSM
• Testing Castor now with ALICE Data Challenges
• Would also be interested in testing InterStage (?)

7
ROOT in ALICE

• All ROOT sub-systems used
• Histogramming -- data analysis
• Object containers
• 2-d and 3-d graphics
• Persistency (serialisation and trees)
• Communication via sockets and shared memory
• C interpreter (framework control)
• Shared libs dynamic compilation linking
• Automatic html documentation from code

8
Upon which commercial, public domain, or HEP
community software is the experiment relying?

• Tools that we are not relying upon
• NAG library
• CLHEP
• STDHEP
• LHC (data presentation)
• Objectivity /Espresso
• HEPODBMS

9
Upon which commercial, public domain, or HEP
community software is the experiment relying?

• NAG
• Some basic functionality in ROOT
• Plans to evaluate GSL (Gnu Scientific Library)
• CLHEP
• More functionality in ROOT
• Classes are non persistent (big problem!)
• Following the discussions between CLHEP and ZOOM
• STDHEP
• Classes with same (proposed) signature in ROOT
• Collaboration with the STDHEP group

10
LHC

• Alices position clearly stated since 1998
• We invested little in LHC evaluation
• Non-convincing prototypes
• Iris Explorer not the way to go
• No scripting language or interpreter
• Far too late now (remember 5 years to reach
  maturity)
• We are using ROOT and we are very happy with that

11
Objectivity

• Conversion of exercise AliRoot hits into
  Objectivity to
• Output the same data in ROOT and Objy
• Evaluate the optimisation effort for ROOT and
  Objy
• Evaluate the ease of use for non-experts
• Test the flexibility of our framework
• Evaluate the performance (speed and space)
• Result cross-checked with RD45 members

12
Objectivity

• Results
• One man year spent in evaluation
• We could use Objy in our framework
• But it will force us to implement transient AND
  persistent versions of each class
• ROOT performance is better (file size, RT)
• Very difficult to tune Objy performance
• Easy test for Objy (local I/O, single
  architecture)
• See Babar performance problems Solaris vs Linux
• Concept of page not suited to high band WAN access

13
Objectivity

• Last LCB referee report (McCubbin)
• Doubts on the technology and on the company
• STAR conclusion on Objectivity (Wenaus)
• Our initial pursue of Objectivity was a mistake
• BABAR position on Objectivity (Jacobsen)
• Not all is sweetness and light
• Replication and distribution not proven concepts
• ROOT I/O used for physics data analysis on
  mini-DSTs

14
Objectivity vs ROOT File Size
15
Objectivity vs ROOT Write Speed
16
Objectivity vs ROOT Read full event
17
Objectivity vs ROOT Selective read
18
Espresso?

• The project
• Estimated 15 man years is unreasonable
• Less people could be found for HEP-wide critical
  MSS/HSM project!
• Why no discussions with the ROOT team?
• Despite ALICE request at LCB
• Far too late now anyway
• Remember more than 5 years for a mature system

19
Espresso?

• The product
• Use of Espresso will force us to implement
  transient AND persistent versions of each class
• Can we expect a performance improvement ?
• Concept of page not suited to high band WAN
  access
• But more important than anything else
• do we really need it?

20
ROOT persistency

• Handles objects!
• Rich set of browsing tools
• Efficient from small objects (histos) to very
  large collections
• Can be integrated with a RDBMS to handle
  association between events and files
• Has a multithreaded daemon for remote file access

21
Persistency the dream (P.Sphicas, CHEP 2000)

• The DB should make access to data anywhere
  anytime transparent and efficient
• Data should be accessible at the object level at
  any stage
• The database should optimise data access patterns
  used by the users
• The MSS/HSM should be transparent to the users

22
More on dreams
Selection Parameters
CPU
Local
DB1
Federation
DB2
DB3
OODB
Remote
DB4
DB5
DB6
23
Persistency Reality Check

• The OODBMS market is not taking off
• The only candidate is from a small company
• Improvements have been poor in 5 years
• In fact only doubts have grown
• Experiments are abandoning Objectivity
• Tests not conclusive for simple local use
• See BABAR
• MSS/HMS do not seem to match Objy
• Do we need it?

24
OODBMS HEP

• Do we really want to
• impose Objy on all institutes (see RedHat 6.1
  crisis)
• put the data in an opaque format? (beware of
  escrow!)
• help Objy?
• access everything from everywhere?
• use a page-based system on WAN? (see RD45
  results)
• store everything (also histos) into a federation?

25
OODBMS HEP

• Commercial OODBMS are tuned mostly for
  transaction management
• Light weight queries, medium-size objects,
  concurrent write/read, page-based, mostly random
  access
• We have different requirements
• Heavy queries, largely variable size objects,
  read only, stream based, sequential and random
  access
• The flexibility and modularity of ROOT allows
  building parallel distributed applications

26
More on requirements

• HEP/LHC requirements on data access are not all
  solved by OODBMS
• At least for ALICE
• We need to manage local and remote CPU and
  storage resources taking into account data
  locality
• Much more to it than data reclustering!
• We will build a prototype based on the Parallel
  ROOT Facility

27
The PROOF workflow
Slave 1
Slave N
Master
Tree-gtDraw()
Tree-gtDraw()
Initialisation
Packet generator
Initialisation
GetNextPacket()
GetNextPacket()
0,100
Process
100,100
Process
GetNextPacket()
GetNextPacket()
200,100
Process
300,40
Process
GetNextPacket()
GetNextPacket()
340,100
Process
Process
440,50
GetNextPacket()
GetNextPacket()
490,100
Process
590,60
Process
GetNextPacket()
SendObject(histo)
Wait for next command
Add histograms
Wait for next command
Display histograms
28
More on requirements
Selection Parameters
TagDB
CPU
Procedure
PROOF
Local
DB1
CPU
RDB
Proc.C
DB2
Remote
Proc.C
DB3
CPU
Proc.C
DB4
Proc.C
CPU
DB5
Proc.C
CPU
DB6
CPU
29
What service and support agreements are either in
place, or need to be put in place? How will this
be done? What is the role of CERN IT in this?

• ALICE needs support agreement for ROOT
• Already requested to CERN management (no answer)
• The support of the ROOT team and of the ROOT user
  community is of excellent quality
• But more man-power is needed
• Needs new support model for Geant4
• Open Source model
• CERN has pioneered some aspects of Open Source
  with CERNLIB and GEANT3, and with success
• Why this is not done for GEANT4?

30
What service and support agreements are either in
place, or need to be put in place? How will this
be done? What is the role of CERN IT in this?

• The promised support for FLUKA should materialise
  soon
• Support for products issued from collaboration
  agreements with external providers will be
  negotiated case by case
• Continued CERN IT support for data recording,
  system administration, mass storage management
  and processing at CERN will be vital

31
What service and support agreements are either in
place, or need to be put in place? How will this
be done? What is the role of CERN IT in this?

• CERN-IT must support experiment computing
• This must include FLUKA and ROOT
• ALICE had very little software support from IT
• This situation can and should improve
• IT system support is good
• It should continue and improve
• Operation of the ALICE computing farm is very
  good
• IT contribution to the ALICE MSS project is
  remarkable

32
What service and support agreements are either in
place, or need to be put in place? How will this
be done? What is the role of CERN IT in this?

• More support should be given to C programming
• Far too much hype on Java
• LHC/ROOT e-mails 2000 50/1400, 1999 16/8000
• JAS/ROOT e-mails 2000 20/1400, 1999 16/8000
• Detachment of IT/API people to the experiments
  for short term projects should be planned

33
What risk factors does the experiment face in its
use of such commercial, public domain or HEP
community software? What plans does it have to
mitigate the risks?

• ALICE is ready to use G4 but is worried
• Modularity of G4
• Maintenance and support scenario
• Physics validation
• Long term development
• ROOT long term support
• A small and motivated local team (3 staffs) could
  be enough
• We do not have yet a valid candidate for MSS/HSM

34
What risk factors does the experiment face in its
use of such commercial, public domain or HEP
community software? What plans does it have to
mitigate the risks?

• ALICE is ready to use G4 but is worried
• We have a program of integration and physics
  testing of GEANT 4
• We have an in-house in-depth knowledge of GEANT4
• We have integrated G3 in our framework
• ROOT long term support
• Hope that CERN management realises the importance
  of ROOT for LHC
• FNAL did for its experiments

35
ALICE GEANT4

• ALICEs position
• Need two MCs for comparison and cross validation
• GEANT3, seamlessly integrated in the framework
• FLUKA, not yet integrated in the framework, but
  it will
• We want to have one single framework!
• We want to integrate GEANT4 in our framework and
  phase out GEANT3
• Thanks to our framework, the phasing out of
  GEANT3 implies no transition and can be done
  seamlessly
• But we need to be able to use GEANT4 as a modular
  toolkit as is the case with GEANT3

36
GEANT4 integration into AliRoot

• UserActions cannot easily call managers and
  services

G4 services
G4Manager
UserAction
G4Manager
UserAction
G4Control
G4Manager
UserAction
G4Manager
UserAction
37
GEANT4 integration into AliRoot

• Services cannot be easily called by user
• Difficult to integrate in a framework
• Cannot take it as the framework, if we want a
  single framework for simulation and
  reconstruction
• User actions can alter the actions of the program
• But only in a limited way
• Not easy to be extended by the user
• The managers are tightly coupled and essentially
  non-modular

38
GEANT4 integration into AliRoot

• ALICE invested two people for two years
• But GEANT4 integration into AliRoot is hindered
  by its lack of modularity
• Can this situation be changed?
• Yes, but we need to improve the design of GEANT4
• Can GEANT4 be used in an environment for
  simulation-reconstruction?

39
What risk factors does the experiment face in its
use of such commercial, public domain or HEP
community software? What plans does it have to
mitigate the risks?

• We do not have yet a valid candidate for MSS/HSM
• Prototyping campaign together with IT
• Eurostore II participation

40
Is the experiment relying on technology, or
functionality, which is assumed, will be provided
by others and which currently is still in the RD
stage --- e.g. grid services? If so, what are the
risk factors?

• Networking infrastructure
• GRID middleware
• MSS/HSM

41
Is the experiment relying on technology, or
functionality, which is assumed, will be provided
by others and which currently is still in the RD
stage --- e.g grid services? If so, what are the
risk factors?

• Networking infrastructure
• Cost and bandwidth
• GRID middleware
• Will this technology be stable enough to be the
  groundwork of our production infrastructure?
• We should have a contingency plan if GRIDs are
  not there
• HSM/MSS
• Restricted market (for the moment, but watch
  internet!) bringing high prices and lack of
  standardisation

42
Given that the manpower from CERN/IT is limited,
how do you identify the areas where you would
most like to see strong CERN/IT involvement and
support? What are the arguments for central CERN
support?

• Non experiment specific areas
• Central data recording and storage and system
  management and networking
• Support for ROOT
• OS (Linux) support
• XML, Geometry data bases and graphics
• Improvement in G4 support and communication

43
Support yes, but how

• IT should not decide what to support
• It should propose and respond to the user needs
  and requirements
• User is the physicist doing analysis, not the
  software developer working in the experiment, who
  is a service provider and not a source of
  requirements!
• Committees should analyse and respond to end-user
  requirements, not set the requirements
• No more we have the right solution but they do
  not understand it!

44
Support yes, but how

• Projects (or equivalent enterprises) should be
  started and stopped in concertation with users
  (see above)!
• Success of a project should be measured by the
  number of users
• Ok, DOS but DOS was a success
• Support and resources should be proportional to
  the success of the project
• But this seems to be difficult see Linux started
  as an underground project

45
Support yes, but how

• IT Application Software experts should be
  detached to experiments for limited projects
• Tangible results!
• Deeper and realistic understanding of the
  experiment requirements
• Abstraction of common elements among experiments
  starting from working prototypes.
• Need Project-Oriented structures

46
Given that the manpower from CERN/IT is limited,
how do you identify the areas where you would
most like to see strong CERN/IT involvement and
support? What are the arguments for central CERN
support?

• Central CERN support for HEP applications (from
  HYDRA to PAW) was an Open Source organisation
  ante literam
• Economy of man-power for development and support
• Increased robustness of products
• Cross fertilisation across experiments
• Reuse of code between different experiments

47
Given that the manpower from CERN/IT is limited,
how do you identify the areas where you would
most like to see strong CERN/IT involvement and
support? What are the arguments for central CERN
support?

• How did it work?
• CERNLIB (PAWGEANT3KERNLIBMATHLIB) have been
  used for LEP-era experiments, LHC experiments and
  they are still used by many existing experiments
  and for CLIC simulations!
• CERNLIB had been adopted by SSC
• ROOT, born out of the CERNLIB experience and
  developed in the same spirit, is the most
  widespread tool in HEP used by all the major
  experiments
• This should be the basis of the future evolution
  of IT software policy