LCG Applications Area Status

1
LCG Applications Area Status

• Torre Wenaus, BNL/CERN
• LCG Applications Area Manager
• http//cern.ch/lcg/peb/applications
• LHCC Meeting
• January 27, 2003

2
Outline

• Applications area scope and organization
• Architecture
• Personnel and planning
• Little on planning since I talked to (most of
  you) about it in Nov
• Status of applications area projects
• SPI
• POOL
• SEAL
• PI
• Math libraries
• Simulation
• Relationship to other LCG activity areas
• Conclusion

3
The LHC Computing Grid Project Structure
Project Overview Board
Software andComputingCommittee(SC2)
Project Leader
ProjectExecutionBoard (PEB)
Requirements, Work plan, Monitoring
GridProjects
Project Work Packages
4
LCG Areas of Work

• Physics Applications Software
• Application Software Infrastructure libraries,
  tools
• Object persistency, data management tools
• Common Frameworks Simulation, Analysis, ..
• Adaptation of Physics Applications to Grid
  environment
• Grid tools, Portals
• Grid Deployment
• Data Challenges
• Grid Operations
• Network Planning
• Regional Centre Coordination
• Security access policy

• Fabric (Computing System)
• Physics Data Management
• Fabric Management
• Physics Data Storage
• LAN Management
• Wide-area Networking
• Security
• Internet Services
• Grid Technology
• Grid middleware
• Standard application services layer
• Inter-project coherence/compatibility

5
Applications Area Organization
Apps Area Leader
Architects Forum
Overall management, coordination, architecture
Project
Project
Project
Project Leaders

Work Package Leaders
WP
WP
WP
WP
WP
WP
WP
Direct technical collaboration between experiment
participants, IT, EP, ROOT, LCG personnel
6
Focus on Experiment Need

• Project structured and managed to ensure a focus
  on real experiment needs
• SC2/RTAG process to identify, define (need-driven
  requirements), initiate and monitor common
  project activities in a way guided by the
  experiments themselves
• Architects Forum to involve experiment architects
  in day to day project management and execution
• Open information flow and decision making
• Direct participation of experiment developers in
  the projects
• Tight iterative feedback loop to gather user
  feedback from frequent releases
• Early deployment and evaluation of LCG software
  in experiment contexts
• Success defined by experiment adoption and
  production deployment

7
Applications Area Projects

• Software Process and Infrastructure (SPI)
  (operating A.Aimar)
• Librarian, QA, testing, developer tools,
  documentation, training,
• Persistency Framework (POOL)
  (operating D.Duellmann)
• POOL hybrid ROOT/relational data store
• Mathematical libraries
  (operating F.James)
• Math and statistics libraries GSL etc. as NAGC
  replacement
• Group in India will work on this (workplan in
  development)
• Core Tools and Services (SEAL)
  (operating P.Mato)
• Foundation and utility libraries, basic framework
  services, system services, object dictionary and
  whiteboard, grid enabled services
• Physics Interfaces (PI)
  (launched V.Innocente)
• Interfaces and tools by which physicists directly
  use the software. Interactive (distributed)
  analysis, visualization, grid portals
• Simulation
  (launch planning in progress)
• Geant4, FLUKA, simulation framework, geometry
  model,
• Generator Services
  (launch as part of simu)
• Generator librarian, support, tool development

Bold Recent developments (last 3 months)
8
Project Relationships
9
Candidate RTAG timeline from March
Blue RTAG/activity launched or (light blue)
imminent
10
LCG Applications Area Timeline Highlights
POOL V0.1 internal release
Architectural blueprint complete
Applications
POOL first production release
Distributed production using grid services
Distributed end-user interactive analysis
Full Persistency Framework
LCG TDR
LCG
50 prototype (LCG-3)
LCG-1 reliability and performance targets
First Global Grid Service (LCG-1) available
LCG launch week
11
Architecture Blueprint

• Executive summary
• Response of the RTAG to the mandate
• Blueprint scope
• Requirements
• Use of ROOT
• Blueprint architecture design precepts
• High level architectural issues, approaches
• Blueprint architectural elements
• Specific architectural elements, suggested
  patterns, examples
• Domain decomposition
• Schedule and resources
• Recommendations

RTAG established in June Expt architects, other
experts After 14 meetings, much email... A
36-page final report Accepted by SC2 October 11
http//lcgapp.cern.ch/project/blueprint/
12
Principal architecture requirements

• Long lifetime support technology evolution
• C today support language evolution
• Seamless distributed operation
• Usability off-network
• Component modularity, public interfaces
• Interchangeability of implementations
• Integration into coherent framework and
  experiment software
• Design for end-users convenience more than the
  developers
• Re-use existing implementations
• Software quality at least as good as any LHC
  experiment
• Meet performance, quality requirements of
  trigger/DAQ software
• Platforms Linux/gcc, Linux/icc, Solaris, Windows

13
Component Model

• Communication via public interfaces
• APIs targeted to end-users, embedding frameworks,
  internal plug-ins
• Plug-ins
• Logical module encapsulating a service that can
  be loaded, activated and unloaded at run time
• Granularity driven by
• component replacement criteria
• dependency minimization
• development team organization

14
Software Structure
ROOT, Qt,
Implementation- neutral services
Grid middleware,
STL, ROOT libs, CLHEP, Boost,
15
Distributed Operation

• Architecture should enable but not require the
  use of distributed resources via the Grid
• Configuration and control of Grid-based operation
  via dedicated services
• Making use of optional grid middleware services
  at the foundation level of the software structure
• Insulating higher level software from the
  middleware
• Supporting replaceability
• Apart from these services, Grid-based operation
  should be largely transparent
• Services should gracefully adapt to unplugged
  environments

16
Managing Objects

• Object Dictionary
• To query a class about its internal structure
• Essential for persistency, data browsing, etc.
• The ROOT team and LCG plan to develop and
  converge on a common dictionary (common interface
  and implementation) with an interface
  anticipating a C standard (XTI) (Timescale
  1yr?)
• Object Whiteboard
• Uniform access to application-defined transient
  objects, including in the ROOT environment
• Object definition based on C header files
• Used by ATLAS and CMS

17
Other Architectural Elements

• Python-based Component Bus
• Plug-in integration of components providing a
  wide variety of functionality
• Component interfaces to bus derived from their
  C interfaces
• Scripting Languages
• Python and CINT (ROOT) to both be available
• Access to objects via object whiteboard in these
  environments
• Interface to the Grid
• Must support convenient, efficient configuration
  of computing elements with all needed components

18
Domain Decomposition
Products mentioned are examples not a
comprehensive list
Grey not in common project scope (also event
processing framework, TDAQ)
19
Use of ROOT in LCG Software

• Among the LHC experiments
• ALICE has based its applications directly on ROOT
• The 3 others base their applications on
  components with implementation-independent
  interfaces
• Look for software that can be encapsulated into
  these components
• All experiments agree that ROOT is an important
  element of LHC software
• Leverage existing software effectively and do not
  unnecessarily reinvent wheels
• Therefore the blueprint establishes a
  user/provider relationship between the LCG
  applications area and ROOT
• LCG AA software will make use of ROOT as an
  external product
• Draws on a great ROOT strength users are
  listened to very carefully!
• So far so good the ROOT team has been very
  responsive to needs for new and extended
  functionality coming from POOL

20
Blueprint RTAG Outcomes

• SC2 decided in October
• Blueprint is accepted
• RTAG recommendations accepted to
• Start common project on core tools and services
• Start common project on physics interfaces

21
Applications Area Personnel Status

• 18 LCG apps hires in place and working 1 last
  week, 1 in Feb
• Manpower ramp is on target (expected to reach
  20-23)
• Contributions from UK, Spain, Switzerland,
  Germany, Sweden, Israel, Portugal, US, India, and
  Russia
• 12 FTEs from IT (DB and API groups)
• 12 FTEs from CERN EP/SFT, experiments
• CERN established a new software group as the EP
  home of the LCG applications area (EP/SFT)
• Led by John Harvey
• Taking shape well. Localized in B.32
• Soon to be augmented by IT/API staff working in
  applications area they will move to EP/SFT
• Will improve cohesion, sense of project
  participation, technical management effectiveness

22
Applications Area Personnel Summary
Details at http//lcgapp.cern.ch/project/mgmt/AppM
anpower.xls
People FTEs
Total LCG hires working, total 19 19.0
Working directly for apps area projects 15 15.0
ROOT 2 2.0
Grid integration work with experiments 2 2.0
Contributions from
IT/DB 3 2.1
IT/API 11 9.7
EP/SFT experiments total 22 11.9
Working directly for apps area projects 19 9.9
Architecture, management 5 2.0
Total directly working on apps area projects 48 36.7
Overall applications area total 55 42.7
23
Current Personnel Distribution
24
FTE-years
25
Personnel Resources Required and Available
Estimate of Required Effort
SPI
60
Math libraries
50
Now
40
Physicist interface
FTEs
30
Generator services
20
Simulation
10
SEAL
0
POOL
Jun-03
Jun-04
Sep-02
Dec-02
Mar-03
Sep-03
Dec-03
Mar-04
Sep-04
Dec-04
Mar-05
Quarter ending
Blue Available effort
Future estimate based on 20 LCG, 12 IT, 28 EP
experiments
26
Schedule and Resource Tracking (example)
27
MS Project Integration POOL Milestones
28
Apps area planning materials

• Planning page linked from applications area page
• Applications area plan spreadsheet overall
  project plan
• http//lcgapp.cern.ch/project/mgmt/AppPlan.xls
• High level schedule, personnel resource
  requirements
• Applications area plan document overall project
  plan
• http//lcgapp.cern.ch/project/mgmt/AppPlan.doc
• Incomplete draft
• Personnel spreadsheet
• http//lcgapp.cern.ch/project/mgmt/AppManpower.xls
  
• Currently active/foreseen apps area personnel,
  activities
• WBS, milestones, assigned personnel resources
• http//atlassw1.phy.bnl.gov/Planning/lcgPlanning.h
  tml

29
L1 Milestones (1)
30
L1 Milestones (2)
31
Software Process and Infrastructure Project

• Alberto Aimar, CERN IT/API

Software development Support

• General Service for Software projects

• a. Provide general services needed by each
  project
• CVS repository, Web Site, Software Library
• Mailing Lists, Bug Reports, Collaborative
  Facilities
• b. Provide components specific to the software
  phases
• Tools, Templates, Training, Examples, etc.

32
SPI Services

• CVS repositories
• One repository per project
• Standard repository structure and include
  conventions
• Will eventually move to IT CVS service when it is
  proven
• AFS delivery area, Software Library
• /afs/cern.ch/sw/lcg
• Installations of LCG-developed and external
  software
• LCG Software Library toolsmith started in
  December
• Build servers
• Machines with needed Linux, Solaris
  configurations
• Project portal (similar to SourceForge)
  http//lcgappdev.cern.ch
• Very nice new system using Savannah
  (savannah.gnu.org)
• Used by POOL, SEAL, SPI, CMS,
• Bug tracking, project news, FAQ, mailing lists,
  download area, CVS access,

33
(No Transcript)
34
(No Transcript)
35
SPI Components

• Code documentation, browsing Doxygen, LXR,
  ViewCVS
• Testing Framework CppUnit, Oval
• Memory Leaks Valgrind
• Automatic Builds NICOS (ATLAS)
• Coding and design guidelines RuleChecker
• Standard CVS organization
• Configuration management SCRAM
• Acceptance of SCRAM decision shows the system
  works
• Project workbook

• All components and services should be in place
  mid-Feb
• Missing at this point
• Nightly build system (Being integrated with POOL
  for testing)
• Software library (Prototype being set up now)

36
POOL Project

• Dirk Duellmann, CERN IT/DB

• Pool of persistent objects for LHC
• Targeted at event data but not excluding other
  data
• Hybrid technology approach
• Object level data storage using file-based object
  store (ROOT)
• RDBMS for meta data file catalogs, object
  collections, etc (MySQL)
• Leverages existing ROOT I/O technology and adds
  value
• Transparent cross-file and cross-technology
  object navigation
• RDBMS integration
• Integration with Grid technology (eg EDG/Globus
  replica catalog)
• network and grid decoupled working modes
• Follows and exemplifies the LCG blueprint
  approach
• Components with well defined responsibilities
• Communicating via public component interfaces
• Implementation technology neutral

37
POOL Release Schedule

• End September - V0.1 (Released Oct 2)
• All core components for navigation exist and
  interoperate
• Assumes ROOT object (TObject) on read and write
• End October - V0.2 (Released Nov 15)
• First collection implementation
• End November - V0.3 (Released Dec 18)
• First public release
• EDG/Globus FileCatalog integrated
• Persistency for general C classes (not
  instrumented by ROOT), but very limited
  elementary types only
• Event metadata annotation and query
• End February V0.4
• Persistency of more complex objects, eg. with STL
  containers
• Support object descriptions from C header files
  (gcc-xml)
• June 2003 Production release

Principal apps area milestone defined in March
LCG launch Hybrid prototype by year end
38
Dictionary Reflection / Population / Conversion
New in POOL 0.3
In progress
39
POOL Milestones
40
Core Libraries and Services (SEAL) Project

• Pere Mato, CERN EP/SFT/LHCb

• Launched in Oct
• 6-member (3 FTE) team initially build up to 8
  FTEs by the summer
• Growth mainly from experiment contributions
• Scope
• Foundation, utility libraries
• Basic framework services
• Object dictionary (taken over from POOL)
• Grid enabled services
• Purpose
• Provide a coherent and complete set of core
  classes and services in conformance with overall
  architectural vision (Blueprint RTAG)
• Facilitate the integration of LCG and non-LCG
  software to build coherent applications
• Avoid duplication of software use community
  standards
• Areas of immediate relevance to POOL given
  priority
• Users are software developers in other projects
  and the experiments

41
SEAL Work Packages

• Foundation and utility libraries
• Boost, CLHEP, experiment code, complementary
  in-house development
• Participation in CLHEP workshop this week
• Component model and plug-in manager
• The core expression in code of the component
  architecture described in the blueprint. Mainly
  in-house development.
• LCG object dictionary
• C class reflection, dictionary population from
  C headers, ROOT gateways, Python binding
• Basic framework services
• Object whiteboard, message reporting, component
  configuration, event management
• Scripting services
• Python bindings for LCG services, ROOT
• Grid services
• Common interface to middleware
• Education and documentation
• Assisting experiments with integration

42
SEAL Schedule

• Jan 2003 - Initial work plan delivered to SC2 on
  Jan 10th and approved
• Including contents of version v1 alpha
• March 2003 - V1 alpha
• Essential elements with sufficient functionality
  for the other existing LCG projects (POOL, )
• Frequent internal releases (monthly?)
• June 2003 - V1 beta
• Complete list of the currently proposed elements
  implemented with sufficient functionality to be
  adopted by experiments
• June 2003 - Grid enabled services defined
• The SEAL services which must be grid-enabled are
  defined and their implementation prioritized.

43
Estimation of Needed Resources for SEAL
Current resources should be sufficient for v1
alpha (March)
44
Math Libraries Project

• Fred James, CERN EP/SFT

• Many different libraries in use
• General purpose (NAG-C, GSL, ..)
• HEP-specific ( CLHEP, ZOOM, ROOT)
• Modern libraries dealing with matrices and
  vectors (Blitz, Boost..)
• Financial considerations can we replace NAG with
  open source
• RTAG yes
• Do comparative evaluation of NAG-C and GSL
• Collect information on what is used/needed
• Evaluation of functionality and performance
• HEP-specific libraries expected to evolve to meet
  future needs

45
Math library recommendations status

• Establish support group to provide advice and
  info about existing libraries, and identify and
  develop new functionality
• Group established in October
• Which libraries and modules are in use by
  experiments
• Little response to experiment requests for info
  group in India is scanning experiment code to
  determine usage
• Detailed study should be undertaken to assess
  needed functionality and how to provide it,
  particularly via free libraries such as GSL
• Group in India is undertaking this study
• Initial plan of work developed with Fred James in
  December
• Targets completion of first round of GSL
  enhancements for April
• Based on priority needs assessment
• Work plan needs to be presented to the SC2 soon
• Scheduled tomorrow, but will be late

46
Physicist Interface (PI) Project

• Vincenzo Innocente, CERN EP/SFT/CMS

• Interfaces and tools by which physicists will
  directly use the software
• Planned scope
• Interactive environment physicists desktop
• Analysis tools
• Visualization
• Distributed analysis, grid portals
• Currently developing plans and trying to clarify
  the grid area
• Completed survey of experiments on their needs
  and interests
• Talking also to grid projects, other apps area
  projects, ROOT,
• Initial workplan proposal will be presented to
  PEB, SC2 this week
• Will plan inception workshops for identified work
  areas
• Identify contributors, partners, users
  deliverables and schedules personnel assignments

47
Proposed Near Term Work Items for PI

• Abstract interface to analysis services
• GEANT4 and some experiments do not wish to depend
  on a specific implementation
• One implementation must be ROOT
• Request for a coherent LCG analysis tool-set
• Interactive analysis environment
• Access to experiment objects (event-data,
  algorithms etc)
• Access to high level POOL services (collections,
  metaData)
• Transparent use of LCG and grid services
• With possibility to expose them for debugging
  and detailed monitoring
• GUI (pointclick) and scripting interface
• Interactive event and detector visualization
• Integrated with the analysis environment
• Offering a large palette of 2D and 3D rendering

48
Simulation Project

• Mandated by SC2 in Dec to initiate simulation
  project, following RTAG recommendations
• Project being organized now
• Discussions with experiments, G4, FLUKA, ROOT,
  on organization and roles in progress
• Probable that I will lead the overall project
  during a startup period, working with a slate of
  strong subproject leaders
• Scope (these are the tentative subprojects)
• Generic simulation framework
• Multiple simulation engine support, geometry
  model, generator interface, MC truth, user
  actions, user interfaces, average GEANE
  tracking, utilities
• ALICE virtual MC as starting point if it meets
  requirements
• Geant4 development and integration
• FLUKA integration
• Physics validation
• simulation test and benchmark suite
• Fast (shower) parameterisation
• Generator services

49
Collaborations

• LCG apps area needs to collaborate well with
  projects broader than the LHC
• See that LHC requirements are met, provide help
  and support targeted at LHC priorities, while
  being good collaborators (e.g. not assimilators)
• e.g. CLHEP discussions at workshop this week
• e.g. Geant4 in context of simulation project
• We also welcome collaborators on LCG projects
• e.g. (renewed?) interest from BaBar in POOL
• We also depend on the other LHC activity areas
• Grid Technology
• Ensuring that the needed middleware is/will be
  there, tested, selected and of production grade
• AA distributed software will be robust and usable
  only if the grid middleware it uses is so
• Grid Deployment AA software deployment
• Fabrics CERN infrastructure, AA development and
  test facilities

50
A note on my own time

• Nearing the one year mark of doing the apps area
  leader job with 75 of my time, resident at CERN
• Other 25 I am an ATLAS/BNL person in the US (1
  week/mo)
• Working well, and sustainable (the LCG job I
  mean)
• From my perspective at least!
• I will be lightening my ATLAS/US load, which will
  be welcome
• Expect to hand over the US ATLAS Software Manager
  job to a highly capable person in a few days or
  weeks
• Expect to hand over the ATLAS Planning Officer
  job shortly to someone else
• Remaining formal US responsibility is BNL Physics
  Applications Software Group Leader, for which I
  have a strong deputy (David Adams)

51
Concluding Remarks

• Expected area scope essentially covered by
  projects now defined
• Manpower is in quite good shape
• Buy-in by the experiments is good
• Direct participation in leadership, development,
  prompt testing and evaluation, RTAGs
• EP/SFT group taking shape well as a CERN hub
• Participants remote from CERN are contributing,
  but it isnt always easy
• POOL and SPI are delivering, and the other
  projects are ramping up
• Persistency prototype released in 2002, as
  targeted in March
• Important benchmark to come delivering
  production-capable POOL scheduled for June