EGEE

1
EGEE Status and Future of the Worlds Largest
Grid Infrastructure

• Dieter Kranzlmüller
• Deputy Project Director EGEE
• CERN, Switzerland

2
Cracow Grid Workshop Series
3
Presentation overview

• EGEE operates a Production Infrastructure
• Operations
• Middleware
• Applications
• EGEE as Partner
• The Next Phase EGEE-II

4
The EGEE project

• Objectives
• consistent, robust and secure service grid
  infrastructure for many applications
• improving and maintaining the middleware
• attracting new resources and users from industry
  as well as science
• Structure
• 71 leading institutions in 27 countries,
  federated in regional Grids
• leveraging national and regional grid activities
  worldwide
• funded by the EU with 32 M Euros for first 2
  years starting 1st April 2004

5
Achievements

• Infrastructure
• gt170 sites
• gt15 000 CPUs
• gt5 PB storage
• gt10 000 concurrent jobs
• gt60 Virtual Organisations

Were already exceeding deployment
expectations!!!
6
EGEE services

• Production service
• Based on the LCG-2 service
• With new resource centres and new applications
  encouraged to participate
• Stable, well-supported infrastructure, running
  only well-tested and reliable middleware
• Pre-production service
• Run in parallel with the production service
  (restricted nr of sites)
• First deployment of new versions of the
  middleware
• Applications test-bed
• GILDA testbed
• https//gilda.ct.infn.it/testbed.html
• Complete suite of Grid elements and
  applications
• Testbed, CA, VO, monitoring
• Everyone can register and use GILDA for training
  and testing

15 sites on 3 continents(all of them GÉANT sites)
7
Production Service

• Scale of the production service
• 16K CPUs/170 sites
• Other national regional grids 60 sites,
  6000 processors
• ? greatly exceeds no of sites planned for the
  end of EGEE
• Interoperability demonstrated with OSG (ongoing
  work with ARC)
• gt 2.2 million jobs Jan-Oct 2005
• Daily averages, sustained over a month 2200 10
  100
• ? 6 M kSI2K.cpu.hours ? 700 cpu years

8
Pre-production service

• Preproduction Service
• 14 sites
• Certification Service
• Certification testbed resides at CERN with 4
  virtual sites

9
Achievements

• Infrastructure
• gt170 sites
• gt15 000 CPUs
• gt5 PB storage
• gt10 000 concurrent jobs
• gt60 Virtual Organisations
• Middleware
• Now at gLite release 1.4
• Focus on basic services, easy installation and
  management
• Industry friendly open source license

10
EGEE Middleware
www.gLite.org

• First release of gLite end of March 2005
• Now at gLite v1.4
• Guiding principles
• Lightweight services
• Interoperability Co-existence with deployed
  infrastructures
• Robust Performance Fault Tolerance
• Portable
• Service oriented architecture
• Site autonomy
• Open source license
• Development / Integration / Testing
• Workload Management
• Information Systems
• Security
• Data Management
• ? gLite offers a complete data management
  solution in a distributed environment building on
  existing technology

11
Architecture Design

• Design team includes
• Representatives from middleware providers (AliEn,
  Condor, EDG, Globus,)
• Colleagues from the Operations activity
• Partners from related projects (e.g. OSG)
• gLite development takes into account input and
  experiences from applications, operations,
  related projects
• Effective exchange of ideas, requirements,
  solutions and technologies
• Coordinated development of new capabilities
• Open communication channels
• Joint deployment and testing of middleware
• Early detection of differences and disagreements

gLite is not just a software stack, it is a
new framework for international collaborative
middleware development
12
Software stack and origin of services in release
1 (simplified)

• Computing Element
• Gatekeeper (Globus)
• Condor-C (Condor)
• CE Monitor (EGEE)
• Local batch system (PBS, LSF, Condor)
• Workload Management
• WMS (EDG)
• Logging and bookkeeping (EDG)
• Condor-C (Condor)
• Information and Monitoring
• R-GMA (EDG)

• Storage Element
• gLite-I/O (AliEn)
• Reliable File Transfer (EGEE)
• GridFTP (Globus)
• SRM Castor (CERN), dCache (FNAL, DESY), other
  SRMs
• Catalog
• File/Replica Metadata Catalogs (EGEE)
• Security
• GSI (Globus)
• VOMS (DataTAG/EDG)
• Authentication for C and Java based (web)
  services (EDG)

Now doing rigorous scalability and performance
tests on pre-production service
13
Code Usage

• gLite core (supported by EGEE)
• 200 packages
• 23 logical deployment modules? flexible
  deployment
• 700.000 LOCs
• mostly in Java, C, and C
• gLite releases have been produced
• Tested, Documented, with Installation and Release
  notes
• Subsystems used on
• Service Challenges
• Pre-Production Services
• Production Service
• And by other communities (e.g. DILIGENT)

14
The Full Picture
Applications
High Level Grid Middleware
Basic Grid Middleware
Grid sites connected by Research Networks
15
Achievements

• Infrastructure
• gt170 sites
• gt15 000 CPUs
• gt5 PB storage
• gt10 000 concurrent jobs
• gt60 Virtual Organisations
• Middleware
• Now at gLite release 1.4
• Focus on basic services, easy installation and
  management
• Industry friendly open source license
• Applications gt20 applications from 7 domains
• High Energy Physics
• Biomedicine
• Earth Sciences
• Computational Chemistry

Additional 8 applications from 4 domains are in
evaluation stage

• Astronomy
• Geo-Physics
• Financial Simulation

16
NA4 Application Support

• More than 20 applications from 7 domains
• High Energy Physics
• 4 LHC experiments (ALICE, ATLAS, CMS, LHCb)
• BaBar, CDF, DØ, ZEUS
• Biomedicine
• Bioinformatics (Drug Discovery, GPS_at_,
  Xmipp_MLrefine, etc.)
• Medical imaging (GATE, CDSS, gPTM3D, SiMRI 3D,
  etc.)
• Earth Sciences
• Earth Observation, Solid Earth Physics,
  Hydrology, Climate
• Computational Chemistry
• Astronomy
• MAGIC
• Planck
• Geo-Physics
• EGEODE
• Financial Simulation
• E-GRID
• Another 8 applications from 4 domains are in
  evaluation stage

17
Applications Example WISDOM

• Grid-enabled drug discovery process for neglected
  diseases
• In silico docking
• compute probability that potential drugs dock
  with target protein
• To speed up and reduce cost required to develop
  new drugs
• WISDOM (World-wide In Silico Docking On Malaria)
• First biomedical data challenge
• 46 million ligands docked in 6 weeks
• Target proteins from malaria parasite
• Molecular docking applications Autodock and
  FlexX
• 1 million virtual ligands selected
• 1TB of data produced
• 1000 computers in 15 countries
• Equivalent to 80 CPU years
• Never done for a neglected disease
• Never done on a large scale production
  infrastructure

18
Applications Example MAGIC

• Major Atmospheric Gamma Imaging Cherenkov
  telescope (MAGIC)
• Origin of VHE ?-rays (30 GeV TeV)
• Active Galactic Nuclei (AGN)
• Supernova Remnants
• Unidentified EGRET sources
• Gamma Ray Bursts
• Huge hadronic background ? MC simulations
• to simulate the background of one night, 70 CPUs
  (P4 2GHz) need to run 19200 days
• Observation data are big too!
• MAGIC Grid
• Use the three national Grid centres as backbone
• All are members of EGEE
• MAGIC telescope II will start 2007 (scalability)
• ? Towards a virtual observatory for VHE ? -rays

19
EGEE User Forum

• 3-day workshop
• Presentations by users from different thematic
  areas
• EGEE presentations
• Goals
• Understand expectations, status and evolution
  across EGEE and related projects
• Promote cross-application fertilisation
• 1-3 March 2006
• CERN, Switzerland
• http//egee-intranet.web.cern.ch/egee-intranet/Use
  r-Forum/index.html

20
User information support

• More than 150 training events across many
  countries
• gt1200 people trained
• induction application developer advanced
  retreats
• Material archive online with 250 presentations
• Public and technical websites as well as
  dissemination material constantly evolving to
  expand information and keep it up to date
• 4 conferences organized
• 300 _at_ Cork
• 400 _at_ Den Haag
• 450 _at_ Athens
• 460 _at_ Pisa

21
EGEE and Industry

• Industry as
• partner to increase know-how on Grid
  technologies
• user for RD applications
• provider of established Grid services, such as
  call centres, support centres and computing
  resource provider centres
• Industry Forum
• Raise awareness of the project among industries
• Encourage businesses to participate
• ? ability to experience EGEE Grid in early
  stages

22
Why work with EGEE?

• A global Grid infrastructure helps to provide
  easier access for
• Small research groups
• Scientists from many different fields
• Remote and still developing countries
• to new technologies
• Produce and store massive amounts of data
• Transparent access to millions of files across
  different administrative domains
• Low cost access to large computing resources
• Mobilise quickly large amounts of CPU on short
  notice
• High-end facilities
• and helps to find new ways to collaborate
• Develop applications using distributed complex
  workflows
• Eases distributed collaborations
• New modes of community building
• Easier access to higher education

23
EGEE as partner

• Ongoing collaborations
• with non EU partners in EGEE US, Israel, Russia,
  Korea, Taiwan
• with other European infrastructure projects
• GÉANT
• DEISA
• SEE-GRID
• with other European grid projects
• DILIGENT
• GRIDCC
• with non-European projects
• OSG OpenScienceGrid (USA)
• NAREGI (Japan)
• EGEE as incubator
• gt10 related projects have been created

24
Related projects
25
Toward EGEE-II

• EGEE-II proposal submitted to the EU
• On 8 September 2005
• Proposed start 1 April 2006
• Natural continuation of EGEE
• Emphasis on providing an infrastructure for
  e-Science
• ? increased support for applications
• ? increased multidisciplinary Grid
  infrastructure
• ? more involvement from Industry
• Expanded consortium
• gt 90 partners in 32 countries (Non-European
  partners in USA, Korea and Taiwan)
• related projects
• world-wide Grid infrastructure
• increased international collaboration

26
Project Activities

• Similar structure to EGEE with increased focus
  on
• Services
• New countries joining
• New activity for sw integration and testing
• Networking
• Increase support for applications
• Reinforce outreach, dissemination and training
• Closer links to industry (CERN openlab project)
• Extend coverage to all regions
• Coordination
• Technical Coordination Group (TCG)
• User Information Group (UIG)

27
Project Activities (II)

• Networking Activities (NA1-NA5)
• Management, Dissemination, Training, User
  support, Cooperation
• ? Growth of the infrastructure user base
• Service Activities
• Operate and maintain a working, sustainable
  infrastructure
• New SA3 Integration, Testing and Certification
• ? EGEE-II as core service provider
• Joint Research Activities
• Reengineering of Grid middleware (more mature)
• Quality Assurance for whole projectincl.
  Security Coordination Group (SCG)

28
EGEE gt EGEE-II

• EGEE Project
• 01.04.04-31.03.06
• Number of Partners 71
• Countries 27
• Total Budget 32 M
• CE Federation Requested Contribution 1.7 M

• EGEE-II Proposal
• 01.04.06-31.03.08
• Number of Partners 92
• Countries 32
• Total Budget 37 M ( 15)
• CE Federation Requested Contribution 2.7 M (
  60)

29
The ACEs Model
CzechRepublic
Hungary
Slovenia
Slovakia
Poland
30
CE Federation Contribution

• CrossGrid Flood Application ported to gLite/EGEE
• Best Demo at 3rd EGEE Conference, Athens
• Selected for focused EGEE Review in December 2005
• VOCE
• Discussed as model for regional
  contribution/coordination
• glogin ? gLite
• P-GRADE as EGEE Portal
• Porting gLite to IA-64

31
Summary

• Grids represent powerful new tools for science
• The potential return to international scientific
  communities is enormous and opens the path for
  commercial and industrial applications
• Many applications are already benefiting from
  Grid technologies
• ? Window of opportunity to move grids from
  research to production, as networks did a few
  years ago
• EGEE offers
• A mechanism for linking together the people,
  resources and data of many scientific communities
• A set of middleware for gridfying applications
  with documentation, training and support
• Regular forums for linking with grid experts,
  other communities and industry
• EGEE-II will further extend support for user
  communities and applications
• ? Success will lead to the adoption of grids as
  the main computing infrastructure for science
• ? If we succeed then the potential return to
  international scientific communities will be
  enormous and open the path for commercial and
  industrial applications

32
Contacts

• EGEE Website
• http//www.eu-egee.org
• How to join
• http//public.eu-egee.org/join/
• EGEE Project Office
• projectoffice_at_eu-egee.org