Title: EU DataGrid progress and EGEE plans
1EU DataGrid progress and EGEE plans
- Fabrizio Gagliardi
- EDG Project Leader - EGEE Proposal Coordinator
- Fabrizio.Gagliardi_at_cern.ch
2The EU DataGrid Project
- 9.8 M Euros EU funding over 3 years, twice as
much from partners - 90 for middleware and applications (HEP, Earth
Obs. and Bio Med.) - Three year phased developments demos
(2001-2003) - 2nd annual project review successfully passed in
Feb 2003! - Total of 21 partners
- Research and Academic institutes as well as
industrial companies - Related projects and activities
- DataTAG (2002-2003)
- CrossGrid (2002-2004)
- GRIDSTART (2002-2004)
- Grace (2002-2004)
3Applications
- Genomic Exploration
- Earth Observation
- High Energy Physics
more and more scientists begin to use the EDG
middleware and testbed, relying on Grid
technology to solve huge data challenges
4Project priorities refocused at EDG5
After initial middleware development and testbed
deployment, effort has been refocused on quality
and stability
- Quality Policy Statement published
- http//eu-datagrid.web.cern.ch/eu-datagrid/WP12/de
fault.htm - List of priorities defined at a project retreat
- http//documents.cern.ch/age?a021130
- Followed-up at the Budapest project conference
- http//www.tomiexpress.hu/datagrid/
- Show-stoppers found by users on the application
testbed were the highest priority - Incremental improvements driven by the needs of
the applications (HEPCAL, Bio and EO use cases)
5 Atlas CMS Evaluations
- RESULTS
- Could distribute and run CMS s/w in EDG
environment - Generated 250K events for physics with 10,000
jobs in 3 week period - OBSERVATIONS
- Were able to quickly add new sites to provide
extra resources - Fast turnaround in bug fixing and installing new
software - Test was labour intensive (since software was
developing and the overall system was fragile) - EDG 2.0 should fix the major problems providing a
system suitable for full integration in
distributed production
- RESULTS
- Atlas software was used in the EDG Grid
environment - Several hundred simulation jobs of length 4-24
hours were executed , data was replicated using
grid tools - Results of simulation agreed with non-Grid
runs - OBSERVATIONS
- Good interaction with EDG middleware providers
and with WP6/8 - With a substantial effort it was possible to
perform the jobs - Showed up bugs and performance limitations (fixed
or to be fixed in EDG 2.0) - We need EDG 2.0 release for use in large scale
data challenges
6EDG Application Testbed for the EU review
- Since Last Year
- Improved software (EDG 1.4.3).
- Doubled sites. More waiting
- Australia, Taiwan, USA (U. Wisc.), UK Sites,
INFN, French sites, CrossGrid, - Significantly more CPU/Storage.
- Hidden Infrastructure
- MDS Hierarchy, Resource Brokers, User Interfaces,
VO Replica Catalogs, VO Membership Servers,
Certificate Authorities
7EDG GEANT
- Connections of the different nodes of the EDG
testbed are made possible by the EU-funded GEANT
project
- connecting more than 30 countries across
Europe - speeds of up to 10 Gbit/s
- high data throughput
- Quality of Service
- EDG and GEANT the first major production
quality tests of the network
- speed
- reliability
- monitoring capabilities
8Related Grid Projects
Through links with sister projects, there is
the potential for a truely global scientific
applications grid
9Current Plans
- Second EU review successfully passed in February
2003 - Major re-orientation of the project towards
production accepted - Need to develop further plans in view of future
project EGEE - Continue collaboration with US and across
Atlantic testbeds (with DataTAG) - Need to accommodate other applications
- Plan long term support of EDG developments
- Major proposal for further EU funding (EGEE)
10EGEE visionEnabling Grids for E-science in
Europe
- Goal
- Create a wide European Grid production quality
infrastructure on top of present and future EU RN
infrastructure - Build on
- EU and EU member states major investments in Grid
Technology - International connections (US and AP)
- Several pioneering prototype results
- Larg Grid development team (gt60 people)
- Requires major EU funding effort
- Approach
- Leverage current and planned national and
regional Grid programmes (e.g. LCG) - Work closely with relevant industrial Grid
developers, NRENs and US-AP projects
Applications
EGEE
Geant network
11Why EGEE? The Historical Analogy
- Prior to the EU Geant program ,there was in
Europe a multitude of exploratory projects in
networking technology. Geant was truly production
oriented, and brought European telecom operators
actively into the picture - In a similar way, EGEE can ensure preservation
of current investments in European Grid RD,
extending the present infrastructure and
focussing all activities towards establishing a
production quality Grid
12Why EGEE? The Societal Impact
Access to a production quality GRID will change
the way science and much else is done in Europe
An international network of scientist will be
able to model a new flood of the Danube in real
time, using meteorological and geological data
from several centers across Europe.
A team of engineering students will be able to
run the latest 3D rendering programs from their
laptops using the Grid.
A geneticist at a conference, inspired by a talk
she hears, will be able to launch a complex
biomolecular simulation from her mobile phone.
13Why EGEE? The Political Context
- Current Grid RD projects run out within 18
months - The EGEE partners have already made major
progress in aligning national and regional Grid
RD efforts, in preparation for EGEE - Launching EGEE now will preserve the current
strong momentum of the European Grid community,
and the enthusiasm of the hundreds of young
European researchers already involved in EU Grid
projects (gt150 in EDG only)
14EGEE Approach
- Most of partners built national and regional Grid
federations to participate in EGEE - Condition to participate in EGEE is to have
already an established Grid activity or be an
established Grid technology centre - EGEE overall project funding needs will require
staggered proposals to respond to several
separate EU calls - EU FP6 Communication Network Development Call
opened on December 17th, proposal submitted on
May 6th
15The EGEE consortium of Grid Federations
Total of 70 full partners covering entire EU and
beyond
16Integrated Infrastructure Initiative (I3)
- Response to FP6 call Communication Network
Development Grids - Three lines of EU funding (with current funding
breakdown) - Networking Activities
- Management coordination (7 of total funding)
- Dissemination and outreach (7 of total funding)
- Application support (10 of total funding)
- Specific Service Activities
- Provision and procurement of Grid services (52
of total funding) - Joint Research Activity
- Engineering development to improve the services
provided by the Grid infrastructure (24 of total
funding)
17Networking activities
General management and coordination of the
project User groups, Industry Forum Dissemination,
outreach, training and pilot applications
18Networking activities
- User Training and Induction User Support and
Consultancy Applications Interface. The lead
partner for the Training activity is the UK
(Edinburgh) - Dissemination and Outreach The lead partner for
the Dissemination activity is the European
Research Network organisation Terena (Amsterdam) - Both lead partners will rely on a supporting
network in the partner regions, partially funded
by the project. The User Support and Consultancy
is envisaged as a distributed effort
19Networking activities
- The Applications Interface Activity has three
components - two Pilot Application Centres high energy
physics (LCG www.cern.ch/lcg ) and bio/medical - one more generic component dealing with the
longer term recruitment and support of other
communities - The Pilot Application Centres are based at CERN
and in France, with participation to their
activities by other partners envisaged - The third, more generic component is distributed
more evenly among the partners, in order to
ensure as broad a contact with scientific
communities as possible
20Specific service activities
- Integration of national and international Grid
infrastructures
21Specific service activities
- The structure of the Grid services will
comprise - EGEE Operations Management at CERN
- EGEE Core Infrastructure Centres in the UK,
France, Italy, Germany and CERN (leveraging LCG
at the start), responsible for managing the
overall Grid infrastructure - Regional Operations Centres, responsible for
coordinating regional resources, regional
deployment and support of services in all other
countries
22Joint research activity
- Hardening and re-engineering of Middleware
23Joint research activity
- Middleware Centres for key services Resource
Access (Italy) Data Management (CERN)
Information Collection and Accounting (UK)
Resource Brokering (Italy) Closely connected to
this middleware development is a Quality
Assurance team (France) and a Grid Security team
(Northern Europe Consortium). A Middleware
Integration team and Middleware Testing Centre
will be located at CERN. - Middleware services and related support teams
require critical mass. With matching funding
from the host institutions, this translates to
over 40 FTEs for all middleware activities
(compare with about 60 FTEs in EDG).
24Summary of budget requirements for EGEE
- EGEE will integrate existing computer resources,
no hardware requests. The bulk of the budget is
for human resources - Building strong teams where already existing
expertise, to progress as quickly as possible.
Host partners for these teams will provide
matching funding and demonstrate adequate
expertise for the job - The partners estimated a total funding need of
about 100 M corresponding to about 33 M EU
funding. Not included in this are US and Russian
contributions (4 M ), and overall project
management, corresponding to 7 of the EU funding
over two years
25EGEE and Industry
- Industrial participation encouraged both as
potential end-users and IT technology and service
suppliers - Normally through national and regional Grid EGEE
federations - EGEE will maintain an Industry Forum to keep
selected Industrial and Commercial interested
parties in close contact - Services developed in first EGEE 2 years phase
(2004-5) might be tendered to Industry in second
phase (2006-7)
26Conclusions
- The EU DataGrid project has successfully
fulfilled its role of EU Grid flagship project in
collaboration with several other EU and
international projects - Essential to keep the momentum and the current
lead in production Grids in Europe - Important to build an international cooperation
between European and US/AP Grid infrastructure
projects - The scientific user communities are already
international (HEP is an excellent example) and
so the computing resources and most of the
experimental instruments - EGEE proposes the right framework and plans to
accomplish the above objectives - EGEE executive summary document linked to the
last project conference agenda (http//agenda.cern
.ch/fullAgenda.php?idaa03816)