GridLab 2003/4

1
GridLab 2003/4 Steady leadership in changing
times!
Jarek Nabrzyski Project Coordinator
Poznan Supercomputing and Networking Center
2
GridLab Project

• Funded by the EU (5 M), January 2002 December
  2004
• Application and Testbed oriented
• Cactus Code, Triana Workflow, all the other
  applications that want to be Grid-enabled
• Main goal to develop a Grid Application Toolkit
  (GAT) and set of grid services and tools
• Resource management (GRMS),
• Data management (GDMS),
• Monitoring (Mercury) and information services,
• Adaptive components (Pythia),
• Mobile user support and remote visualization,
• Security services (GAS),
• Portals (GridSphere),
• ... and test them on a real testbed with real
  applications

3
GridLab Members

• PSNC (Poznan) - coordination
• AEI (Potsdam)
• ZIB (Berlin)
• Univ. of Lecce
• Cardiff University
• Vrije Univ. (Amsterdam)
• SZTAKI (Budapest)
• Masaryk Univ. (Brno)
• NTUA (Athens)
• Sun Microsystems GmbH
• HP France
• ANL (Chicago, I. Foster)
• ISI (LA, C.Kesselman)
• UoWisconsin (M. Livny)

• collaborating with
• Users!
• EU Astrophysics Network,
• DFN TiKSL/GriKSL
• NSF ASC Project
• other Grid projects
• Globus, Condor,
• GrADS,
• PROGRESS, Clusterix
• GriPhyn/iVDGL,
• European Grid Projects (GRIDSTART)
• KISTI, Canadian Grid, LSU

4
Review Issues and Concerns

• It is GridLabs first direct contact with the new
  PO and 2 out of 3 reviewers are new to the
  project,
• Difficult to review a project of this size and
  complexity in several hours,
• Decision to present the project from the
  integration angle,
• Problems with synchronization of deliverables
  with the last-minute amendment (details in the WP
  presentations)
• Presentations Short time per workpackage, short
  time for the demo,
• Innovative GAT, the portal and services presented
  in a practical demo during SC2003 in Phoenix
• The more features we add (to every single WP out
  of 12) the more complex the integration gets

5
GridLab Aims

• Get Computational Scientists using the Grid and
  Grid services for real, everyday, production work
  (AEI Relativists, EU Network, Grav Wave Data
  Analysis, Cactus User Community, ...),
• Make it easier for applications to make flexible,
  efficient, robust, use of the resources available
  to their virtual organizations
• Dream up, prototype, and test new application
  scenarios which make adaptive, dynamic, wild, and
  futuristic uses of resources.

6
What GridLab isnt

• We are not developing low level Grid
  infrastructure - we do more actually, we want to
  be infrastructure independent,
• We do not want to repeat work which has already
  been done (want to incorporate and assimilate it
  )
• Globus APIs,
• OGSA/OGSI/,
• ASC Portal (GridSphere/Orbiter),
• GPDK,
• GridPort,
• DataGrid,
• GriPhyn,
• ...

7
Project Goals for 2003

• Integration of all the GridLab software,
• Zakopane Scenario driving all the developments
  and integration,
• First release of GridLab software scheduled for
  December 2003,
• Make the testbed more stable and ready for any
  demo action and/or development,
• SC2003 was chosen to demo the GridLab software,
• Strong dissemination of the project in order to
  get attention of interesting Grid applications,
• Start the GAT and other GridLab standardization
  work under GGF,
• Strengthen management of the project, with clear
  responsibilities to ensure better control of
  resources, fast identification of the risks,
  ensure progress,
• GridLab Open Source License,
• Prepare the plan for exploitation of the results,
• Adjust the Technical Description of Work to the
  current needs of the project

8
What have we done during last 8 months?

• Response to the reviewers report sent to all
• Open Source License
• GT2 vs. GT3 issues (finally it was wise decision)
• Strategy for dissemination in 2004
• Deliverables, some delays here (see details in
  the WP presentations)
• Contract amendment is on the way
• GridLab meeting in Eger (April) and Olomouc
  (October)
• More developments and integration
• More and more WPs brought to the scenario
• Today most of the WPs were integrated with each
  other
• SC2003 in Phoenix (2 big demos, many small
  demos)
• Painful demo, but successful GRMS portal GAT
  application Testbed WP5 information service
  Adaptive with some last minute problems with
  other services
• The devil is in the details - the most true
  sentence.

9
Major achievements

• Integration efforts and devotion of project
  staff,
• Meeting in Eger in March/April, meeting in
  Olomouc in October,
• Many small integration meetings
  (Poznan-Berlin-Potsdam, Potsdam-Budapest,
  Potsdam-Prague, Cardiff, GGF meetings and at
  other confs.),
• Getting a major attention towards the GAT of many
  application communities,
• GRIDSTART, GGF, application communities and 6FPP
  (EGEE, DEISA, HPC-Europa, )
• Expectations are high!
• Many successful implementations of services
• Single GridLab services being used by other
  projects,
• Most of the services are fully integrated with
  each other, but can be configured as stand alone
• GAT can connect to most of the services. All will
  be available in a month.

10
Major achievement (cont.)

• GridLab middleware being exploited by other
  projects and production testbeds
• GridSphere tens of projects (Grid Infrastructure
  Project, GEON, HPC-Europa, Physics Portal
  Development Project, )
• GRMS (Clusterix, SGI Grid, Progress, Canadian
  Grid)
• Mercury Monitor (DataGrid, P-GRADE)
• WP8 Visualization - GriKSL
• Open Source License
• GridSphere has its own Open Source license

11
Major achievementsStandards, collaboration

• Global Grid Forum
• Positions in GGF
• Alexander Reinefeld - GGF Advisory Board
• Jarek Nabrzyski step down from the SC and AD
  for Apps and now take care of the GSA RG),
• Ed Seidel, Thilo Kielman chairing the Apps RG
• Thilo Kileman chairing Grid CPR WG
• Andre Merzky chairing the Persistent Archive WG
  and secretary of Data Area and Persistent
  Environments
• GAT architecture put at the Apps and APM groups
• Many WG talks about GridLab
• Putting the GAT to the BOF in October 2003
• Very successful
• Participation in Gridstart!
• GridLab chairs at least half of the TWGs
• Other
• Projects GriPhyN, GRIKSL, CrossGrid, Reality
  Grid (and some others) have a certain level of
  collaboration with GridLab
• Companies Grid Systems, Platform Computing, IBM,
  HP, Sun

12
Major achievementsDissemination

• Improved the www.gridlab.org
• Several new papers accepted for books, journals
  and conferences
• Many talks have been given at major conferences
  and events
• Events organized
• GridLab Conference and Workshop in Eger,
  March-April 2003
• Grid Portals Conference, August 2003, Edinburgh
• Grid Application Workshop in Czestochowa,
  September 2003
• GridLab Conference and Workshop in Olomouc,
  October 2003
• Grid Resource Management Workshop, Poznan,
  October 2003
• Workshops at international conferences and
  symposia
• Tutorials on major conferences, also via
  AccessGrid
• 2 GridLab brochures
• In 2004 we will produce 14 different GridLab
  brochures featuring all the services, tools and
  advanced scenarios (see first three issues)

13
Book on Grid Resource Management

• 4 GridLab chapters
• Application requirements for resource brokering
• Multicriteria resource management (GRMS)
• Triana peer-to-peer RM
• Workflow management
• ISI/Argonne/Wicsonsin contributions
• Published in November by Kluwer Academic
  Publishers

14
Technical Progress

• GAT API specification and GAT adaptor developers
  guide
• GAT tested by Cactus and a GAT subset
  implementation for Triana (GAP)
• Strenghten the collaboration with ISI, ANL,
  Wisconsin
• New Portal Framework (GridSphere) - 1st release
  in 2003 (2nd in 2004) - being used by tens of
  projects!
• Participation in the Grid Technology Repository
  (Jarek Nabrzyski serves as the Int. Advisory
  Board member)
• Strong participation in the GRIDSTART project
  (GridLab was one of the main contributors to the
  Technical Roadmap)
• Services ready and interoperable in spite of
  changing requirements
• GAT ready for standardization process, already
  under the community comment process

15
Technical Progress (cont.)

• Extending the GridLab scenario
• Successful meeting in Eger (new application users
  involved, new developments) and in Olomouc
  (integration)
• Problems/Issues
• GAT development delayed - underestimated efforts
  needed for the GAT development
• Should be fully ready in a month
• Will be revisited and refactored! Users input is
  needed. Good docs is needed.
• Hiring a new person, experienced with Cactus
  development to speed up the CGAT integration.
• GAT documentation is behind! We are shifting
  efforts a bit to speed up the process.
• Some requirements and scenarios have changed
  dramatically new ones appeared (WP7)
• Some unexpected problems with hackers on one of
  the test machines

16
Issues

• GAT development delayed - underestimated efforts
  needed for the GAT development
• Should be fully ready in a month
• Will be revisited and refactored! Users input is
  needed. Good docs is needed.
• Hiring a new person, experienced with Cactus
  development to speed up the CGAT integration.
• GAT documentation is behind! We are shifting
  efforts a bit to speed up the process.
• Some deliverables are delayed
• New Annex1 caused some problems
• GAT development delayed (CGAT, GAT documentation)
• Some requirements and scenarios have changed
  dramatically new ones appeared (WP7)
• Some unexpected problems with hackers on one of
  the test machines

17
Plans for 2004

• Continue the development of GAT and services
• more info in the following presentations
• More complicated scenarios
• collaborative environments,
• submitting, controling and steering jobs from
  mobile devices,
• more dynamic behavior of applications
• Two more GridLab Workshop Meetings (Lecce 16-22
  May, Zakopane early December)
• Organizing the GridLab/GT3.2/WSRF integration
  meeting withUS partners (to ensure GAT
  compatibility)
• Preparing for the Supercomputing demos
• GGF BOF (GAT) and finally GGF WG
• GGF Scheduling Architecture Working Group now
  started (GRMS)
• Exploitation of the project results
• Close work with GridLabs commercial partners
• Global Grid Application Alliance with GridLabs
  leadership
• GGF activities (long term)
• GridSuit (based on GridLab plus Gridstart Open
  Source, PSNC plus partners)
• Open Source commercial support
• European Grid Support Centre (PSNC)

18
Summary

• GAT is important
• We will put yet more effort into the GAT
• GAT will be revisited after finalizing the 1st
  release (in one month)
• Refactoring
• Lecce Meeting to bring application people, TWG8,
  tutorial, exploitation
• Documentation!
• GGF/Gridstart
• There was a great interest and we are talking to
  many groups
• The project works better than ever!
• Project is unique in this sense (this might be an
  impression of ours only)
• Integration was much more difficult than expected
• We had to deal with many (too many) unexpected
  problems in 3rd party software.

19
Collaboration with ISI
20
Collaboration with Argonne

• OGSA C bindings
• GridLab/GT3.2 Inegration Workshop (4 Argonne GT3
  developers/experts are coming to help us with the
  integration)
• Help with developing the GT3.2 adaptors
• GAT review

21
Collaboration with Condor Group
22
Condor cont.
23
Exploitation Plan

• Jarek Nabrzyski

24
Pillars of exploitation

• Close work with GridLabs commercial partners
• Global Grid Application Alliance with GridLabs
  leadership
• GGF activities (long term)
• GridSuite (based on GridLab plus Gridstart Open
  Source, PSNC plus partners)
• European Grid Support Centre (PSNC, HP)
• European Grid Technology Broker

25
Exploitation of the results through commercial
partners

• HP
• Grid Resource Toolkit. HP investigates which GL
  software can be adopted.
• Distribution through server/software distribution
• Sun
• This meeting!
• Actions to be taken
• GridLab will work closely with the companies to
  ensure that the companies evaluate the software
  and make right decisions on which software is
  supposed to be part of their distributions
• Responsible WP managers will provide software
  plus documentation to a contact person at the
  vendor side.

26
Global Grid Application Alliance

• Global
• Europe - GridLab
• US - Ed Seidel
• Asia Pacific - both Satoshis
• Korea - KISTI
• Tasks
• Bring the GAT and services to the community
• Train people to make then use the GAT
• People will write adaptors for their favorite
  services
• Work on the new scenarios that the Alliance will
  bring
• Work on portable applications on a global scale
  using the GridLab technologies
• Actions
• The initial meeting of the founders took already
  place
• SC will be responsible for the actions meetings
  soon to be scheduled

27
GGF, GRIDSTART activities

• SAGA
• GSA
• and more will be continued beyond the timeframe
  of the project
• Gridstart TWGs will be used to work with other
  Grid projects in order to deploy the GAT and
  GridLab tools in these projects
• TWG chairs (from GridLab) are responsible to make
  it happen (GridLab WP managers, SC members)

28
GridSuite at PSNC

• GridSuite (based on GridLab plus Gridstart open
  source, PSNC plus partners) - Grid out-of-the-box
  complete solution
• Open Source commercial support
• PSNC has declared to hire a team of 6-8 persons
  to build the GridSuite this year
• PSNC will work with GridLab partners to provide
  an expertise to build the GridSuite and to design
  a business plan for the effort.
• Release roadmap is under way.
• This might be a background for the commercial
  company selling the Grid services for European
  industry
• Actions to be taken
• PSNC will discuss this with the GridLab Steering
  Committee soon,
• This will be more open to other GridLab partners
  (not limited to PSNC)
• Strategy is being developed by PSNC now
• Initial money from the European infrastructure
  funds (under way) and PSNCs own funds.

29
European Grid Support Centre

• European Grid Support Centre (HP, PSNC plus UK
  eScience, CERN, PDC)
• Institutions own funds
• The task is to support different user groups
  (currently DataGrid, EGEE, GridLab, NorduGrid)
• The tools and the procedures are now being
  established.
• The ground for GridLabs support for its users
  and infrastructure builders,
• Currently PSNC and HP from GridLab only. More
  will be involved with increase of the number of
  users.
• Actions
• Build the procedures and assure that GridLab
  users know about the Center and use its
  services,
• To be discussed by the SC in more details in 6
  weeks time,
• Extension of the group with a time and load
  (currently one person half-time at PSNC,
  similarly at HP, 3-4 persons at PSNC soon).

30
Exploitation by partners - general strategy

• Every GL partner will work with the users that
  the SC agrees to work with closely,
• In such a case particular institution will make
  sure that appropriate resources are assigned to
  work with the user in order to deploy GridLab
  software in the users environment,
• The appropriate resolutions will be worked out in
  details and decided by the Steering Committee by
  the existence of the GAT release, no later than
  by the end of April.
• At the beginning we will focus on strategic users
  to ensure the added value to the project.
• We will work closely with the GridLab application
  communities first of all (Cactus and Triana) on
  the GAT usage.

31
Beyond the project

• Seek for local funding by every partner
• Survive the gap after the GridLab
• Continue work, bring more users, get new
  requirements, analyze scenarios, work with the
  GGAA (Global Grid Application Alliance)