GridLab: A Grid Application Toolkit and Testbed

1
GridLabA Grid Application Toolkit and Testbed

• Jaroslaw Nabrzyski
• GridLab Project Manager
• Poznan Supercomputing and Networking Center,
  Poland
• naber_at_man.poznan.pl
• office_at_gridlab.org

www.GridLab.org
2
GridLab Project

• EU Funded (5M)
• 2002-2004
• Application and Testbed orientated
• Applications and grid application developers are
  in the center of the project

3
GridLab Members

• Also collaborating with
• Users!!!
• (EU Astrophysics Network,
• High Performance Computing Group, Ottawa)
• DFN TiKSL/GriKSL
• NSF ASC Project
• GrADS Project
• GriPhyn/iVDGL

• PSNC (Poznan) - coordination
• AEI (Potsdam)
• ZIB (Berlin)
• Univ. of Lecce
• Cardiff University
• Vrije Univ. (Amsterdam)
• SZTAKI (Budapest)
• Masaryk Univ. (Brno)
• NTUA (Athens)
• Sun
• Compaq (HP)
• ANL (Chicago, I. Foster)
• ISI (LA, C.Kesselman)
• UoWisconsin (M. Livny)

4
GridLab on a map
5
GridLab Aims

• Get Computational Scientists using the Grid and
  Grid services for real, everyday, production work
  (AEI Relativists, EU Network, Grav Wave Data
  Analysis, Cactus User Community).
• Make it easier for applications to make flexible,
  efficient, robust, use of the resources available
  to their virtual organizations.
• Dream up, prototype, and test new application
  scenarios which make adaptive, dynamic, wild, and
  futuristic uses of resources.

6
What GridLab Isnt

• Dont want to develop low level Grid
  Infrastructure
• Dont want to repeat work which has already been
  done (want to incorporate and assimilate it
  Globus APIs, OGSA, ASC Portal (GridSphere/Orbiter)
  , GPDK, GridPort, DataGrid, GriPhyn)

7
GridLab end user requirements

• Give us an application oriented environment,
• Applications running on resources of one or more
  virtual organisations,
• Flexible, easy-to-use, simple interfaces to
  resources, jobs, and data (including compiling,
  tracking jobs, cataloguing data)
• Give us means to make efficient and effective use
  of resources,
• Fault tolerance is a key requirement,
• The ability to work in a disconnected
  environment,
• Mobile working,

8
GridLab end user requirements (cont)

• The ability to run in environments as
  minimalistic as laptops with no grid
  infrastructure to fully deployed Virtual
  Organisations
• Complexity should be hidden as much as possible,
• Provide a collaborative infrastructure,
• The infrastructure must cater for all classes of
  applications, from lightweight to largescale,
• The infrastructure must provide capabilities to
  customise choice of service implementation (e.g.
  using efficiency, reliability, first succeeding,
  all).

9
Solution

• Grid Application Toolkit or GAT
• Provides a layer between applications and
  emerging grid technologies. Provides an
  application developer orientated API, allowing
  the flexible use of different tools and services,
  as well as providing protection from developing
  software.
• GridLab Testbed/VO
• Diverse controllable environment for developing
  and testing applications and tools, software
  maintained by people who know it.

End Users
GAT Tool Developers
GAT-API Developers
Grid Infrastructure Developers
10
General Architecture
11
What is the GAT ?

• Set of application developer APIs for Grid tools,
  services and software libraries, (and example
  implementations) that support the development of
  grid-enabled aplications (open source!)
• Usable from any high level application (any
  generic code, Cactus, Triana, Portals, Scripts,
  )
• More or less
• Set of calls GAT_ToolOrService(arguments)
• Your chosen tools/services Resource broker,
  information server, application manager, grid
  monitoring, data manager, notification, etc, etc
• Set of APIs for dealing with the GAT
  (registration, information, errors, fault
  tolerance)

12
GAT Architecture
Cactus, Triana, Generic Codes, Portals, Scripts
Applications
GAT-API
High level calls based on functionality GAT_FileCo
py, GAT_ResourceFind
GridLab Services
Resource Broker, Monitoring System, Data Manager
Grid Core Services
GRAM, GridFTP, GridSSH, MDS but not limited to
Globus! (Triana and JXTA)
13
Proposed GAT Implementation
Applications
Cactus, Triana, Generic Codes, Portals, Scripts
GAT-API
GAT Engine
Library containing registration routines and
empty implementations
GAT Adaptors
Layer providing access to services
Grid Infrastructure (GAT/GridLabbase/core grid
services)
Resource brokers, basic grid services (GRAM,
GSIFTP, MDS)
14
GAT Example

• How to move the file?
• Available software
• User authentification
• Disk properties (disksize,
• user quota, inodes)
• Network bandwidth

Application (Cactus Migration Tool, User
Portal, Grid Script) wants to move a file between
two other machines
Machine A Gridftp Gridscp
Machine B Gridscp
15
GAT Example
GAT_FileCopy(GAT, ThisMachineThisFile, ThatMa
chineThatFile)
GAT API call from Cactus Application, or Portal,
or Python script
GAT Engine
Engine decides (by different means) how to
respect this request
GAT_GFTPAdaptor IGAT_FileCopy IGAT_FileMove IGAT_M
kDir
GridFTP Adaptor says it knows how to both Copy
and Move files between machines
GridSSH Adaptor only knows how to Copy files
GAT_GSCPAdaptor IGAT_FileCopy
16
Implementation

• GridLab is an early OGSA adopter, however
• We start with the web services implementation
  (all the GridLab services will be web services
  based and in the next step OGSA compliant)
• Several implementations of GAT will be developed

17
Workpackages

• WP1 Grid Application Toolkit (AEI)
• This is a key component of GridLab - link between
  Grid middleware and applications, usable by any
  conforming application or middleware component.
  Requiring input from, and connecting to, most
  other workpackages and components.
• WP2 Cactus Grid Application Toolkit (AEI)
• Provides an extended GAT interface for Cactus, a
  very general toolkit framework supporting
  different Grid applications, from astrophysics to
  chemical engineering. Cactus will be one of the
  primary application drivers for the GAT, and the
  project generally.
• WP3 Work-flow Application Toolkit (CARDIFF)
• Will develop Grid capabilities for a widely used
  dataflow programming environment, Triana, used in
  gravitational wave and other data analysis areas.

18
Workpackages (cont.)

• WP4 Grid Portals (AEI)
• Will be highly application driven, aimed at
  providing uniform, flexible and intuitive user
  access to Grid resources from anywhere, as well
  as administration tools for maintaining a Grid
  environment.
• WP5 Testbed management (MU)
• Will administrate and maintain an active
  development testbed across roughly a dozen EU
  sites (leveraging the work of the EGrid),
  deploying technologies as they are developed by
  the project. This workpackage will also
  coordinate with sites in the USA-based NCSA
  Alliance and others to test and develop
  interoperability.
• WP6 Security (PSNC)
• Will develop the required security mechanisms and
  will ensure the integration of all the
  technologies developed under other WPs, taking
  into account the various local security
  requirements and state of the art solutions.

19
Workpackages (cont.)

• WP7 Adaptive Application Components (VU)
• develops a set of components and APIs to be
  plugged into the toolkit, for example to take
  monitoring information and implement basic
  techniques for short-term forecasting and
  behavior adaptation/optimization.
• WP8 Data Handling and Visualization (ZIB)
• will provide Grid aware techniques for data
  management, analysis, and visualization, needed
  especially for applications that make use of
  multiple sites in a dynamic, time dependent
  manner, leaving data unpredictably scattered
  across the Grid.
• WP9 Resource Management (PSNC)
• will develop resource need estimators, resource
  brokers, and other tools, for both Grid users and
  the applications themselves to make intelligent
  decisions about which Grid resources should be
  used at any instant in the lifetime of a
  simulation.

20
Workpackages (cont.)

• WP10 Information Services (ISUFI)
• will extend existing Grid middleware toolkits
  with dynamic features needed by applications to
  select appropriate Grid resources and to provide
  simulation information to collaborative user
  groups.
• WP11 Monitoring (SZTAKI)
• will develop new components that will fit in the
  general Grid monitoring architecture to support
  application steering, adaptive monitoring, and
  automatic analysis and prediction of performance
  data.
• WP12 Access for mobile users (PSNC)
• will develop and test Grid access and monitoring
  technologies through a variety of mobile devices,

21
Workpackages (cont.)

• WP13 Information Dissemination and Exploitation
  (PSNC)
• will ensure the active dissemination of the
  project results through a variety of channels,
  including active participation in international
  organizations (e.g. GGF), co-development with
  other Grid projects in the USA and EU,
  participation in international conferences,
  training programs, instruction of GridLab
  technologies into various communities, and
  introduction into the commercial vendor world.
• WP14 Project Management (PSNC)
• day-to-day scientific, financial and
  administrative management of the project,
  including careful orchestration and monitoring of
  work across groups, major project decisions,
  liaisons with external projects and with the
  international advisory board, reporting

22
More info / summary

• www.GridLab.org
• gridlab_at_gridlab.org, news_at_gridlab.org
• office_at_gridlab.org
• Youre welcome to join our testbed
• We are open for collaborations!