SZTAKI results with respect to the UK OGSA Testbed project

1
SZTAKI results with respect to the UK OGSA
Test-bed project

• Gergely Sipos, Szabolcs Illés, Szabolcs Nemeth,
  Tamas Boczko, Gabor Dozsa and Peter Kacsuk
• 2004. aprilis

2
The common goal of UoW and SZTAKI
The Grid
Grid service
Grid service

Grid service
P-GRADE Portal
Legacy code
GEMLCA
3
The common goal of UoW and SZTAKI
The Grid
This problem to be solved by SZTAKI
Grid service
Grid service

Grid service
P-GRADE Portal
These services can be Web, Jini, Globus,
services
Such a client program is really the definition of
a GS workflow.
Legacy code
GEMLCA
This problem to be solved by UoW
4
The deployment, management and usage of a Grid
service
P-GRADE/ P-GRADE Portal
5
The deployment, management and usage of a Grid
service
P-GRADE/ P-GRADE Portal
A Grid resource
WS/Jini/GTx wrapper service
Sequential or parallel program
Its frame can be generated by P-GRADE, but the
program logic must be provided by the user.
6
A real scenario from the future 1.
P-GRADE Portal
A parallel PVM job that can find the maximum of a
function given as an input parameter
Genetic algoritm
site boundary
7
A real scenario from the future 1.
P-GRADE Portal
Grid resource
Genetic algoritm WS wrapper
Genetic algoritm
The job does not start yet! It starts only when
its wrapper receives an input function from a
client!
site boundary
UDDI registry
8
The solution
P-GRADE/ P-GRADE Portal
GS-job pair hosting provider
Job Wrapper GS
Sequential or parallel program
GS client
9
The first operating version
10
General properties of the version

• A bash script that can generate a GT3 wrapper
  service for a P-GRADE MPI program
• The GT3 wrapper GS can start its back-end job
  with the input files sent by the client.
• The service can send back output files of the job
  to the client.

11
The system during operation
Developers host
Generator script
Submitter script template
Default job config
GT3 GS template
12
The system during operation
Developers host
n0
Parallel MPI program
Generator script
Submitter script
Deployment by hand
Parallel MPI program
Actual job config
start
Submitter script template
Default job config
Wrapper GT3 GS
Parallel MPI program
Archived input files
GT3 GS template
NFS
Input files
nN
n1
NFS
NFS
Client host
Input files
GT3 GS client
MPI process
MPI process
Local FS

13
The system during operation
Developers host
n0
Parallel MPI program
Generator script
Submitter script
Deployment by hand
Parallel MPI program
Actual job config
start
Submitter script template
Default job config
Wrapper GT3 GS
Parallel MPI program
Archived input files
GT3 GS template
NFS
Input files
nN
n1
NFS
NFS
Client host
Input files
GT3 GS client
Local FS

14
Defects of the version

• Only MPI jobs can be wrapped, and only into GT3
  services
• (consequence GT3 has to be installed on the
  developers host)
• The GS submitter script job triplet has to
  be deployed manually
• The GS does not apply BatchExecutor service to
  execute the job, it uses directly local
  resources.
• Only an appropriate client program can use the
  generated GS (the GS uses private protocol for
  file transfer)
• The client is blocked while the job runs

15
Status of the implementation

• A Jini based Batch job executor service is
  already developed
• It wraps a Condor pool into a Jini service
• It can execute P-GRADE jobs with given input
  parameters, files, environm.
• It can transfer the job input files and the
  result files between the Condor pool and the
  submitter entity
• In GT3 environments the MMJFS is this batch
  executor service.
• Norbert Purger is almost finished a WS for the
  same purpose.

16
The P-GRADE Portal
17
The layers of the service-oriented global Grid
Provides interfaces to grid services
Client browser
Client browser
Client browser
Client browser
Client browser
Client browser




Grid Portal
Grid Portal
Grid Portal
Makes worldwide discovery possible
Global information system
Make local service discovery possible
grid information system

GS
GS
grid information system
grid information system




GS
GS
GS
GS
GS
GS
grid-level functionalities
Internet
Communication layer
host
host
host
host
host
host
host
host
host
host

The provider devices
18
General properties of the P-GRADE Portal

• Portal for computational grids
• Supported services functionalities
• GT2 GRAM job execution
• Mercury job monitoring
• PROVE execution visualization
• MyProxy proxy credential management
• Contains a built-in workflow editor

19
The layers of the P-GRADE Portal today
Client browser
P-GRADE Portal
GT2 GRAM
GT2 GRAM

Mercury PROVE
Mercury PROVE

MyProxy
MyProxy

The supported technologies
Internet
host
host
host
host
host
host
host
host
host
host

20
The layers of the P-GRADE Portal today
Client browser
Tomcat
GridSphere
Makes the low-level protocols invisible
Java 2 SDK API
Mercury Java API
CoG Java API
shell scripts
DAGMan
Condor-G
GT2 GRAM
GT2 GRAM

Mercury PROVE
Mercury PROVE

MyProxy
MyProxy

Internet
host
host
host
host
host
host
host
host
host
host

21
Workflow support by the P-GRADE portal
Job Monitoring
Workflow Monitoring
22
Further Developments 1.
Client browser
P-GRADE Portal (computational portal)
Connect the portal tonew services that apply
different technologies
GT2 Job executor
Mercury PROVE
MyProxy certificate storage
GT3 Job executor
GEMLCA
Internet
host
host
host
host
host
host
host
host

23
Further Developments 2.
Client browser
P-GRADE Portal (computational Grid portal)
Make the Portal cooperate with information
system(s)
Information system
GT2/GT3 Job executor
GEMLCA
MyProxy certificate storage
Internet
host
host
host
host
host
host
host
host

24
PROVE as Grid Service
25
Prove as servlets
Client host
Java applet
Local or remote file
Prove service host
Servlet engine
Java servlets
HTTP server
C library
26
Evaluation of the servlet implementation

• Advantages
• no special client side program has to be
  installed gt a java-enabled web browser is enough
  at the client.
• Useful to apply in Grid portals gt this version
  is used in the P-GRADE portal
• Drawbacks
• servlet containers provide stateless connection
  so they always initiate a new servlet instance
  for every incoming request gt large overhead in
  case of multiple requests
• There is no notification infrastructure gt update
  initiated from the collector service is
  impossible

27
Prove as Grid Service
Client host
GT3
Graphical Java application
Prove service host
GT3 container
Prove Factory service
HTTP server
Prove service
Local or remote file
C library
GT3
28
Advantages of the GT3 implementation

• GT3 containers provide stateful connection gt no
  need to initiate a new PROVE server instance for
  multiple requests gt zooming is much simpler than
  in the servlet case
• There is a notification infrastructure gt update
  initiated from the collector service is possible

29
Sunnary Main results

• GT-3 version of P-GRADE
• P-GRADE portal
• GT-3 and WS version of the PROVE performance and
  execution visualization tool