GAPI BoF APME Area Grid Application Programming Interface

1
G-API BoF (APME Area)Grid Application
Programming Interface

• Ian Taylor (U-Cardiff)
• Stephen Pickles (U-Manchester)
• Tom Goodale (LSU)
• Andre Merzky (ZIB)

2
Outline

• Motivation
• Present / Discuss Charter Draft
• The Seven Questions...
• Organization of WG
• Organization of Doc writing

3
Motivation

• Application Programmer Scenario (Ian Taylor)

4
Pseudo Code

• Target Languages
• Fortran
• C
• C
• Java

5
Copy a File -- Globus / Gass
int CopyFile (const char source,
const char target) globus_result_t
result globus_url_t
source_url globus_io_handle_t
dest_io_handle globus_ftp_client_operationat
tr_t source_ftp_attr globus_gass_transfer_requ
estattr_t source_gass_attr globus_gass_copy_att
r_t source_gass_copy_attr
globus_gass_copy_handle_t
gass_copy_handle globus_gass_copy_handleattr_t
gass_copy_handleattr globus_ftp_client_han
dleattr_t ftp_handleattr globus_io_attr_t
io_attr int
output_file -1 if (
globus_url_parse (source_URL, source_url) !
GLOBUS_SUCCESS ) printf ("can not parse
source_URL \"s\"\n", source_URL) return
(-1)
6
if ( source_url.scheme_type !
GLOBUS_URL_SCHEME_GSIFTP
source_url.scheme_type ! GLOBUS_URL_SCHEME_FTP
source_url.scheme_type !
GLOBUS_URL_SCHEME_HTTP
source_url.scheme_type ! GLOBUS_URL_SCHEME_HTTPS
) printf ("can not copy from s -
unsupported protocol\n", source_URL) return
(-1) globus_gass_copy_handleattr_init
(gass_copy_handleattr) globus_gass_copy_attr_i
nit (source_gass_copy_attr)
globus_ftp_client_handleattr_init
(ftp_handleattr) globus_io_fileattr_init
(io_attr) globus_gass_copy_attr_set_io
(source_gass_copy_attr, io_attr)
globus_gass_copy_handleattr_set_ftp_attr
(gass_copy_handleattr,
ftp_handleattr) globus_gass_copy_handle_init
(gass_copy_handle, gass_copy_handleattr)
if (source_url.scheme_type
GLOBUS_URL_SCHEME_GSIFTP
source_url.scheme_type GLOBUS_URL_SCHEME_FTP
) globus_ftp_client_operationattr_init
(source_ftp_attr) globus_gass_copy_attr_set_
ftp (source_gass_copy_attr,
source_ftp_attr)
else globus_gass_transfer_requestatt
r_init (source_gass_attr, source_url.scheme)
globus_gass_copy_attr_set_gass
(source_gass_copy_attr,
source_gass_attr)

7
output_file globus_libc_open ((char)
target, O_WRONLY O_TRUNC O_CREAT,
S_IRUSR S_IWUSR
S_IRGRP S_IWGRP) if ( output_file -1 )
printf ("could not open the destination
file \"s\"\n", target) return (-1)
if ( globus_io_file_posix_convert (output_file,
GLOBUS_NULL, dest_io_handle) !
GLOBUS_SUCCESS) printf ("Error converting
the file handle\n") return (-1)
result globus_gass_copy_register_url_to_handle
( gass_copy_handle,
(char)source_URL,
source_gass_copy_attr,
dest_io_handle, my_callback,
NULL) if ( result ! GLOBUS_SUCCESS
) printf ("error s\n",
globus_object_printable_to_string
(globus_error_get (result))) return (-1)
globus_url_destroy (source_url)
return (0)
8
Copy a File OGSA
Package org.globus.ogsa.gui import
java.io.BufferedReader import java.io.File impor
t java.io.FileReader import java.net.URL import
java.util.Date import java.util.Vector import
javax.xml.rpc.Stub import org.apache.axis.message
.MessageElement import org.apache.axis.utils.XMLU
tils import org.globus.axis.gsi.GSIConstants imp
ort org.globus.ogsa.ServiceProperties import
org.globus.ogsa.base.multirft.MultiFileRFTDefiniti
onServiceGridLocator import org.globus.ogsa.base.
multirft.RFTOptionsType import
org.globus.ogsa.base.multirft.RFTPortType import
org.globus.ogsa.base.multirft.TransferRequestEleme
nt import org.globus.ogsa.base.multirft.TransferR
equestType import org.globus.ogsa.base.multirft.T
ransferType
9
import org.globus.ogsa.impl.security.authenticatio
n.Constants import org.globus.ogsa.impl.security.
authorization.NoAuthorization import
org.globus.ogsa.utils.AnyHelper import
org.globus.ogsa.utils.GetOpts import
org.globus.ogsa.utils.GridServiceFactory import
org.globus.ogsa.utils.MessageUtils import
org.gridforum.ogsi.ExtendedDateTimeType import
org.gridforum.ogsi.ExtensibilityNotSupportedFaultT
ype import org.gridforum.ogsi.ExtensibilityType
import org.gridforum.ogsi.ExtensibilityTypeFaultTy
pe import org.gridforum.ogsi.Factory import
org.gridforum.ogsi.HandleType import
org.gridforum.ogsi.InfinityType import
org.gridforum.ogsi.LocatorType import
org.gridforum.ogsi.OGSIServiceGridLocator import
org.gridforum.ogsi.ServiceAlreadyExistsFaultType
import org.gridforum.ogsi.TerminationTimeType imp
ort org.gridforum.ogsi.WSDLReferenceType import
org.gridforum.ogsi.holders.ExtensibilityTypeHolder
import org.gridforum.ogsi.holders.LocatorTypeHol
der import org.gridforum.ogsi.holders.Termination
TimeTypeHolder import org.globus.gsi.proxy.Ignore
ProxyPolicyHandler import org.w3c.dom.Document i
mport org.w3c.dom.Element
10
public class RFTClient public static void
copy (String source_url, String target_url)
try File requestFile new File
(source_url) BufferedReader reader
null try reader new
BufferedReader (new FileReader (requestFile))
catch (java.io.FileNotFoundException fnfe)
Vector requestData new Vector ()
requestData.add (target_url)
TransferType transfers1 new
TransferTypetransferCount RFTOptionsType
multirftOptions new RFTOptionsType ()
multirftOptions.setBinary
(Boolean.valueOf (
(String)requestData.elementAt (0)).booleanValue
()) multirftOptions.setBlockSize
(Integer.valueOf (
(String)requestData.elementAt (1)).intValue
()) multirftOptions.setTcpBufferSize
(Integer.valueOf (
(String)requestData.elementAt (2)).intValue
()) multirftOptions.setNotpt
(Boolean.valueOf (
(String)requestData.elementAt (3)).booleanValue
()) multirftOptions.setParallelStreams
(Integer.valueOf (
(String)requestData.elementAt (4)).intValue
()) multirftOptions.setDcau(Boolean.valueOf
( (String)requestData.elementAt
(5)).booleanValue ())
11
int i 7 for (int j 0 j lt
transfers1.length j)
transfers1j new TransferType ()
transfers1j.setTransferId (j)
transfers1j.setSourceUrl
((String)requestData.elementAt (i))
transfers1j.setDestinationUrl
((String)requestData.elementAt (i))
transfers1j.setRftOptions
(multirftOptions)
TransferRequestType transferRequest new
TransferRequestType () transferRequest.setT
ransferArray (transfers1) int
concurrency Integer.valueOf
((String)requestData.elementAt(6)).intValue()
if (concurrency gt transfers1.length)
System.out.println ("Concurrency
should be less than the number"
"of transfers in the
request") System.exit (0)
12
transferRequest.setConcurrency
(concurrency) TransferRequestElement
requestElement new TransferRequestElement ()
requestElement.setTransferRequest
(transferRequest) ExtensibilityType
extension new ExtensibilityType ()
extension AnyHelper.getExtensibility
(requestElement) OGSIServiceGridLocator
factoryService new OGSIServiceGridLocator ()
Factory factory factoryService.getFactoryPor
t (new URL (source_url))
GridServiceFactory gridFactory new
GridServiceFactory (factory) LocatorType
locator gridFactory.createService (extension)
System.out.println ("Created an instance of
Multi-RFT") MultiFileRFTDefinitionServiceG
ridLocator loc new
MultiFileRFTDefinitionServiceGridLocator ()
RFTPortType rftPort loc.getMultiFileRFTDefiniti
onPort (locator)
((Stub)rftPort)._setProperty (Constants.AUTHORIZAT
ION,
NoAuthorization.getInstance())
((Stub)rftPort)._setProperty (GSIConstants.GSI_MOD
E,
GSIConstants.GSI_MODE_FULL_DELEG)
((Stub)rftPort)._setProperty (Constants.GSI_SEC_CO
NV,
Constants.SIGNATURE) ((Stub)rftPort)._setPr
operty (Constants.GRIM_POLICY_HANDLER,
new IgnoreProxyPolicyHand
ler ()) catch (Exception e)
System.err.println (MessageUtils.toString (e))

13
Copy a File G-API
include ltGAPI.hgt int CopyFile (const char
source_url, const char
target_url) try GAPI_File file
new GAPI_File (source_url) file-gtcopy
(target_url) catch (GATException e)
printf (e.ErrorString ()) return
(e.ErrorCode ()) return (0)
14
Coverage of API

• What Aspects covered
• Files and Streams
• Job submission
• Monitoring
• Steering
• ...
• What Aspects NOT covered
• MPI
• RPC
• ...

15
Draft Charter
The Grid Application API Working Group will
produce a GGF proposed recommendation for a high
level API specification to enable application
programmers to transparently recognise and
exploit a Grid environment. The proposed API will
provide a simple abstraction layer allowing
scientific applications easy access to the Grid
services necessary to grid-enable their
applications.
16
In the Grid community the demand for real
applications using the emerging Grid
infrastructure is steadily increasing. Over the
last years much of the Grid middleware advanced
from a purely experimental and research state to
mature and widely supported software systems. The
complexity of these systems, both in terms of
administration and application usability, remains
high. One of these aspects, the usability for
application programmers, is the focus of this WG.
17
The G-API will lower the barrier for scientific
application developers to make use of the grid by
providing a small, consistent API for the
operations of interest. It is not, however, a
goal of the G-API to cover the full range of
interactions of applications with the Grid, or to
cover all types of applications using Grids.
18
The G-API aims to simplify the utilisation of
Grids for those applications which require a
limited subset of Grid capabilities, and whose
developers do not want to get involved with Grid
technologies in great detail. We believe that
this is a common characteristic of a significant
fraction of scientific applications.
19
The general aim is to free application
pro-grammers from the need to directly code
against lower level Grid Services or libraries,
and the need to integrate complicated and
redundant software components reacting on the
dynamic nature of Grid environments. With an
abstract, application focused Grid API,
application programmers will be able to develop
portable software which can be effectively
deployed in both Grid and traditional
environments, and which will be able to take
advantage of different middleware and services
easily.
20
The goals of the group are to specify the needs
of application programmers in Grid environments,
and hence derive an abstract API specification,
and a set of language specific APIs derived from
this. The group will promote these APIs as GGF
recommendations. Typical operations an
application developer wants to invoke should be
enabled with a minimum of API calls, by hiding
complexities such as, for example, -
service discovery and utilisation -
resource management - QoS issues
- authorisation -
authentication - ...
21
For example, the process of copying of a remote
file may involve interaction with a replica
location service, a data transport service, a
resource management service, and a security
service. It may involve communication via
LDAP/LDIF, GRAM, HTTP, and GSI, as protocols or
protocol extensions. All the application
programmer should see are calls very much like
fileCopy (source, destination)
Although this example is simplified, it
illustrates the motivation for our work.
22
The APIs specified by this WG will deliver a
similar level of abstraction for a range of basic
operations which need to be grid aware, such as
- file access - communication primitives -
process management - monitoring - steering -
...
23
Specific Goals
The goal of the WG is to define a set of APIs as
GGF recommendations, which enable scientific
applications to be grid-enabled. The WG will
work with the application groups to identify
these operations, and the languages which should
be supported.
24
Milestones

• - Use cases and application scenarios survey
• - Requirements document
• - operations which should be supported
• - programming languages
• - organisation of API documents
• (e.g. one per subsystem, or one
  complete document)
• - Draft abstract API document(s)
• - Final abstract API document(s)
• - Draft language-specific API document(s)
• - Final language-specific API document(s)

25
The Seven Questions

• Focus
• Clarity/Relevance
• Necessity
• Appropriateness
• Overlap
• List of groups
• Critical Mass
• Demand

26
Focus

• See Charterapplication oriented API for
  scientific applications

Clarity / Relevance

• Simplify development and deployment of scientific
  applications on the Grid

27
Necessity

• We do not know of similar efforts.
• We do know application groups with demand.
• Groups are already working on this.

Appropriateness

• Clearly Grid-related API.
• Leverages GGF results to End-Users.

28
Overlap

• No overlap in Scope !!
• Common aspects with many GGF APIs.
• Need coordination with various GGF groups
• APPS
• ACE
• Life-Sciences
• Astronomy
• UPDT
• DRMAA
• Grid-RPC
• Grid-CPR
• ...

29
Critical Mass

• GridStart strong support at GS meeting
• RealityGrid eScience project in UK.
• GridLab 14 Institutions/Companies all over
  Europe.
• Support from individuals and projects.
• Support from the APPS-RG in GGF.

Demand

• Strong feedback from other EU projects.
• Strong feedback from related apps groups
  (Environmental, Physics, Biochemistry, Astronomy,
  ...).
• Gets developed anyway lets make it Standard!

30
Organization of WG

• Charter
• Milestones
• Chair(s)
• Secretar y,ies
• Mailing List majordomo_at_ggf.orgSubscribe gapi-wg
• Web Page
• Next Meeting

31
Name of Group

• Grid Application Programming Interface (G-API)
• Grid-Lite
• Application Level Grid Programming Interface
• Grid Capability Abstraction API
• Computational Science API
• ???

32
Organization of Doc Writing

• Use Cases and Application Scenario Survey
• G-API requirement specification
• G-API specification