Introduction to OGSA and OGSI

1
Introduction to OGSA and OGSI

• ? ??
• ???? ????

2
Objectives

• GT2 and Web Services
• OGSA OGSI
• WSDL GSDL
• Standard Interfaces Behaviors
• OGSA Examples
• Conclusion
• GGF and Resources

3
The Globus Toolkit v2 in One Slide

• Grid protocols (GSI, GRAM, ) enable resource
  sharing within virtual orgs toolkit provides
  reference implementation ( Globus Toolkit
  services)

• Protocols (and APIs) enable other tools and
  services for membership, discovery, data mgmt,
  workflow,

4
GT-Based Grid Tools Applications

• Data Grids
• Distributed management of large quantities of
  dataphysics, astronomy, engineering
• High-throughput computing
• Coordinated use of many computers
• Collaborative environments
• Authentication, resource discovery, and resource
  access
• Portals
• Thin client access to remote resources services
• And combinations of the above

5
Globus Toolkit Evaluation ()

• Good technical solutions for key problems, e.g.
• Authentication and authorization
• Resource discovery and monitoring
• Reliable remote service invocation
• High-performance remote data access
• This and good engineering is enabling progress
• Good quality reference implementation,
  multi-language support, interfaces to many
  systems, large user base, industrial support
• Growing community code base built on tools

6
Globus Toolkit Evaluation (-)

• Protocol deficiencies, e.g.
• Heterogeneous basis HTTP, LDAP, FTP
• No standard means of invocation, notification,
  error propagation, authorization, termination
• Significant missing functionality, e.g.
• Databases, sensors, instruments, workflow,
• Virtualization of end systems (hosting envs.)
• Little work on total system properties, e.g.
• Dependability, end-to-end QoS,
• Reasoning about system properties

7
Other Grid Tools Critique

• A wide variety of wonderful tools have been
  constructed
• Use of common Grid infrastructure (i.e. Globus
  Toolkit) has accelerated development and
  deployment
• - Still relatively little reuse of constituent
  components across projects
• - Interoperability does not extend to important
  issues like error notification

8
Grid Evolution Open Grid Services Architecture

• Refactor Globus protocol suite to enable common
  base and expose key capabilities
• Service orientation to virtualize resources and
  unify resources/services/information
• Embrace key Web services technologies for
  standard IDL, leverage commercial efforts
• Result standard interfaces behaviors for
  distributed system managementthe Grid Service
• Introduced at GGF4 in February 2002

9
OGSA Structure

• A standard substrate the Grid Service
• OGSI Open Grid Service Infrastructure
• Standard interfaces and behaviors that address
  key distributed system issues
• A refactoring and extension of the Globus Toolkit
  protocol suite
• supports standard service specifications
• Resource mgt, dbms, workflow, security,
• Target of current planned GGF efforts
• and arbitrary application-specific services
  based on these other definitions

10
Web Services

• Increasingly popular standards-based framework
  for accessing network applications
• W3C standardization
• Broad industry support
• Independent of implementation technology
• WSDL Web Services Description Language
• Interface Definition Language for Web services
• SOAP Simple Object Access Protocol
• XML-based RPC protocol
• Common WSDL target
• Many more specifications in the pipeline

11
Open Grid Services Infrastructure Transient
Service Instances

• Web services address discovery invocation of
  persistent services
• Interface to persistent state of entire
  enterprise
• In Grids, must also support transient service
  instances, created/destroyed dynamically
• Interfaces to the states of distributed
  activities
• e.g. workflow, video conf., dist. data analysis
• Significant implications for how services are
  managed, named, discovered, and used
• In fact, much of OGSA (and Grid) is concerned
  with the management of service instances

12
OGSI Grid Service Specification

• Defines WSDL conventions and GSDL extensions
• For describing and structuring services
• Working with W3C WSDL working group to drive GSDL
  extensions into WSDL
• Defines fundamental interfaces (using WSDL) and
  behaviors that define a Grid Service
• A unifying framework for interoperability
  establishment of total system properties

13
WSDL Conventions GSDL Extensions

• portType (WSDL v1.1)
• Defines an interface a named set of related
  operations
• serviceData (GSDL extension) data values
• serviceDataDescription (GSDL extension)
• description of data associated with service
• portType inheritance (WSDL v1.2 draft)
• Interface aggregation portType can comprise one
  or more other portTypes

14
Service Description

• Describes how client interacts with a service,
  independent of any particular service instance
• In WSDL, this is the portType (and the messages
  and types that it implies)
• Primary purposes
• Discovery find services of interest
• Tooling generate client proxies server code
• Any number of service instances may bind to a
  particular service description

15
Discovery

• Discovery drove many details of OGSI
• Examples Find me a service that
• supports a particular set of operations.
• can create a service that supports operations.
• will respond as I expect to an op request.
• I can use.
• is currently suspended waiting for input.
• has 10MB bandwidth to my machine.
• has 5ms latency to any copy of my database.
• has various combinations of these

16
Tooling

• Standard WSDL has most of what is needed for code
  generation
• Client proxies in various languages
• Server skeletons
• One critical missing bit portType inheritance
• Allows interface composition to define services
• WSDL v1.1 ltservicegt element is ambiguous about
  the relationship of its ports
• Are all ports in a service related?
• Mixes service definition with service binding
• But WSDL v1.2 draft has this!

17
Capturing Semantics

• Service description captures interface syntax
• But capturing semantic meaning is critical for
  discovery
• Not only does the service accept an operation
  request with a particular signature
• But it should also respond as expected
• As expected is usually defined offline in
  specifications
• Approach name everything
• Name implies semantics
• Semantics is either formally defined (e.g.
  semantic web), or informally (e.g. specs)

18
Fundamental Interfaces and Behaviors

• OGSI defines basic patterns of interaction, which
  can be combined with each other with custom
  patterns in a myriad of ways
• Grid Service Specification focuses on
• Atomic, composable patterns in the form of
  portTypes service data element types
• A model for how these are composed
• Complete service descriptions are left to other
  groups that are defining real services
• More on this later

19
The Grid Service Interfaces/Behaviors Service
Data
GridService (required)

• Required
• Introspection (service data)
• Explicit destruction
• Soft-state lifetime

other interfaces (optional)

• Optional
• Service creation- Notification
• Registration
• Collections
• application-specific interfaces

Service data element
Service data element
Service data element
Implementation

• Binding properties
• Authentication
• Reliable invocation
• Transactions
• QoS

Hosting environment/runtime (C, J2EE, .NET, )
20
Standard Interfaces Behaviors Four
Interrelated Concepts

• Naming and bindings
• Every service instance has a unique name, from
  which can discover supported bindings
• Lifecycle
• Service instances created by factories
• Destroyed explicitly or via soft state
• Information model
• Service data associated with Grid Service
  instances, operations for accessing this info
• Notification
• Interfaces for registering existence and
  delivering notifications

21
OGSA Interfaces and Operations Defined to Date

• GridService (Required)
• findServiceData
• destroy
• setTerminationTime
• NotificationSource
• subscribeServiceData
• SubscriptionManagement
• modifySubscription
• NotificationSink
• deliverNotification

• Factory
• createService
• Registration
• registerService
• HandleResolver
• findByHandle

22
Naming and Bindings

• Every service instance has unique and immutable
  names Grid Service Handle (GSH)
• Just a URI, where scheme defines resolver
• Handle must be resolved to a Grid Service
  Reference (GSR) to use service
• Includes binding information may expire
• Separation of name from implementation
  facilitates service evolution
• The HandleResolver interface allows client to ask
  a resolver to map from a GSH to a GSR
• Some handle schemes may not use this

23
Lifetime Management

• GS instances created by factory or manually
  destroyed explicitly or via soft state
• Negotiation of initial lifetime with a factory
  (service supporting Factory interface)
• GridService interface supports
• destroy operation for explicit destruction
• setTerminationTime operation for keepalive
• Soft state lifetime management avoids
• Explicit client teardown of complex state
• Resource leaks in hosting environments

24
Factory

• Factory interfaces createService operation
  creates a new Grid Service instance
• Reliable creation (once-and-only-once) with help
  from binding
• createService operation can be extended to accept
  service-specific creation parameters
• Returns a Grid Service Handle (GSH)
• A globally unique URL
• Uniquely identifies the instance for all time
• Based on name of a home handleMap service

25
Factories as Templates

• Factories has an extensibility argument that
  captures all the interesting, service specific,
  input/output parameters
• Service data used to describe what a particular
  factory supports in its extensibility arguments
• Single Factory portType is a template

26
Factories and Virtualization

• Consider a factory to create a given service
• createService expects particular input args
• GS interfaces permit various implementations,
  translucent to the client
• Simple factory might create service within its
  own hosting environment
• Factory might discover an appropriate resource to
  host the service, and delegate request to another
  factory
• Factory may decompose request into multiple
  service creations, create aggregating service

27
Service Data

• A Grid Service instance maintains a set of
  service data elements
• Described in WSDL extension
• XML element encapsulated in standard container
  name, type, lifetime, etc.
• Includes basic introspection information,
  interface-specific data, and application state
• Pull and push models for information query
• GridServicefindServiceData operation
• Pull queries this information via extensible
  query language
• NotificationSourcesubscribeServiceData
• Push Subscribe to notification of changes to
  information

28
Why Service Data?

• Discovery often requires instance-specific,
  perhaps dynamic information
• Service data offers a general solution
• Every service must support some common service
  data, and may support any additional service data
  desired
• Not just meta-data, but also instance state
• Part of the MDS-2 model contained in OGSI
• Defines standard data model, and query op
• Complements soft-state registration and
  notification

29
Service Data Lifetime Annotations

• goodFrom Declares the time from which the value
  of the SDE carried in its extensibility element
  is said to be valid. This is typically the time
  at which the contained element was created or
  aggregated.
• goodUntil Declares the time until which the
  value of the SDE carried in its extensibility
  elements is said to be valid. This value MUST be
  greater than the goodFrom time.
• availableUntil Declares the time until which
  this named SDE is expected to be available.
  Prior to this time, a client SHOULD be able to
  query for an updated value of this SDE. This
  value MUST be greater than the goodFrom time.

30
Service Data Example

• ltgsdlserviceData
• namemynsMyServiceData
• goodFrom2002-04-27T102000.000-0600
  
• goodUntil2002-04-27T112000.000-060
  0
• availableUntil2002-04-28T102000.000
  -0600gt
• ltn1e1gt
• ltn1e2gt
• abc
• lt/n1e2gt
• ltn1e3 gsdlgoodUntil2002-04-27T1030
  00.000-0600gt
• def
• lt/n1e3gt
• ltn1e4 gsdlavailableUntil2002-04-27T2
  02000.000-0600gt
• ghi
• lt/n1e4gt
• lt/n1e1gt
• lt/gsdlserviceDatagt

31
findServiceData

• Standard query operation against an instances
  service data elements
• Simple by name query language required
• Can support Xpath, Xquery, etc.
• Simple, extensible query operation
• Not meant to be the end-all, be-all of query
  interfaces
• Expect other groups to define query interfaces
  designed to handle other data types (e.g.
  relational), large responses (e.g. iterator-based
  interface), etc.

32
setServiceData

• Some service data is modifiable by client
• ltserviceData name modifiabletrue\gt
• setServiceData operation allows those SDEs to be
  set
• Service data values (i.e. state) may change as a
  result of other operations, or non-operation
  driven events
• Uses expressions to define nature of set
• Single SDE, multiple SDEs, delect SDE
• Extensible part of an SDE (XUpdate), etc.

33
subscribeServiceData

• Standard subscription operation against an
  instances service data elements
• Simple by name subscription language required
• e.g. Send me a message whenever this service
  data element changes
• Push model counterpart to the findServiceData
  pull model
• A persistent query against service data

34
Notification Interfaces

• NotificationSource for client subscription
• Subscription expression describes which service
  data element changes are of interest
• Creates a subscription manager service
• Manages the lifetime and properties of
  subscription
• NotificationSink for asynchronous delivery of
  notification messages
• Simple, flexible base with wide variety of uses
• Dynamic discovery/registry services, monitoring,
  application error notification,
• Intermediaries filter, aggregate, archive,
• Can integrate commercial messaging svcs

35
Subscription Expressions

• Any service that implements the
  NotificationSource must support
  subscribeByServiceDataName expressions
• ltgsdlsubscribeByServiceDataName
• nameqname // name from serviceData
  declaration
• minIntervalnonNegativeInteger?
• maxIntervalnonNegativeInteger? /gt
• What would richer query languages look like?
• How to express temporal aspects of a query when
  the subscription language is complex?
• Maybe related to RDBMS triggers?

36
Subscription Manager

• subscribeServiceData is really a factory
  operation it creates a service instance
• Instance supports SubscriptionManagement
• Manages lifetime of subscription through normal
  Grid Service (soft-state) lifetime management
• Allows subscription to be changed

37
Notification Example

• Notifications can be associated with any
  (authorized) service data elements

Grid Service
Grid Service
Notification Sink
DBaccess
Name, lifetime, etc.
Name, lifetime, etc.
Notification Source
DB info
38
Notification Example

• Notification subscription includes a subscription
  expression, describing which service data element
  change are of interest

Grid Service
Grid Service
Notification Sink
DBaccess
Notify me of new data about membrane proteins
Name, lifetime, etc.
Name, lifetime, etc.
Notification Source
DB info
39
Notification Example

• Creates subscription manager, and returns handle
  of manager to subscriber

Grid Service
Grid Service
Notification Sink
DBaccess
Handle to subscription manager
Name, lifetime, etc.
Name, lifetime, etc.
Notification Source
DB info
Grid Service
Name, lifetime, etc.
Subscription Management
Subscription info
40
Notification Example

• Use normal service lifetime management to manage
  subscription lifetime

Grid Service
Grid Service
Notification Sink
DBaccess
Name, lifetime, etc.
Name, lifetime, etc.
Notification Source
DB info
Grid Service
SetTerminationTime
Name, lifetime, etc.
Subscription Management
Subscription info
41
Notification Example

• Notification messages will be sent to Sink when
  service data changes

Grid Service
Grid Service
Notification Sink
DBaccess
Messages
Name, lifetime, etc.
Name, lifetime, etc.
Notification Source
DB info
Grid Service
Name, lifetime, etc.
Subscription Management
Subscription info
42
Registration

• The Registration interface may be used to
  register Grid Service instances with a registry
• A set of Grid Services can periodically register
  their GSHs into a registry service, to allow for
  discovery of services in that set
• Just registration of existence, not of
  information
• Registrations maintained in the service data of
  the registry service
• Standard discovery mechanisms can then be used to
  discover registered services
• Service data schema may be registry specific

43
Example Building Registries

• Options for building registries
• Need for radically different query capabilities
• Topology of services used for discovery
• e.g hierarchical, p2p
• Can illuminate important aspects of OGSI
• Composition of interfaces
• Service data
• Multiple protocol bindings

44
Architecting Registries

• There is no single registry that can serve all
  purposes
• A Virtual Organization (community) must architect
  registries that are appropriate to their
  needs
• But there are common primitives that can be used
  to architect many different registries
• Service data
• Notification
• Soft-state Registration

45
Need for Different Queries

• Need registries that can answer radically
  different queries
• Find me all Redhat Linux 7.2 machines which are
  available for my use with a load lt 0.3.
• Requires a registry that can deal with dynamic
  information
• Find me both an available cluster and my
  project database servers with good network
  connectivity between them.
• Requires a registry that can join information
  from multiple services

46
Use of Service Data

• A registrys service data should be architected
  to support query requirements
• Customized service data XML types
• More powerful (e.g. Xpath, Xquery) or custom
  query languages
• A registry is defined largely by its service data
  and query language

47
WS-Inspection

• Standard format for a list of services
• May be appropriate service data format for some
  simple registries

lt?xml version"1.0"?gt ltinspectionxmlns"http//s
chemas.xmlsoap.org/ws/2001/10/inspection/"gt
ltservicegt ltdescription
referencedNamespacehttp//schemas.xmlsoap.org/wsd
l/ location"http//example.com/stockquote.
wsdl" /gt lt/servicegt lt/inspectiongt
48
Discovery Topologies

• GS patterns can be applied in various ways to
  build discovery topologies
• Hierarchical with caching
• Hierarchical with forwarding
• Peer-to-peer mesh

49
Hierarchical with Caching

• Soft-state registration up the tree
• Various services in a VO are configured to send
  soft-state registrations to registries
• Registry services may be configured to send
  soft-state registrations to higher-level
  registries
• findServiceData Subscription down the tree
• Registry gathers information that it cares about
  from each registered service
• Registry caches that information as its service
  data, with some refresh policy

50
Hierarchical with Forwarding

• Soft-state registration up the tree
• Forwards query subscription down the tree
• When registry receives query from client, it
• Queries lower level registries services
• Collects results, formats them appropriately
• Replies to client query

51
Peer-to-peer Mesh

• Each service also acts as a registry
• It supports queries against service data that is
  appropriate for discovery
• Each service soft-state registers with a set of
  peers, to establish the mesh
• When queries, the service forwards the query to
  any services that are registered with it
• Perhaps caches any subsequent responses

52
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
BioDB 1
Compute Service Provider
User Application
. . .
. . .
I want to create a personal database containing
data on e.coli metabolism
Database Service
Database Factory
BioDB n
Storage Service Provider
53
OGSA Example Data Mining for Bioinformatics
Find me a data mining service, and somewhere to
store data
Community Registry
Mining Factory
Database Service
BioDB 1
Compute Service Provider
User Application
. . .
. . .
Database Service
Database Factory
BioDB n
Storage Service Provider
54
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
GSHs for Mining and Database factories
BioDB 1
Compute Service Provider
User Application
. . .
. . .
Database Service
Database Factory
BioDB n
Storage Service Provider
55
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
Create a data mining service with initial
lifetime 10
BioDB 1
Compute Service Provider
User Application
. . .
. . .
Create a database with initial lifetime 1000
Database Service
Database Factory
BioDB n
Storage Service Provider
56
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
Create a data mining service with initial
lifetime 10
BioDB 1
Miner
Compute Service Provider
User Application
. . .
. . .
Create a database with initial lifetime 1000
Database Service
Database Factory
BioDB n
Database
Storage Service Provider
57
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
Query
BioDB 1
Miner
Compute Service Provider
User Application
. . .
. . .
Query
Database Service
Database Factory
BioDB n
Database
Storage Service Provider
58
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
Query
BioDB 1
Miner
Keepalive
Compute Service Provider
User Application
. . .
. . .
Query
Database Service
Database Factory
Keepalive
BioDB n
Database
Storage Service Provider
59
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
BioDB 1
Miner
Keepalive
Compute Service Provider
User Application
. . .
. . .
Results
Database Service
Database Factory
Keepalive
Results
BioDB n
Database
Storage Service Provider
60
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
BioDB 1
Miner
Compute Service Provider
User Application
. . .
. . .
Database Service
Database Factory
Keepalive
BioDB n
Database
Storage Service Provider
61
OGSA Example Data Mining for Bioinformatics
Community Registry
Mining Factory
Database Service
BioDB 1
Compute Service Provider
User Application
. . .
. . .
Database Service
Database Factory
Keepalive
BioDB n
Database
Storage Service Provider
62
Conclusion

• OGSI defines
• How to describe Grid Services using WSDL
  conventions and GSDL extensions
• Fundamental interfaces (using WSDL) and behaviors
  that define a Grid Service
• OGSI used to define stateful service instances
  for a myriad of uses
• OGSA defines a standard set of OGSI-compliant
  interfaces for various functions

63
OGSA Standards Efforts in GGF

• OGSI (IInfrastructure)
• Defining core Grid Service Specification
• (At least) three implementation efforts
• Other WGs formed or being formed
• OGSA architecture
• OGSA Platform model binding
• OGSA Security architecture roadmap
• Initially based on WS Security, expect evolution
• OGSA Database Access and Integration
• OGSA resource models based on CIM schema
• Resource usage records and protocols
• Java Interfaces for OGSI

64
Grid Services Resources

• OGSI documents
• www.gridforum.org/ogsi-wg
• OGSA documents
• www.gridforum.org/ogsa-wg
• Globus Toolkit v3 implementation
• www.globus.org