Grid%20Middleware%20and%20Globus%20Toolkit%20Architecture

1
Grid Middleware and Globus Toolkit Architecture
Lisa Childers Argonne National Laboratory Universi
ty of Chicago
2
Overview

• Grid Middleware
• The problem supporting Virtual Organizations
• Requirements
• Capabilities
• The Globus Approach

3

• A fundamental problem of Grid Computingsupporti
  ng virtual organizations

4
Virtual Organizations
5
Virtual Organizations

• Distributed resources and people

R
R
R
R
R
R
R
R
R
R
R
R
R
6
Virtual Organizations

• Distributed resources and people
• Linked by networks, crossing admin domains

R
R
R
R
R
R
R
R
R
R
R
R
R
R
7
Virtual Organizations

• Distributed resources and people
• Linked by networks, crossing admin domains
• Sharing resources, common goals

R
R
R
R
R
R
R
R
R
R
R
R
R
R
VO-A
VO-B
8
Virtual Organizations

• Distributed resources and people
• Linked by networks, crossing admin domains
• Sharing resources, common goals
• Dynamic

R
R
R
R
R
R
R
R
R
R
R
R
R
R
VO-A
VO-B
9
Virtual Organizations

• Distributed resources and people
• Linked by networks, crossing admin domains
• Sharing resources, common goals
• Dynamic
• Fault tolerant

R
R
R
R
R
R
R
R
R
R
R
R
R
R
VO-A
VO-B
10

• movie time!
• (a short tour of the accelerator at CERN)

11

• 10,000 Encyclopedia Britannica's per second

12
Overview

• Grid Middleware
• The problem supporting Virtual Organizations
• Requirements
• Capabilities
• The Globus Approach

13
Support for Heterogeneous Environments

• Resource virtualization
• Common management capabilities
• Attribute-based resource discovery
• Standard protocols and schemas

14
Cross-Organizational Resource Sharing

• Global namespace
• Metadata services
• Site autonomy
• Resource usage data

15
Optimization of Resource Usage

• Dynamic resource allocation(supply-side
  management)
• Dynamic workload prioritization(demand-side
  management)

16
Quality of Service (QoS)Assurance

• Service-level agreement and attainment
• Migration

17
Administration

• Policy-based management mechanisms
• Problem-determination mechanisms
• Scalable management architecture

18
High Availability

• Disaster recovery mechanisms
• Mechanisms for fault management

19
Job Execution

• Support for a variety of remote job types
• Remote job management
• Job scheduling
• Resource provisioning

20
Data Services

• Abstractions that provide uniform access and
  integration to various types of data
• Mechanisms to keep data consistent across
  replicas and caches
• Mechanisms to persist data
• Mechanisms for data location management

21
Security

• Authentication and authorization mechanisms
• Support for multiple security infrastructures
• Perimeter security mechanisms that support local
  infrastructure
• Isolation
• Delegation of access rights
• Support for dynamic negotiation of security
  policies
• Monitoring in support of intrusion detection and
  secure logging

22
Overview

• Grid Middleware
• The problem supporting Virtual Organizations
• Requirements
• Capabilities
• The Globus Approach

23
Grid Infrastructure a Conceptual View
24
Execution Services

• To instantiate and manage units of work
• Behavior includes
• Finding and selecting execution candidate
  locations
• Preparing for execution
• Initiating execution
• Managing execution

25
Data Services

• To move, access and manage data resources
• Behavior includes
• Move data
• Manage replicated copies
• Run queries and updates
• Transform data into new formats
• Maintain metadata

26
Resource Management Services

• Management of low-level resources
• Monitoring, setup and control, discovery
• Management of the capabilities
• Functional interface management (e.g. create and
  destroy jobs)
• Domain-independent management
• System management of the many services on the
  Grid (e.g., system-wide backups)

27
Security Services

• To facilitate the enforcement of security-related
  policy within Virtual Organizations
• Behavior includes
• Verifying proof of an asserted identity
• Identity mapping
• Policy-based access control decision-making
• Audit and secure logging
• Privacy

28
Self-Management Services

• To reduce the cost and complexity of owning and
  operating IT infrastructure
• Behavior includes
• Negotiating mechanisms for resource usage

29
Information Services

• To efficiently access information about
  applications, resources and services
• Behavior includes
• Monitor services and resources
• Log system activities
• Cache and publish metadata

30
Overview

• Grid Middleware
• The problem supporting Virtual Organizations
• Requirements
• Capabilities
• The Globus Approach

31
The Role of the Globus Toolkit

• The Globus Toolkit is a collection of solutions
  to problems commonly found in collaborative
  distributed applications
• Heterogeneity
• A focus, in particular, on wrapping heterogeneity
  for application developers
• Abstractions
• Supporting general-case patterns and
  interactions, not specific to a particular
  application domain
• Standards
• We capitalize on and encourage use of existing
  standards (IETF, W3C, OASIS, GGF)
• GT also includes reference implementations of
  new/proposed standards in these organizations

32
Layers in the Grid
33
Without the Globus Toolkit
ComputeServer
A
SimulationTool
ComputeServer
B
WebBrowser
WebPortal
RegistrationService
Camera
TelepresenceMonitor
DataViewerTool
Camera
Application Developer 10
Off the Shelf 12
Globus Toolkit 0
Grid Community 0
Database service
C
ChatTool
DataCatalog
Database service
D
CredentialRepository
Database service
E
Certificate authority
Resources implement standard access management
interfaces
Collective services aggregate /or virtualize
resources
Users work with client applications
Application services organize VOs enable access
to other services
34
With the Globus Toolkit
ComputeServer
GlobusGRAM
SimulationTool
ComputeServer
GlobusGRAM
WebBrowser
CHEF
Globus IndexService
Camera
TelepresenceMonitor
DataViewerTool
Camera
Application Developer 2
Off the Shelf 9
Globus Toolkit 4
Grid Community 4
Database service
GlobusDAI
CHEF ChatTeamlet
GlobusRLS
Database service
GlobusDAI
MyProxy
Database service
GlobusDAI
CertificateAuthority
Resources implement standard access management
interfaces
Collective services aggregate /or virtualize
resources
Users work with client applications
Application services organize VOs enable access
to other services
35
The Globus ToolkitStandard Plumbing for the
Grid

• Today the majority of the GT public interfaces
  are usable by application developers and system
  integrators
• Relatively few end-user interfaces
• In general, not intended for direct use by end
  users (scientists, engineers, marketing
  specialists)
• Not turnkey solutions, but building blocks
  tools for application developers system
  integrators
• Some components (e.g., file transfer) go farther
  than others (e.g., remote job submission) toward
  end-user relevance
• Better to reuse than reinvent!
• Plenty of interesting unsolved problems to work
  on
• Compatibility with other Grid systems comes for
  free

36
Sidebar The Globus Commitment to Open Source

• To allow for inspection
• for consideration in standardization processes
• To encourage adoption
• in pursuit of ubiquity and interoperability
• To encourage contributions
• harness the expertise of the community
• http//dev.globus.org

37
Globus Toolkit byDomain Areas

• Core runtime
• Infrastructure for building new services
• Security
• Apply uniform policy across distinct systems
• Execution management
• Provision, deploy, manage services
• Data management
• Discover, transfer, access large data
• Information services
• Discover monitor dynamic services

38
Globus Toolkit by Protocol Type

• Web service protocols
• WSDL, SOAP
• WS Addressing, WSRF, WSN
• WS Security, SAML, XACML
• WS-Interoperability profile
• Non Web service protocols
• Standards-based, such as GridFTP
• Custom

39
Globus Toolkit version 2 (GT2)
Web ServicesComponents
Pre-WS Authentication Authorization
GridFTP
C Common Libraries
Grid Resource Alloc. Mgmt (GRAM)
Monitoring Discovery (MDS)
Non-WS Components
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
40
Globus Toolkit version 3 (GT3)
OGSA-DAI
CommunityAuthorization
Web ServicesComponents
WS Authentication Authorization
Reliable File Transfer
Grid Resource Alloc. Mgmt (WS GRAM)
MDS3
Java WS Core
Pre-WS Authentication Authorization
GridFTP
C Common Libraries
Grid Resource Alloc. Mgmt (GRAM)
Monitoring Discovery (MDS)
Non-WS Components
Replica Location
eXtensible IO (XIO)
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
41
Globus Toolkit version 4 (GT4)
Grid Telecontrol Protocol
Depre-cated
Community Scheduling Framework
Data Replication
Delegation
Python WS Core
WebMDS
Workspace Management
OGSA-DAI
CommunityAuthorization
Trigger
C WS Core
Web ServicesComponents
Authentication Authorization
Reliable File Transfer
Grid Resource Allocation Management
Index
Java WS Core
Pre-WS Authentication Authorization
GridFTP
Pre-WS Grid Resource Alloc. Mgmt
Pre-WSMonitoring Discovery
C Common Libraries
Non-WS Components
Replica Location
eXtensible IO (XIO)
Credential Mgmt
www.globus.org
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
42
Globus Toolkit Common Capabilities
Globus Toolkit v4 www.globus.org
Data Replication
Replica Location
Grid Telecontrol Protocol
CredentialMgmt
OGSA-DAI
Community Scheduling Framework
Delegation
Python Runtime
WebMDS
Reliable File Transfer
CommunityAuthorization
Trigger
C Runtime
Workspace Management
GridFTP
Authentication Authorization
Grid Resource Allocation Management
Index
Java Runtime
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
43
GT4 Common Runtime

• Providing the common interfaces and capabilities
  for the toolkit as a whole
• Supports GT services (GRAM, RFT, Delegation,
  etc.) as well as user-developed services
• Leverages existing WS standards
• WS-I Basic Profile WSDL, SOAP, etc.
• WS-Security, WS-Addressing
• Adds support for emerging WS standards
• WS-Resource Framework, WS-Notification
• Java, Python, C hosting environments

44
FileTransferService (without WSRF)
FileTransferService
Client
move (A to B) transferID
move
whatHappen
state
tellMeWhen
cancel

• Developer reinvents wheel for each new service
• Custom management and identification of state
  transferID
• Custom operations to inspect state synchronously
  (whatHappen) and asynchronously (tellMeWhen)
• Custom lifetime operation (cancel)

45
Uniform Interface for Common Interaction Patterns

• State representation
• Resource
• Resource Property
• State identification
• Endpoint Reference
• State Interfaces
• GetRP, QueryRPs, GetMultipleRPs, SetRP
• Notification Interfaces
• Subscribe
• Notify
• Lifetime Interfaces
• SetTerminationTime
• ImmediateDestruction
• ServiceGroups

Service
GetRP
GetMultRPs
EPR
EPR
EPR
SetRP
QueryRPs
Subscribe
SetTermTime
Destroy
46
FileTransferService (with WSRF)
FileTransferService
Client
createResource (A to B) EPR
createResource
getRP
queryRPs
destroy

• Developer specifies custom method to
  createResource and leaves the rest to WSRF
  standards
• State exposed as Resource Resource Properties
  and identified by Endpoint Reference (EPR)
• State inspected by standard interfaces (GetRP,
  QueryRPs)
• Lifetime management by standard interfaces
  (Destroy)

47
Uniform Interfaces for Common Requirements

• Naming and bindings (basis for virtualization)
• Every resource can be uniquely referenced, and
  has one or more associated services for
  interacting with it
• Lifecycle (basis for fault resilient state mgmt)
• Resources created by services following factory
  pattern
• Resources destroyed immediately or scheduled
• Information model (basis for monitoring,
  discovery)
• Resource properties associated with resources
• Operations for querying and setting this info
• Asynchronous notification of changes to
  properties
• Service groups (basis for registries, collective
  svcs)
• Group membership rules membership management
• Base Fault type

48
Globus Toolkit version 4 (GT4)
Grid Telecontrol Protocol
Depre-cated
Community Scheduling Framework
Data Replication
Delegation
Python WS Core
WebMDS
Workspace Management
OGSA-DAI
CommunityAuthorization
Trigger
C WS Core
Web ServicesComponents
Authentication Authorization
Reliable File Transfer
Grid Resource Allocation Management
Index
Java WS Core
Pre-WS Authentication Authorization
GridFTP
Pre-WS Grid Resource Alloc. Mgmt
Pre-WSMonitoring Discovery
C Common Libraries
Non-WS Components
Replica Location
eXtensible IO (XIO)
Credential Mgmt
www.globus.org
Data Mgmt
Security
CommonRuntime
Execution Mgmt
Info Services
49
Wrap-up

• Grid Middleware
• The problem supporting Virtual Organizations
• Requirements
• Capabilities
• The Globus Approach