Grid Computing

1
Grid Computing Web ServicesA Natural
Partnership

• Ian Foster
• Mathematics and Computer Science Division
• Argonne National Laboratory
• and
• Department of Computer Science
• The University of Chicago

Dave Angulo Department of Computer Science The
University of Chicago and Mathematics and
Computer Science Division Argonne National
Laboratory
Address of Poznan Supercomputing Networking
Center Poznan, Poland February 7, 2002
2
Partial Acknowledgements

• Open Grid Services Architecture work is performed
  by
• Ian Foster, Globus Co-PI _at_ Argonne/UofC
• Carl Kesselman, Globus Co-PI _at_ USC/ISI
• Steve Tuecke, Globus Toolkit Architect _at_ANL
• Jeff Nick, Steve Graham, Jeff Frey _at_ IBM
• Globus Toolkit RD involves many fine scientists
  engineers at ANL, USC/ISI, and elsewhere (see
  www.globus.org)
• Strong collaborations with many outstanding EU,
  UK, US Grid projects
• Support from DOE, NASA, NSF, Microsoft

3
Abstract

• "Grid" computing has emerged as an important new
  field
• Distinguished from conventional distributed
  computing by focus on
• Large-scale resource sharing
• Innovative applications
• High-performance orientation (in some cases)
• In this talk, this new field is defined
• First, "Grid problem reviewed, which Ian Foster
  defines as
• flexible, secure, coordinated resource sharing
• among dynamic collections of individuals,
  institutions, and resources (referred to as
  virtual organizations)
• Challenges in such settings
• authentication
• authorization
• resource access
• resource discovery
• and other challenges

4
Abstract (Cont.)

• This class of problem addressed by Grid
  technologies
• Major Grid projects worldwide reviewed
• Describe their contributions to the realization
  of this architecture.
• Future Architecture Overview
• Open Grid Services Architecture is presented

5
Partial Acknowledgements

• Globus ToolkitTM
• RD involves
• many fine scientists engineers at ANL/UofC,
  USC/ISI, and elsewhere (see www.globus.org)
• Led by
• Ian Foster _at_ Argonne/UofC
• Carl Kesselman _at_ USC/ISI
• Open Grid Services Architecture work performed by
• Ian Foster, Globus Co-PI _at_ Argonne/UofC
• Carl Kesselman, Globus Co-PI _at_ USC/ISI
• Steve Tuecke, Globus Toolkit Architect _at_ANL
• Jeff Nick, Steve Graham, Jeff Frey _at_ IBM
• Strong collaborations with many outstanding EU,
  UK, US Grid projects
• Support from DOE, NASA, NSF, Microsoft, IBM

6
Grid Computing
7
The Grid Problem

• Resource sharing coordinated problem solving
  in dynamic, multi-institutional virtual
  organizations

8
Why Grids?

• A biochemist exploits 10,000 computers to screen
  100,000 compounds in an hour
• 1,000 physicists worldwide pool resources for
  petaflop analyses of petabytes of data
• Civil engineers collaborate to design, execute,
  analyze shake table experiments
• Climate scientists visualize, annotate, analyze
  terabyte simulation datasets
• A home user invokes architectural design
  functions at an application service provider
• An application service provider purchases cycles
  from compute cycle providers

9
Elements of the Problem

• Resource sharing
• Computers, storage, sensors, networks,
• Sharing always conditional issues of trust,
  policy, payment,
• Coordinated problem solving
• Beyond client-server distributed data analysis,
  computation,
• Dynamic, multi-institutional virtual orgs
• Community overlays on classic org structures
• Large or small, static or dynamic

10
Grids Why Now?

• Moores law improvements in computing produce
  highly functional end systems
• The Internet and burgeoning wired and wireless
  provide universal connectivity
• Network exponentials produce dramatic changes in
  geometry and geography

11
Grids Why Now?

• Moores law improvements in computing produce
  highly functional endsystems
• The Internet and burgeoning wired and wireless
  provide universal connectivity
• Network exponentials produce dramatic changes in
  geometry and geography
• 9-month doubling double Moores law!
• 1986-2001 x340,000 2001-2010 x4000?

12
The Grid World Current Status

• Dozens of major Grid projects in scientific
  technical computing/research education
• Deployment, application, technology
• Considerable consensus on key concepts and
  technologies
• Globus Toolkit has emerged as de facto standard
  for major protocols services
• Global Grid Forum has emerged as a significant
  force
• And first Grid proposals at IETF

13
Selected Major Grid Projects
New
New
14
Selected Major Grid Projects
New
New
New
New
New
15
Selected Major Grid Projects
New
New
16
Selected Major Grid Projects
New
New
Also many technology RD projects e.g., Condor,
NetSolve, Ninf, NWS See also www.gridforum.org
17
Grid Communities ApplicationsData Grids for
High Energy Physics
www.griphyn.org www.ppdg.net
www.eu-datagrid.org
18
Grid Communities and ApplicationsMathematicians
Solve NUG30

• Communityan informal collaboration of
  mathematicians and computer scientists
• Condor-G delivers 3.46E8 CPU seconds in 7 days
  (peak 1009 processors) in U.S. and Italy (8
  sites)
• Solves NUG30 quadratic assignment problem

14,5,28,24,1,3,16,15, 10,9,21,2,4,29,25,22, 13,26,
17,30,6,20,19, 8,18,7,27,12,11,23
www.mcs.anl.gov/metaneos Argonne, Iowa, NWU,
Wisconsin
19
Grid Communities and ApplicationsNetwork for
Earthquake Eng. Simulation

• NEESgrid national infrastructure to couple
  earthquake engineers with experimental
  facilities, databases, computers, each other
• On-demand access to experiments, data streams,
  computing, archives, collaboration

NEESgrid Argonne, Michigan, NCSA, UIUC, USC
www.neesgrid.org
20
The 13.6 TF TeraGridComputing at 40 Gb/s
Site Resources
Site Resources
26
HPSS
HPSS
4
24
External Networks
External Networks
8
5
Caltech
Argonne
External Networks
External Networks
NCSA/PACI 8 TF 240 TB
SDSC 4.1 TF 225 TB
Site Resources
Site Resources
HPSS
UniTree
TeraGrid/DTF NCSA, SDSC, Caltech, Argonne
www.teragrid.org
21
Intl. Virtual Data Grid Lab.
www.ivdgl.org
22
Access Grid

• Collaborative work among large groups
• 50 sites worldwide
• Use Grid services for discovery, security
• www.scglobal.org

Access Grid Argonne, others
www.accessgrid.org
23
Grid Architecture Globus Toolkit

• The question
• What is needed for resource sharing coordinated
  problem solving in dynamic virtual organizations
  (VOs)?
• The answer
• Major issues identified membership, resource
  discovery access, ,
• Grid architecture captures core elements,
  emphasizing pre-eminent role of protocols
• Globus Toolkit has emerged as de facto standard
  for major protocols services

24
The Critical Role of Protocols

• Need for interoperability when different groups
  want to share resources
• E.g., IP lets me talk to your computer, but how
  do we establish maintain sharing?
• How do I discover, authenticate, authorize,
  describe what I want to do, etc., etc.?
• Need for shared infrastructure services to avoid
  repeated development, installation, e.g.
• One port/service for remote access to computing,
  not one per tool/application
• X.509 enables sharing of Certificate Authorities

25
Grid Architecture
For more info www.globus.org/research/papers/anat
omy.pdf
26
Globus Project and Toolkit

• Globus Project
• RD project at ANL, U.Chicago, USC/ISI
• Emphasis on identifying and defining core
  protocols and services
• O(40) researchers developers
• Globus Toolkit
• A major product of the Globus Project
• Open source software reference implementation of
  core protocols services
• Growing open source developer community

27
Globus Toolkit Evaluation (1)

• Good technical solutions for key problems, e.g.
• Authentication and authorization
• Resource discovery and monitoring
• Reliable remote service invocation
• High-performance remote data access
• This good engineering is enabling progress
• Good quality reference implementation,
  multi-language support, interfaces to many
  systems, large user base, industrial support
• Growing community code base built on tools

28
Globus Toolkit Evaluation (2)

• Protocol deficiencies, e.g.
• Heterogeneous basis HTTP, LDAP, FTP
• No standard means of error propagation
• Significant missing functionality, e.g.
• Databases, sensors, instruments
• Programming tools workflow,
• Virtualization of end systems (hosting envs.)
• Little work on total system properties, e.g.
• Dependability, end-to-end QoS,
• Reasoning about system properties

29
Web Services

• Increasingly popular standards-based framework
  for accessing network applications
• W3C standardization Microsoft, IBM, Sun, others
• WSDL Web Services Description Language
• Interface Definition Language for Web services
• SOAP Simple Object Access Protocol
• XML-based RPC protocol common WSDL target
• WS-Inspection (WSIL)
• Conventions for locating service descriptions
• UDDI Universal Desc., Discovery, Integration
• Directory for Web services

30
Transient Service Instances

• Web services address discovery invocation of
  persistent services
• In Grids, must also support transient service
  instances, created/destroyed dynamically
• E.g., to manage eBusiness workflow, video
  conference, or distributed data analysis
• Significant implications for how services are
  managed, named, discovered, and used
• In fact, much of our work is concerned with the
  management of service instances

31
Open Grid Services Architecture

• Service orientation to virtualize resources
• From Web services
• Standard interface definition mechanisms
  multiple protocol bindings, multiple
  implementations, local/remote transparency
• Building on Globus Toolkit
• The Grid service defines standard semantics for
  service interactions
• Factory, registry, and mapper services
• Reliable and secure transport
• Multiple hosting targets J2EE, .NET, C, etc.

32
OGSA Service Model

• System comprises (a typically few) persistent
  services (potentially many) transient services
• All services adhere to specified Grid service
  interfaces and behaviors
• Reliable invocation, lifetime management,
  discovery, authorization, notification,
  upgradeability, concurrency, manageability
• Interfaces for managing Grid service instances
• Factory, registry, mapper
• Heavily leverage Globus Toolkit technology
• gt Reliable secure mgmt of distributed state

33
The Grid Service

• A (potentially transient) Web service with
  specified interfaces behaviors, including
• Creation (Factory)
• Global naming (GSH) references (GSR)
• Lifetime management
• Registration Discovery
• Authorization
• Notification
• Concurrency
• Manageability

34
Factory

• A Grid service with Factory interface can be
  requested to create a new Grid service instance
• Reliable creation (once-and-only-once)
• Create operation can be extended to accept
  Grid-service-specific creation parameters
• Returns a Grid Service Handle (GSH)
• A globally unique URL
• Uniquely identifies the instance for all time
• Based on name of a home mapper service

35
Mapper

• A GSH is a stable name for a Grid service, but
  does not allow client to actually communicate
  with the Grid service
• A Grid Service Reference (GSR) is a WSDL document
  that describes how to communicate with the Grid
  service
• Contains protocol binding, network address,
• May expire (I.e. GSR information may change)
• The Mapper interface allows a client to map from
  a GSH to a GSR
• http get on GSH also returns a GSR

36
Lifetime Management

• GS instances created by factory or manually
  destroyed explicitly or via soft state
• Negotiation of initial lifetime with Factory
• SoftStateDestruction interface supports
• GetTerminationTime message for inquiry
• Notification interface also allows for lifetime
  notification
• SetTerminationTime message for keepalive
• Soft state lifetime management avoids
• Explicit client teardown of complex state
• Resource leaks in hosting environments
• ExplicitDestruction interface also available

37
Discovery

• A Grid service instance may maintain a set of
  service information
• XML fragments encapsulated in standard ltname,
  type, TTL-infogt containers
• Discovery interface allows clients to query the
  Grid service instance for this information
• Query operation, plus supporting operations
• Extensible query language support
• See also Notification interfaces
• Allows notification of service existence and
  about service information

38
Registry

• The Registry interface may be used to discover a
  set of Grid service instances
• Returns a WS-Inspection document containing the
  GSHs of a set of Grid services
• Also returns policy associated with the set
• Also available through Discovery interface
• The RegistryManagement interface allows for
  soft-state registration of a Grid service
• A set of Grid services can periodically register
  their GSHs into a registry service, to allow for
  discovery of services in that set

39
Authorization

• Protocol binding handles authentication during
  invocation of Grid service operation
• Gives service URI for authenticated subject
• Grid service instance should apply authorization
  policy on all operations
• May be site-, service-, instance-, etc., specific
• OGSA defines standard interfaces for remote
  management of access control policy
• OperationAuthorizationManagement
• SubjectEquivalency

40
Notification Interfaces

• NotificationSource for client subscription
• One or more notification generators
• Generates notification message of a specific type
• Typed interest statements E.g., Filters, topics,
  
• Supports messaging services, 3rd party filter
  services,
• Soft state subscription to a generator
• NotificationSink for asynchronous delivery of
  notification messages
• A wide variety of uses are possible
• E.g. Dynamic discovery/registry services,
  monitoring, application error notification,

41
Use of Web Services (1)

• A Grid service interface is a WSDL portType
• A Grid service definition is a WSDL extension
  (serviceType) containing
• A set of one or more portTypes supported by the
  service
• portType serviceType compatibility statements,
  to support upgradability
• For discovery of compatible services when
  interfaces are upgraded
• Implementation version information

42
Use of Web Services (2)

• A GSR is a WSDL document with extensions
• Extension to service element to reference
  serviceType
• Service element extensions to carry the GSH, and
  the expiration time of the GSR
• A GSH is an URL, with the following properties
• Globally unique for all time
• http get on GSH .wsdl returns GSR
• Can derive GSH to Mapper from it
• Registry returns WS-Inspection documents

43
Using OGSAto Construct Grid Environments
In each case, Registry handle is effectively the
unique name for the virtual organization.
44
OGSA and the Globus Toolkit

• Technically, OGSA enables
• Refactoring of protocols (GRAM, MDS-2,
  etc.)while preserving all GT concepts/features!
• Integration with hosting environments
  simplifying components, distribution, etc.
• Greatly expanded standard service set
• Pragmatically, we are proceeding as follows
• Develop open source OGSA implementation
• Globus Toolkit 3.0 supports Globus Toolkit 2.0
  APIs
• Partnerships for service development
• Also expect commercial value-adds

45
Globus Toolkit Refactoring

• Grid Security Infrastructure (GSI)
• Used in Grid service network protocol bindings
• Meta Directory Service 2 (MDS-2)
• Native part of each Grid service
• Discovery, Registry, RegistryManagement,
  Notification
• Grid Resource Allocation Mngt (GRAM)
• Gatekeeper -gt Factory for job mgr instances
• GridFTP
• Refactor control channel protocol
• Other services refactored to used Grid services

46
Timeline

• Summer 2002 Alpha releases of high-level Grid
  Services
• Late 2002, Early 2003 Alpha release of new core
  Grid Services (MDS, GRAM, GridFTP)

47
Migration Paths

• Globus ToolkitTM evolutionary in nature
• Toolkit implementation may change
• Underlying model of Grid Computing remains the
  same
• Capabilities of future Toolkits will be superset
  of todays Toolkit
• New implementations integrate better with
  existing commodity technologies
• In cases of radical departure from current
  implementations, migration paths will be provided
• possibly maintain compatible APIs
• possibly create gateways to todays protocols

48
SummaryEvolution of Grid Technologies

• Initial exploration (1996-1999 Globus 1.0)
• Extensive appln experiments core protocols
• Data Grids (1999-?? Globus 2.0)
• Large-scale data management and analysis
• Open Grid Services Architecture (2001-??, Globus
  3.0)
• Integration w/ Web services, hosting
  environments, resource virtualization
• Databases, higher-level services
• Radically scalable systems (2003-??)
• Sensors, wireless, ubiquitous computing

49
Summary

• The Grid problem Resource sharing coordinated
  problem solving in dynamic, multi-institutional
  virtual organizations
• Grid architecture Protocol, service definition
  for interoperability resource sharing
• Globus Toolkit a source of protocol and API
  definitionsand reference implementations
• And many projects applying Grid concepts (
  Globus technologies) to important problems
• Open Grid Services Architecture represents (we
  hope!) next step in evolution

50
For More Information

• The Globus Project
• www.globus.org
• Grid architecture
• www.globus.org/research/papers/anatomy.pdf
• Open Grid Services Architecture
• www.globus.org/research/papers/ogsa.pdf
• www.globus.org/research/papers/gsspec.pdf