ECECS Lecture 18 Grid Computing

1
ECECS Lecture 18Grid Computing

• Citation B.Ramamurthy/Suny-Buffalo

2
Globus Material

• The presentation is based on the two main
  publications on grid computing given below
• The Physiology of the Grid, An Open Services
  Architecture for Distributed Systems Integration,
  by Ian Foster, Carl Kesselman, Jeffrey Nick, and
  Steven Tuecke, 2002.
• The Anatomy of the grid, Enabling Scalable
  Virtual Organization, Ian Foster, Carl Kesselman,
  Steven Tuecke, 2001.
• URLhttp//www.globus.org/research/papers.html

3
Grid Technology

• Grid technologies and infrastructures support the
  sharing and coordinated use of diverse resources
  in dynamic, distributed virtual organizations.
• Grid technologies are distinct from technology
  trends such as Internet, enterprise, distributed
  and peer-to-peer computing. But these
  technologies can benefit from growing into the
  problem space addressed by grid technologies.

4
Virtual Organization Problem Space

• An industrial consortium formed to develop a
  feasibility study for a next generation
  supersonic aircraft undertakes a highly accurate
  multidisciplinary simulation of the entire
  aircraft.
• A crisis management teams responds to a chemical
  spill by using local weather and soil models to
  estimate the spread of the spill, planning and
  coordinating evacuation, notifying hospitals and
  so forth.
• Thousands of physicists come together to design,
  create, operate and analyze products by pooling
  together computing, storage, networking resources
  to create a Data Grid.

5
Resource Sharing Requirements

• Members should be trustful and trustworthy.
• Sharing is conditional.
• Should be secure.
• Sharing should be able to change dynamically over
  time.
• Need for discovery and registering of resources.
• Can be peer to peer or client/server.
• Same resource may be used in different ways.
• All these point to well defined architecture and
  protocols.

6
Grid Definition

• Architecture identifies the fundamental system
  components, specifies purpose and function of
  these components, and indicates how these
  components interact with each other.
• Grid architecture is a protocol architecture,
  with protocols defining the basic mechanisms by
  which VO users and resources negotiate ,
  establish, manage and exploit sharing
  relationships.
• Grid architecture is also a services
  standards-based open architecture that
  facilitates extensibility, interoperability,
  portability and code sharing.
• API and Toolkits are also being developed.

7
Grid Services Architecture
High-energy physics data analysis
Collaborative engineering
On-line instrumentation
Applications
Regional climate studies
Parameter studies
Grid Fabric Layer
Transport
Multicast
. . .
Instrumentation
Control interfaces
QoS mechanisms
8
Architecture
Internet
GRID
Application
Application
Collective
Resource
Transport
Connectivity
Internet
Fabric
Link
9
Fabric Layer

• Fabric layer Provides the resources to which
  shared access is mediated by Grid protocols.
• Example computational resources, storage
  systems, catalogs, network resources, and
  sensors.
• Fabric components implement local, resource
  specific operations.
• Richer fabric functionality enables more
  sophisticated sharing operations.
• Sample resources computational resources,
  storage resources, network resources, code
  repositories, catalogs.

10
Connectivity Layer

• Communicating easily and securely.
• Connectivity layer defines the core communication
  and authentication protocols required for
  grid-specific network functions.
• This enables the exchange of data between fabric
  layer resources.
• Support for this layer is drawn from TCP/IPs IP,
  TCL and DNS layers.
• Authentication solutions single sign on, etc.

11
Resources Layer

• Resource layer defines protocols, APIs, and SDKs
  for secure negotiations, initiation, monitoring
  control, accounting and payment of sharing
  operations on individual resources.
• Two protocols information protocol and management
  protocol define this layer.
• Information protocols are used to obtain the
  information about the structure and state of the
  resource, ex configuration, current load and
  usage policy.
• Management protocols are used to negotiate access
  to the shared resource, specifying for example
  qos, advanced reservation, etc.

12
Collective Layer

• Coordinating multiple resources.
• Contains protocols and services that capture
  interactions among a collection of resources.
• It supports a variety of sharing behaviors
  without placing new requirements on the resources
  being shared.
• Sample services directory services,
  coallocation, brokering and scheduling services,
  data replication service, workload management
  services, collaboratory services.

13
Applications Layer

• These are user applications that operate within
  VO environment.
• Applications are constructed by calling upon
  services defined at any layer.
• Each of the layers are well defined using
  protocols, provide access to useful services.
• Well defined APIs also exist to work with these
  services.
• A toolkit Globus implements all these layers and
  supports grid application development.

14
Globus Toolkit Services

• Security (GSI)
• PKI-based Security (Authentication) Service
• Job submission and management (GRAM)
• Uniform Job Submission
• Information services (MDS)
• LDAP-based Information Service
• Remote file management (GASS)
• Remote Storage Access Service
• Remote Data Catalogue and Management Tools
• Support by Globus 2.0 released in 2002

15
High-level services

• Part II

16
Sample of High-Level Services

• Resource brokers and co-allocators
• DUROC, Nimrod/G, Condor-G, GridbusBroker
  Communication I/O libraries
• MPICH-G, PAWS, RIO (MPI-IO), PPFS, MOL
• Parallel languages
• HPC, CC, Nimrod Parameter Specification
• Collaborative environments
• CAVERNsoft, ManyWorlds
• Others
• MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow

17
The Nimrod-G Grid Resource Broker

• A resource broker for managing, steering, and
  executing task farming (parameter sweep/SPMD
  model) applications on the Grid based on deadline
  and computational economy.
• Based on users QoS requirements, our Broker
  dynamically leases services at runtime depending
  on their quality, cost, and availability.
• Key Features
• A single window to manage control experiment
• Persistent and Programmable Task Farming Engine
• Resource Discovery
• Resource Trading
• Scheduling Predications
• Generic Dispatcher Grid Agents
• Transportation of data results
• Steering data management
• Accounting
• Uses Globus MDS, GRAM, GSI, GASS

18
Condor-G Condor for the Grid

• Condor is a high-throughput scheduler
• Condor-G uses Globus Toolkit libraries for
• Security (GSI)
• Managing remote jobs on Grid (GRAM)
• File staging remote I/O (GSI-FTP)
• Grid job management interface scheduling
• Robust replacement for Globus Toolkit programs
• Globus Toolkit focus is on libraries and
  services, not end user vertical solutions
• Supports single or high-throughput apps on Grid
• Personal job manager which can exploit Grid
  resources

19
Production Grids Testbeds

• Production deployments underway at
• NSF PACIs National Technology Grid
• NASA Information Power Grid
• DOE ASCI
• European Grid
• Research testbeds
• EMERGE Advance reservation QoS
• GUSTO Globus Ubiquitous Supercomputing Testbed
  Organization
• Particle Physics Data Grid
• World-Wide Grid (WWG)

20
Production Grids Testbeds
NASAs Information Power Grid
The Alliance National Technology Grid
GUSTO Testbed
21
World Wide Grid (WWG)
Australia
North America
GMonitor
MelbourneMonash U VPAC, Physics
ANL SGI/Sun/SP2 NCSA Cluster Wisc
PC/cluster NRC, Canada Many others
GridbusNimrod-G
MEG Visualisation
Solaris WS
Internet
_at_ SC 2002/Baltimore
Europe
Grid MarketDirectory
ZIB T3E/Onyx AEI Onyx CNR Cluster CUNI/CZ
Onyx Pozman SGI/SP2 Vrije U Cluster Cardiff
Sun E6500 Portsmouth Linux PC Manchester
O3K Cambridge SGI Many others
Asia
AIST, Japan Solaris Cluster Osaka University
Cluster Doshia Linux cluster Korea Linux cluster
22
Example Applications Projects (via Nimrod-G or
Gridbus)

• Molecular Docking for Drug Discovery
• Docking molecules from chemical databases with
  target protein
• Neuro Science
• Brain Activity Analysis
• High Energy Physics
• Belle Detector Data Analysis
• Natural Language Engineering
• Analyzing audio data (e.g., to identify emotional
  state of a person!)

23
Example Application Projects

• Computed microtomography (ANL, ISI)
• Real-time, collaborative analysis of data from
  X-Ray source (and electron microscope)
• Hydrology (ISI, UMD, UT also NCSA, Wisc.)
• Interactive modeling and data analysis
• Collaborative engineering (tele-immersion)
• CAVERNsoft _at_ EVL
• OVERFLOW (NASA)
• Large CFD simulations for aerospace vehicles

24
Example Application Experiments

• Distributed interactive simulation (CIT, ISI)
• Record-setting SF-Express simulation
• Cactus
• Astrophysics simulation, viz, and steering
• Including trans-Atlantic experiments
• Particle Physics Data Grid
• High Energy Physics distributed data analysis
• Earth Systems Grid
• Climate modeling data management

25
The Globus Advantage

• Flexible Resource Specification Language which
  provides the necessary power to express the
  required constraints
• Services for resource co-allocation, executable
  staging, remote data access and I/O streaming
• Integration of these services into high-level
  tools
• MPICH-G grid-enabled MPI
• globus-job- flexible remote execution commands
• Nimrod-G Grid Resource broker
• Gridbus Grid Business Infrastructure
• Condor-G high-throughput broker
• PBS, GRD meta-schedulers

26
Resource Management

• Resource Specification Language (RSL) is used to
  communicate requirements
• The Globus Resource Allocation Manager (GRAM) API
  allows programs to be started on remote
  resources, despite local heterogeneity
• A layered architecture allows application-specific
  resource brokers and co-allocators to be defined
  in terms of GRAM services

27
Resource Management Architecture
RSL specialization
RSL
Application
Information Service
Queries
Info
Ground RSL
Simple ground RSL
Local resource managers
GRAM
GRAM
GRAM
LSF
EASY-LL
NQE
28
GRAM Components
MDS client API calls to locate resources
Client
MDS Grid Index Info Server
Site boundary
MDS client API calls to get resource info
GRAM client API calls to request resource
allocation and process creation.
MDS Grid Resource Info Server
Query current status of resource
GRAM client API state change callbacks
Globus Security Infrastructure
Local Resource Manager
Allocate create processes
Request
Job Manager
Create
Gatekeeper
Process
Parse
Monitor control
Process
RSL Library
Process
29
A simple run

• raj_at_belle raj globus-job-run belle.anu.edu.au
  /bin/date
• Mon May 3 150542 EST 2004

30
Resource Specification Language (RSL)

• Common notation for exchange of information
  between components
• Syntax similar to MDS/LDAP filters
• RSL provides two types of information
• Resource requirements Machine type, number of
  nodes, memory, etc.
• Job configuration Directory, executable, args,
  environment
• API provided for manipulating RSL

31
RSL Syntax

• Elementary form parenthesis clauses
• (attribute op value value )
• Operators Supported
• lt, lt, , gt, gt , !
• Some supported attributes
• executable, arguments, environment, stdin,
  stdout, stderr, resourceManagerContact,resourceMa
  nagerName
• Unknown attributes are passed through
• May be handled by subsequent tools

32
Constraints

• globusrun -o -r belle.anu.edu.au
  "(executable/bin/date)"
• For example
• (countgt5) (countlt10)
• (max_time240) (memorygt64)
• (executablemyprog)
• Create 5-10 instances of myprog, each on a
  machine with at least 64 MB memory that is
  available to me for 4 hours

33
Disjunction

• For example
• (executablemyprog)
• ( ((count5)(memorygt64))
• ((count10)(memorygt32)))
• Create 5 instances of myprog on a machine that
  has at least 64MB of memory, or 10 instances on a
  machine with at least 32MB of memory

34
Multirequest

• A multi-request allows us to specify multiple
  resource needs, for example
• ( (count5)(memorygt64)
• (executablep1))
• ((networkatm) (executablep2))
• Execute 5 instances of p1 on a machine with at
  least 64M of memory
• Execute p2 on a machine with an ATM connection
• Multirequests are central to co-allocation

35
Co-allocation

• Simultaneous allocation of a resource set
• Handled via optimistic co-allocation based on
  free nodes or queue prediction
• In the future, advance reservations will also be
  supported
• globusrun and globus-job- will co-allocate
  specific multi-requests
• Uses a Globus component called the Dynamically
  Updated Request Online Co-allocator (DUROC)

36
DUROC Functions

• Submit a multi-request
• Edit a pending request
• Add new nodes, edit out failed nodes
• Commit to configuration
• Delay to last possible minute
• Barrier synchronization
• Initialize computation
• Bootstrap library
• Monitor and control collection

37
DUROC Architecture
Controlled Jobs
Subjob status
Controlling Application
RSL multi-request
Edit request
Barrier
38
RSL Creation Using globus-job-run

• globus-job-run can be used to generate RSL from
  command-line args
• globus-job-run dumprsl \ - host1
  -np N1 -s executable1 args1 \ -
  host2 -np N2 -s executable2 args2 \
  ... gt rslfile
• -np number of processors
• -s stage file
• argument options for all RSL keywords
• -help description of all options

39
Job Submission Interfaces

• Globus Toolkit includes several command line
  programs for job submission
• globus-job-run Interactive jobs
• globus-job-submit Batch/offline jobs
• globusrun Flexible scripting infrastructure
• Other High Level Interfaces
• General purpose
• Nimrod-G, Condor-G, PBS, GRD, etc
• Application specific
• ECCE, Cactus, Web portals

40
globus-job-run

• For running of interactive jobs
• Additional functionality beyond rsh
• Ex Run 2 process job w/ executable staging
• globus-job-run - host np 2 s myprog arg1 arg2
• Ex Run 5 processes across 2 hosts
• globus-job-run \
• - host1 np 2 s myprog.linux arg1 \
• - host2 np 3 s myprog.aix arg2
• For list of arguments run
• globus-job-run -help

41
globus-job-submit

• For running of batch/offline jobs
• globus-job-submit Submit job
• Same interface as globus-job-run
• Returns immediately
• globus-job-status Check job status
• globus-job-cancel Cancel job
• globus-job-get-output Get job stdout/err
• globus-job-clean Cleanup after job

42
globusrun

• Flexible job submission for scripting
• Uses an RSL string to specify job request
• Contains an embedded globus-gass-server
• Defines GASS URL prefix in RSL substitution
  variable
• (stdout(GLOBUSRUN_GASS_URL)/stdout)
• Supports both interactive and offline jobs
• Complex to use
• Must write RSL by hand
• Must understand its esoteric features
• Generally you should use globus-job- commands
  instead

43
Resource Brokers
Run a distributed interactive simulation
involving 100,000 entities
Supercomputers providing 100 GFLOPS, 100 GB, lt
100 msec latency
DIS-Specific Broker
Information Service
Supercomputer resource broker
80 nodes on Argonne SP, 256 nodes on CIT
Exemplar 300 nodes on NCSA O2000
Simultaneous start co-allocator
"Run SF-Express on 80 nodes
"Run SF-Express on 256 nodes
Run SF-Express on 300 nodes
NCSA Resource Manager
Argonne Resource Manager
CIT Resource Manager
44
Brokering via Lowering

• Resource location by refining a RSL expression
  (RSL lowering)
• (MFLOPS1000) Þ
• ( (archsp2)(count200)) Þ
• ( ( (archsp2) (count120)
• (resourceManagerContactanlsp2))
• ( (archsp2) (count80)
• (resourceManagerContactuhsp2)))

45
Remote I/O and Staging

• Tell GRAM to pull executable from remote location
• Access files from a remote location
• stdin/stdout/stderr from a remote location

46
What is GASS?

• (a) GASS file access API
• Replace open/close with globus_gass_open/close
  read/write calls can then proceed directly
• (b) RSL extensions
• URLs used to name executables, stdout, stderr
• (c) Remote cache management utility
• (d) Low-level APIs for specialized behaviors

47
GASS Architecture
(executablehttps//)
main( ) fd globus_gass_open()
read(fd,) globus_gass_close(fd)
(b) RSL extensions
GRAM
GASS Server
HTTP Server
(a) GASS file access API
FTP Server
Cache
(c) Remote cache management
(d) Low-level APIs for customizing cache GASS
server
globus-gass-cache
48
GASS File Naming

• URL encoding of resource names
• https//quad.mcs.anl.gov9991/bester/myjob
• protocol server address
  file name
• Other examples
• https//pitcairn.mcs.anl.gov/tmp/input_dataset.1
• https//pitcairn.mcs.anl.gov2222/./output_data
• http//www.globus.org/bester/input_dataset.2
• Supports http https
• Support ftp gsiftp.

49
GASS RSL Extensions

• executable, stdin, stdout, stderr can be local
  files or URLs
• executable and stdin loaded into local cache
  before job begins (on front-end node)
• stdout, stderr handled via GASS append mode
• Cache cleaned after job completes

50
GASS/RSL Example

• (executablehttps//quad1234//myexe)
  (stdinhttps//quad1234//myin)
  (stdout/home/bester/output) (stderrhttps//qua
  d1234/dev/stdout)

51
Example GASS Applications

• On-demand, transparent loading of data sets
• Caching of data sets
• Automatic staging of code and data to remote
  supercomputers
• (Near) real-time logging of application output to
  remote server

52
GASS File Access API

• Minimum changes to application
• globus_gass_open(), globus_gass_close()
• Same as open(), close() but use URLs instead of
  filenames
• Caches URL in case of multiple opens
• Return descriptors to files in local cache or
  sockets to remote server
• globus_gass_fopen(), globus_gass_fclose()

53
GASS File Access API (cont)

• Support for different access patterns
• Read-only (from local cache)
• Write-only (to local cache)
• Read-write (to/from local cache)
• Write-only, append (to remote server)

54
globus_gass_open()/close()
no
Download File into cache
URL in cache?
yes
open cached file,add cache reference
globus_gass_close()
globus_gass_open()
55
GASS File API Semantics

• Copy-on-open to cache if not truncate or
  write-only append and not already in cache
• Copy on close from cache if not read only and not
  other copies open
• Multiple globus_gass_open() calls share local
  copy of file
• Append to remote file if write only append e.g.,
  for stdout and stderr
• Reference counting keeps track of open files

56
globus-gass-server

• Simple file server
• Run by user wherever necessary
• Secure https protocol, using GSI
• APIs for embedding server into other programs
• Example
• globus-gass-server r w -t
• -r Allow files to be read from this server
• -w Allow files to be written to this server
• -t Tilde expand (/ ? (HOME)/)
• -help For list of all options

57
GRAM GASS Putting It Together
1. Derive Contact String 2. Build RSL string 3.
Startup GASS server 4. Submit to request 5.
Return output
5
5
4
5
5
3
4
4
gatekeeper
58
Example A Simple Broker

• Select machines based on availability
• Use MDS queries to get current host loads
• Look at output and figure out what machines to
  use
• Generate RSL based on selection
• globus-job-run -dumprsl can assist
• Execute globusrun, feeding it the RSL generated
  in previous step

59
GRAM GASS

• Using RSL with globusrun
• Running globus-gass-server
• Modifying a program to use globus_gass_open() to
  read files remotely from a GASS server

60
Globus Components In Action
Local Machine
User Proxy Cert
X509 User Cert
Machines
RSL string
mpirun
grid-proxy-init
RSL multi-request
globusrun
RSL single request
RSL parser
DUROC
GASS Server
GRAM Job Manager
GRAM Job Manager
GRAM Gatekeeper
GRAM Gatekeeper
GSI
GASS Client
GASS Client
GSI
PBS
Unix Fork
App
App
Nexus
Nexus
AIX
MPI
Solaris
MPI
Remote Machine
Remote Machine
61
GRAM Components
MDS client API calls to locate resources
Client
MDS Grid Index Info Server
Site boundary
MDS client API calls to get resource info
GRAM client API calls to request resource
allocation and process creation.
MDS Grid Resource Info Server
Query current status of resource
GRAM client API state change callbacks
Globus Security Infrastructure
Local Resource Manager
Allocate create processes
Request
Job Manager
Create
Gatekeeper
Process
Parse
Monitor control
Process
RSL Library
Process
62
MDS Monitoring and Discovery Service

• Learn how to use the MDS to locate and determine
  characteristics of resources
• Locate resources
• Where are resources with required architecture,
  installed software, available capacity, network
  bandwidth, etc.?
• Determine resource characteristics
• What are the physical characteristics,
  connectivity, capabilities of a resource?

63
The Need for Information

• System information is critical to operation of
  the grid and construction of applications
• How does an application determine what resources
  are available?
• What is the state of the computational grid?
• How can we optimize an application based on
  configuration of the underlying system?
• We need a general information infrastructure to
  answer these questions

64
Using Information forResource Brokering
Info service location selection
Metacomputing Directory Service
Resource Broker
What computers? What speed? When available?
20 Mb/sec
GRAM
Globus Resource Allocation Managers
50 processors storage from 1020 to 1040 pm
Fork LSF EASYLL Condor etc.
65
Examples of Useful Information

• Characteristics of a compute resource
• IP address, software available, system
  administrator, networks connected to, OS version,
  load
• Characteristics of a network
• Bandwidth and latency, protocols, logical
  topology
• Characteristics of the Globus infrastructure
• Hosts, resource managers

66
Grid Information Service

• Provide access to static and dynamic information
  regarding system components
• A basis for configuration and adaptation in
  heterogeneous, dynamic environments
• Requirements and characteristics
• Uniform, flexible access to information
• Scalable, efficient access to dynamic data
• Access to multiple information sources
• Decentralized maintenance

67
MDS

• Store information in a distributed directories
• Directory stored in collection of LDAP servers
• Each server optimized for particular function
• Directory can be updated by
• Information providers and tools
• Applications (i.e., users)
• Backend tools which generate info on demand
• Information dynamically available to
• Tools
• Applications

68
Directory Service Functions

• White Pages
• Look up the IP number, amount of memory, etc.,
  associated with a particular machine
• Yellow Pages
• Find all the computers of a particular class or
  with a particular property
• Temporary inconsistencies are often considered
  okay
• In a distributed system, you often do not know
  the state of a resource until you actually use it
• Information is often used as hints
• Information itself can contain ttl, etc.

69
MDS Approach

• Based on LDAP
• Lightweight Directory Access Protocol v3 (LDAPv3)
• Standard data model
• Standard query protocol
• Globus specific schema
• Host-centric representation
• Globus specific tools
• GRIS, GIIS
• Data discovery, publication,

Application
Middleware
LDAP API
GRIS
GIIS


SNMP
NWS
NIS
LDAP
70
MDS Components

• Uses standard LDAP servers
• OpenLDAP, Netscape, Oracle, etc
• Tools for populating maintaining MDS
• Integrated with Globus Toolkit server release,
  not of concern to most Globus users
• Discover/update static and dynamic info
• APIs for accessing updating MDS contents
• C, Java, PERL (LDAP API, JNDI)
• Various tools for manipulating MDS contents
• Command line tools, Shell scripts GUIs

71
Anonymous Grid info search

• grid-info-search -x -h belle.anu.edu.au
• .
• Mds-Computer-isa i686
• Mds-Computer-platform i686
• Mds-Computer-Total-nodeCount 1
• Mds-Cpu-Cache-l2kB 512
• Mds-Cpu-features fpu vme de pse tsc msr pae mce
  cx8 apic sep mtrr pge mca cmo
• v pat pse36 clflush dts acpi mmx fxsr sse sse2
  ss ht tm
• Mds-Cpu-Free-15minX100 400
• Mds-Cpu-Free-1minX100 400
• Mds-Cpu-Free-5minX100 400
• Mds-Cpu-model Intel(R) Xeon(TM) CPU 2

72
Summary

• MDS provides the information needed to perform
  dynamic resource discovery and configuration
• Critical component of resource brokers
• MDS is base on existing directory service
  standards (LDAPv3)