Recent Advances in Grid Computing and Business Models: A Gridbus Perspective

1
Recent Advances in Grid Computing and Business
Models A Gridbus Perspective
Grid Business Symposium 2005, Seoul, Korea

• Rajkumar Buyya

Grid and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of MelbourneMelbourne,
Australiawww.gridbus.org
2
Outline

• Introduction
• Utility Networks and Grid Computing
• Global Grids and Challenges
• Grid Initiatives
• World-wide with Australia and India Perspective
• Introduction to Gridbus Project and Grid Economy
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Economy-based Scheduling in Data Grids
• Summary

3
4 Essential Utilities and Delivery Networks
4
(5) IT services as the fifth utility (water,
electricity, gas, telephone, IT)
eScience eBusiness eGovernment eHealth Multilingua
l eEducation
5
A Bird Eye View of World-Wide Grid Environment
Grid Information Service
Grid Resource Broker
Application
R2
R3
R4
R5
RN
Grid Resource Broker
R6
R1
Resource Broker
Grid Information Service
6
Grid Resource Sharing Synergies

• Computational Services CPU cycles
• SETI_at_Home, NASA IPG, TeraGrid, I-Grid,
• Data Services
• Data replication, management, secure access--LHC
  Grid/Napster
• Application Services
• Access to remote software/libraries and license
  managementNetSolve
• Interaction Services
• eLearning, Virtual Tables, Group Communication
  (Access Grid), Gaming
• Knowledge Services
• The way knowledge is acquired and manageddata
  mining.
• Utility Computing Services
• Towards a market-based Grid computing Leasing
  and delivering Grid services as ICT utilities.

Utility Grid
Knowledge Grid
Interaction Grid
ASP Grid
Data Grid
Computational Grid
7
Grid Challenges
8
Some Grid Initiatives Worldwide

• Australia
• Nimrod-G
• Gridbus
• DISCWorld
• GrangeNet.
• APACGrid
• ARC eResearch
• Brazil
• OurGrid, EasyGrid
• LNCC-Grid many others
• China
• ChinaGrid Education
• CNGrid - application
• Europe
• UK eScience
• EU Grids..
• and many more...
• India
• I-Grid

• USA
• Globus
• NASA IPG
• AccessGrid
• TeraGrid
• Cyberinfrasture
• Industry Initiatives
• IBM On Demand Computing
• HP Adaptive Computing
• Sun N1
• Microsoft - .NET
• Oracle 10g
• Satyam Grid Practice
• Infosys, Wipro, TCS
• StorageTek Grid..
• Public Forums
• Global Grid Forum
• Australian Grid Forum
• Conferences

27 million
1.3 billion 3 yrs
2? billion
120million 5 yrs
450million 5 yrs
486million 5 yrs
1.3 billion (Rs)
1 billion 5 yrs
http//www.gridcomputing.com
9
Grid (Market) Participant Types and Application
Category
free trading
Publiccomputing (SETI_at_Home, Alchemi, UD)
Private enterprises(IBM, Sun, Satyam)
SharingModel
National provider(KGrid, TeraGrid,
Garuda/IndiaGrid, UKGrid)
regulation
scientific
commercial
Application Category
10
Grid Computing in Australia(Courtesy Jihyoun
Park, SNU Visitor to Melbourne)
Academia
Government
Collaboration
Industry
11
Academic activities

• 1 University laboratories for Grid computing
• - Uni. of Melbourne(GRIDS lab) Gridbus
  (GridSim, GMD, GridBank, Alchemi, ..), Master of
  Engineering in Distributed Computing
• - Monash Uni. GriddlsS (Legacy SW to the
  computational grid), Nimrod-G
• - Australian national Uni. (Internet Futures
  Group)
• - Sydney Uni.(ViSLAB) high performance
  visualization computing
• - Uni. of Adelaide (DHPC Group) DISCWorld
• - Queensland Uni. of Technology (PLAS) G2 (.NET
  based)
• 2 Grid Infrastructure Projects
• APACGrid, National Neurosciece Facility,
  Australian Virtual Observatory, several state
  level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC)
• 3 Grid Applications
• Asia Pacific Bioinformatics Network/ Virtual
  Drug Design Molecular Modeling for Drug Design
  on P2P Grid/ HEPGrid High Energy Physics and the
  Grid Network/ Access Grid/Australian
  Computational Earth Systems Simulator/.
• Recently 30 more applications are funded as
  part of ARC e-Research
• Govt. has formed National e-Research
  Coordination Committee.

12
Grid Computing in India
13
Grid Computing in India Academic and Industrial
Activities

• Academic and Government Initiatives
• TIFR, IITM, Anna University, IITD, UoH, etc.
• C-DACs Garuda Ministry of IT
• Software Companies in India
• Top 4 Indian IT Companies Satyam, Infosys, TCS
  (Tata Consultancy Service), and Wipro.
• Oracle 10g, IBM, HP, Sun ertc. have a large Grid
  development centers in Bangalore, India.
• Satyam is leading the pack in Grid Business push
• Grid Practice Centre with top management support.
• Singned MoU with Melbourne University and
  extensively using Gridbus in powering
  applications.
• Also contributing the development of Gridbus
  technologies (e.g., Alchemi) SEI CMM Level 5
  principles.
• Application Verticals Manufacturing, Security,
  Life Sciences, Finance

14
(No Transcript)
15
Australian and Indian Grid Efforts Compared
Korea Is it like Australia or India?
16
The Gridbus Project _at_ MelbourneEnable Leasing
of ICT Services on Demand
Distributed Data
WWG
Gridbus
World Wide Grid! ?On Demand Utility Computing
17
The Gridbus Project http//www.gridbus.org

• A multi-institutional Open Source RD Project
  with focus on
• Architecture, Specification, and Open Source
  Reference Implementation.
• Service-Oriented Grid, Utility Computing
  Distributed Data and Computation Economy
• Scaling from Desktops, Clusters, Cluster
  Federation, Enterprise Grids to Global Grids.
• Alchemi Harnessing .NET/Windows-based Resources
• Grid Market Directory and Web Services
• Grid Bank Accounting and Transaction Management
• Visual Tools for Creation of Distributed
  Applications
• Workflow Composition and Deployment Services
• Data Grid Brokering and Grid Economy Services
• Data Replication Strategies
• GridSim Toolkit Enhanced to support Data Grid,
  Reservation, etc.
• Libra SLA-based Allocation of Cluster Resources
• Coupling of Clusters and Computational Economy
• WWG Global Data Intensive Grid Testbed
• Application Enabler Projects
• High-Energy Physics , Astronomy, Brain Activity
  Analysis Osaka U., Natural Language
  Processing, Portfolio Analysis Spain, BioGrid
  - WEHI (via APACGrid), SensorGrid (NICTA),
  Medical Imaging (HFI)
• Supported by

18
Grid Economy Methodology for Sustained Resourced
Sharing and Managing Supply-and-Demand for
Resources
19
New challenges of Grid Economy

• Resource Owners
• How do I decide prices ? (economic models?)
• How do I specify them ?
• How do I translate price to resource allocation ?
• How do I enforce them ?
• How do I advertise attract consumers ?
• How do I do accounting and handle payments?
• ..
• Resource Consumers
• How do I decide expenses ?
• How do I express QoS requirements ?
• How do I trade between timeframe cost ?
• How do I map jobs to resources to meet my QoS
  needs?
• ..
• They need mechanisms and technologies for value
  expression, value translation, and value
  enforcement.

20
Grid Entities and Architecture
GSP site scheduler
Grid consumer
Resource owners
broker
GSP global scheduler
GSP site scheduler
Resource owners
Market Maker
accounting
End users
Private enterprises
National providers
21
A Reference Service-Oriented Architecture for
Utility Grids
Data Catalogue
Grid Bank
Information Service
Grid Market Services
Sign-on
HealthMonitor
Info ?
Grid Node N

Grid Explorer

Secure
ProgrammingEnvironments
Job Control Agent
Grid Node1
Applications
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services

Deployment Agent
JobExec
Resource Allocation
Storage
Grid Resource Broker

R1
R2
Rm
Grid Middleware Services
Grid Consumer
Grid Service Providers
22
Gridbus and Complementary Technologies
realizing Utility Grid
Grid Applications

Portals
Science
Commerce
Engineering
Collaboratories

X-Parameter Sweep Lang.
Workflow
ExcellGrid
Gridscape
MPI
User-LevelMiddleware (Grid Tools)

Grid Brokers
Gridbus Data Broker
Workflow Engine
Nimrod-G
Core Grid Middleware
Grid Exchange Federation
Grid MarketDirectory
Globus
Unicore
Grid Storage Economy
GridBank

Alchemi
NorduGrid
XGrid
GRIDSIM
.NET
JVM
Condor
SGE
Tomcat
PBS
Libra
Grid Economy
Grid Fabric Software
Mac
AIX
Solaris
Windows
Linux
IRIX
OSF1
Grid Fabric Hardware
Worldwide Grid
23
Alchemi .NET-based Enterprise Grid Platform
Web Services
Alchemi Manager
Web Services
Internet
Alchemi Users
Internet

• SETI_at_Home like Model
• General Purpose
• Dedicated/Non-dedicate workers
• Role-based Security
• .NET and Web Services
• C Implementation
• GridThread and Job Model Programming
• Easy to setup and use
• Widely in use!

Alchemi Worker Agents
24
Some Users of Alchemi
Tier Technologies, USA Large scale document
processing using Alchemi framework
Satyam Computers Applied Research Laboratory,
India Micro-array data processing using Alchemi
framework
CSIRO, Australia Natural Resource Modeling
The University of Sao Paulo, Brazil The Alchemi
Executor as a Windows Service
stochastix GmbH, Germany Asynchronous Excel Tasks
using ManagedXLL and Alchemi .Net Grid Computing
framework.
The Friedrich Miescher Institute (FMI) for
Biomedical Research, Switzerland Patterns of
transcription factors in mammalian genes
Many users in Universities See next for an
example.
25
On Demand Assembly of Services Putting Them All
Together
Data Source
(Instruments/distributed sources)
Cluster Scheduler
PE
Grid Service Provider (GSP) (e.g., CERN)
26
The Gridbus Grid Service Broker for Data Grid
Applications

• Builds on the Nimrod-G Computational Grid Broker
  and Computational Economy Buyya, Abramson,
  Giddy, Monash University, 1999-2001
• And
• Extends its notion for Data and Service Grids

27
Gridbus Broker Architecture
Gridbus Client
Gridbus Client
Gribus Client
(Bag of Tasks Applications)
App, T, , Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Trading Manager
RecordKeeper
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS

GE GIS, NWS
Grid Info Server
RM TS
G

Data Catalog
Data Node
C

U
G
Unicore enabled node.
Globus enabled node.
L
A
RM Local Resource Manager, TS Trade Server
Alchemi enabled node.
28
Gridbus Broker and Remote Service Access Enablers
Credential Repository MyProxy
Portlets
Data Store
Access Technology
SRB
Grid FTP
29
Gridbus Services for eScience applications

• Application Development Environment
• XML-based language for composition of task
  farming (legacy) applications as parameter sweep
  applications.
• Task Farming APIs for new applications.
• Web APIs (e.g., Portlets) for Grid portal
  development.
• Threads-based Programming Interface
• Workflow interface and Gridbus-enabled workflow
  engine.
• Resource Allocation and Scheduling
• Dynamic discovery of optional computational and
  data nodes that meet user QoS requirements.
• Hide Low-Level Grid Middleware interfaces
• Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

30
Click Here for Demo
Figure 3 Logging into the portal.
Drug Design Made Easy!
31
Economy-based Data Grid Scheduling
CLICK HERE TO SKIP IF RUNNING OUT of TIME

• High Energy Physics as eScience Application Case
  Study

32
Australian Belle Data Grid Testbed
33
Case Study Event Simulation and Analysis
B0-gtDD-Ks

• Simulation and Analysis Package - Belle Analysis
  Software Framework (BASF)
• Experiment in 2 parts Generation of Simulated
  Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) were
distributed among the five nodes
34
Resources Used and their Service Price
Organization   Node details Role Cost (in G/CPU-sec)
CS,UniMelb belle.cs.mu.oz.au4 CPU, 2GB RAM, 40 GB HD, Linux Broker host, Data host, NWS server N.A. (Not used as a compute resource)
Physics, UniMelb fleagle.ph.unimelb.edu.au1 CPU, 512 MB RAM, 40 GB HD, Linux Replica Catalog host, Data host, Compute resource, NWS sensor 2
CS, University of Adelaide belle.cs.adelaide.edu.au 4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux Data host, NWS sensor N.A. (Not used as a compute resource)
ANU, Canberra belle.anu.edu.au4 CPU, 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor 4
Dept of Physics, USyd belle.physics.usyd.edu.au4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor 4
VPAC, Melbourne brecca-2.vpac.org 180 node cluster (only head node used), Linux Compute resource, NWS sensor 6
35
Network Cost (in Grid /Currency!)
36
Deploying Application Scenario

• A data grid scenario with 100 jobs and each
  accessing remote data of 30MB
• Deadline 3hrs.
• Budget G 60K
• Scheduling Optimisation Scenario
• Minimise Time
• Minimise Cost
• Results

37
Grid and Gridbus Technologies for Various Grid
(Market) Types
free trading
Publiccomputing (Alchemi)
Private enterprises (Libra, Gridbus, Globus)
SharingModel
National provider(Globus, Gridbus,..)
regulation
scientific
commercial
Application Category
38
(5) IT services as the fifth utility (water,
electricity, gas, telephone, IT)
eScience eBusiness eGovernment eHealth Multilingua
l eEducation
39
Summary and Conclusion

• Grids exploit synergies that result from
  cooperation of autonomous entities
• Resource sharing, dynamic provisioning, and
  aggregation at global level.
• Grid Economy provides incentive needed for
  sustained cooperation.
• Grid Network has potential to serve as
  Cyberinfrastructure for Utility Computing
• Grids offer enormous opportunities for realizing
  eScience and eBusiness at global level.

40
Any Questions ?
Gridbus Project - http//www.gridbus.org
41
Thanks for your attention!
The Gridbus Cooperation! http//www.gridbus.com
42
Backup Slides
43
Some Open Research Challenges

• Value expression, translation, and enforcement
  mechanisms and supporting Grid technologies for
• different economic models for spot markets and
  futures
• application models
• Dynamic Pricing Schemes
• Interaction Protocols for Service Negotiation
• Micro payments and Digital Currencies
• Scheduling Algorithms
• Programming Environments for Building Information
  Utility Applications
• Last, but not least
• Dispute Managements and Legal Issues
• Taxation (consult your National Taxation Office)
• State, national, and international boundaries
• Tax returns!

44
This talk is designed to answer

• How can Grid technologies support the emergence
  and operation of virtual enterprises?
• How can Grid shared resources be treated,
  brokered, and marketed as ICT commodities or
  futures among networked organisations?
• What kind of Grid architecture is needed for
  handling such market mechanisms in an automated
  fashion?
• How can Grid economies map the evolution of
  networked business models?

45
What do Grids aim for and how to support them.

• Grids aim at exploiting synergies that result
  from cooperation of autonomous distributed
  entities. Synergies include
• Resource sharing
• On-demand Virtual Enterprises creation
• Aggregation of resources on demand.
• For this cooperation to be sustainable,
  participants needs to have (economic) incentive.
• Therefore, incentive mechanisms should be
  considered as one of key design parameters of
  Grid computing.

46
Grid Market (Participant) Types and Application
Category
free trading
Publiccomputing
Private enterprises
SharingModel
National provider
regulation
scientific
commercial
Application Category
47
Appropriate Market Model for different market
types
high
Variable price auction
Commodity market
Willingness to Pay
Posted price oligopoly
low
weak
strong
Demand elasticity
48
Realising Market-based Grid Minimal New
Components

• Grid Market Directory Services
• Grid Trading Services
• for different economic models
• Grid Metering Services
• Grid Accounting and Payment Services
• Grid Service Broker

49
Deadline (D) and Budget (B) Constrained
Scheduling Algorithms
Algorithm Execution Time (D) Execution Cost (B) Compute Grid Data Grid
Cost Opt Limited by D Minimize Yes Yes
Cost-Time Opt Minimize if possible Minimize Yes
Time Opt Minimize Limited by B Yes Yes
Conservative-Time Opt Minimize Limited by B, jobs have guaranteed minimum budget Yes