Grid%20Economics%20and%20Business%20Models:%20A%20Gridbus%20Perspective

1
Grid Economics and Business Models A Gridbus
Perspective
Grid User Meet, Bengaluru, India

• Rajkumar Buyya

Grid and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of MelbourneMelbourne,
Australiawww.gridbus.org
2
Outline

• Introduction
• Utility Networks and Grid Computing
• Global Grids and Challenges
• Security. Resource management, pricing and
  service models
• Service Oriented Grids and Grid Economy
• SOGA, Grid Market Directory, Grid Bank, Broker
• Grid Service Broker
• Architecture, Design and Implementation
• Performance Evaluation Experiments in Creation
  and Deployment of Applications on Global Grids
• A Case Study in High Energy Physics
• Summary

3
4 Essential Utilities and Delivery Networks
4
(5) IT services as the fifth utility (water,
electricity, gas, telephone, IT)
eScience eBusiness eGovernment eHealth Multilingua
l eEducation
5
Leading to Grid (computing) ParadigmCyberinfra
structure for sharing resources

• Inspired by Power Grid!
• A service-oriented/utility computing paradigm
  that enables seamless sharing of geographically
  distributed, autonomous resources (for PROFIT).
• This was the original aim of building Internet
  although it ended up in giving birth to email!

6
A Bird Eye View of World-Wide Grid Environment
Grid Information Service
Grid Resource Broker
Application
R2
R3
R4
R5
RN
Grid Resource Broker
R6
R1
Resource Broker
Grid Information Service
7
Various Types of Grid Services

• Computational Services CPU cycles
• SETI_at_Home, NASA IPG, TeraGrid, I-Grid,
• Data Services
• Data replication, management, secure access--LHC
  Grid/Napster
• Application Services
• Access to remote software/libraries and license
  managementNetSolve
• Interaction Services
• eLearning, Virtual Tables, Group Communication
  (Access Grid), Gaming
• Knowledge Services
• The way knowledge is acquired and manageddata
  mining.
• Utility Computing Services
• Towards a market-based Grid computing Leasing
  and delivering Grid services as ICT utilities.

Utility Grid
Knowledge Grid
Interaction Grid
ASP Grid
Data Grid
Computational Grid
8
Worldwide Grid Spending

• After the year 2006, business popularity of Grid
  computing is expected to be accelerate
  exponentially
• Especially, the financial services and ERP
  services is expected to take major parts in the
  expense

Source Insight Research Corp.
9
Grid Challenges
10
Some Grid Initiatives Worldwide

• Australia
• Nimrod-G
• Gridbus
• DISCWorld
• GrangeNet.
• APACGrid
• ARC eResearch
• Brazil
• OurGrid, EasyGrid
• LNCC-Grid many others
• China
• ChinaGrid Education
• CNGrid - application
• Europe
• UK eScience
• EU Grids..
• and many more...
• India
• Garuda

• USA
• Globus
• NASA IPG
• AccessGrid
• TeraGrid
• Cyberinfrasture
• Industry Initiatives
• IBM On Demand Computing
• HP Adaptive Computing
• Sun N1
• Microsoft - .NET
• Oracle 10g
• Satyam Grid Practice
• Infosys, Wipro, TCS
• StorageTek Grid..
• Public Forums
• Global Grid Forum
• Australian Grid Forum
• Conferences

27 million
1.3 billion 3 yrs
2? billion
120million 5 yrs
450million 5 yrs
486million 5 yrs
1.3 billion (Rs)
1 billion 5 yrs
http//www.gridcomputing.com
11
Grid (Market) Participant Types and Application
Category
free trading
Publiccomputing (SETI_at_Home, Alchemi, UD)
Private enterprises(Satyam, IBM, Sun)
SharingModel
National provider(KGrid, TeraGrid,
Garuda/IndiaGrid, UKGrid, AusGrid)
regulation
scientific
commercial
Application Category
12
mix-and-match (service)
Object-oriented
Internet/partial-P2P
Grid Computing Approaches
Network enabled Solvers
Economic-based Utility / Service-Oriented
Computing
Nimrod-G
13
The Gridbus Project _at_ MelbourneEnable Leasing
of ICT Services on Demand
Distributed Data
WWG
Gridbus
World Wide Grid! ?On Demand Utility Computing
14
(No Transcript)
15
Grid Economy Methodology for Sustained Resourced
Sharing and Managing Supply-and-Demand for
Resources
16
New challenges of Grid Economy

• Resource Owners
• How do I decide prices ? (economic models?)
• How do I specify them ?
• How do I translate price to resource allocation ?
• How do I enforce them ?
• How do I advertise attract consumers ?
• How do I do accounting and handle payments?
• ..
• Resource Consumers
• How do I decide expenses ?
• How do I express QoS requirements ?
• How do I trade between timeframe cost ?
• How do I map jobs to resources to meet my QoS
  needs?
• ..
• They need mechanisms and technologies for value
  expression, value translation, and value
  enforcement.

17
Grid Entities and Architecture
GSP site scheduler
Grid consumer
Resource Provider
broker
GSP global scheduler
GSP site scheduler
Resource Provider
Market Maker
accounting
End users
Private enterprises
National providers
18
A Reference Service-Oriented Architecture for
Utility Grids
Data Catalogue
Grid Bank
Information Service
Grid Market Services
Sign-on
HealthMonitor
Info ?
Grid Node N

Grid Explorer

Secure
ProgrammingEnvironments
Job Control Agent
Grid Node1
Applications
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services

Deployment Agent
JobExec
Resource Allocation
Storage
Grid Resource Broker

R1
R2
Rm
Grid Middleware Services
Grid Consumer
Grid Service Providers
19
Gridbus and Complementary Technologies
realizing Utility Grid
Grid Applications

Portals
Science
Commerce
Engineering
Collaboratories

X-Parameter Sweep Lang.
Workflow
ExcellGrid
Gridscape
MPI
User-LevelMiddleware (Grid Tools)

Grid Brokers
Gridbus Data Broker
Workflow Engine
Nimrod-G
Core Grid Middleware
Grid Exchange Federation
Grid MarketDirectory
Globus
Unicore
Grid Storage Economy
GridBank

Alchemi
NorduGrid
XGrid
GRIDSIM
.NET
JVM
Condor
SGE
Tomcat
PBS
Libra
Grid Economy
Grid Fabric Software
Mac
AIX
Solaris
Windows
Linux
IRIX
OSF1
Grid Fabric Hardware
Worldwide Grid
20
On Demand Assembly of Services Putting Them All
Together
21
Alchemi .NET-based Enterprise Grid Platform
Web Services
Alchemi Manager
Web Services
Internet
Alchemi Users
Internet

• SETI_at_Home like Model
• General Purpose
• Dedicated/Non-dedicate workers
• Role-based Security
• .NET and Web Services
• C Implementation
• GridThread and Job Model Programming
• Easy to setup and use
• Widely in use!

Alchemi Worker Agents
22
Some Users of Alchemi
Tier Technologies, USA Large scale document
processing using Alchemi framework
Satyam Computers Applied Research Laboratory,
India Micro-array data processing using Alchemi
framework
CSIRO, Australia Natural Resource Modeling
The University of Sao Paulo, Brazil The Alchemi
Executor as a Windows Service
stochastix GmbH, Germany Asynchronous Excel Tasks
using ManagedXLL and Alchemi .Net Grid Computing
framework.
The Friedrich Miescher Institute (FMI) for
Biomedical Research, Switzerland Patterns of
transcription factors in mammalian genes
Many users in Universities See next for an
example.
23
The Gridbus Grid Service Broker for Data Grid
Applications

• Builds on the Nimrod-G Computational Grid Broker
  and Computational Economy Buyya, Abramson,
  Giddy, Monash University, 1999-2001
• And
• Extends its notion for Data and Service Grids

24
Gridbus Broker Architecture
Gridbus Client
Gridbus Client
Gribus Client
(Bag of Tasks Applications)
App, T, , Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Trading Manager
RecordKeeper
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS

GE GIS, NWS
Grid Info Server
RM TS
G

Data Catalog
Data Node
C

U
G
Unicore enabled node.
Globus enabled node.
L
A
RM Local Resource Manager, TS Trade Server
Alchemi enabled node.
25
Gridbus Broker and Remote Service Access Enablers
Credential Repository MyProxy
Portlets
Data Store
Access Technology
SRB
Grid FTP
26
Gridbus Services for eScience applications

• Application Development Environment
• XML-based language for composition of task
  farming (legacy) applications as parameter sweep
  applications.
• Task Farming APIs for new applications.
• Web APIs (e.g., Portlets) for Grid portal
  development.
• Threads-based Programming Interface
• Workflow interface and Gridbus-enabled workflow
  engine.
• Resource Allocation and Scheduling
• Dynamic discovery of optional computational and
  data nodes that meet user QoS requirements.
• Hide Low-Level Grid Middleware interfaces
• Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

27
Click Here for Demo
Drug Design Made Easy!
28
Economy-based Data Grid Scheduling
CLICK HERE TO SKIP IF RUNNING OUT of TIME

• High Energy Physics as eScience Application Case
  Study

29
Case Study High Energy Physics and Data Grid

• The Belle Experiment
• KEK B-Factory, Japan
• Investigating fundamental violation of symmetry
  in nature (Charge Parity) which may help explain
  the universal matter antimatter imbalance.
• Collaboration 400 people, 50 institutes
• 100s TB data currently

30
Australian Belle Data Grid Testbed
VPACMelbourne
31
Case Study Event Simulation and Analysis
B0-gtDD-Ks

• Simulation and Analysis Package - Belle Analysis
  Software Framework (BASF)
• Experiment in 2 parts Generation of Simulated
  Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) were
distributed among the five nodes
32
Belle Data Grid (GSP CPU Service Price G/sec)
G4
NA
G4
G6
VPACMelbourne
G2
Datanode
33
Belle Data Grid (Bandwidth Price G/MB)
32
33
36
G4
31
30
34
NA
38
31
G4
G6
VPACMelbourne
G2
Datanode
34
Network Cost (in Grid /Currency!)
35
Deploying Application Scenario

• A data grid scenario with 100 jobs and each
  accessing remote data of 30MB
• Deadline 3hrs.
• Budget G 60K
• Scheduling Optimisation Scenario
• Minimise Time
• Minimise Cost
• Results

36
Time Minimization in Data Grids
37
Results Cost Minimization in Data Grids
38
Observation
Organization   Node details Cost (in G/CPU-sec) Total Jobs Executed Total Jobs Executed
Organization   Node details Cost (in G/CPU-sec) Time Cost
CS,UniMelb belle.cs.mu.oz.au4 CPU, 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- --
Physics, UniMelb fleagle.ph.unimelb.edu.au1 CPU, 512 MB RAM, 40 GB HD, Linux 2 3 94
CS, University of Adelaide belle.cs.adelaide.edu.au 4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- --
ANU, Canberra belle.anu.edu.au4 CPU, 2GB RAM, 40 GB HD, Linux 4 2 2
Dept of Physics, USyd belle.physics.usyd.edu.au4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux 4 72 2
VPAC, Melbourne brecca-2.vpac.org 180 node cluster (only head node used), Linux 6 23 2
39
Grid and Gridbus Technologies for Various Grid
(Market) Types
free trading
Publiccomputing (Alchemi)
Private enterprises (Libra, Gridbus, Globus)
SharingModel
National provider(Globus, Gridbus,..)
regulation
scientific
commercial
Application Category
40
(5) IT services as the fifth utility (water,
electricity, gas, telephone, IT)
eScience eBusiness eGovernment eHealth Multilingua
l eEducation
41
Summary and Conclusion

• Grids exploit synergies that result from
  cooperation of autonomous entities
• Resource sharing, dynamic provisioning, and
  aggregation at global level.
• Grid Economy provides incentive needed for
  sustained cooperation.
• Grid Network has potential to serve as
  Cyberinfrastructure for Utility Computing
• Grids offer enormous opportunities for realizing
  eScience and eBusiness at global level.

42
Any Questions ?
Gridbus Project - http//www.gridbus.org
43
Thanks for your attention!
The Gridbus Cooperation! http//www.gridbus.com
44
Backup Slides
45
This talk is designed to answer

• How can Grid technologies support the emergence
  and operation of virtual enterprises?
• How can Grid shared resources be treated,
  brokered, and marketed as ICT commodities or
  futures among networked organisations?
• What kind of Grid architecture is needed for
  handling such market mechanisms in an automated
  fashion?
• How can Grid economies map the evolution of
  networked business models?

46
Realising Market-based Grid Minimal New
Components

• Grid Market Directory Services
• Grid Trading Services
• for different economic models
• Grid Metering Services
• Grid Accounting and Payment Services
• Grid Service Broker

47
Deadline (D) and Budget (B) Constrained
Scheduling Algorithms
Algorithm Execution Time (D) Execution Cost (B) Compute Grid Data Grid
Cost Opt Limited by D Minimize Yes Yes
Cost-Time Opt Minimize if possible Minimize Yes
Time Opt Minimize Limited by B Yes Yes
Conservative-Time Opt Minimize Limited by B, jobs have guaranteed minimum budget Yes