Title: The Gridbus Middleware: Creating and Managing Utility Grids for Powering e-Science and e-Business Applications
1The Gridbus MiddlewareCreating and Managing
Utility Grids for Powering e-Science and
e-Business Applications
Grid Computing and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of Melbourne,
Australiaww.gridbus.org
Gridbus Sponsors
2Outline
- Introduction to the University
- Melbourne, GRIDS Lab, and Opportunities
- Recap of the First Lecture
- What are Grids, Challenges, Middleware Solutions
- Service-Oriented Grid Architecture and Gridbus
Solutions - Market-based Management, Grid Market Directory,
Grid Bank - Grid Service Broker
- Architecture, Design and Implementation
- Performance Evaluation Experiments in Creation
and Deployment of Applications on Global Grids - A Case Study in High Energy Physics
- Summary and Conclusion
3Melbourne!How does it compares to Osaka city?
- Melbourne is Australias 2nd largest and the most
multicultural city. - Melbournes residents represent 110 nationalities
and speak 151 languages - Greater Melbourne population is 3.5 million
- Extensive parks and gardens, restaurants, cafes,
bustling market places, theatre, art and
entertainment - Outstanding Educational and Research Institutes
-
- Often rated as Worlds most livable city
4University of Melbourne at a Glance
- Established 1853
- One of the oldest universities in Australia and
the Asia Pacific region. - A comprehensive university (Arts, Science,
Engineering, Economics, Business, Agriculture,
Medicine) - Engineering started in 1861.
- Currently a total of 43,000 students
- 30 of them are International.
5Global Standing
- Ranked as Australias 1 University in Research.
- Ranked 19 in Top200 Universities Worldwide by
Times Higher Education. - UoM is the only Australian University in TOP 20
in the world. - Engineering and ICT 18 in the world.
- Strong in Biomedical Research - 10 in the world.
- Internationally renowned academics who are
leading researchers in their respective fields
including 4 Nobel Laureates
6GRIDS Lab _at_ Melbourne
Education
R D
- Youngest and one of the rapidly growing research
labs in our School/University - Founded in 2002
- Houses
- Research Fellows (3)
- Research Programmers (3)
- PhD candidates (10)
- Honours/Masters students (5)
- Funding
- National and International organizations
- Australian Research Council
- Many industries (Sun, StorageTek, Microsoft, IBM,
Microsoft) - University-wide collaboration
- Faculties of Science, Engineering, and Medicine
- Many national and international collaborations.
- Academics
- Industries
- Software
- Widely in academic and industrial users.
- Publication
Community Services e.g., IEEE TC for Scalable
Computing
7GRIDS Lab Research Probeshttp//www.gridbus.org/r
esearch_probes.html
- Grid Economy and Scheduling
- Data Grid Brokering and Scheduling
- Workflow Scheduling and Grid Economy
- Cluster Economy and Scheduling
- .NET Based Grid Computing
- Grid Market Directory and Service Publication
- Grid Simulation (GridSim)
- Resource Usage Accounting
- Meta Search Engine and Web Services
- Web-based Grid Portals
- Gridscape Grid Monitoring Portal and its
Integration with Google Maps - P2P Compute Power Market
- Distributed Application Composition
- Sensor Grids and Open Sensor Web Architecture
8GRIDS Lab Books and Publication
9GRIDS Lab Team Members Origin
- Australia
- India
- China
- Singapore
- Indonesia
- Czech Republic
- Brazil
- Bangladesh
- Korea
- Japan
- Many short term visiting researchers from USA and
Europe
10Join us and experience
- Melbourne Advantage
- multicultural environment
- Worlds most liveable city
- The University
- Australias 1 University
- Many international students and professors
- One of the best places in the world to conduct
Grid computing research! - Melbourne Welcome you.
11Outline
- Introduction to the University
- Melbourne, GRIDS Lab, and Opportunities
- Recap of the First Lecture
- What are Grids, Challenges, Middleware Solutions
- Service-Oriented Grid Architecture and Gridbus
Solutions - Market-based Management, Grid Market Directory,
Grid Bank - Grid Service Broker
- Architecture, Design and Implementation
- Performance Evaluation Experiments in Creation
and Deployment of Applications on Global Grids - A Case Study in High Energy Physics
- Summary and Conclusion
12Computer Systems Single -gt Global
Computer Systems
Distributed Systems
Single System
(multiple systems)
PC/Workstation
SMP/NUMA
Vector
Mainframe
Client Server
Clusters
Grids
Peer-to-Peer
Control and Management
Centralised
Decentralised
13What is Grid?(there are several academic
definitions, here is ours)
- A type of parallel and distributed system that
enables the sharing, exchange, selection,
aggregation of geographically distributed
autonomous resources - Computers PCs, workstations, clusters,
supercomputers, laptops, notebooks, mobile
devices, PDA, etc - Software e.g., ASPs renting expensive special
purpose applications on demand - Catalogued data and databases e.g. transparent
access to human genome database - Special devices/instruments e.g., radio
telescope SETI_at_Home searching for life in
galaxy. - People/collaborators.
- depending on their availability, capability,
cost, and user QoS requirements.
Widearea
14How Are Grids Used?
Utility computing
High-performance computing
Collaborative design
Financial modeling
High-energy physics
E-Business
Drug discovery
Life sciences
Data center automation
E-Science
Natural language processing Data Mining
Collaborative data-sharing
15Grid Challenges
16Open-Source Grid Middleware Projects
17Layers of Grid Architecture Middleware
Grid applications Web Portals, Applications,
User level
Grid programming environment and tools Languages,
API, libraries, compilers, parallelization tools
User-LevelMiddleware
Resource management and scheduling
Adaptive Management
Job submission, info services, Storage access,
Trading, Accounting, License
CoreMiddleware
Security Services Authentication, Single sign-on,
secure communication
Grid resources Desktops, servers, clusters,
networks, applications, storage, devices
resource manager monitor
System level
18The Gridbus Project _at_ MelbourneEnable Leasing
of ICT Services on Demand
WWG
Gridbus
Pushes Grid computing into mainstream computing
19 20Outline
- Introduction to the University
- Melbourne, GRIDS Lab, and Opportunities
- Recap of the First Lecture
- What are Grids, Challenges, Middleware Solutions
- Service-Oriented Grid Architecture and Gridbus
Solutions - Market-based Management, Grid Market Directory,
Grid Bank - Grid Service Broker
- Architecture, Design and Implementation
- Performance Evaluation Experiments in Creation
and Deployment of Applications on Global Grids - A Case Study in High Energy Physics
- Summary and Conclusion
21What does Grid players want?
- Grid Consumers
- Execute jobs for solving varying problem size and
complexity - Benefit by utilizing distributed resources wisely
- Tradeoff timeframe and cost
- Strategy minimise expenses
- Grid Providers
- Contribute resources for executing consumer jobs
- Benefit by maximizing resource utilisation
- Tradeoff local requirements market opportunity
- Strategy maximise return on investment
22What does Grid players require?
- They need tools and technologies that help them
in value expression, value translation, and value
enforcement. - Grid Service Consumers (GSCs)
- How do I express QoS requirements ?
- How do I trade between timeframe cost ?
- How do I map jobs to resources to meet my QoS
needs? - How do I manage Grid dynamics and get my work
done? -
- Grid Service Providers (GSPs)
- How do I decide service pricing models ?
- How do I specify them ?
- How do I translate them into resource allocations
? - How do I enforce them ?
- How do I advertise attract consumers ?
- How do I do accounting and handle payments?
23Solution 1 Market-Oriented Grid Computing - (a)
Sustained Resourced Sharing and (b) Effective
Management of Shared Resources
Grid Economy
24Solution 2 Service Oriented Architecture (SOA)
- A SOA is a contractual architecture for offering
and consuming software as services. - There are four entities that make up an SOA
- service provider,
- service registry, and
- service consumer (also known as service
requestor). - The functions or tasks that the service provider
offers, along with other functional and technical
information required for consumption, are defined
in - the service definition or contract.
registry
contract
provider
consumer
25Service-Oriented Grid Architecture
Data Catalogue
Grid Bank
Information Service
Grid Market Services
Sign-on
HealthMonitor
Info ?
Grid Node N
Grid Explorer
Secure
ProgrammingEnvironments
Job Control Agent
Grid Node1
Applications
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services
Deployment Agent
JobExec
Resource Allocation
Storage
Grid Resource Broker
R1
R2
Rm
Core Middleware Services
Grid Servuce Consumer
Grid Service Providers
26Gridbus and Complementary Technologies
realizing Utility Grid
Grid Applications
Portals
Science
Commerce
Engineering
Collaboratories
X-Parameter Sweep Lang.
Workflow
ExcellGrid
Gridscape
MPI
User-LevelMiddleware
Grid Brokers
Gridbus Data Broker
Workflow Engine
Nimrod-G
Grid Exchange Federation
Grid MarketDirectory
Globus
Unicore
Grid Storage Economy
GridBank
Core Grid Middleware
Alchemi
NorduGrid
XGrid
Grid Economy
.NET
JVM
Condor
SGE
Tomcat
PBS
Libra
Grid Fabric Software
Mac
AIX
Solaris
Windows
Linux
IRIX
OSF1
Grid Fabric Hardware
Worldwide Grid
27(No Transcript)
28Grid Market Directory (GMD)
29Grid Bank Market-based Grid Access Management
GridBank Server
GridCheque Resource Usage (GSC Account Charge
GridCheque
Establish Service Costs
Applications
Grid Trade Server
GridBank Charging Module
Grid Resource Broker (GRB)
GridBank Payment Module
GridCheque
Resource Usage
Grid Agent
Grid Resource Meter
User
Deploy Agent and Submt Jobs
Usage Agreement
R1
R2
R3
R4
Grid Service Consumer (GSC)
User
Grid Service Provider (GSP)
30Outline
- Introduction to the University
- Melbourne, GRIDS Lab, and Opportunities
- Recap of the First Lecture
- What are Grids, Challenges, Middleware Solutions
- Service-Oriented Grid Architecture and Gridbus
Solutions - Market-based Management, Grid Market Directory,
Grid Bank - Grid Service Broker
- Architecture, Design and Implementation
- Performance Evaluation Experiments in Creation
and Deployment of Applications on Global Grids - A Case Study in High Energy Physics
- Summary and Conclusion
31Grid Service Broker (GSB)
- A resource broker for scheduling task farming
data Grid applications with static or dynamic
parameter sweeps on global Grids. - It uses computational economy paradigm for
optimal selection of computational and data
services depending on their quality, cost, and
availability, and users QoS requirements
(deadline, budget, T/C optimisation) - Key Features
- A single window to manage control experiment
- Programmable Task Farming Engine
- Resource Discovery and Resource Trading
- Optimal Data Source Discovery
- Scheduling Predications
- Generic Dispatcher Grid Agents
- Transportation of data sharing of results
- Accounting
32Gridbus Broker Architecture
Gridbus Client
Gridbus Client
Gribus Client
(Bag of Tasks Applications)
App, T, , Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Trading Manager
RecordKeeper
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS
GE GIS, NWS
Grid Info Server
RM TS
G
Data Catalog
Data Node
C
U
G
Unicore enabled node.
Globus enabled node.
L
A
RM Local Resource Manager, TS Trade Server
Alchemi enabled node.
33Gridbus Broker and Remote Service Access Enablers
Credential Repository MyProxy
Portlets
Data Store
Access Technology
SRB
Grid FTP
34Gridbus Services for eScience applications
- Application Development Environment
- XML-based language for composition of task
farming (legacy) applications as parameter sweep
applications. - Task Farming APIs for new applications.
- Web APIs (e.g., Portlets) for Grid portal
development. - Threads-based Programming Interface
- Workflow interface and Gridbus-enabled workflow
engine. - Resource Allocation and Scheduling
- Dynamic discovery of optional computational and
data nodes that meet user QoS requirements. - Hide Low-Level Grid Middleware interfaces
- Globus (v2, v4), SRB, Alchemi, Unicore, and
ssh-based access to local/remote resources
managed by XGrid, Condor, SGE.
35Click Here for Demo
Figure 3 Logging into the portal.
Drug Design Made Easy!
36Excel Plugin to Access Gridbus Services
37Adaptive Scheduling Steps
Discover More Resources
Discover Resources
Establish Rates
Evaluate Reschedule
Compose Schedule
Meet requirements ? Remaining Jobs, Deadline,
Budget ?
Distribute Jobs
38Deadline (D) and Budget (B) Constrained
Scheduling Algorithms
39Outline
- Introduction to the University
- Melbourne, GRIDS Lab, and Opportunities
- Recap of the First Lecture
- What are Grids, Challenges, Middleware Solutions
- Service-Oriented Grid Architecture and Gridbus
Solutions - Market-based Management, Grid Market Directory,
Grid Bank - Grid Service Broker
- Architecture, Design and Implementation
- Performance Evaluation Experiments in Creation
and Deployment of Applications on Global Grids - A Case Study in High Energy Physics
- Summary and Conclusion
40Case Study High Energy Physics and Data Grid
- The Belle Experiment
- KEK B-Factory, Japan
- Investigating fundamental violation of symmetry
in nature (Charge Parity) which may help explain
why do we have more antimatter in the
universe?. - Collaboration 1000 people, 50 institutes
- 100s TB data currently
41Case Study Event Simulation and Analysis
B0-gtDD-Ks
- Simulation and Analysis Package - Belle Analysis
Software Framework (BASF) - Experiment in 2 parts Generation of Simulated
Data and Analysis of the distributed data -
?Analyzed 100 data files (30MB each) that were
distributed among the five nodes within
Australian Belle DataGrid platform.
42Australian Belle Data Grid Testbed
VPACMelbourne
43Belle Data Grid (GSP CPU Service Price G/sec)
G4
NA
G4
G6
VPACMelbourne
G2
Datanode
44Belle Data Grid (Bandwidth Price G/MB)
32
33
36
G4
31
30
34
NA
38
31
G4
G6
VPACMelbourne
G2
Datanode
45Deploying Application Scenario
- A data grid scenario with 100 jobs and each
accessing remote data of 30MB - Deadline 3hrs.
- Budget G 60K
- Scheduling Optimisation Scenario
- Minimise Time
- Minimise Cost
- Results
46Time Minimization in Data Grids
47Results Cost Minimization in Data Grids
48Observation
49Outline
- Introduction to the University
- Melbourne, GRIDS Lab, and Opportunities
- Recap of the First Lecture
- What are Grids, Challenges, Middleware Solutions
- Service-Oriented Grid Architecture and Gridbus
Solutions - Market-based Management, Grid Market Directory,
Grid Bank - Grid Service Broker
- Architecture, Design and Implementation
- Performance Evaluation Experiments in Creation
and Deployment of Applications on Global Grids - A Case Study in High Energy Physics
- Summary and Conclusion
50The GridSim ToolkitA Java based tool for Grid
Scheduling Simulations
Application, User, Grid Scenarios Input and
Results
. . .
Application Configuration
Resource Configuration
Visual Modeler
Grid Scenario
Output
Grid Resource Brokers or Schedulerss Simulation
GridSim Toolkit
Application Modeling
Information Services
Resource Allocation
Statistics
Job Management
Resource Entities
Add your own policy for resource allocation
Resource Modeling and Simulation (with Time and
Space shared schedulers)
Clusters
Single CPU
Reservation
SMPs
Load Pattern
Network
Basic Discrete Event Simulation Infrastructure
SimJava
Distributed SimJava
Virtual Machine (Java, cJVM, RMI)
Distributed Resources
PCs
Workstations
Clusters
SMPs
51Selected GridSim Users - 2002
52Summary and Conclusion
- Grids exploit synergies that result from
cooperation of autonomous entities - Resource sharing, dynamic provisioning, and
aggregation at global level ?Great Science and
Great Business! - Grids have emerged as enabler for
Cyberinfrastructure that powers e-Science and
e-Business applications. - SOA Market-based Grid Management Utility
Grids - Grids allow users to dynamically lease Grid
services at runtime based on their quality, cost,
availability, and users QoS requirements. - Delivering ICT services as computing utilities.
- Grids offer enormous opportunities for realizing
e-Science and e-Business at global level. - Use our Gridbus technology to realise this and
make money!
53Melbourne welcomes you
- Melbourne as a place
- multicultural environment
- Worlds most liveable city
- The University
- Australias 1 University
- Many international students and professors
- One of the best places in the world to conduct
Grid computing research! - My Contact Details
- rbuyya_at_unimelb.edu.au
- www.buyya.com
- 61-3-83441244
54You could be a co-author of my next book!
55Thanks for your attention!
We Welcome Cooperation in Research and
Commercialisation! http/www.gridbus.org
http//www.gridbus.com
56Big Question?
- Can computational grids drive the economy of the
21st century similar to the electric power grid
that drove the economy of the 20th century?
57(No Transcript)