The Gridbus Middleware: Creating and Managing Utility Grids for Powering e-Science and e-Business Applications - PowerPoint PPT Presentation

1 / 57
About This Presentation
Title:

The Gridbus Middleware: Creating and Managing Utility Grids for Powering e-Science and e-Business Applications

Description:

Melbourne is Australia's 2nd largest and the most multicultural city. Melbourne's residents represent 110 nationalities and speak 151 languages ... – PowerPoint PPT presentation

Number of Views:129
Avg rating:3.0/5.0
Slides: 58
Provided by: grid6
Category:

less

Transcript and Presenter's Notes

Title: The Gridbus Middleware: Creating and Managing Utility Grids for Powering e-Science and e-Business Applications


1
The Gridbus MiddlewareCreating and Managing
Utility Grids for Powering e-Science and
e-Business Applications
  • Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS)
LaboratoryDept. of Computer Science and Software
EngineeringThe University of Melbourne,
Australiaww.gridbus.org
Gridbus Sponsors
2
Outline
  • Introduction to the University
  • Melbourne, GRIDS Lab, and Opportunities
  • Recap of the First Lecture
  • What are Grids, Challenges, Middleware Solutions
  • Service-Oriented Grid Architecture and Gridbus
    Solutions
  • Market-based Management, Grid Market Directory,
    Grid Bank
  • Grid Service Broker
  • Architecture, Design and Implementation
  • Performance Evaluation Experiments in Creation
    and Deployment of Applications on Global Grids
  • A Case Study in High Energy Physics
  • Summary and Conclusion

3
Melbourne!How does it compares to Osaka city?
  • Melbourne is Australias 2nd largest and the most
    multicultural city.
  • Melbournes residents represent 110 nationalities
    and speak 151 languages
  • Greater Melbourne population is 3.5 million
  • Extensive parks and gardens, restaurants, cafes,
    bustling market places, theatre, art and
    entertainment
  • Outstanding Educational and Research Institutes
  • Often rated as Worlds most livable city

4
University of Melbourne at a Glance
  • Established 1853
  • One of the oldest universities in Australia and
    the Asia Pacific region.
  • A comprehensive university (Arts, Science,
    Engineering, Economics, Business, Agriculture,
    Medicine)
  • Engineering started in 1861.
  • Currently a total of 43,000 students
  • 30 of them are International.

5
Global Standing
  • Ranked as Australias 1 University in Research.
  • Ranked 19 in Top200 Universities Worldwide by
    Times Higher Education.  
  • UoM is the only Australian University in TOP 20
    in the world.
  • Engineering and ICT 18 in the world.
  • Strong in Biomedical Research - 10 in the world.
  • Internationally renowned academics who are
    leading researchers in their respective fields
    including 4 Nobel Laureates

6
GRIDS Lab _at_ Melbourne
Education
R D
  • Youngest and one of the rapidly growing research
    labs in our School/University
  • Founded in 2002
  • Houses
  • Research Fellows (3)
  • Research Programmers (3)
  • PhD candidates (10)
  • Honours/Masters students (5)
  • Funding
  • National and International organizations
  • Australian Research Council
  • Many industries (Sun, StorageTek, Microsoft, IBM,
    Microsoft)
  • University-wide collaboration
  • Faculties of Science, Engineering, and Medicine
  • Many national and international collaborations.
  • Academics
  • Industries
  • Software
  • Widely in academic and industrial users.
  • Publication

Community Services e.g., IEEE TC for Scalable
Computing
7
GRIDS Lab Research Probeshttp//www.gridbus.org/r
esearch_probes.html
  • Grid Economy and Scheduling
  • Data Grid Brokering and Scheduling
  • Workflow Scheduling and Grid Economy
  • Cluster Economy and Scheduling
  • .NET Based Grid Computing
  • Grid Market Directory and Service Publication
  • Grid Simulation (GridSim)
  • Resource Usage Accounting
  • Meta Search Engine and Web Services
  • Web-based Grid Portals
  • Gridscape Grid Monitoring Portal and its
    Integration with Google Maps
  • P2P Compute Power Market
  • Distributed Application Composition
  • Sensor Grids and Open Sensor Web Architecture

8
GRIDS Lab Books and Publication
9
GRIDS Lab Team Members Origin
  • Australia
  • India
  • China
  • Singapore
  • Indonesia
  • Czech Republic
  • Brazil
  • Bangladesh
  • Korea
  • Japan
  • Many short term visiting researchers from USA and
    Europe

10
Join us and experience
  • Melbourne Advantage
  • multicultural environment
  • Worlds most liveable city
  • The University
  • Australias 1 University
  • Many international students and professors
  • One of the best places in the world to conduct
    Grid computing research!
  • Melbourne Welcome you.

11
Outline
  • Introduction to the University
  • Melbourne, GRIDS Lab, and Opportunities
  • Recap of the First Lecture
  • What are Grids, Challenges, Middleware Solutions
  • Service-Oriented Grid Architecture and Gridbus
    Solutions
  • Market-based Management, Grid Market Directory,
    Grid Bank
  • Grid Service Broker
  • Architecture, Design and Implementation
  • Performance Evaluation Experiments in Creation
    and Deployment of Applications on Global Grids
  • A Case Study in High Energy Physics
  • Summary and Conclusion

12
Computer Systems Single -gt Global
Computer Systems
Distributed Systems
Single System
(multiple systems)
PC/Workstation
SMP/NUMA
Vector
Mainframe
Client Server
Clusters
Grids
Peer-to-Peer
Control and Management
Centralised
Decentralised
13
What is Grid?(there are several academic
definitions, here is ours)
  • A type of parallel and distributed system that
    enables the sharing, exchange, selection,
    aggregation of geographically distributed
    autonomous resources
  • Computers PCs, workstations, clusters,
    supercomputers, laptops, notebooks, mobile
    devices, PDA, etc
  • Software e.g., ASPs renting expensive special
    purpose applications on demand
  • Catalogued data and databases e.g. transparent
    access to human genome database
  • Special devices/instruments e.g., radio
    telescope SETI_at_Home searching for life in
    galaxy.
  • People/collaborators.
  • depending on their availability, capability,
    cost, and user QoS requirements.

Widearea
14
How Are Grids Used?
Utility computing
High-performance computing
Collaborative design
Financial modeling
High-energy physics
E-Business
Drug discovery
Life sciences
Data center automation
E-Science
Natural language processing Data Mining
Collaborative data-sharing
15
Grid Challenges
16
Open-Source Grid Middleware Projects
17
Layers of Grid Architecture Middleware
Grid applications Web Portals, Applications,
User level
Grid programming environment and tools Languages,
API, libraries, compilers, parallelization tools
User-LevelMiddleware
Resource management and scheduling
Adaptive Management
Job submission, info services, Storage access,
Trading, Accounting, License
CoreMiddleware
Security Services Authentication, Single sign-on,
secure communication
Grid resources Desktops, servers, clusters,
networks, applications, storage, devices
resource manager monitor
System level
18
The Gridbus Project _at_ MelbourneEnable Leasing
of ICT Services on Demand
WWG
Gridbus
Pushes Grid computing into mainstream computing
19

20
Outline
  • Introduction to the University
  • Melbourne, GRIDS Lab, and Opportunities
  • Recap of the First Lecture
  • What are Grids, Challenges, Middleware Solutions
  • Service-Oriented Grid Architecture and Gridbus
    Solutions
  • Market-based Management, Grid Market Directory,
    Grid Bank
  • Grid Service Broker
  • Architecture, Design and Implementation
  • Performance Evaluation Experiments in Creation
    and Deployment of Applications on Global Grids
  • A Case Study in High Energy Physics
  • Summary and Conclusion

21
What does Grid players want?
  • Grid Consumers
  • Execute jobs for solving varying problem size and
    complexity
  • Benefit by utilizing distributed resources wisely
  • Tradeoff timeframe and cost
  • Strategy minimise expenses
  • Grid Providers
  • Contribute resources for executing consumer jobs
  • Benefit by maximizing resource utilisation
  • Tradeoff local requirements market opportunity
  • Strategy maximise return on investment

22
What does Grid players require?
  • They need tools and technologies that help them
    in value expression, value translation, and value
    enforcement.
  • Grid Service Consumers (GSCs)
  • How do I express QoS requirements ?
  • How do I trade between timeframe cost ?
  • How do I map jobs to resources to meet my QoS
    needs?
  • How do I manage Grid dynamics and get my work
    done?
  • Grid Service Providers (GSPs)
  • How do I decide service pricing models ?
  • How do I specify them ?
  • How do I translate them into resource allocations
    ?
  • How do I enforce them ?
  • How do I advertise attract consumers ?
  • How do I do accounting and handle payments?

23
Solution 1 Market-Oriented Grid Computing - (a)
Sustained Resourced Sharing and (b) Effective
Management of Shared Resources
Grid Economy
24
Solution 2 Service Oriented Architecture (SOA)
  • A SOA is a contractual architecture for offering
    and consuming software as services.
  • There are four entities that make up an SOA
  • service provider,
  • service registry, and
  • service consumer (also known as service
    requestor).
  • The functions or tasks that the service provider
    offers, along with other functional and technical
    information required for consumption, are defined
    in
  • the service definition or contract.

registry
contract
provider
consumer
25
Service-Oriented Grid Architecture
Data Catalogue
Grid Bank
Information Service
Grid Market Services
Sign-on
HealthMonitor
Info ?
Grid Node N

Grid Explorer

Secure
ProgrammingEnvironments
Job Control Agent
Grid Node1
Applications
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services

Deployment Agent
JobExec
Resource Allocation
Storage
Grid Resource Broker

R1
R2
Rm
Core Middleware Services
Grid Servuce Consumer
Grid Service Providers
26
Gridbus and Complementary Technologies
realizing Utility Grid

Grid Applications
Portals
Science
Commerce
Engineering
Collaboratories

X-Parameter Sweep Lang.
Workflow
ExcellGrid
Gridscape
MPI
User-LevelMiddleware

Grid Brokers
Gridbus Data Broker
Workflow Engine
Nimrod-G
Grid Exchange Federation
Grid MarketDirectory
Globus
Unicore
Grid Storage Economy
GridBank

Core Grid Middleware
Alchemi
NorduGrid
XGrid
Grid Economy
.NET
JVM
Condor
SGE
Tomcat
PBS
Libra
Grid Fabric Software
Mac
AIX
Solaris
Windows
Linux
IRIX
OSF1
Grid Fabric Hardware
Worldwide Grid
27
(No Transcript)
28
Grid Market Directory (GMD)
29
Grid Bank Market-based Grid Access Management
GridBank Server
GridCheque Resource Usage (GSC Account Charge
GridCheque
Establish Service Costs
Applications
Grid Trade Server
GridBank Charging Module
Grid Resource Broker (GRB)
GridBank Payment Module
GridCheque
Resource Usage
Grid Agent
Grid Resource Meter
User
Deploy Agent and Submt Jobs
Usage Agreement
R1
R2
R3
R4
Grid Service Consumer (GSC)
User
Grid Service Provider (GSP)
30
Outline
  • Introduction to the University
  • Melbourne, GRIDS Lab, and Opportunities
  • Recap of the First Lecture
  • What are Grids, Challenges, Middleware Solutions
  • Service-Oriented Grid Architecture and Gridbus
    Solutions
  • Market-based Management, Grid Market Directory,
    Grid Bank
  • Grid Service Broker
  • Architecture, Design and Implementation
  • Performance Evaluation Experiments in Creation
    and Deployment of Applications on Global Grids
  • A Case Study in High Energy Physics
  • Summary and Conclusion

31
Grid Service Broker (GSB)
  • A resource broker for scheduling task farming
    data Grid applications with static or dynamic
    parameter sweeps on global Grids.
  • It uses computational economy paradigm for
    optimal selection of computational and data
    services depending on their quality, cost, and
    availability, and users QoS requirements
    (deadline, budget, T/C optimisation)
  • Key Features
  • A single window to manage control experiment
  • Programmable Task Farming Engine
  • Resource Discovery and Resource Trading
  • Optimal Data Source Discovery
  • Scheduling Predications
  • Generic Dispatcher Grid Agents
  • Transportation of data sharing of results
  • Accounting

32
Gridbus Broker Architecture
Gridbus Client
Gridbus Client
Gribus Client
(Bag of Tasks Applications)
App, T, , Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Trading Manager
RecordKeeper
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS

GE GIS, NWS
Grid Info Server
RM TS
G

Data Catalog
Data Node
C

U
G
Unicore enabled node.
Globus enabled node.
L
A
RM Local Resource Manager, TS Trade Server
Alchemi enabled node.
33
Gridbus Broker and Remote Service Access Enablers
Credential Repository MyProxy
Portlets
Data Store
Access Technology
SRB
Grid FTP
34
Gridbus Services for eScience applications
  • Application Development Environment
  • XML-based language for composition of task
    farming (legacy) applications as parameter sweep
    applications.
  • Task Farming APIs for new applications.
  • Web APIs (e.g., Portlets) for Grid portal
    development.
  • Threads-based Programming Interface
  • Workflow interface and Gridbus-enabled workflow
    engine.
  • Resource Allocation and Scheduling
  • Dynamic discovery of optional computational and
    data nodes that meet user QoS requirements.
  • Hide Low-Level Grid Middleware interfaces
  • Globus (v2, v4), SRB, Alchemi, Unicore, and
    ssh-based access to local/remote resources
    managed by XGrid, Condor, SGE.

35
Click Here for Demo
Figure 3 Logging into the portal.
Drug Design Made Easy!
36
Excel Plugin to Access Gridbus Services
37
Adaptive Scheduling Steps
Discover More Resources
Discover Resources
Establish Rates
Evaluate Reschedule
Compose Schedule
Meet requirements ? Remaining Jobs, Deadline,
Budget ?
Distribute Jobs
38
Deadline (D) and Budget (B) Constrained
Scheduling Algorithms
39
Outline
  • Introduction to the University
  • Melbourne, GRIDS Lab, and Opportunities
  • Recap of the First Lecture
  • What are Grids, Challenges, Middleware Solutions
  • Service-Oriented Grid Architecture and Gridbus
    Solutions
  • Market-based Management, Grid Market Directory,
    Grid Bank
  • Grid Service Broker
  • Architecture, Design and Implementation
  • Performance Evaluation Experiments in Creation
    and Deployment of Applications on Global Grids
  • A Case Study in High Energy Physics
  • Summary and Conclusion

40
Case Study High Energy Physics and Data Grid
  • The Belle Experiment
  • KEK B-Factory, Japan
  • Investigating fundamental violation of symmetry
    in nature (Charge Parity) which may help explain
    why do we have more antimatter in the
    universe?.
  • Collaboration 1000 people, 50 institutes
  • 100s TB data currently

41
Case Study Event Simulation and Analysis
B0-gtDD-Ks
  • Simulation and Analysis Package - Belle Analysis
    Software Framework (BASF)
  • Experiment in 2 parts Generation of Simulated
    Data and Analysis of the distributed data

?Analyzed 100 data files (30MB each) that were
distributed among the five nodes within
Australian Belle DataGrid platform.
42
Australian Belle Data Grid Testbed
VPACMelbourne
43
Belle Data Grid (GSP CPU Service Price G/sec)
G4
NA
G4
G6
VPACMelbourne
G2
Datanode
44
Belle Data Grid (Bandwidth Price G/MB)
32
33
36
G4
31
30
34
NA
38
31
G4
G6
VPACMelbourne
G2
Datanode
45
Deploying Application Scenario
  • A data grid scenario with 100 jobs and each
    accessing remote data of 30MB
  • Deadline 3hrs.
  • Budget G 60K
  • Scheduling Optimisation Scenario
  • Minimise Time
  • Minimise Cost
  • Results

46
Time Minimization in Data Grids
47
Results Cost Minimization in Data Grids
48
Observation
49
Outline
  • Introduction to the University
  • Melbourne, GRIDS Lab, and Opportunities
  • Recap of the First Lecture
  • What are Grids, Challenges, Middleware Solutions
  • Service-Oriented Grid Architecture and Gridbus
    Solutions
  • Market-based Management, Grid Market Directory,
    Grid Bank
  • Grid Service Broker
  • Architecture, Design and Implementation
  • Performance Evaluation Experiments in Creation
    and Deployment of Applications on Global Grids
  • A Case Study in High Energy Physics
  • Summary and Conclusion

50
The GridSim ToolkitA Java based tool for Grid
Scheduling Simulations
Application, User, Grid Scenarios Input and
Results
. . .
Application Configuration
Resource Configuration
Visual Modeler
Grid Scenario
Output
Grid Resource Brokers or Schedulerss Simulation
GridSim Toolkit
Application Modeling
Information Services
Resource Allocation
Statistics
Job Management
Resource Entities
Add your own policy for resource allocation
Resource Modeling and Simulation (with Time and
Space shared schedulers)
Clusters
Single CPU
Reservation
SMPs
Load Pattern
Network
Basic Discrete Event Simulation Infrastructure
SimJava
Distributed SimJava
Virtual Machine (Java, cJVM, RMI)
Distributed Resources
PCs
Workstations
Clusters
SMPs
51
Selected GridSim Users - 2002
52
Summary and Conclusion
  • Grids exploit synergies that result from
    cooperation of autonomous entities
  • Resource sharing, dynamic provisioning, and
    aggregation at global level ?Great Science and
    Great Business!
  • Grids have emerged as enabler for
    Cyberinfrastructure that powers e-Science and
    e-Business applications.
  • SOA Market-based Grid Management Utility
    Grids
  • Grids allow users to dynamically lease Grid
    services at runtime based on their quality, cost,
    availability, and users QoS requirements.
  • Delivering ICT services as computing utilities.
  • Grids offer enormous opportunities for realizing
    e-Science and e-Business at global level.
  • Use our Gridbus technology to realise this and
    make money!

53
Melbourne welcomes you
  • Melbourne as a place
  • multicultural environment
  • Worlds most liveable city
  • The University
  • Australias 1 University
  • Many international students and professors
  • One of the best places in the world to conduct
    Grid computing research!
  • My Contact Details
  • rbuyya_at_unimelb.edu.au
  • www.buyya.com
  • 61-3-83441244

54
You could be a co-author of my next book!
55
Thanks for your attention!
We Welcome Cooperation in Research and
Commercialisation! http/www.gridbus.org
http//www.gridbus.com
56
Big Question?
  • Can computational grids drive the economy of the
    21st century similar to the electric power grid
    that drove the economy of the 20th century?

57
(No Transcript)
Write a Comment
User Comments (0)
About PowerShow.com