Nimrod/G and GriddLeS: Grid Programming with Ease - PowerPoint PPT Presentation

Loading...

PPT – Nimrod/G and GriddLeS: Grid Programming with Ease PowerPoint presentation | free to view - id: 237b1-OGY2M



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Nimrod/G and GriddLeS: Grid Programming with Ease

Description:

Dr Kate Smith, Monash University. Neural Network. Optimization. CFD ... Jon Giddy, Cardiff U. Slavisa Garic, DSTC. Colin Enticott, DSTC. Rok Sosic, TurboLinux ... – PowerPoint PPT presentation

Number of Views:211
Avg rating:3.0/5.0
Slides: 50
Provided by: davidab8
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Nimrod/G and GriddLeS: Grid Programming with Ease


1
Nimrod/G and GriddLeS Grid Programming with Ease
  • David Abramson
  • Monash University
  • DSTC

2
This is how it all began ...
3
Air pollution modelling circa 1990
  • Want to control Ozone
  • What happens if we reduce NOx?
  • What happens if we reduce ROC?

4
But, Ozone chemistry is non-linear
5
Distributed computing comes to the rescue .
For each scenario Generate input files Copy
them to remote node Run SMOG model Post-process
output files Copy results back to root
6
Its just too hard!
  • Doing by hand
  • Nightmare!!
  • Programming with (say) MPI
  • Overkill
  • No fault tolerance
  • Codes no longer work as stand alone code.
  • Scientists dont want to know about underlying
    technologies

7
Building on Legacy Software
  • Nimrod
  • Support parametric computation without
    programming
  • High performance distributed computing
  • Clusters (1994 1997)
  • The Grid (1997 - ) (Added QOS through
    Computational Economy)
  • Nimrod/O Optimisation on the Grid
  • Active Sheets Spreadsheet interface
  • GriddLeS
  • General Grid Applications using Legacy Software
  • Whole applications as components
  • Using no new primitives in application

8
Nimrod Robust Design by Parametric Computation
9
Parametric Execution
  • Study the behaviour of some of the output
    variables against a range of different input
    scenarios.
  • Allows real time analysis for many applications
  • More realistic simulations
  • More rigorous science
  • More robust engineering

10
In Nimrod, an application doesnt know it has
been Grid enabled
Input Files Substitution
Output Files
Computational Nodes
Root Machine
11
How does a user develop an application using
Nimrod?
Description of Parameters PLAN FILE
12
EnFuzion TurboLinux Inc
13
Case Studies
14
Case Studies ...
15
Case Studies ...
16
Case Studies ...
17
Nimrod on the Grid Nimrod/G
18
Nimrod/G 2.0 Architecture
Plan File
Nimrod/G Client
Rsrc. Scheduler
Nimrod/G Client
Nimrod/G GUI
Enfuzion API
Run File
Database
Generator
Creator
Level 3
Job Scheduler
Agent Scheduler
Level 2
Level 1
DB Server
Globus Actuator
Condor Actuator
Legion Actuator
Grid Middleware
Grid Information Server(s)
RM TS
G
Agent
Agent
Agent
RM TS
L
Globus enabled node
C
RM TS
Legion enabled node.
Condor enabled node.
RM Local Resource Manager, TS Trade Server
19
A Grid Architecture for Computational Economy
Information Server(s)
Grid Market Services
Sign-on
Health Monitor
Info ?
Grid Node N

Grid Explorer

Application
Secure
Job Control Agent
Grid Node1
Schedule Advisor
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource Reservation
Misc. services

Deployment Agent
JobExec
Resource Allocation
Storage
Grid User
Grid Resource Broker

R1
R2
Rm
Grid Middleware Services
Grid Service Providers
20
A Nimrod/G Client
21
A Multi EnFuzion Client
MEC
ENFDISPATCHER
ENFDISPATCHER
ENFDISPATCHER
22
EnFuzion and MOSIX Clusters
MEC
ENFDISPATCHER
23
Active Sheets Client
24
Scheduling and QOS
25
Resources Selected Price/CPU-sec.
26
Execution _at_ AU Peak Time
27
Execution _at_ AU Offpeak Time
28
Scheduling for Time Optimization
29
Scheduling for Cost Optimization
30
Nimrod/O Design Goals
  • Nimrod allows exploration of design scenarios
  • Search by enumeration
  • Search for local/global minima based on objective
    function
  • How do I minimise the cost of this design?
  • How do I maxmimize the life of this object?
  • Objective function evaluated by computational
    model
  • Computationally expensive

31
Architecture of Nimrod/O
Meta-heuristic Search
Simplex
BFGS
Nimrod Plan File
Nimrod or EnFuzion Dispatcher
Supercomputer or Cluster Pool or Grid
32
An Aerofoil case study
  • Joint work with Clive Fletcher (CANCES)
  • Simple two-dimensional aerofoil was modelled
    using a FLUENT simulation
  • The aerofoil mesh generated by GAMBIT
  • 28089 nodes
  • 49426 elements, made up of 43090 triangular
    elements and 6336 quad elements.
  • Optimization to the design of an aerofoil,
  • maximise the ratio of lift to drag.
  • Complete enumeration infeasible because the
    number of simulations required is excessive

33
The wing
34
Results
35
Dynamic Behaviour
36
Parallel behaviour
37
GriddLeS
  • Significant body of useful applications that are
    not Grid Enabled
  • Lessons from Nimrod
  • Users will avoid rewriting applications if
    possible
  • Applications need to function in the Grid and
    standalone
  • Users are not experts in parallel/distributed
    computing
  • General Grid computations have much more general
    interconnections than possible with Nimrod.
  • Legacy Applications are Components!

38
GriddLeS
  • Specification of the interconnections between
    components
  • Interfaces for discovering resources and mapping
    the computations to them
  • Locate data files in the grid and connect the
    applications to them
  • Schedule computations on the underlying platforms
    and making sure the network bandwidth is
    available and
  • Monitor the progress of the grid computation and
    reassign work to other parts of the Grid as
    necessary.

39
Exemplar Grid Application
Real time data
Data Fusion
General Circulation model


Topography Database
Regional weather model




Vegetation Database

GASS
Photo-chemical pollution model
Particle dispersion model
Bushfire model
Emissions Inventory
40
Standard Implementation
Change Models
Real time data
Data Fusion
MPI
MPI
MPI
General Circulation model


GASS/GridFTP/GRC
Topography Database
Regional weather model
MPI




Vegetation Database

GASS
Photo-chemical pollution model
Particle dispersion model
Bushfire model
GASS
Emissions Inventory
MPI
41
GriddLeS Grid Application


Topography Database
General Circulation model
Regional weather model





Vegetation Database
Emissions Inventory
Particle dispersion model
Photo-chemical pollution model
Bushfire model



42
GriddLeS Grid Application


Topography Database
General Circulation model
Regional weather model





Vegetation Database
Emissions Inventory
Particle dispersion model
Photo-chemical pollution model
Bushfire model



43
Interprocess Communication in GriddLeS
Writer Application
Reader Application
fd open(blah, w) write(fd, ..)
fd open(blah, r) read(fd, ..)
44
Exploiting File IO
Local File
read()
write()
seek()
Remote File
close()
open()
FileMultiplexer
Cache
Remote Application Process
Legacy Application
45
Locating Files in the Grid
  • GriddLeS Name Server (GNS)
  • Translates local file names into
  • Local file names
  • Global file names
  • Resolves replication
  • Interprocess pipes

GNS
fd open(blah, r) read(fd, ..)
fd open(blah, w) write(fd, ..)
46
Exploiting Middleware
  • Globus
  • GRAM
  • GridFTP
  • Replication Catalogue
  • Security
  • Security
  • Condor
  • Bypass
  • CDS
  • Buffer management
  • Nimrod/G
  • GRACE
  • APST
  • NWS

47
Scheduling
  • Processes need to be assigned to resources
  • Files need to be located
  • Computational economy supports
  • Deadline scheduling
  • Cost minimization
  • Build on Nimrod/G Scheduler, APST

48
Acknowledgements
  • Funding
  • CRC for Distributed Systems Technology
  • Australian Research Council
  • Turbo Linux
  • Hewlett Packard
  • Dedication
  • Melfyn Lloyd
  • Research Director
  • DSTC
  • 1994 - 2002
  • Colleagues
  • Jon Giddy, Cardiff U.
  • Slavisa Garic, DSTC
  • Colin Enticott, DSTC
  • Rok Sosic, TurboLinux
  • Andrew Lewis, QPSF
  • Rajkumar Buyya, UoMelb
  • Tom Peachy, Monash
  • Paul Roe, QUT
  • Ian Foster, ANL

49
Further Information
  • Nimrod
  • www.csse.monash.edu.au/davida/nimrod
  • DSTC
  • www.dstc.edu.au
  • Globus
  • www.globus.org
  • TurboLinux
  • www.turbolinux.com
  • Our Cluster
  • http//hathor.csse.monash.edu.au
  • Nimrod/G available from
  • www.csse.monash.edu.au/davida/nimrod
  • Nimrod/O available from
  • www.csse.monash.edu.au/davida/nimrodo
  • GriddLeS
  • www.csse.monash.edu.au/davida/griddles
About PowerShow.com