Grid eXplorer (GdX) An Instrument for eXploring the GRID

1
Grid eXplorer (GdX)An Instrument for eXploring
the GRID

• F. Cappello, O. Richard, P. Sens
• (fci_at_lri.fr, olivier.richard_at_imag.fr,
  pierre.sens_at_lip6.fr)

LRI, ID-Imag, Lip6, etc.
2
What we know about Grid/P2P?

• Infrastructure features
• Large scale distributed systems
• Dynamic infrastructures (network, nodes)
• Dynamic workloads
• Heterogeneous (parallel uniproc. nodes)
• High Computation/Communication perf. Ratio

3
What we know about Grid/P2P?

• Application features
• Large data sets (data bases, physics
• instruments, generation of very large results)
• Huge computations (in terms of operations)
• Long run times
• Large number of data operations (movements,
  transactions, etc.)

4
What we know about Grid/P2P?

• Utilization
• A large number of users (researcher, industry,
  individual, etc.)
• A large set of applications (research, business)
• High throughput (file download, multi-parameters,
  computation, etc.)
• Real time (Sensors, Entertainment, HPC, etc.)

5
What are open issues?

• Security (GSI, CAS)
• Data Storage/consultation/movement
• Multi users/ Multi applications scheduling
• Coordination (virtual, ephemeral infrastructure)
• Programming
• Fault Tolerance!
• Scalability
• Performance
• Easy/efficient deployment techniques
• Application characterization techniques
• Etc.

6
What is Grid today?

• Middleware Globus, Legion, Netsolve, Unicore,
  DIET, Condor, XtremWeb, Boinc, NWS
• ? they are actually working!
• Testbed DataGRID, TeraGRID, Etoile, Grads, XW,
  Boinc
• ? Difficult to build (debug, human factor, etc.)
• Dedicated applications (SETI, Kazaa, Jabber,
  etc.)
• ? they are working already (at large scale!),
• ? BUT they address much less issues!

7
Why Grid is so difficult?

• Grids are very complex systems (we still have
  problems with large scale parallel computers!, we
  have less control on Grid resources! )
• Many issues are addressed simultaneously
• ? We need a methodology enabling the study of
  Grid issues, independently but realistically.

8
What are the current approaches?

• Simulators SimGRID, MicroGRID, etc.
• ? they have strong limitations (scalability,
• ! than execution of real codes, validation)
• Experimental testbed (is there any?)
• ? Most testbed are for production, each testeb
  is specific, representativeness?
• We have no way to test
• ideas independently, at a significant scale,
• with realistic parameters and behaviors!

9
What is missing?

• A full fledge scientific environment
• (reproducible realistic experimental conditions)
• Probes measuring the performance of real
  resources and networks (Ganglia, NWS, la
  grenouille)
• Fully experimental testbed, (GRID 5000 would
  remove this lack)
• ? Not enough, we need instruments with
  parametrisable reproducible experimental
  conditions

10
Methodology

• When physicists cant measure a phenomenon
• because a) the item to measure is not
  reachable or b) the phenomenon is hidden by
  others,
• They build instruments!
• And they observe

11
Grid eXplorer
A tool set for conducting experiments
measurements
An experimental conditions database
or generation
An experimental hardware platform
12
Grid eXplorerAnalogy with physic instruments
A set of sensors Inside real platform
A tool set for Observation Results Analysis
A Hardware Platform for Emulation Simulation
A database of Real platform
Validation on real platforms
13
Grid eXplorer (GdX) Instrument for Grid
exploration

• An instrument for exploring Grid middleware,
  algorithms, performance and applications under
  reproducible experimental conditions
• A tool set for emulation/simulation of large
  scale distributed systems
• 1K CPU clusters configurable network OS
• For Grid, P2P, etc.
• Potentially connected to Grid testbeds

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
14
Grid eXplorer (GdX) eXperimental conditions
database

• A set of sensors (Nodes, Networks)
• ? Academic Networks (x K nodes, GRID 5000)
• ? ADSL (la grenouille ? 60 K nodes)
• A common format for traces
• A tool set for accessing, managing traces
• Tools for trace analysis

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
15
Grid eXplorer (GdX) Hardware platform for
eXperiments

• 1K CPU clusters
• configurable network
• configurable OS
• Multi-users
• Potentially located/managed by IDRIS or CEA

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
16
Grid eXplorer (GdX) Tool set

• Emulators (folding 10K virtual nodes on 1K nodes)
• Parallel simulators (difficult!)
• Virtual GRID environment (1k virtual nodes on 1k
  nodes)
• Measurement tools,
• Visualization tools.

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
17
Grid eXplorer (GdX) eXperiments

• Real scale pseudo-emulation (11 scale)
• Large scale emulation (101 scale)
• Large scale simulation (1001 scale)
• Application test (bottleneck discovery,
  performance evaluation, optimization)
• Infrastructure test (bottleneck discovery,
  performance evaluation, optimization)
• Connection to Grid 5000, International Grids,
  etc.

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
18
Grid eXplorer (GdX) Instrument related projects

• Instrument projects
• Cluster management (ID Imag),
• Sensor event data base (LRI-fci, la grenouille),
• Event analysis (ID Imag),
• Emulators (LRI-fci, etc.),
• Experimental conditions injection,
• Result visualization (???),
• Result analysis,
• Validation methodology,
• User validation (physicists, biologists,
  chemists, etc.)

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
19
Grid eXplorer (GdX) Grid/P2P research projects

• Grid
• DataGRID technology (???)
• GRIDRPC (LIP, Desprez?)
• Code coupling (IRISA?)
• P2P
• Large scale desktop Grids (LRI-fci)
• Large scale storage (Laria, Irisa?)
• Large scale data consistency (Lip6-P. sens)
• Large scale fault tolerance (LRI-Beauquier)
• Network
• Emulation of latency, packet loss, bandwidth
• Emulation of topology

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
20
Grid eXplorer (GdX) User applications

• Bio Grid (Genomining, IBBMC, etc.)
• Medi GRID (???)
• Geo GRID (IFP, CGG)
• Power GRID (EDF)
• Industry GRID (EADS, Alcatel Space)
• Academic GRID (Orsay, Berkeley, SDSC,
  Suitzerland, etc.)
• Industry Partners (GridXpert???, IBM???)

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
21
Grid eXplorer (GdX)Project organisation

• A Virtual laboratory
• A director (or a group),
• A scientific director, an administrative director
• A scientific council (with physicists),
• A Researcher group
• A group of engineers,
• A user group
• A set of scientific events

F. Cappello, O. Richard, P. Sens ---oo
Draft oo--- Contact us for experiment
proposal
22
Grid eXplorer GdX

• A long term effort
• A medium term milestone 4 years
• ? A fully functional prototype
• Many scientific issues (large scale emulation,
  experimental conditions injection, distance to
  reality, etc.)
• A tool for Grid users or potential users
• A tool for Grid/P2P developers