The Ibis Project: Simplifying Grid Programming

1
The Ibis ProjectSimplifying Grid Programming
DeploymentHenri Balbal_at_cs.vu.nlVrije
Universiteit Amsterdam
2
The Promise of the Grid

• Efficient and transparent (i.e. easy-to-use)
  wall-socket computing over a distributed set of
  resources Sunderam ICCS2004, based on
  Foster/Kesselman

3
Reality Problems of the Grid

• Performance scalability
• Heterogeneous
• Low-level changingprogramming interfaces
• writing deploying grid applications is hard

• Connectivity issues
• Fault tolerance
• Malleability

!
Wide-Area Grid Systems
User
4
The Ibis Project

• Goal
• drastically simplify grid programming/deployment
• write and go!

5
Approach

• Minimal assumptions about execution environment
• Virtual Machines (Java) deal with heterogeneity
• Use middleware-independent APIs
• Different programming abstractions
• Designed for dynamic/hostile grid environments
• Modular and flexible can replace Ibis components
  by external ones

6
Global picture
7
Grid programming

• Programming models
• Low-level message passing (RMI, MPJ)
• High-level divide-and-conquer (Satin)
• IPL (Ibis Portability Layer)
• Java-centric run-anywhere library, sent with
  application
• Can handle fault-tolerance, malleability
• SmartSockets
• Solve connectivity problems automatically
  (firewalls, NAT, addressing problems)

8
Grid deployment

• Zorilla P2P system
• Jobs management, gossiping,clustering, flood
  scheduling
• JavaGAT Java Grid Application Toolkit
• Make applications independent of underlying grid
• Used for file copying, resource discovery, job
  submission monitoring, user authentication
• API is currently standardized (SAGA)

9
Java GAT
Grid Application
File.copy(...)?
submitJob(...)?
GAT
Remote Files
Monitoring
Info service
Resource Management
GAT Engine
GridLab
Globus
Unicore
SSH
P2P
Local
Intelligentdispatching
globus
gridftp
van Nieuwpoort et al., SC07
10
Ibis applications

• e-Science (VL-e)
• Brain MEG-imaging
• Mass spectroscopy
• Multimedia content analysis, video processing
• Various parallel applications
• SAT-solver
• N-body
• Grammar learning

11
European users

• D-Grid Workflow engine for astronomy
• U. Erlangen grid file system
• INRIA ProActive on Ibis RMI
• U. Patras Jylab scientific computing system
• HITACHI Peta-scale data management

12
DAS-3
272 nodes(AMD Opterons) 792 cores 1TB
memory LAN Myrinet 10G Gigabit
Ethernet WAN (StarPlane) 20-40 Gb/s OPN
See CCGrid08 session 11 (Verstoep)
13
Gene sequence comparison in Satin (on DAS-3)
Speedup on 1 cluster
Run times on 5 clusters

• Divideconquer scales much better than
  master-worker
• 78 efficiency on 5 clusters (with 1462
  WAN-msgs/sec)

14
Multimedia content analysis

• Analyzes video streams to recognize objects
• Extract feature vectors from images
• Describe properties (color, shape)
• Data-parallel task implemented with C/MPI
• Compute on consecutive images
• Task-parallelism on a grid

See CCGrid08 SCALE challenge sessions 1216
15
High Resolution Video Processing

• Realtime processing of CineGrid movie data
• 3840x2160 (4xHD) _at_ 30 fps 1424 MB/sec
• Multi-cluster processing pipeline
• Using DAS-3, StarPlane and Ibis

16
Summary

• Goal Simplify grid programming/deployment
• Key ideas in Ibis
• Virtual machines (JVM) deal with heterogeneity
• High-level programming abstractions (Satin)
• Handle fault-tolerance, malleability,
  connectivity problems automatically (Satin,
  SmartSockets)
• Middleware-independent APIs (JavaGAT)
• Modular

17
Acknowledgements

• Past members
• John Romein
• Gosia Wrzesinska
• Rutger Hofman
• Maik Nijhuis
• Olivier Aumage
• Fabrice Huet
• Alexandre Denis

• Current members
• Rob van Nieuwpoort
• Jason Maassen
• Thilo Kielmann
• Frank Seinstra
• Niels Drost
• Ceriel Jacobs
• Kees Verstoep
• Roelof Kemp
• Kees van Reeuwijk

18
More information

• Ibis can be downloaded from
• http//www.cs.vu.nl/ibis
• Papers
• Satin PPoPP07, SmartSockets HPDC07,
  Gossiping HPDC07, JavaGAT SC07,MMCA IEEE
  Multimedia07
• Ibis tutorials
• CCGrid08