Best Practice Project Rapid Prototyping of Usable Middleware Peter Coveney Centre for Computational Science University College London

1
Best Practice ProjectRapid Prototyping of
Usable MiddlewarePeter CoveneyCentre for
Computational ScienceUniversity College London
2
Scientists developing middleware!

• Rapid prototyping of usable grid middleware
  (EPSRC funded, starts April 2005)
• Partners include Manchester, Southampton
  (Comb-e-Chem), Oxford (IB),
• Robust application hosting under WSRFLite (OMII
  funded)
• Combined value 500K cash 100K in kind support
• OMII Open Middleware Infrastructure
  Institute (UK)
• www.omii.ac.uk

3
Talk contents

• Grid computing--what is it?
• Problems with existing grid middleware
• The case for lightweight middleware
• Robust application hosting
• Enabling grid-based computational science
• Materials science example

4
Grid Computing
My preferred definition Grid computing is
distributed computing performed transparently
across multiple administrative domains
See Phil Trans R Soc London A (2005)
5
Grid Computing
Problem No so-called Grid we use today
fulfils this definition
6
TeraGyroid Grid
Starlight (Chicago)
Netherlight (Amsterdam)
10 Gbps
ANL
PSC
Manchester
Caltech
BT provision
NCSA
Daresbury
2 x 1 Gbps
production network
MB-NG
SJ4
SDSC
Phoenix
Visualization
UCL
Computation
Access Grid node
Service Registry
Network PoP
Dual-homed system
7
UK NGS
Leeds
Manchester
Starlight (Chicago)
US TeraGrid
Netherlight (Amsterdam)
Oxford
RAL
SDSC
NCSA
PSC
UCL
UKLight
AHM 2004
Local laptops,PDAs, and Manchester vncserver
Both the US TeraGrid and UK NGS use GT2 middleware
All sites connected by production network (not
all shown)
Computation
Steering clients
Network PoP
Service Registry
8
Problems for users

• lack of a common API for usable core
  functionality  (e.g.  file-transfer)  across
  distinct  grid applications and domains
• heterogeneous software stacks make  grid-applicati
  onportability a nightmare for users
• security -- high barrier for getting certificates
  accepted beyond the issuing domain (more
  tomorrow)
• non-uniform scheduling and  job-launching
  resources and often incompatible policies in
  different admin domains
• complex grid middleware detrimental to scientific
  research, and contrary to the stipulated goals of
  grid computing

9
Grid computing headaches

• Deployment
• It takes a long time and much effort by many
  people to get applications properly deployed
• Lots of things can go wrong
• Most people give up -- ROI too low
• Lack of persistence of grid infrastructure
  capabilities
• Security issues (more in tomorrows talk)
• Clunky, not very usable
• Existing model not taken seriously by people who
  care about it

10
How we build services on GT2 grids

• Globus Toolkit 2 has limited usable
  functionality, so we
• Track specs standards
• Provide functionality as easily as possible
• Put this on top of GT2 grid middleware
• We do NOT wait for heavyweight/generic solutions
  provided by others
• GT3 (obsolescent)
• GT4 (yes, but when?)
• Its a recipe for being sidelined indefinitely
• Lightweight middleware makes provision of a
  service oriented architecture a pleasant
  experience for all

11
Lightweight middleware

• What do we mean by lightweight?
• Minimal dependencies on third-party software
• Small learning-curve for new users obviate the
  need to learn new programming methods
• Interoperable with other WSRG implementations
• Easy to write, and so to adapt to new specs, etc.
• Original use of OGSI compliant services
• Now have WSRFLite (interoperable with other
  WSRG implementations)
• Tracks the evolving WSRF standards, implementing
  stable areas of the specifications

12
Lightweight middleware

• OGSILite/WSRFLite
• by Mark McKeown of Manchester University
• Lightweight OGSI/WSRF implementation, written in
  Perl
• uses existing software (eg for SSL) where
  possible simple installation
• Necessary for all RealityGrid grid work
• Using OGSILite (2003)
• Grid-based job submission and steering
  retrofitted onto the LB2D workstation class
  simulation code within a week
• Standards compliance we were able to steer
  simulations from a web browser, with no custom
  client software needed
• Now developing extended capabilities using
  WSRFLite on US TeraGrid UK NGS
• We have developed WEDS--a web service hosting
  environment for distributed simulation

13
About WEDS

• Developed to make life easy for application
  scientists for once
• Easy to deploy sits inside a WSRFLite
  container, has no additional software
  requirements
• Provides all the tools and glue required to
• expose an unaltered binary application as a
  service
• create and interact with service instances
• Broker service manages creation of services, to
  load balance across a pool of machines
• For grid deployment, needs
• security solution (WS-Security, TLS) and
• grid job submission tools (from OMII_1, others
  from GridSAM project)

See Coveney et al., 2004, NeSC Tech Rpt
14
WEDS Architecture

• Each resource runs a WSRFLite container
  containing a WEDS machine service and factory
  services for each hosted application.
• Each machine that a user wishes to use is
  registered with a broker service
• The user contacts the broker with the details of
  the job to run
• The broker match-makes the job details with the
  capabilities advertised by each machine service
  and decides where to invoke the service
• The broker passes back the contact details of the
  service instance to the client

Client
Broker
Machine Service
Service Factory
Wrapper Service
Invoked Application
Managed resource
15
About WEDS

• Can interact flexibly with OMII middleware
• OMII interface to WEDS resources
• WEDS broker will soon interact with GT2, OMII
  resources
• Delegation of file-transfer to existing
  transports (HTTP(S), FTP, GridFTP, etc)
• Provides C and Fortran API to allow an
  application programmer to expose a richer service
  interface to the application.
• Already hosted LB2D, DL_MESO, NAMD, LAMMPS, CPMD
• RealityGrid steering will be incorporated as
  those tools move towards WSRF compliance

16
OGSA/WSRF compliance

• In the main, the hosting environment is WSRF- and
  OGSA- compliant
• BUT we have to go outside these specifications
  (with DataProxy) because they require binary data
  to be moved within XML files!
• W3C has spotted the problem and is now proposing
  recommendations

17
Transferring binary data

• World Wide Web Consortium Issues Three Web
  Services Recommendations
• http//www.w3.org/ -- 25 January 2005 -- The
  World Wide Web Consortium (W3C) has published
  three new Web Services Recommendations
  XML-binary Optimized Packaging (XOP), SOAP
  Message Transmission Optimization Mechanism
  (MTOM), and Resource Representation SOAP Header
  Block (RRSHB). These recommendations provides
  ways to efficiently package and transmit binary
  data included or referenced in a SOAP 1.2
  message.
• Web Services Applications Need Effective,
  Standard Methods for Handling Binary Data

18
Transferring binary data

• One of the biggest technical and performance
  issues for Web services occurs when a user or
  application is handling large binary files.
  Encoding binary data as XML produces huge files,
  which absorbs bandwidth and measurably slows down
  applications. For some devices, it slows down so
  much that the performance is considered
  unacceptable.
• http//www.w3.org/2005/01/xmlp-pressreleas
  e

19
Robust application hosting

• Developing our lightweight hosting tools to meet
  the needs of applications scientists
• No preconceptions about the 'right way' to do
  things or pre-determined adherence to particular
  specifications or work flows
• Gain experience by working with real-world
  problems, refactoring design as required
• Projects/people we are collaborating with as
  end-users
• --Daniel Mason (Imperial) -- polystyrene-surface
  interactions (see demo)
• --CCP5s DL-MESO Project (Rongshan Qin, DL) --
  mesoscale modelling/simulation
• --Jonathan Essex (Southampton) -- NAMD for
  protein modelling
• --Integrative Biology EPSRC e-Science Project
  project
• --IBiS (Integrative Biological Simulation) BBSRC
  Bioinformatics e-Science Project
• Close collaboration with OMII and its middleware

20
(No Transcript)
21
Lightweight hosting of Polysteer application
22
Visualisation

• Showing each atom is unreadable
• Potentials treat CHx, Bz as single entities
• We visualise ellipsoids rather than spheres

23
Summary

• Lightweight middleware greatly facilitates
  deployment of applications on grids
• Were now working with several computational
  user communities from physics through to biology
• All our middleware will be in the public domain

24
Acknowledgements

• Matt Harvey, Laurent Pedesseau, Mark Mc Keown,
  Stephen Pickles, Daniel Mason, Jonathan Chin
• EPSRC
• OMII