R-GMA: Production Services for Information and Monitoring in the Grid John Walk / RAL

1
R-GMA Production Services for Information and
Monitoring in the Grid John Walk / RAL
UK e-Science AHM2004, Nottingham
www.eu-egee.org
EGEE is a project funded by the European Union
under contract IST-2003-508833
2
Whats the talk about?

• Overview of R-GMA
• Practical experience with R-GMA
• Building a production-quality Web Services
  version of R-GMA

3
Who are we?

• The system (R-GMA)
• Grid Information and Monitoring System created in
  the European DataGrid (EDG) project
• Now being taken forward as part of the
  production-quality middleware being developed in
  the EGEE project
• The people (JRA1-UK)
• 9 Full-time at Rutherford Appleton Laboratory
  (CCLRC)
• 1 Full-time at Queen Mary, University of London
• Contributions from Heriot-Watt University
  (Edinburgh) and Trinity College Dublin
• We can be e-mailed at
• jra1-uk_at_physics.gla.ac.uk

4
Relational Grid Monitoring Architecture
Publish Data
Producer Service
Producer application
API
Register
Registry Service
Data
Query
Locate
Mediator
Send Query
Consumer Service
Consumer application
API
Receive Data

• An implementation of the GGF Grid Monitoring
  Architecture (GMA).

5
Relational Grid Monitoring Architecture
Publish Tuples
Producer Service
Producer application
API
Register
SQL INSERT
Registry Service
Tuples
Query
SQL SELECT
Locate
Mediator
Send Query
Consumer Service
Consumer application
API
Schema Service
Receive Tuples
SQL CREATE TABLE

• An implementation of the GGF Grid Monitoring
  Architecture (GMA)
• All data modelled as tables a single schema
  gives the impression of one (loosely coupled)
  virtual database for VO

6
Secondary Producers

• Used to republish data
• To co-locate information to speed up queries
• To reduce network traffic
• To use different producer properties

PP
PP
SP
PP
PP Primary Producer SP Secondary Producer
PP
7
A unified approach to information and monitoring

• Provides information on both
• The Grid itself
• Mainly for the middleware packages, but the user
  may also query it to understand the status of the
  Grid
• Grid applications
• For users
• Flexible infrastructure
• All data has a timestamp enabling its use for
  monitoring
• Dynamic addition and deletion of information
  producers
• Dynamic creation and deletion of tables in schema
• Able to cope with failures in a distributed
  environment

8
Resilience

• Soft state registration protects services and
  registry from defunct producers and consumers
• Registry and schema are replicated to avoid
  single point of failure (and improve scalability)
• All services tolerate failure of remote service
  or resource by seeking an alternative, trying
  again later, or failing gracefully if these are
  not possible

9
Grid Application CMS Job Monitoring
http//cmsinfo.cern.ch/
10
R-GMA in CMS
C

• In testing
• Target 2000-job batch
• Exceeded in 2004

11
Grid Middleware Network Monitoring
http//ccwp7.in2p3.fr/
12
R-GMA in EDG WP7 Network Monitoring
Network metrics include
Network Managers Resource Brokers Replica
Managers
TCP throughput
SP
MySQL Database
ICMP packet loss
Transfer times
P
P
PingER IPerf UDPmon
Network Monitoring Nodes
Storage Elements
13
Enabling Grids for E-science in Europe

• EGEE will integrate current national,
  regional and thematic Grid efforts to create a
  seamless European Grid infrastructure for the
  support of the European Research Area
  (http//www.eu-egee.org)
• R-GMA is the Information Services component of
  the gLite middleware being developed for EGEE
• Main efforts are
• Producing production-quality software
• Migrating to Web Services
• Improving portability
• Duration of project is two years, with first
  release of software due after first year (March
  2005)

14
Main achievements so far (JRA1-UK)

• Specification complete
• Design complete
• WSDL produced for user interfaces
• Prototype of new API available (Java, C, C,
  Python), as a wrapper around the old one
  (feedback welcome)
• Registry replication completed outline design
  for schema replication produced

15
Next steps

• Complete move to WS-I compliant Web Services
• Introduce multi-VO support
• Complete security design work
• Complete the schema replication
• Port to Win32 (EGEE secondary platform)
• Watch developments elsewhere and seek to
  inter-work (e.g. contribution to OGSA
  collaboration with LB work)

16
R-GMA Web Services
AXIS SERVLET
SOAP/HTTP Messages
Primary Producer Service
User Application
R-GMA API
SOAP STUB
Secondary Producer Service
On-demand Producer Service
SOAP/HTTP Messages And SOAP Faults
Consumer Service
Created from WSDL using gSOAP or Axis
Tomcat Servlet Container

• API available for Java, C, C and Python
• Users may by-pass API if they wish, but API is
  the easiest way to use R-GMA services
• Registry and Schema also implemented as Web
  Services, but not accessed directly by users,
  except for administration

17
R-GMA Security

• Many issues are EGEE-wide and are being addressed
  by JRA3 (Security)
• User requirements vary widely from HEP (would
  probably do without it) to BioMed (very stringent
  confidentiality requirements)
• Will allow installations to turn security off
  where performance is paramount
• Some R-GMA users require access to data to be
  controlled down to sub-table level. Proposal is
  to use parameterized views, e.g.
• SELECT from Job where OwnerDN DNDN

18
Further Information

• JRA1-UK web site
• http//hepunx.rl.ac.uk/egee/jra1-uk/
• E-mail
• jra1-uk_at_physics.gla.ac.uk

Thanks to the EU and our national funding
agencies for their support of this work