Workload Management, Portal and Real Time Monitoring

1
Workload Management, Portal and Real Time
Monitoring

• Keith Sephton, Barry MacEvoy, Gidon Moont, Steve
  Mcgough, David Colling
• (with contributions from Mona Aggarwal and
  Olivier van der Aa)

2
Contents

• WMS testing
• SGE support
• Real Time Monitor
• Portal
• GridCC
• No time to cover anything in any detail so ask

3
WMS testing

• gLite WMS known to have scaling issues
• So far have been testing the WMS on the PPS.
  Previously reported to GridPP (http//www.gridpp.a
  c.uk/gridpp16/GridPP16_WMS.ppt) This has not been
  as effective as hoped as too far removed from the
  development cycle and meant to provide a service.
• So now moved to the SA3 certification activity.
  Developed/developing stress tests (rather than
  functional tests) that are designed to test the
  performance to the point of destruction.
• Monitoring load on the WMS machine as well as the
  job times
• Testing performance (success/failure, submission
  time, matching time etc) as function of input
  (requirements, ranking, number/size of input
  files etc )
• Testing both the normal and web service
  submission, individual and bulk submission etc
• Jobs actually run on the PPS

4
WMS Testing

• Much closer connection to the developers
• Fortnightly releases from the developers (YAIM
  releases interleaved with manual/patch releases)
• First release end of next week.
• Currently finishing a set of tests and their
  automatic publishing finish by the end of the
  week so if you see me tapping away at the back
  this is what I am doing.

5
SGE Porting

• Porting being carried out at several sites
• More coordinated than it used to be between
  different groups. Now part of SA3.
• UK (Keith Sephton) provided a new version of the
  information provider in January.
• Now UK working on the blah implementation release
  due by the end of this month
• May move this from bash to perl.
• Then deploy on the certification testbed.

6
Real Time Monitor

• New 3D version
• 75 WMS/RB
• Used SC06 etc. Will be in the Science Museum
  April-gtSept
• Some problems with various graphics drivers
• Still going to maintain applet version
• Publishing the information gathered
• Part of the WLCG monitoring grup
• Data to be published in an agreed format. Already
  publish some information as XML (with XSLT).

7
Real Time Monitor

• Still analyse analysis on data collected
• Still produce nightly summary

8
Efficiency per CE per quarter
9
Efficiency per Tier per quarter
10
Portal

• Have helped CALICE to send Mokka jobs to the
  Grid. The portal has proven useful as a check
  for real availability (lcg-infosites can include
  incorrectly configured sites) from which correct
  Requirements can then be created.
• MICE is still interested in using the Grid, but
  has not yet got software ready to run.

11
Java Proxy Software

• The Java Web Start (JWS) VOMS Proxy Manager has
  been re-engineered into a core library with
  command-line and GUI front ends.
• Code now provides full security checks on the
  VOMS enabled certificates.
• Components are being used by other projects.
• Work is being done to help PERMIS use this to
  create a webservices based client/server.

12
GridCC

• Concentrates on instrumentation on the Grid
• Workflows with QoS guarentees
• Performance repository
• Workflow resolver and planner
• Hard and Soft QoS
• Project ends August 07
• Maybe follow on project (outside of GridPP scope)

13
An example application
This high level workflow actually contains many
basic components e.g. configuring the monitors,
moving the data etc
14
Conclusions

• For the moment
• UK is active in WMS testing, WMS support for
  SGE, Portal, RTM and GridCC
• Future of these activities is unclear

15

• Backup Slides

16
Some UK Metrics

• Made using root. data from 1jan 2006 to 31 de
  2006. plots extracted by Olivier, data gathered
  by Gidon

17
Resource Usage in the Last 8 Months
18
Efficiency per Tier per quarter
19
Efficiency per ce per quarter
20
Total Hours per Tier
21
Total Hours Fractions per Tier
22
Succes/Failed Hours per CE
23
VO Usage in UK
24
Web Service interface
GridCC as seen by the Execution Services
Standard gLite
Standard gLite
Information
Standard gLite
Standard gLite