JRA Status and Progress Analysis of communication patterns and plan for adaptation of components

1
JRA Status and Progress Analysis of
communication patternsand plan for adaptation of
components

• 2nd All-Hands-Meeting (AHM)
• Alexey Cheptsov, Kiril Dichev, Rainer Keller,
• on behalf of the JRA
• 20.10.2008, Poznan, Poland

2
JRA Progress Report1.05.2008 15.10.2008

• OUTLINE
• Overview
• Performed JRA Tasks and Activities
• Progress of Partners
• HLRS
• ELETTRA
• PSNC
• Plans and Deadlines for the next period
  (1.11.2008 15.01.2009)

3
JRA Tasks 1/2

• Task JRA1.1 Identification of generic interface
  remote sensor/devices and implementation of
  support for sensors and devices used in NA3
  (ELETTRA)
• Task JRA1.2
• Subtask JRA1.2.1 Supporting Interactive jobs for
  experiment workflows (PSNC)
• Subtask JRA1.2.2 Application developers support
  (performance analysis, remote debugging) (USTUTT)
• Subtask JRA1.2.3 Advanced visualization
  techniques (GUP UM)
• Subtask JRA1.2.4 Collaboration tools (ELETTRA)
• Task JRA1.3 Integration of all enhanced
  components and applications into a common
  framework (USTUTT, PSNC,GUP/LMU,ELETTRA)

4
JRA Tasks 2/2
Time schedule of JRA tasks
5
JRA Activities 1/4

• Deliverable DJRA1.1 Analysis of provided
  applications
• Delivered (as discussed in the Trieste meeting)
• Prepared the Activity Report for the 2nd Quarter
• Deliverable DJRA1.2 Analysis of communication
  patterns and plan for adaptation of components
• Analysis of communication patterns of
  applications provided by DORII scientific
  communities is performed (ELETTRA was responsible
  for remote instruments, HLRS analyzed
  message-passing communication patterns in
  parallel applications).
• The DORII architecture is updated (ELETTRA)
• The functionality of the common library for DORII
  is elaborated (ELETTRA, PSNC)
• The plan for adaptation of middleware products
  with regard to the DORII common library is worked
  out (ELETTRA, PSNC)
• Version of the report is sent out for the
  internal review (HLRS)

6
JRA Activities 2/4

• Regular EVO Video Meetings were organized

• Skype Teleconferences took place for discussions
  of current status between partners (ELETTRA,
  PSNC, HLRS, OGS, UC)

7
JRA Activities 3/4

• Participating in scientific conferences
• ASIM, Berlin 1.-2.10.2008 conference for
  simulation technology (HLRS)Presentation of the
  DORII project at the Poster Session
• EGEE08, Istanbul, 25.9.2008 DORII Joint session
  (PSNC, ELETTRA, IFCA, TUM, HLRS et
  al)Presentation of DORII support of scientific
  communities
• Future events
• SC08, Austin, 17.11.2008 GASC Workshop
  (HLRS)Presentation of the DORII infrastructure
  and proposed application support.
• ICT, Lyon, 25.-27.11.2008General Workshop on
  the DORII e-Infrastructure
• HPC-Europa, 4.11.-16.12.2008 Joint research
  project (OGS, HLRS)Exchange of scientists, here
  to support and enhance applications in the DORII
  e-Infrastructure.

8
JRA Activities 4/4

• Prepared Papers
• A contribution from JRA to the article on behalf
  of the DORII concern for the Tridentcom
  conference The DORII Project Test Bed
  Distributed eScience Applications at Work is
  prepared
• A common article on behalf of JRA for the
  infrastructure section of the CMST journal of
  PSNC - Grid aspects of the remote
  instrumentation infrastructure's deployment (in
  the frame of the DORII project) under
  development
• A common OGS-HLRS article for the application
  section of the CMST journal of PSNC Towards
  porting the OPATM-BFM application to a grid
  e-Infrastructure optimization of communication
  patterns is being finalized

9
JRA Progress HLRS

• Exemplary porting of application (OGS, HLRS)

The scheme of the application analysis (in the
frame of the DORII project)
10
JRA Progress HLRS

• OPATM-BFM application (OGS)Detailed analysis of
  the internal message/passing communication pattern

11
JRA Progress HLRS

• Porting of OPATM-BFM application to Grid (OGS,
  HLRS)Analysis of application characteristics and
  scalability

12
JRA Progress HLRS

• OPATM-BFM application (OGS)
• The svn-repository on the HLRS GForge server is
  used for management of the application source
  code for distributed development
• Disk I/O operations
• Use of parallel MPI I/O for accessing / storing
  the input and output data
• Substitution of NetCDF with Parallel NetCDF
  library (built on top of MPI I/O)Currently up to
  70 time reduction (on standard clusters using
  NFS)
• Message-passing communication patterns
• Usage of MPI collective communication operations
  instead of point-to-point MPI
• Encapsulate multiple point-to-point and
  collective messages into one
• Currently up to 50 time reduction (on standard
  clusters using IB)

13
JRA Progress HLRS

• OPATM-BFM application (OGS)
• Porting to the Grid
• The application is adapted for the usage on a
  standard cluster of workstations (the test bed
  cluster cacau of HLRS is used for this purpose)
• Until the DORII infrastructure was under
  construction, usage of I2G-resourses
• Organization of data management on SEs
• Adaptation of application for software packages
  available on CEs (compilers, numerical libraries
  etc.)
• Future activities
• Porting to the DORII Grid, testing the
  infrastructure
• Use of tools provided by the DORII architecture
  for the application development and management on
  the Grid

14
JRA Progress HLRS

• SWAN application (UC)
• Porting to the Grid
• Sequential version is working on the test cluster
  at HLRS
• Future activities
• Testing the parallel implementation with Open MPI
• Porting to the DORII infrastructure, usage of
  tools provided by the DORII architecture

15
JRA Progress ELETTRA

• Presented by Roberto via EVO

16
DORII Deployment of Remote Instrumentation
InfrastructureWorkpackage JRAVCE, IE, CommonLib

• Andrea Del Linz
• Milan Prica
• George Kourousias
• Roberto Pugliese
• Sincrotrone Trieste SCpA (ELETTRA)
• DORII AHM Poznan, PL, Oct 2008

17
DORII Architecture
Legenda IE - instrument element CE - computing
element SE - storage element WMS - workload
managementsystem(CE broker) LFC - location file
catalog (SE broker) DB - database element IS -
information system VOMS - VO management
system OtherS - other services (e.g. OGC) VCR -
virtual control room G-Eclipse - developer
workbenck Wf-Editor - workflow editor Wf-Monitor
- workflow monitor Wf-Man System - workflow
management system Script Man - script
manager Apps Man - application manager
User
Apps Man
Application Developer
Wf-Editor
VCR
G-Eclipse
TunnelNon-Grid Interactive
Wf-Monitor
Script Man
Native AppLauncher
Wf-Man System
IE
CE
SE
WMS
LFC
DB
IS
VOMS
OtherS
Grid / eInfrastructure
18
VCR status

• Ant-installer!
• Supports gLite 3.x (MyPROXY, VOMS, IS, WMS, LFC,
  CE, SE)
• IE current version plus new wsdl
• Skype
• Logbook, Calendar, Chat, Scripts, Applications,
  Tunnels

19
VCR development plan

• New VCR is based on Gridsphere 3.x and GWT
• Strategy port VCR portlets to JSR168 GS3 and
  decide case by case what should be implemented
  using GWT
• GWT native registration, approval, approval
  workflow, ldap integration, certificate
  management (mid dec 08)
• i2login integration old VCR (mid nov 08)
• All old VCR portlets to JSR168 (end Jan 09)
• EVO integration (?, webservie api not yet
  available)
• Common library adaptation (?)
• Workflow integration (?)
• OGC integration (?)

20
IE status

• New WSDL
• subscribe/unsubscribe
• Lock
• gLite security (wsdl now integrates delegation)

21
IE development plan

• VOMS integration (mid dec 08)
• BCS integration (end oct 08)
• WSDL IM plugin (?)
• Labview plugin (data socket) now bridge via Tango
  plug-in Tango/Labview bridge (?)
• Ant-Installer / rpm (?)
• CIMA plugin (?)
• EPICS plugin (?)
• OGC service integration (?)

22
Common JAVA LIBRARY

• The common library is a java library to access
  programmatically all the services of the DORII
  architecture
• The library combines the CoGRIDCC library with
  the g-Eclipse gLite library
• It seams that the easier way to combine the
  libraries is to implement it as a set of OGSi
  bundles.
• This requires considerable effort on the VCR side
  and a complete rewrite of the CoGRIDCC library
• The potential advantage is flexibility and
  modularity
• Detailed plan should be decided at the AHM. It is
  clear that the release will be after the first
  deployment

23
Common JAVA LIBRARYsteps

• Refactor the g-Eclipse gLite OGSi bundle(s)
• What, how, when
• Refactor VCR g-Lite specifics as an OGSi bundle
• MyProxy, IE, BDII (?)
• How, when
• Refactor the VCR to access OGSi bundles
• How, when
• Implement the whole VCR as an OGSi bundle (or
  more than one?)
• How, when
• Is this possible with GS3.x and GWT?

24
Open Issues

• Common Library release plan
• Interactive Application support in the VCR
• More IE plug-ins (Labview/IVI, EPICS, )
• OGC services integration (CYCLOPS MoU)
• Implementing the SOS (sensor observation service)
  using the IE as SPS (sensor planning service) and
  SCS (sensor coverage service) are implemented by
  CYCLOPS using respectively gLite WMS and SE
• VCR scripting environment (?)
• Workflow management (another solution can be
  gridified-taverna integrated with the common
  library and with the VCR as in the case of the
  jython scripts see above) and integration with
  the VCR
• gEclipse access to the IE (?)

25
FYI, Common Lib Interfaces

• IE
• consists of a collection of services that
  provide the functionality to configure, partition
  and control a set of instruments.
• From the VCR, an instrument element is
  accessed through the InstrumentElement interface
  that defines the Instrument Element Service.

26
FYI, Common Lib Interfaces

• IM
• The public interface InstrumentManager
  describes how to interact with instruments. The
  Instrument Manager is the implementation of a
  common interface to a real instrument or a
  simulation of an instrument. An IM is completely
  described by parameters, attributes, commands,
  and finite state machine
• Parameters hold configuration information
  Attributes hold instrument variables Command hold
  actions Finite state machine adds constraints
  over the commands

27
FYI, VCR component

• The new VCR has been installed for DORII VO at
• https//dorii-vcr.grid.elettra.trieste.it/gridsph
  ere/gridsphere

• An account can be requested from the same VCR
  home page.

28
JRA Progress PSNC

• Presented by Marcin Lawenda

29
DORIIWorkflow Management System

• Marcin Lawenda
• lawenda_at_man.poznan.pl
• PSNC
• AHM, Poznan, 20.10.2008

30
Workflow work conception
31
Status

• Prepared conception of the workflow integration
• Analysis of other workflow notations
• Analysis of application demands in terms of the
  workflow
• First application to implement in the workflow
  was choosen Oceanographic and coastal
  observation and we started implementation

32
FYI, Overview
The following components of the DORII
architecture will be used from g-Eclipse VOMS,
IS BDII, WMS, LFC, SE The description of APIs
will be extracted from g-Eclipse is collected on
DORII Wiki
33
JRA Plans and Deadlines

• Plan for adaptation of components

34
JRA Plans and Deadlines

• Plan for adaptation of components

35
JRA Plans and Deadlines

• Plans and deadlines for the next period
• Milestones
• Deliverables
• DJRA1.3 First prototype of integrated products
  from previous projects (M12)