WP3 New Grid Services and Tools Task 3'4 Optimization and Data Access

1
WP3 New Grid Services and Tools Task 3.4
Optimization and Data Access

• Deliverable D3.5

Lukasz Dutka, Jacek Kitowski, Renata Slota and
Darin Nikolow dutka_at_agh.edu.pl CYFRONET AGH
Krakow, Poland
2
Goals

• A1 Unified Data Access Platform building the
  unified and highly expandable data access
  platform for grid storage nodes, selected
  database, optimized data access and predicted
  data access factors for data,
• A2 Data Access Estimators development of
  data access estimators for HSM system, disk
  drives and data bases, compatible with the
  unified platform,
• A3 Data Access Handlers development of data
  access handlers compatible with the platform and
  realizing some optimization and unification
  tasks,
• A4 Integration with EDG integration of the
  unified data access platform with EDG-Replica
  Manager,
• A5 Speedup access to tape resident files
  build of middleware for faster access to large
  tape resident files.

3
UDAL Deployment
Selects replica on the base of gathered costs
Application of UDAL for Estimation of Data Access
Cost in Heterogeneous Environment
Application of UDAL for Data Access Stored in
Heterogeneous Storage
4
STATUS

• A1 Unified Data Access Platform
• The provided components, constitute a highly
  expandable access platform.
• A2 Data Access Estimators
• A model of data access factors estimator for data
  located on HSM (DiskXtender and Castor) has been
  worked out. Data access estimator for disk drives
  has also been implemented. In addition there are
  ongoing works aiming to develop estimator for
  Castor HSM, however its current status is not
  sufficient to include it into the prototype
  bundle.

5
STATUS CONTD

• A3 Data Access Handlers
• To make GridFTP Wrapper usable there have been
  implemented some data access handlers. Moreover,
  those data handlers were designed and developed
  according to the requirements of Component-Expert
  Architecture and are compatible with CEXS.
• A4 Integration with EDG
• The integration process with EDG WP2 has been
  finished. All assumed aims have been achieved.
  Some results of our cooperation were described in
  the joint paper presented during Grid Conference
  at Phoenix and published in the conference
  proceedings.
• A5 Speedup access to tape resident files
• The middleware for faster access to large tape
  resident files has been developed and tested. But
  it is not included because it has requires
  CG_CGFTP which is not finished yet.

6
INTERNAL TESTS

• In summary the following test have been
  completed
• extensive verification of component selection
  decisions. These analyses check the component
  selection decisions taken by CEXS together with
  STEL, which in this context provides current
  internal configuration status of the storage.
• extensive tests verifying communication of DAES
  and CEXS and data access estimators. During these
  tests the quality of the estimations were also
  checked.
• replica selection experiments. During these tests
  the inter-component (inter-project) communication
  was tested. Additionally the first replica
  selection experiments were done.
• integration of GridFTP server with the unified
  data access platform. The tests allowed checking
  communication between GridFTP server, GridFTP
  Wrapper, CEXS and the sample data handlers.
• Our modules were also tested by the partners form
  EDG in the black box way. Respective modules were
  installed in some machines at CERN for over 6
  months period were used without restart. . These
  modules have been used for inter-project tests
  with results described in the paper presented
  during Grid Conference in Phoenix.
• The result of the tests where also described in
  an internal EDG report prepared by task EDG-WP2.

7
ISSUES

• Problems during an application of implementation
  of GSI secure communication.
• problems usually were related to gSOAP,
• pthread library
• pipes between processes.
• Fortunately, all the obstacles have been
  overcome.
• During integration of EDG Replica Manager we came
  across some differences in realization of base
  data types in SOAP protocol between gSOAP
• These obstacles have been overcome as well.

8
SUMMARY AND FUTURE PLANS

• Components in the current prototype are stable,
  some detailed performance analysis will be done
  in the future.
• The integration with EDG Replica Manager still
  needs more tests
• Task 3.4 is going to provide in the nearest
  future new elements
• extended GridFTP service being able communicating
  and exploiting functionality of CEXS and entire
  Component-Expert Architecture (on-progress)
• estimator for Castor HSM
• works are on progress,
• better integration with the Globus GSI
  infrastructure
• faster data handlers for larger files stored on
  HSM systems works on progress,
• additional data handlers optimizing data access
  and coping with data transfer limitations,

9
(No Transcript)