D4Science and ETICS Building and Testing gCube and gCore

1
D4Science and ETICSBuilding and Testing gCube
and gCore
EGEE08 Conference 25 September 2008 Istanbul
(Turkey)
Pedro Andrade CERN
www.d4science.eu
2
D4Science

• D4Science is a FP7 project establishing a
  knowledge infrastructure of data, services and
  processing resources through the provision of
  Virtual Research Environments
• D4Science simplifies the discovery, access and
  production of existing and new data through a
  collaborative environment
• D4Science continues and consolidates the work
  started in the DILIGENT project opening it to
  new communities
• D4Science provides a production environment for
  ESA FAO

3
gCube

• gCube is the software implementing the D4Science
  resources sharing vision via Virtual Research
  Environments
• gCube software
• Provides a multi-domain Service Oriented
  Infrastructure
• Gathers data, services and hardware as system
  resources
• Supports on-demand virtual collaborations
• Reduces the cost of managing research
  infrastructures
• Tailored for e-Science (content oriented, data
  intensive)
• Is complaint with consolidated and emerging
  standards

4
gCube

• gCube resources
• Computing and storage
• VRE services
• content management
• search and index
• annotations
• Collections
• raw data
• content
• metadata
• Processes to manage other resources

5
gCore

• gCore is a framework developed to reduce the
  complexity of designing and implementing gCube
  SOI complaint services
• gCore is composed by
• gCore Framework An application framework to
  consolidate and develop existing/new gCube
  services
• gCore Container A self-contained software
  container distributed to facilitate the
  deployment of gCube services

6
D4Science in ETICS

• Two projects
• org.gcube 24 subsystems
• org.gcore 2 subsystems
• 225 components
• 160 software components
• 51 services, 84 libraries, and 25 portlets
• Including services samples, external libs, test
  libs (18 components)
• 65 distribution components
• 64 gCube Software Archives
• gCore Hosting Node
• Around 500 configurations

7
D4Science Release Cycle
etics component configurations
etics subsystem configurations
etics-test
etics project configurations
etics-build
8
Build

• The objective of the build activity is to ensure
  api-level compliance of components and the
  creation of components artifacts.
• ETICS is exploited to
• Model build scripts and manage build dependencies
• Execute on-demand remote builds (via WA and CLI)
• Execute nightly local builds for development
  configurations
• Produce distribution artifacts (via dedicated
  components)
• Feed the D4Science reporting application with xml
  raw reports

9
Deployment Testing

• The objective of the deployment testing activity
  is to ensure that the gCube software archives
• Can be remotely and automatically deployed
• Once deployed, that it starts and is activated
  correctly
• ETICS is exploited to
• Manage the executing the deployment tests
• Define the deployment test tool dependencies
• Generate deployment test reports

D4Science and ETICS
25 September 2008, Istanbul (Turkey)
10
Deployment Test
ETICS Infrastructure
D4Science Private Resources
Service 2
Service 1
Service A
Condor Node
Condor Node
Condor Node
Standard Remote Test
ETICS Server
Condor Node
Condor Node
Condor Node
Co-scheduled Remote Test
Service 2
Service 1
Service A
11
Static Analyzers

• The objective of the adoption of static analyzers
  is to improve the quality of gCore and gCube
  software
• ETICS is planned to be exploited to
• Run jFindbugs and jDepend (under adoption)
• Other analyzers (not exploited via ETICS)
• Check coverage of javadoc documentation
  (checkstyle)
• Monitor the content of code repository
• e.g. pre-compiled artifacts, etc..
• Certification of distribution components
• check structure, correctness, etc..

12
Current Releases
13
Current Releases
14
Improvements

• Parametric tests on remote infrastructure
• Single remote test with multiple parameterized
  tests
• Co-scheduling mechanism to be further exploited
• Condor nodes firewall limitations
• Notification mechanism
• ETICS is designed to support distributed teams
• No notifications between different ETICS roles
• components availability
• successful build/test execution

15
Improvements

• ETICS configurations contains many information
• Can an ETICS API be useful ?
• Reports summary information
• Summary page with built, test, and plug-ins
  execution
• Organized by configuration
• Plug-ins availability
• Info about existing ones?
• How to add external plug-ins?

16
Conclusions

• Solid exploitation
• DILIGENT/D4Science using ETICS since the early
  days
• ETICS is the default tool in D4Science build an
  test tasks
• ETICS completely adopted by all D4Science members
• Solid collaboration
• D4Science and ETICS plans kept aligned
• Efficient support from ETICS