CVS setup at CCIN2P3 and Datagrid edgbuild tools

1
CVS setup at CC-IN2P3 and Datagrid edg-build
tools

• CVS management, centralized code checking and
  automatic building of rpm packages from CVS
• Yannick Patois
• E-mail patois_at_in2p3.fr

2
Table Of Content

• Datagrid CVS tools (edg-build)
• General idea
• Code checking
• RPM building
• Package publishing
• RPM distribution management

• CVS service at CC-IN2P3 (Loïc Tortay)
• How it started
• Authorized user access in read/write setup
• Anonymous access setup
• CVSweb

3
CVS service at CC-IN2P3 - starting point

• To share code development among local users
• CVS repository on AFS
• But a project (Datagrid) needed a CVS repository
  open to more people
• Lab policy wouldn't allow local login for
  everyone
• Needed another solution
• Once created and tested for Datagrid, this CVS
  service was extender to other experiments
  cvs.in2p3.fr was created.
• The presented setup has been set in place by Loïc
  Tortay at CC-IN2P3.

• CVS.IN2P3.FR
• Dedicated authentication mechanism (not part of
  the generic users account policy)
• Offer open CVS services to experiements.

4
CVS at CC-IN2P3 - Read/Write access setup

• Authentication by ssh keys
• Interactive login is disabled
• CVS operations are performed through a specific
  pseudo-shell that only allow CVS operations to be
  executed.

• rcvssh remote CVS shell
• Build from 'anoncvssh' (OpenBSD project)
• Authentication managed above (by sshd)
• The shell ensure that the command and arguments
  are in a predefined set of allowed commands (cvs
  commands in our case)
• Can chroot the user to a subtree (anonymous
  access)
• Verbose login facility

5
CVS setup at CC-IN2P3 - Anonymous access

• Anonymous access is done on a chrooted cvs server
• Use a different port than the authenticated
  access mechanism
• The dedicated ssh daemon uses specifics
  configurations parameters (allow password
  authentication, allow empty password, only one
  account allowed)

Picture taken from http//www.winternet.com/mikel
r/ with kind courtesy of Mike Reed.
6
CVS setup at CC-IN2P3 - CVSweb and other tools

• CVSweb setup
• Module based configuration
• Web access aliased cvs.in2p3.fr/experiment
• Apache access rights can be managed by experiment
• CVS commit info on mailing list

7
CVS Services at CCIN2P3

• cvs_at_in2p3 Service just started a few weeks ago
• A few experiments (EROS, SUPERNOVAE, ..) uses it
• Datagrid CVS repository
• Around 120 accounts
• 360Mb of sources
• A few hundreds connexions per day (more than one
  thousand if accounting for automatic connexion
  -like autobuild)

8
Autobuild tools The Datagrid collaboration

• European grid project
• A Grid for
• HEP
• Biomedical applications (genomic and imaging)
• Earth study (ESA)
• Constraint
• Distributed development
• Several dependent modules (around 30)

9
The Datagrid Software Repository

• The Datagrid Software Repository
• Hosted at CC-IN2P3 (Lyon)CNRS - France
• Main services
• Source code management for developers (CVS)
• Packages distribution (RPMS)
• The autobuild an check system (automatic building
  of packages)
• Packages distribution Management

http//datagrid.in2p3.fr/autobuild/rh6.2/
10
Autobuild system - generalities
Publication Machine IBM-AIX CVS
repository Packages (rpm) repository
Build Machine RedHat Linux Building, package
managing

• Context
• Run on RH6.2 and RH7.2, have been tested on
  Solaris
• CVS and package repository machine is remote (btw
  it's an AIX machine).

• Autobuild
• Around 3000 SLOC of python
• One external Java program called
• A lot of os.system() calls to various unix
  utilities (from mkdir to cvs)

11
Autobuild system what it does

• Statistical informations about the project
• RPMS number/size
• SLOCs for every edg code
• Auto-documentation pages
• Processed user doc (LaTeX files, ...)
• Autogenerated code doc (Doxygen...)
• Build platform informations
• Installed rpms, environments...

• CVS module processing for each module
• Conformance check
• README and INSTALL files
• CVS tags
• Build
• autotools
• make
• make doc
• make check
• make install
• make dist
• make rpm
• Publish build RPMs

12
Autobuild result sample
13
Edg-release rpm lists
14
Autobuild present and futur state

• Autobuild program (called edg-build) can be
  downloaded from http//datagrid.in2p3.fr
• The code is most likely still very specific to
  Datagrid own needs, but is improving.
• Current evolution plan
• Build on demand
• Email to developers on build failure
• Package managing capability (partially
  implemented)

15
Creating RPM lists from dependencies I (idea)

• Problem Managing huge lists of RPM
  (edg-release) can be difficult.
• Idea Creating them from the RPM
  inter-dependencies

Practical implementation
Datagrid RPMS
Striped RPMS (only headers)
Build Machine
Local copy
Custom list RPM database
RPM task (virtual rpm pkgs)
Custom everything RPM database
Rpm -Va
Full RPM list
Validation
16
Creating RPM List from Dependencies II (example)
Redhat-base-edg-0.0.1-1.i386.rpm redhat-kernel-edg
-0.0.1-1.i386.rpm task-build-wp6-0.0.1-1.i386.rpm
Name task-build-wp6 ... This is for
edg-example Requires libtool Requires
automake Requires autoconf Requires
make Requires c Requires doxygen Requires
sgml-tools Requires tetex-dvips Requires
ghostscript ...
Full RPM List