Johan Muskens

1
Outline

• Subversion server
• What is it?
• Why is it useful?
• How to use
• Sandpit cluster
• What is it?
• Why is it useful?
• How to use

2
Subversion What is it?

• Version control system
• Successor to CVS

Subversion client 1 C\My Documents\
Subversion client 2 D\My Documents\
Subversion server https//svn.win.tue.nl/repos/san

• education
• courses
• 2IC10
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

• courses
• 2IC10
• 2IF20

• education
• courses
• 2IC10
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

3
Subversion Why is it useful?

• Sharing knowledge
• Now documents, code, etc. are only available on
  pc of individual group members
• These artifacts should be access-able for all
• Collaboration
• Writing documents together
• No need for mailing documents to each other over
  and over again
• Backup

4
Subversion How to use?

• Important actions
• Check-out
• Update
• Add
• Remove
• Move
• Commit

5
Check-out

• Create a local working copy of a sub-tree of the
  repository

Subversion client C\My Documents\
Subversion server https//svn.win.tue.nl/repos/san

• education
• courses
• 2IC10
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

• courses
• 2IC10
• 2IF20

6
Update

• Synchronize updates on the repository with your
  local copy.

Subversion client C\My Documents\
Subversion server https//svn.win.tue.nl/repos/san

• education
• courses
• 2IC10
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

• courses
• 2IC10
• 2IF20

7
Add

• Put a (new) file under version control

Subversion client C\My Documents\
Subversion server https//svn.win.tue.nl/repos/san

• education
• courses
• 2IC10
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

• courses
• 2IC10
• New file
• 2IF20

!! File will be added to repository
during next commit.
8
Remove

• Remove a file from version control (The history
  will remain !!).

Subversion client C\My Documents\
Subversion server https//svn.win.tue.nl/repos/san

• education
• courses
• 2IC10
• file
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

• courses
• 2IC10
• file
• 2IF20

!! File will be removed to repository
during next commit.
9
Move

• Move a file from one location in the repository
  to a different one.

Subversion client C\My Documents\
Subversion server https//svn.win.tue.nl/repos/san

• education
• courses
• 2IC10
• file
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

• courses
• 2IC10
• file
• 2IF20
• file

!! File will be moved to repository
during next commit.
10
Commit

• Changes in your local working copy are
  synchronized with the repository

Subversion client C\My Documents\
Subversion server https//svn.win.tue.nl/repos/san

• education
• courses
• 2IC10
• New file
• 2IF20
• File
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

• courses
• 2IC10
• New file
• 2IF20
• File

11
Our Repository

• https//svn.win.tue.nl/repos/san/
• education
• courses
• 2IC10
• 2IF20
• student_projects
• research
• personal
• jmuskens
• mchaudron
• projects
• robocop
• space4u
• www

12
Tools

• Windows client
• Tortoise SVNClient
• Integrated in Windows Explorer
• Unix client
• Command line (svn add/remove/)
• Graphical client ?

13
Things to remember

• Changes are not effectuated at the server until
  you do a commit
• You can only add, remove, things when you have
  a local copy of a sub-tree of the repository
• You can have multiple local copies of possibly
  different sub-trees of the repository
• You can put files in a local working copy without
  putting them under version control
• History remains
• Once you add something to subversion, traces
  remain (even when you do a remove).

14
Guidelines

• Make it available for all
• Also old project documentation
• Add sources, not the derived artefacts
• Dont use versions in the document names
• Subversion does the versioning for you !
• Dont alter the existing directory structure
• How you structure your personal, course or
  project sub-tree is up to you
• Keep it clean

15
SANdpitSAN distributed parallel integrated
terminal
to dig up information

• Richard Verhoeven

16
Content

• Hardware
• Software
• General, OpenPBS, LAM/MPI
• Usage
• Courses
• Projects
• Users
• Data access

17
Old cluster

• One master with 17 nodes (pacluster)
• 533 MHz Pentium III, 256 MB, 10 GB, 100 Mbit
• RedHat 7.3 with security fixes
• Failing power supplies, fans, disks, video cards
• Only 13 nodes kept running to some extend

18
New cluster

• One master with 17 nodes (sandpit)
• 3 GHz Pentium 4 with hyper-threading
• 400 GB disk in master, 40 GB disk per node
• 1 GB memory
• 2 Gigabit NICs
• 19 rack mounted

19
Hardware
Internet
2x200 GB
40 GB
20
Software

• Gentoo Linux distribution
• Compile from source
• Optimized for platform
• Large collection of standard software
• Text editors (vi, emacs, nano)
• Compilers (gcc, g, g77, gcj, icc, javac)
• Text processing (perl, LaTeX, ps2???)
• Graphic processing (netpbm, gstreamer)
• GUI apps (gnome, mozilla)
• More software is installed upon request

21
Directory structure

• /home/user (190 GB)
• /local/user (190 GB)
• /wslocal/sb01/user

• /home/user
• /wslocal/sandman/user
• /local/user (30 GB)
• /wslocal/sb01/user

sb01 .. sb17
sandpit / sandman
890 GB shared disk space for users
22
OpenPBS

• Portable Batch System
• Jobs are submitted to the server with qsub
• Status of queue is available through qstat
• Scheduler allocates nodes to specific jobs.
• Four queues for jobs
• small (lt 20 min, highest priority)
• medium (lt 2 hours)
• large (lt 12 hours)
• verylarge (lt 3 days)
• Maui scheduler

23
LAM/MPI

• MPI message passing interface
• LAM/MPI open-source MPI implementation
• lamboot create a multicomputer
• mpirun run parallel program
• lamhalt terminate multicomputer
• Under PBS
• Create script with lamboot mpirun lamhalt
• Create multicomputer from allocated nodes

24
Courses

• Student accounts
• Course specific software
• Special PBS queue
• Only for students and instructors
• Only active queue during instruction hours
• Cluster remains accessible for others
• Getting started manual

25
Projects

• Group of users
• Project related directory
• Software repository
• Documentation
• Guest accounts
• Faculty, university, national, international

26
Users

• SAN members
• Copied from UNIX systems
• Faculty members
• mCRL group
• Students of courses
• Guest researchers
• Use ssh to login and rsh/ssh to move between
  nodes.

27
Data Access

• Central directories are not accessible
• UNIX directories not NFS mountable (anymore)
• Window G disk not SAMBA mountable (yet)
• Use ssh and scp

28
Data Backup

• Cluster not under the backup policy of BCF
• Data is often temporary
• Too much data for offsite backups
• Options
• Copy data to different nodes
• Copy data regularly to central network disk
• Integrate backup into PBS scripts
• Check data into Subversion

29
Usage policy

• No enforcement (yet)
• By-passing PBS is possible
• No disk quota
• No accounting of processor time
• Fair use
• Clean up temporary data regularly
• Dont abuse the resources
• Personal projects are allowed as long as they
  dont interfere with education or research.
• Otherwise
• Enforcement will be added or account disabled

30
Possible extension

• Use OpenMosix
• Extension to move processes between nodes
• Add real-time boot configuration
• Reboot nodes when job scheduler asks for it
• Use second network interface
• Small ring
• Second connection to the switch
• Add cross-compiler tool chains
• Familiar Linux for iPAQ
• TinyOS for sensor motes
• SymbianOS for mobile phones

31
What happened with

• the old cluster?
• Already reused in other projects
• Several nodes revisited to get them working
• Create heterogeneous network setup
• Ethernet, wifi, firewire, bluetooth, sensors
• Combine or upgrade test environments
• Give to family and friends

Moet weg tegen elk aannemelijk bod
32
Questions?