Quattor Cookbook C. Loomis (LAL-Orsay) Clermont-Ferrand Quattor Tutorial

1
Quattor Cookbook C. Loomis (LAL-Orsay)Clermont-
Ferrand Quattor Tutorial
EGEE is a project funded by the European Union
under contract IST-2003-508833
2
Contents

• Overview
• Quattor Server
• OS Installation (RH7.3)
• Quattor Services
• Pan Templates
• Quattor Clients
• Manual Installation
• Automatic (PXE) Installation
• Writing NCM Components
• Summary

3
Overview

• Purpose
• Install basic quattor server.
• Understand use pan templates.
• Install manage clients with quattor.
• Write simple NCM component.
• Ground Rules
• Originally intended as informal one-on-one
  tutorial, i.e. expect glitches.
• Feel free to interrupt, correct, praise,
  complain, etc.
• I'm not a quattor developer!
• Only useful features (IMHO) are covered.
• Cookbook doesn't replace the complete quattor
  documentation.

4
Quattor Resources

• Website (http//quattor.web.cern.ch/quattor/)
• Overview
• Design and architecture
• Current status
• Bug reports (via savannah)
• Documentation (http//quattor.web.cern.ch/quattor/
  documentation.htm)
• Installation guide
• Component writers' guide
• PAN language reference
• Tutorials
• Quattor Packages (http//quattor.web.cern.ch/quatt
  or/software.htm)
• RedHat Linux 7.3
• Others (FC1, RHES, ...) not yet available

5
Quattor Server

• Requirements
• RedHat Linux 7.3
• Daemons (Apache, DHCP, TFTP)
• Disk 2.5 GB for system, 2.5 GB per client OS, 5
  GB for LCG-2
• Basic Quattor Server
• All quattor services on one machine.
• Some advanced services are not deployed.
• Should be OK for most sites.
• Small on order of 100 machines.
• Have not tested this myself!
• LCG-2 installation
• Templates components not complete.
• Community effort trying to make this happen.

6
RedHat 7.3 Installation

• Collect necessary information
• IP addresses (quattor server, DNS, gateway, NTP)
• MAC addresses, network mask, broadcast
• Standard server installation
• Edit partitions for gt10 GB in /var.
• No firewall.
• Additional packages
• Web server group
• DHCP server (system environment/daemons/dhcp)
• TFTP server (system environment/daemons/tftp-serve
  r)
• NTP client (system environment/daemons/ntp)
• mtools (applications/system/mtools)

7
RedHat 7.3 Updates

• Quattor server itself not managed by quattor!
• Use standard tools for server system updates.
• RedHat no longer supports RH7.3.
• Fedora Legacy Project provides security patches.
• Follow the instructions at the link below to
  enable yum
• http//www.fedoralegacy.org/docs/yum-rh7x.php
• Recommend
• Enabling kernel updates.
• Enabling automatic updates.
• Using appropriate mirrors.
• Update system and reboot!
• yum update
• reboot

8
Downloads

• Complete OS for kickstart installs
• Available on web server for clients.
• Download RH7.3 images (valhalla.iso).
• Copy image contents to /var/www/html/rh73/.
• Quattor packages
• Alpha-2
• edg-cdb2sql package is not needed for this
  tutorial
• http//quattorsw.web.cern.ch/quattorsw/software/qu
  attor/release/alpha_2/i386-rh7.3/
• External
• Need to upgrade syslinux others can be installed
• http//quattorsw.web.cern.ch/quattorsw/software/qu
  attor/external/RPMS/RH73/
• Download RPMs to temporary area.

9
Downloads (cont.)

• LCG-2
• Use LCG-2 machine types as examples.
• Get all LCG-2 rpms for all machines
• http//grid-deployment.web.cern.ch/grid-deployment
  /download/RpmDir/index_LCG-2_0_0.html
• Put into temporary area with sufficient space.

10
Quattor Server Services

• Central Configuration Database (CDB)
• Contains configuration information for all
  quattor clients.
• Configuration done via Pan templates.
• Translated to low-level XML format for
  distribution to clients.
• Software Repository (SWRep)
• Contains all software packages for clients OS,
  LCG-2, quattor, ...
• Automated Installation Infrastructure (AII)
• Uses standard OS tool for initial installation.
• Quattor is used to update system after the first
  boot.

11
Quattor Client Services

• Configuration Cache Manager (CCM)
• Downloads caches low-level XML profile.
• Node Configuration Manager (NCM)
• Runs necessary configuration components when
  profile changes.
• Software Package Management Agent (SPMA)
• Installs updates software packages.

12
CDB Configuration

• Ensure that Apache is running
• chkconfig httpd on
• service httpd start
• Check that the server responds to a browser.
• Verify configuration file
• top is location of database (/var/lib/cdb)
• pan is location of executable
  (/opt/edg/bin/panc)
• cake is location of cake executable
  (/opt/edg/libexec/cake)
• Initialize CDB
• /opt/edg/sbin/edg-cdb-setup
• Creates hld and lld directories in /var/lib/cdb.
• Links /var/www/html/profiles to
  /var/lib/cdb/lld/xml.

13
CDB Configuration (cont.)

• CDB notifications (for later)
• Add server_module host.example.org for
  synchronization.
• Or server_module none for no notifications.
• Two management tools
• cdb-simple-cli (local management)
• cdbop (remote management)

14
CDB Client Configuration

• Simple command line (cdb-simple-cli)
• Set password for cdb account.
• Session-based tool (cdbop)
• Add userpasswd lines to /etc/httpd/conf/.passw
  d file.
• Crypt password perl -e 'print crypt(password,
  be)'
• Replace /etc/httpd/conf/cdb.allow contents with
  line cdb admin.
• Copy /usr/share/doc/cdb-cli-ltvergt/cdbop.cf to
  /etc/cdbop.cf.
• Add quattor server name to /etc/cdbop.cf.
• Test
• cdb-simple-cli list
• cdbop then user, password, open, list,
  exit.
• Commands should work but return nothing (as there
  are no templates yet).

15
SW Repository Configuration (Server)

• Copy example configuration file
• /usr/share/doc/edg-swrep-server-ltvergt/edg-swrep-se
  rver.cfg to /etc/swrep/.
• Edit parameters
• name Cookbook Software Repository
• owner grid.support_at_example.org
• url http//quattor.example.org/swrep
• aclfile /etc/swrep/edg-swrep.acl
• rootdir /var/www/html/swrep
• enable-sync no

16
SW Repository Configuration (Client)

• Copy example configuration file
• /usr/share/doc/edg-swrep-ltvergt/edg-swrep-client.cf
  g to /etc/swrep/.
• Edit parameters
• repository quattor_server_at_example.org
• runsync no
• ssh-params modification needed only if using
  kerberos

17
SW Repository Authorization

• Generate ssh keys if necessary
• ssh-keygen -b 4096 -t rsa
• Copy id_rsa.pub to /var/swrep/.ssh/authorized_keys
  2
• Add 'environmentSSH_USERswrep ' before
  ssh-rsa
• ACL (/etc/swrep/edg-swrep.acl) should already
  have
• swrep/ as the first line

18
SW Repository Organization

• Organize as you want. Typically something like
• edg-swrep-client addplatform i386_rh73
• edg-swrep-client addarea i386_rh73 /base
• edg-swrep-client addarea i386_rh73 /updates
• But for this tutorial will use
• edg-swrep-client addarea i386_rh73 /lcg2
• edg-swrep-client addarea i386_rh73 /quattor
• Adding/removing packages
• edg-swrep-client put i386_rh73 /tmp/eg.rpm /lcg2
• edg-swrep-client list i386_rh73
• edg-swrep-client remove i386_rh73 /lcg2 eg.rpm
• If all worked, then SW repository server and
  client are OK.

19
Fill SW Repository

• Load all LCG-2 rpms into repository
• edg-swrep-client put i386_rh73 ltxgt.rpm /lcg2
• Load all quattor rpms into repository
• edg-swrep-client put i386_rh73 ltxgt.rpm /quattor
• Generate a repository template
• edg-swrep-client template i386_rh73
• Rename appropriately (see structure template
  line of generated file).

20
Pan Overview

• Pan Language
• Used to specify machine configuration.
• Flexible, hierarchical organization of
  information.
• Allows arbitrary validation of configuration.
• Organization (pro_declaration_structure_.tpl)
• /hardware
• cpus
• ...
• /software
• ...
• /system
• network
• ...

21
Pan Types

• Built-in Types
• long
• double
• string
• Standard types
• pro_declaration_type_validation_function_.tpl
• pro_declaration_type_.tpl
• ip, ipv4, ipv6, fqdn, hostname, shorthostname
• URI, absoluteURI, hostURI
• asndate, isodate, date (deprecated)
• email, hwaddr (MAC)

22
Standard Templates

• Load standard types into CDB
• cdb-simple-cli --add pro_declaration_type.tpl
• Load LCG-2 templates
• cdb-simple-cli --add pro_software_lcg2_.tpl
• Load repository template
• cdb-simple-cli --add repository.tpl

23
Hardware Templates

• Create hardware templates (CPU, RAM, disk, NIC)
• e.g. pro_hardware_cpu_GenuineIntel_Xeon_2400.tpl
• e.g. pro_hardware_ram_2048.tpl
• e.g. pro_hardware_harddisk_scsi_32.tpl
• e.g. pro_hardware_card_nic_broadcom_NetXtreme.tpl
• Create machine type
• Dual Xeon with 2GB RAM, 32GB SCSI disk, 3 network
  interfaces
• e.g. pro_hardware_dell_poweredge_1750_v00.tpl
• Load hardware templates
• cdb-simple-cli --add hardware.tpl

24
Software Templates

• SW repository templates
• Typically generated with edg-swrep-client
  command.
• e.g. repository_lal_repository_i386_rh73.tpl
• Can create templates for standard web servers.
• e.g. standard LCG-2 package repository
• Package lists
• Templates exist for LCG-2.0.0.
• Machine types pro_software_lcg2_machine_.tpl
• Base RH7.3 system pro_software_lcg2_machine_base.
  tpl
• User interface pro_software_lcg2_machine_ui.tpl
• Detailed service lists pro_software_lcg2_service_
  .tpl

25
System Templates

• Specify host information in object template
• Hostname
• NIC parameters
• Kernel version
• Create common system template (pro_system_common.t
  pl).
• Cluster name
• Root mail address
• Disk partitions
• Components
• ...

26
Object Templates

• Object template corresponds to client machines.
• e.g. profile_grid06.tpl
• Validation
• All object templates are validated for ALL
  template changes!
• Ensures confidence in generated machine profiles.
• Can be very painful to get right the first time.
• Must have working profile before trying client.

27
Manual Client Installation

• Prepare machine
• Install RedHat Linux 7.3
• server installation
• No firewall
• Make sure enough space (10 GB) on / partition!
• Optionally update with yum (see previous
  directions).
• Don't need to setup for automatic updates.
• Download quattor rpms
• Alpha-2 and external from previous links.
• Install all RPMs except syslinux and edg-cdb2sql.

28
Manual Client Configuration

• Configuration Cache Manager (CCM)
• Copy /usr/share/doc/edg-ccm-ltvergt/eg/edg-ccm.conf
  to /etc.
• Edit profile parameter
• change http//cdb/host.domain.xml
• to http//quattor.example.org/profiles/profile_h
  ost.xml
• Comment out context.
• Run /usr/sbin/edg-ccm-initialise.
• Node Configuration Deployer (NCD)
• Defaults in /etc/ncm-ncd.conf should be OK.
• Software Package Management Agent (SPMA)
• Set userpkgs in /etc/spma.conf to no.
• Other defaults should be OK.

29
Manual Client Configuration (cont.)

• Configuration Distribution Protocol Daemon (CDP)
• Copy /usr/share/doc/edg-cdp-listend-ltvergt/examples
  /edg-cdp-listend.conf to /etc.
• Change fetch parameter to /usr/sbin/edg-ccm-fetc
  h.
• Other defaults should be OK.
• Configuration Dispatch Daemon (cdispd)
• Defaults in /etc/ncm-cdispd.conf should be OK.

30
Manual Client Update

• Fetch the XML profile from the server.
• edg-ccm-fetch
• Start configuration dispatch daemon.
• service ncm-cdispd start
• Check log files for changes
• ncm-cdispd /var/log/ncm-cdispd.log
• SPMA /var/log/spma.log
• NCM components /var/log/ncm/
• On quattor server
• Edit /opt/edg/etc/cdb.conf to add client
  server_module.
• Change package list and verify changes are
  propagated to client.

31
Automatic Client Installation

• AII works from quattor server.
• Uses native machine installation tools.
• For linux kickstart.
• Bootstrap using PXE.
• DHCP server needed (IP address, kernel location)
• TFTP server needed (boot kernel)
• HTTP server needed (OS images, packages)

32
DHCP Setup

• Copy /usr/share/doc/aii-ltvergt/eg/dhcpd.conf to
  /etc.
• Edit to ensure that common setting are
  appropriate for site.
• Individual hosts will be added updated by AII
  scripts.

33
TFTP Setup

• Create directories /osinstall/nbp and
  /os/install/nbp/rh73.
• Copy the pxelinux.0 file to /osinstall/nbp.
• cp -f /usr/lib/syslinux/pxelinux.0 /osinstall/nbp
• Edit /etc/xinetd.d/tftp
• disable no
• server_args -s /osinstall/nbp
• Edit /etc/hosts.allow to permit clients to
  connect!
• Copy boot files to /osinstall/nbp/rh73
• cp /mnt/cdrom/images/pxeboot/ /osinstall/nbp/rh73
  
• Adjust location to where RH7.3 installation disk
  is located.
• Restart the xinetd service to activate changes!

34
Kickstart Setup

• Create directory /osinstall/nbp/ks.
• Make kickstart files available via web server.
• ln -s /osinstall/ks /var/www/html/ks
• Copy the pxelinux.0 file to /osinstall/nbp.
• Contents of installation disks must be available
  from web server.
• cp -r /mnt/cdrom/ /var/www/html/rh73
• Copy acknowledgement script to cgi location.
• cp /usr/sbin/aii-installack.cgi /var/www/cgi-bin
• Allow apache to run AII commands without
  password.
• Add line apache ALL(ALL) NOPASSWD ALL to
  /etc/sudoers.

35
Edit AII Templates

• Edit pro_software_component_aii.tpl
• File contains lots of local machine information.
• time zone
• keyboard type
• language
• Partitioning used for the local disks.
• Location of profiles
• /software/components/aii/osinstall/ks/cdbserver
• This must be the URL where the machine profiles
  are located.
• Make sure that updated component is loaded into
  CDB.
• Make sure /usr/lib/aii/osinstall/rh73.kstpl file
  is OK.
• Can make modifications to post install script
  here if necessary.
• Contains names of additional RPMs which must be
  installed.

36
AII Client Installation

• Edit /etc/aii-shellfe.conf setting cdburl to
  the CDB URL on server!
• Signal AII to do configuration for new client.
• aii-shellfe --configure quattor_client.example.org
  
• Mark this machine for installation.
• aii-shellfe --install quattor_client.example.org
• Verify changes
• /etc/dhcpd.conf contains correct host information
• Profile available from http//quattor.example.org/
  profiles.
• Kickstart file in /osinstall/ks is correct.
• Install Client
• Reboot client using PXE.
• Initial installation via kickstart reboots on
  completion.
• Quattor finishes installation then reboots.

37
Common Problems

• PXE can't get installation parameters from DHCP
  server.
• DHCP server not configured correctly. (Check IP
  numbers, etc.)
• Firewall is blocking access to DHCP server.
• Another DHCP server is responding.
• TFTP server can't be contacted.
• Check /etc/hosts.allow file.
• Verify all necessary files are available in
  /osinstall area.
• Firewall is blocking access to TFTP server.
• Kickstart fails
• Check information in pro_software_component_aii.tp
  l
• Change hda to sda for SCSI interface.
• Verify URL download location.

38
Writing NCM Components

• Easiest to start from an example.
• Check out quattor (and NCM components) from CVS
• export CVSROOTpserveranonymous_at_isscvs.cern.ch/
  local/reps/elfms
• cvs co elfms/quattor
• Copy ncm-edglcg to ncm-example
• cd elfms/quattor/ncm-components/lcg-2/
• mkdir ncm-example
• cp ncm-edglcg/ ncm-example/
• Rename edit files in ncm-example.
• Component writes params.--alpha, beta, gamma--to
  /etc/example.conf.
• See example files.

39
Incorporating New Component

• Build the necessary RPMs
• make rpm
• Load RPMs into SW repository.
• edg-swrep-client put i386_rh73 \ncm-example-1.0.0
  -1.i386.rpm /quattor
• Generate new repository template and load in CDB
• edg-swrep-client template i386_rh73 gt tmp.tpl
• mv tmp.tpl repository_lal_repository_i386_rh73.tpl
  
• cdb-simple-cli --update \repository_lal_repositor
  y_i386_rh73.tpl

40
Incorporating New Configuration

• Create template to include component
  configuration.
• make rpm
• Add template to quattor client's object template.
• edg-swrep-client put i386_rh73 \ncm-example-1.0.0
  -1.i386.rpm /quattor
• Check that changes are propagated to client.
• ls -l /etc/example.conf
• cat /etc/example.conf
• Make changes to configuration verify changes on
  client.

41
Summary

• Installed quattor server appropriate for small
  site.
• Used pan language to define client configuration.
• Managed manually-installed machine via quattor.
• Installed from scratch machine using quattor.
• Wrote and added a simple NCM component to
  configuration.
• Comments on improving the tutorial are very
  welcome!