Software Versioning and Debugging Tools

1
Part IV AliEn

2
Outline I

• Installation of the AliEn software.
• Authentication Possible problems.
• General description of the shell
• Basic commands.
• Basic functionalities.
• Working with the file catalogue
• Copying files from/to the catalogue.
• File catalogue structure.
• Querying the file catalogue.
• ROOT API
• News on production.

3
Outline II

• Flow of the overall analysis procedure.
• Local analysis
• Creation of tag files.
• Local analysis using the Event Tag System.
• Interactive analysis with AliEn stored files.
• Batch analysis
• Flow of the procedure.
• Description of the files needed.
• Description of the jdl fields.
• Practical examples.
• AliEn plugin

4
Prerequisites

• Did you followed ALL the steps at
• http//alien.cern.ch/twiki/bin/view/Alice/UserRegi
  stration
• Do you have valid usercert.pem and userkey.pem
  files ?
• If not, you will only be able to watch this
  tutorial...

5
Prepare

• Login to trnXXX
• Copy grid certificates to trnXXX
• /afs/cern.ch/user/t/trnXXXgt mkdir .globus
• scp user_at_lxplus.globus/.pem .globus
• Copy tutorial tarball to trnXXX
• wget http//aliceinfo.cern.ch/Offline/Activities/A
  nalysis/TagAnalysis/GridTutorial.tgz
• tar xvzf GridTutorial.tgz
• cd GridTutorial
• source gridenv.sh

6
Installation Getting the installer
7
Installation Selecting the version
8
Installation Reuse of software
9
Installation Workspace directory
10
Installation Detecting the platftom
11
Installation Supported platforms
12
Installation Installation directory
13
Installation Selecting installation directory
14
Installation Overwriting files
15
Installation Selecting packages
16
Installation Progress bar
17
Installation Final window
18
Installation Try it out

• Download the alien installer from
  http//alien.cern.ch.
• Make the file executable.
• Run the installer.
• Select v2-15
• Platform should be i686.
• Select installation directory.
• Select the following packages
• GShell

19
Authentication Preparing the certificates
20
Authentication Changes in env. variables

• AliEn environment
• export ALIEN_ROOTHOME/alien
• export GSHELL_ROOTALIEN_ROOT/api
• export PATHGSHELL_ROOT/binPATH
• Configure ROOT with
• --enable-alien --enable-globus \
• --with-alien-incdirALIEN_ROOT/api/include \
• --with-alien-libdirALIEN_ROOT/api/lib \

21
Authentication Getting a GRID proxy
22
Authentication Getting an AliEn token
23
Authentication Authentication problems I

• Globus related
• Permissions on HOME/.globus/userkey.pem are not
  private to the user chmod 400 userkey.pem
• Your certificate authority is exotic and not
  known to the server.
• Your certificate has expired.
• Clock skew
• Your local computer time is in the future with
  respect to the server's time.
• Your local computer time is more in the past than
  the certificate life time.

24
Authentication Authentication problems II

• alien-token-init related
• You have not gone through all steps of the AliEn
  user registration.
• You have not given the AliEn user name as an
  argument to the token-init command and your local
  user name is not identical to the AliEn user
  name.
• The script wants to bootstrap the installation
  but you don't have write permissions on the
  installation path Avoid bootstrapping by
  setting the GSHELL_ROOT environment variable.

25
Authentication Try it out

• Upload your certificates to your machines
• Store them under e.g /afs/cern.ch/user/t/trn2301/
  .globus/
• Check that the key file is read only.
• Get a valid alien token.
• Use /afs/cern.ch/alice/offline/alien/pro/api/bin
  /alien-token-init ltcertificate_namegt
• If asked if you want to compile the gapi and
  xrootd libs answer YES (everything is already
  precompiled)?
• Source the generated environment file
• . /tmp/gclient_env_UID (where UID is the user
  ID)?

26
Shell Accessing the shell
27
Shell Basic commands I
Tab completion working!!!
28
Shell Basic commands II
29
Shell Basic commands III
30
Shell whereis command
31
Shell Viewing the files I
32
Shell Viewing the files II
33
Shell Editing files
Define your preferred editor via the variable
EDITOR 'emacs' 'emacs -nw' 'xemacs' 'xemacs
-nw' 'pico' 'vi' (DEFAULT)? 'vim' The file is
temporary in /tmp on your local disk and then is
uploaded once you exit the editor!
34
Shell Clear old versions
35
Shell Copying files from/to the F.C.
36
Shell File catalogue structure

• The path name will be
• for real data /alice/data/ltYeargt/ltAcceleratorPe
  riodgt/ltRunNumbergt/
• Like /alice/data/2008/LHC08c/000057647
• for simulated data /alice/sim/ltYeargt/ltProductionT
  ypegt/ltRunNumbergt/
• Like /alice/sim/PDC_08/LHC08b1/300000
• Subdirectories will be called
• raw/
• cond/
• reco/ltPassXgt/ESD/
• reco/ltPassXgt/AOD/
• File names will look like this
  ltxxxxgt.AliESD.root
• For further information see
• http//indico.cern.ch/conferenceDisplay.py?confId
  3280
• http//cern.ch/Oldenburg/MetaData/MetaData.doc

37
Shell Querying the F.C. I
alienshalice 1 find -x pp /alice/sim/PDC_08/
LHC08b1/300000 tag.root gt pp.xml
Redirect the output to the xml collection.
38
Shell Try it out I

• Access the alien shell.
• Check your user name by typing whoami.
• List the contents of your home directory.
• Print the working directory.
• Create the following directory structure in your
  AliEn space
• HOME/bin (if it is not there)?
• HOME/Tutorial/ and HOME/Tutorial/output
• Get the information of the file (whereis)
  /alice/cern.ch/user/a/agheata/Tutorial/AliAnalysis
  TaskPt.cxx

39
Shell Try it out II

• Go to GridTutorial/Tags directory
• Access the shell again and query the f.c. and get
  all the tag files (.tag.root) under
  /alice/sim/PDC_08/LHC08b1/300000/
• Get the output on your terminal (find /alice/)?
• Get the output on your terminal in a xml format
  (find -x tag /alice/)?
• Redirect the results to the tag.xml collection
  the output will not be stored in your AliEn
  working directory but rather in your local
  working directory.
• Repeat the previous exercise limiting the number
  of output files to 10 (find -l 10 ...).

40
Production status
41
ROOT ROOT API
42
ROOT - Connecting
43
ROOT Accessing a GRID file
alien///alice/sim/PDC_08/LHC08b1/300000/001/AliE
SDs.root
44
ROOT Try it out

• Source the environment file created by
  alien_token_init
• . /tmp/gclient_env_UID
• Once finished, type root and then
• root 0 TGridConnect(alien//)

45
Analysis Flow of the analysis procedure
Query the file catalog
46
Analysis Event Tag System
47
Local analysis Creation of tag files
Setup par archive Load the needed libraries
AliESDTagCreator t new AliESDTagCreator()
t-gtSetStorage(0)
t-gtReadLocalCollection(/home/pchrist/PDC06/pp14Te
V/)
t-gtMergeTags(ESD)
48
Local analysis Local analysis with tags I
Setup par archive Load the needed libraries
AliRunTagCuts runCuts new AliRunTagCuts() AliL
HCTagCuts lhcCuts new AliLHCTagCuts() AliDetec
torTagCuts detCuts new AliDetectorTagCuts() Al
iEventTagCuts evCuts new AliEventTagCuts() evC
uts-gtSetMultiplicityRange(0,1500)
AliTagAnalysis tagAna new AliTagAnalysis(ESD)
tagAna-gtChainLocalTags(.)
analysischain tagAna-gtQueryTags(runCuts,lhcCuts,
detCuts,evCuts)

• The query of the Event Tag System can be done by
• Using the AliTagCuts objects
• Using string statements ((fEventTag.fNumberOfTrac
  ks gt 0)(fEventTag.fNumberOfTracks lt 1500))?

49
Local analysis Local analysis with tags II
AliAnalysisManager manager new
AliAnalysisManager() AliESDInputHandler esdH
new AliESDInputHandler() manager-gtSetInputEventHa
ndler(esdH) AliAnalysisTaskPt task new
AliAnalysisTaskPt(TaskPt) manager-gtAddTask(task
)
AliAnalysisDataContainer cinput1
manager-gtCreateContainer("cchain1",TChainClass()
,AliAnalysisManagerkInputContainer) AliAnalysis
DataContainer coutput1 manager-gtCreateContainer
("chist1", TH1Class(),AliAnalysisManagerkOutpu
tContainer,"Pt.ESD.root")
manager-gtConnectInput(task,0,cinput1) manager-gtCo
nnectOutput(task,0,coutput1) if
(!manager-gtInitAnalysis()) return
manager-gtStartAnalysis(local,chain)
50
Local analysis Try it out

• Run the GridTutorial/tags/runTagCreator.C macro
  to create the tag files.
• Go to your local /home/trn23XX/GridTutorial/Local
  directory and open the runLocal.C file.
• Run the runLocal.C macro with root.
• Impose some selection criteria and rerun the
  example.

51
Interactive analysis Using the tags
Setup par archive Load the needed libraries
TGridConnect("alien//)
TAlienCollection coll TalienCollectionOpen("t
ag100.xml") TGridResult tagResult
coll-gtGetGridResult(",0,0)
AliTagAnalysis tagAna new AliTagAnalysis(ESD)
tagAna-gtChainGridTags(TagResult)
TAG CUTS
TChain chain tagAna-gtQueryTags(runCuts,lhcCuts,
detCutsevCuts)
AliAnalysisManager manager new
AliAnalysisManager() AliAnalysisTaskPt task
new AliAnalysisTaskPt(TaskPt) manager-gtAddTask(
task)
Same code as in two pages back
manager-gtStartAnalysis (local,chain)
52
Interactive analysis Try it out

• Go to your local /home/trn23XX/GridTutorial/Intera
  ctive directory and open the runInteractive.C
  file.
• Change the line where you define the tag
  collection and put the name of the file you
  created by querying the f.c (it should be
  tag.xml).
• Run the runInteractive.C macro with root
• Impose some selection criteria and rerun the
  example.

These examples can also be found
in ALICE_ROOT/PWG2/AnalysisMacros/Interactive
(ESDKine analysis)?
53
Batch analysis Flow of batch analysis
54
Batch analysis Analysis flow with tags
Event list doesnt get lost during splitting
55
Batch analysis Creating a new xml file
Setup par archive Load the needed libraries

• You need to use the tags in a batch session
  because
• They reduce your analysis time which allows you
  to
• lower your TTL (see next slides) and thus make
  sure
• that your job starts early enough (jobs are
  ordered by TTL).
• They provide the analyzed data in the proper
  format
• (TChain TEntryLists) in a totally transparent
  way.

TGridConnect("alien//)
TAlienCollection coll TalienCollectionOpen("t
ag.xml") TGridResult tagResult
coll-gtGetGridResult(",0,0)
AliTagAnalysis tagAna new AliTagAnalysis(ESD)
tagAna-gtChainGridTags(tagResult)
AliRunTagCuts runCuts new AliRunTagCuts() AliL
HCTagCuts lhcCuts new AliLHCTagCuts() AliDetec
torTagCuts detCuts new AliDetectorTagCuts() Al
iEventTagCuts evCuts new AliEventTagCuts() evC
uts-gtSetMultiplicityRange(0,1500)
tagAna-gtCreateXMLCollection(global,runCuts,lhcCu
ts,detCuts,evCuts)
The old xml collection (tag.xml) has information
about the tag files that are qoing to be
queried. The new xml collection (global.xml) has
information about the ESDs that are going to be
analyzed.
56
Batch analysis Files for batch analysis

• Executable
• Par file
• Macro
• Selectors/Tasks
• xml collection
• jdl

57
Batch analysis - Executable
!/bin/bash echo echo
PATH echo ROOTSYS echo LD_LIBRARY_PATH echo
root -b -q
Analysis.C
IT SHOULD BE STORED UNDER HOME/bin IN THE FILE
CATALOG!!!
58
Batch analysis - Macro

• Setup the par file compile and load the
  libESD.so (or any necessary library that is
  needed for the analysis).
• Get the xml collection.
• Convert the collection to a list of files.
• Process the chain with the selector or an
  AliAnalysisManager.

59
Batch analysis JDL fields I

• Executable Compulsory field where we give the
  lfn of the executable that should be stored in
  /bin or V0/bin or HOME/bin.
• Arguments They will be passed to the executable.
• Packages Type packages in the shell to see what
  kind of packages are installed.
• InputFile The files that will be transported to
  the node where the job will run.
• InputData It will require that the job will be
  executed in a site close to the files specified
  here.
• InputDataList The filename in which the Job
  Agent will write the InputData list.
• InputDataListFormat The format of the InputData
  list.

60
Batch analysis JDL fields II

• OutputFile The files that will be registered in
  the catalog once the job finishes.
• OutputArchive What files will be archived in a
  zip file.
• Validationcommand Specifies the script to be
  used as a validation script.
• Email Receive a mail when the job finishes.
• TTL The maximum run time of your job.
• Split Split the jobs in several sub jobs.

61
Batch analysis Job status
62
Batch analysis Submitting jobs
If everything is ok with your jdl then your job
is submitted and a ltJOBIDgt.is assigned to it. You
get a submission error message if i)a file
listed in the jdl is missing ii)a package defined
in the jdl is not listed in the packman
63
Batch analysis Checking the job status I
64
Batch analysis Checking the job status II
65
Batch analysis Checking the priority
66
Batch analysis Checking the job output I
67
Batch analysis Checking the job output II
68
Batch analysis Merging the output

• In order to merge the several output files you
  have to put the following lines in your jdl
• Merge "ltroot file to mergegt/alice/jdl/mergero
  otfile.jdlltmerge-outputfilegt"
• MergeOutputDir"/alice/cern.ch/....../run-xxx/"
• Example 1 you want to merge your output
  histogram file 'histo.root' into a merge file
  'histo-merged.root'. You just add this line to
  your master job
• Merge "histo.root/alice/jdl/mergerootfile.jdl
  histo-merged.root"
• Example 2 you want to merge two different output
  files 'histo.root' 'results.root', then you
  just specify a list of merge jobs in the JDL
• Merge "histo.root/alice/jdl/mergerootfile.jdl
  histo-merged.root" , "results.root/alice/jdl/mer
  gerootfile.jdlresults-merged.root"

69
Batch analysis Try it out I

• Go to your local /home/trn23XX/GridTutorial/Batch
  
• Run the CreateXML.C macro.
• Change selection criteria

70
Batch analysis Try it out II

• Copy the local file global.xml that we just
  created to your AliEn HOME/Tutorial/
• HINT cp file/home/trn23XX/GridTutorial/Batch/glo
  bal.xml global.xml_at_ALICECERNse
• Copy the following local files to your AliEn
  HOME/Tutorial/
• /home/trn23XX/GridTutorial/Batch/STEERBase.par ,
  ESD.par , AOD.par , ANALYSIS.par
  ANALYSISalice.par
• /home/trn23XX/GridTutorial/Batch/AliAnalysisTaskPt
  .h
• /home/trn23XX/GridTutorial/Batch/AliAnalysisTaskPt
  .cxx
• /home/trn23XX/GridTutorial/Batch/runBatch.C
• /home/trn23XX/GridTutorial/Batch/analysisESD.jdl
• /home/trn23XX/GridTutorial/Batch/validation.sh
• Copy the local file /home/trn23XX/GridTutorial/Bat
  ch/root.sh to your AliEn HOME/bin

71
Batch analysis Try it out III

• While being in the AliEn shell, edit the
  analysisESD.jdl and modify the InputFile,
  InputDataCollection, InputDataListFormat and
  OutputDir fields.
• Once finished, submit your batch job by typing
• submit analysisESD.jdl

These examples can also be found
in ALICE_ROOT/PWG2/AnalysisMacros/Batch
72
Batch analysis Try it out IV

• Check your job priority by typing queue
  priority jobs username.
• Display the jdl of your job by typing ps -jdl
  jobid.
• Trace the status of your job by typing ps
  -trace jobid.
• When the job 's status turns to RUNNING you can
  get the stdout and stderr of the job by typing
• spy jobid stdout.
• spy jobid stderr.

73
AliEn plugin

• A helper thay will ease-up running your analysis
  in GRID
• Works as a plugin for the analysis manager (as
  event handlers)
• One has to create and configure a
  AliAnalysisAlien object
• See http//aliceinfo.cern.ch/Offline/Activities/A
  nalysis/AnalysisFramework/AlienPlugin.html
• Creates dataset, JDL, analysis macro,
  executionvalidation scripts
• Submits your job and merges the results

74
Plugin configuration

• Open CreateAlienHandler.C
• Change working/output directories
• Modify number of files/worker
• Make sure the run mode is set to full
• Run macro runGrid.C
• Inspect the job status
• Modify the run mode to terminate once job
  finished
• Run again runGrid.C

75
References I

• Registration Certificates
• http//alien.cern.ch/twiki/bin/view/Alice/UserRegi
  stration
• https//ca.cern.ch/ca/
• AliEn
• http//alien.cern.ch
• Gshell
• http//alien.cern.ch/twiki/bin/view/AliEn/GAPI
• User's guide
• http//project-arda-dev.web.cern.ch/project-arda-d
  ev/alice/apiservice/AA-UserGuide-0.0m.pdf

76
References II

• aliensh Grid Command Online Reference V1.0
• http//project-arda-dev.web.cern.ch/project-arda-d
  ev/alice/apiservice/guide/guide-1.0.htm
• Previous tutorials
• http//aliceinfo.cern.ch/Offline/Analysis/Tutorial
  /
• Event Tag System
• http//pcaliweb02.cern.ch/Offline/Analysis/RunEven
  tTagSystem/EventTags.htmlEvent20tag20system
• https//edms.cern.ch/document/788315/1 (INTERNAL
  NOTE)?

77
References III

• Creation of tag files
• http//pcaliweb02.cern.ch/Offline/Analysis/RunEven
  tTagSystem/EventTagsCreation.htmlCreate20tags20
  howto
• Analysis using the Event Tag System
• http//pcaliweb02.cern.ch/Offline/Analysis/RunEven
  tTagSystem/EventTagsAnalysis.htmlAnalysis20with
  20tags

78
References IV

• File catalog structure Queries
• http//pcaliweb02.cern.ch/Offline/Analysis/RunEven
  tTagSystem/RunTags.htmlRun/File20metadata
• File level metadata
• http//cern.ch/Oldenburg/MetaData/MetaData.doc
• Analysis framework
• http//indico.cern.ch/materialDisplay.py?contribId
  19ampsessionId3ampmaterialIdslidesampconf
  Ida056304