Advanced Topics: MPI jobs

1
Advanced Topics MPI jobs
The EPIKH Project
(Exchange Programme to advance e-Infrastructure
Know-How)
Diego Scardaci diego.scardaci_at_ct.infn.it INFN
Catania Joint CHAIN/GISELA/EPIKH Grid School for
Application Porting 29th November - 9th Dicember
2010
www.epikh.eu
2
Table of Contents

• MPI and its implementations
• Wrapper script for mpi-start
• Hooks for mpi-start
• Defining the job and executable
• Running the MPI job
• References

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
3
MPI and its implementations

• The Message Passing Interface (MPI) is a de-facto
  standard for writing parallel application.
• There are two versions of MPI, MPI-1 and MPI-2
• Two implementations of MPI-1
• LAM
• MPICH.
• Two implementations of MPI-2
• OpenMPI
• MPICH2.
• Individual sites may chose to support only a
  subset of these implementations, or none at all.

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
4
Requirements to submit MPI jobs

• Proxy certificate.
• Permissions to submit job in a specific site.
• The mpi jobs can run only in one Grid site.

3
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
5
Get information from the information sys

• Find the sites that support MPICH and MPICH2

4
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
6
Get information from the information sys

• Find the sites that support MPICH and its
  available CPUs

5
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
7
Get information from the information sys

• Find the sites who have shared home MPI directory
• What happens when there is a MPI shared directory
  in the site?
• The file is compiled in one WN, then the file
  read by the other WNs in the same site.
• What happens when is not MPI shared directory?
• The file is compiled in one WN, then the file
  must be copied to the others WNs.

TagMPI-START
6
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
8

• MPI and its implementations
• Wrapper script for mpi-start
• Hooks for mpi-start
• Defining the job and executable
• Running the MPI job
• References

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
9
mpi-start

• mpi-start is a recommended solution to hide the
  implementation details for jobs submission.
• The design of mpi-start was focused in making the
  MPI job submission as transparent as possible
  from the cluster details!
• It was developed inside the Int.EU.Grid project
• The RPM to be installed in all WNs can be found
  here
• Using the mpi-start system requires the user to
  define a wrapper script that set the environment
  variables and a set of hooks.

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
10
Wrapper script for mpi-start

• !/bin/bash
• Pull in the arguments.
• MY_EXECUTABLEpwd/1
• MPI_FLAVOR2
• Convert flavor to lowercase for passing to
  mpi-start.
• MPI_FLAVOR_LOWERecho MPI_FLAVOR tr
  'upper' 'lower'
• Pull out the correct paths for the requested
  flavor.
• eval MPI_PATHprintenv MPI_MPI_FLAVOR_PATH
• Ensure the prefix is correctly set. Don't rely
  on the defaults.
• eval I2G_MPI_FLAVOR_PREFIXMPI_PATH
• export I2G_MPI_FLAVOR_PREFIX
• Touch the executable.
• It exist must for the shared file system check.
• If it does not, then mpi-start may try to
  distribute the executable
• when it shouldn't.
• touch MY_EXECUTABLE
• Setup for mpi-start.
• export I2G_MPI_APPLICATIONMY_EXECUTABLE
• export I2G_MPI_APPLICATION_ARGS

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
11

• MPI and its implementations
• Wrapper script for mpi-start
• Hooks for mpi-start
• Defining the job and executable
• Running the MPI job
• References

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
12
Hooks for mpi-start /1

• The user may write a script which is called
  before and after the MPI executable is run.
• The pre-hook script can be used, for example, to
  compile the executable itself or download data
• The post-hook script can be used to analyze
  results or to save the results on the grid.
• The pre- and post- hooks script may be defined in
  separate files, but the name of the functions
  named exactly pre_run_hook and post_run_hook

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
13
Hooks for mpi-start /2

• !/bin/sh
• This function will be called before the MPI
  executable is started.
• pre_run_hook ()
• Compile the program.
• echo "Compiling I2G_MPI_APPLICATION"
• Actually compile the program.
• cmd"mpicc MPI_MPICC_OPTS -o
  I2G_MPI_APPLICATION I2G_MPI_APPLICATION.c"
• echo cmd
• cmd
• if ! ? -eq 0 then
• echo "Error compiling program. Exiting..."
• exit 1
• fi
• Everything's OK.
• echo "Successfully compiled I2G_MPI_APPLICATION
  "
• return 0
• This function will be called before the MPI
  executable is finished.

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
14

• MPI and its implementations
• Wrapper script for mpi-start
• Hooks for mpi-start
• Defining the JDL for the job and executable
• Running the MPI job
• References

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
15
Defining the job and executable /1

• Running the MPI job itself is not significantly
  different from running a standard grid
  job.JobType Normal"
• CpuNumber 2
• Executable "mpi-start-wrapper.sh"
• Arguments "mpi-test MPICH"
• StdOutput "mpi-test.out"
• StdError "mpi-test.err"
• InputSandbox "mpi-start-wrapper.sh",
  "mpi-hooks.sh","mpi-test.c"
• OutputSandbox "mpi-test.err","mpi-test.out"
• Requirements Member("MPI-START", other.GlueHos
  tApplicationSoftwareRunTimeEnvironment)
• Member(MPICH", other.GlueHostApplicationSoftw
  areRunTimeEnvironment)
• The JobType must be Normal and the attribute
  CpuNumber must be defined

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
16
Structure of a MPI job in the Grid with mpi-start
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
17
Structure of a MPI job in the Grid with mpi-start
8
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
18
Defining the job and executable /2

• include "mpi.h"
• include ltstdio.hgt
• int main(int argc, char argv)
• int numprocs / Number of processors /
• int procnum / Processor number /
• / Initialize MPI /
• MPI_Init(argc, argv)
• / Find this processor number /
• MPI_Comm_rank(MPI_COMM_WORLD, procnum)
• / Find the number of processors /
• MPI_Comm_size(MPI_COMM_WORLD, numprocs)
• printf ("Hello world! from processor d out of
  d\n", procnum, numprocs)
• / Shut down MPI /
• MPI_Finalize()
• return 0

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
19
Hook Helpers

• The shell variable MPI_START_SHARED_FS can be
  checked to figure out if the current site has a
  shared file system or not.
• The mpi_start_foreach_host shell function can be
  used to iterate over all the available machines
  in the current run.do_foreach_node () the
  first parameter 1 contains the hostname
• post_run_hook ()
• ...
• mpi_start_foreach_host do_foreach_node
• ...

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
20

• MPI and its implementations
• Wrapper script for mpi-start
• Hooks for mpi-start
• Defining the job and executable
• Running the MPI job
• References

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
21
Running the MPI job

• Running the MPI job is no different from any
  other grid job.
• If the job ran correctly, then the standard
  output should contain something like the
  following-ltSTART PRE-RUN HOOKgt-----------------
  --------------------------
• -ltSTOP PRE-RUN HOOKgt-----------------------------
  ----------------
• START
  
• Hello world! from processor 1 out of 2
• Hello world! from processor 0 out of 2
• FINISHED
  
• -ltSTART POST-RUN HOOKgt---------------------------
  ----------------
• -ltSTOP POST-RUN HOOKgt----------------------------
  ---------------

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
22
Running MPI job in the Grid without mpi-start
12
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
23

• MPI and its implementations
• Wrapper script for mpi-start
• Hooks for mpi-start
• Defining the job and executable
• Running the MPI job
• References

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
24
References

• GISELA MPI FAQ
• http//applications.eu-eela.eu/grid_faq_MPICH.php?
  l40n14
• EGEE Mpi guide link
• EGEE MPI WG link
• MPI-START Documentation link
• Site config for MPI link

Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010
25
Thank you for your kind attention !
Any questions ?
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School
for Application Porting, 01-12-2010