Porting MPI Applications use cases: OpenFOAM, Fluent

1
Porting MPI Applications use cases OpenFOAM,
Fluent

• Marcello Iacono Manno
• Consorzio COMETA
• marcello.iacono_at_ct.infn.it
• JOINT EELA2-EGEE3 TUTORIAL4TRAINERS
• Catania, 2008, July 4th

2
Outline

• OpenFOAM to the Grid
• Porting Issues
• Submission procedure
• Developed Tools
• Future Developments
• Fluent
• Porting Issues
• Submission procedure
• Developed Tools
• Future Developments
• Abaqus
• Developed tools
• Future Development

3
Porting (1/5)

• OpenFOAM is an open-source SW for computational
  fluid dynamics (CFD)
• it is a parallel, multi stage application
• several solvers, C-based libraries
• PORTING OPENFOAM to the Grid
• a forerunner experience
• static installation
• the gcc4.2.1/pgcc compiler
• the 32/64 bit issue
• LAM / MPICH
• which mpirun
• machines file
• preparing parallel execution

4
Porting (2/5)

• Installation
• compressed package is about 300 MB
• dynamical installation not feasible
• stable package
• jobs can also be short (test)
• environment problem not solved
• but some further testing is required on this
  particular issue
• static installation
• software manager
• directly on the WNs
• installation job (lcg-asis)
• check for compatibility
• middleware
• other applications

!
5
Static vs. Dynamic Installation

• Two ways to install the software
• Static Installation (local or shared fixed
  installation)
• Advantage speeds up job execution (less work to
  do)
• Disadvantages
• compatibility problems with middleware and/or
  other applications
• more complex modifying and updating the
  application
• requires Software Manager (SWM) role privileges
• Usage
• huge and/or stable SW packages
• only MPI binary executable available (difficultly
  modifiable)
• Dynamic Installation (local installation during
  job pre processing)
• Advantages
• more robust and flexible jobs
• easier modifying and updating
• No SWM privileges required
• Disadvantages slows down job execution (more
  work to do)
• Usage
• little and/or frequently modified SW packages

6
Porting (3/5)

• Gcc4.1.2
• different middleware versions
• OpenFOAM-1.4 requires at least gcc3.4.6 (4.1.2
  suggested)
• old WNs run 32 bits Scientific Linux (SLC)
  3.0.8 with gcc3.2.6
• INCOMPATIBILITY
• 64-bit installation
• full-64 OpenFOAM-1.4 bit is available
• new WNs have a 64 bit-architecture with SLC4
  and gcc4.x.x
• 32/64 bit
• no 64-bit user interface available up to
  yesterday
• pre and post processing must run (as jobs) on
  the WN
• autonomous jobs or part of a greater job (pre /
  post) ?
• alternatively cross-pre-processing on a 64-bit
  machine, then
• transferring the OpenFOAM user directory

!
7
Porting (4/5)

• LAM / MPICH
• different protocols
• OpenFOAM-1.4 is declared MPICH-compliant (by
  official docs)
• only libPstream needed to be recompiled (thanks
  to the forum!)
• no native mpirun allowed (Grid incompatible)
• Which mpirun
• new 64-bit (Grid compliant) mpirun script
• different libraries for MPICH/MPICH2,
  InfiniBand/GigaBit
• device p4 ? ch_p4
• Machines files
• specified by the middleware (transparent to the
  user)
• all the WNs of a CE

!
8
Porting (5/5)

• Preparing parallel execution
• OpenFOAM user workspace is local
• a sub directory called user-1.4 under PWD
• PWD/user-1.4/run/tutorials/ltrootgt/ltcasegt/system
• ltrootgt ? solver (icoFoam)
• ltcasegt ? geometry (cavity)
• system ? running system information
• (file) decomposeParDict ? mesh decomposition
• decomposePar icoFoam cavity
• (file) proc0n ? processing element information
• Current pre-processing
• needs to become a job
• or
• alternatively, a 64-bit UI process

!
9
Submit (1/10)
iacono_at_infn-ui-01 OpenFOAM.grid cat
openfoam.jdl Type "Job" JobType
"MPICH" Executable "/opt/exp_soft/cometa/OpenFO
AM/OpenFOAM-1.4/applications/bin/ linuxGcc4DPOpt/i
coFoam" Arguments ". cavity -parallel" NodeNum
ber 4 StdOutput "openfoam.out" StdError
"openfoam.err" InputSandbox "mpi.pre.sh","mpi.
post.sh","cavity.tar" OutputSandbox
"openfoam.out","openfoam.err","result.tgz" Retr
yCount 7 Requirements (
other.GlueCEUniqueID "infn-ce-01.ct.pi2s2.it21
19/jobmanager-lcglsf-infinite" )
10
Submit (2/10)

• iacono_at_infn-ui-01 OpenFOAM.grid cat mpi.pre.sh
• !/bin/bash
• date
• echo "openfoam pre-processing file"
• hostname -f
• tar xzf cavity.tar
• ls -l
• echo "content of PWD/cavity"
• ls -l PWD/cavity

11
Submit (3/10)
marcello_at_infn-ui-01 OpenFOAM.grid ll
/home/marcello/JobOutput/marcello_-fGr9naNAp6Phj3P
_GacQw total 88 -rw-rw-r-- 1 marcello marcello
0 Jul 2 1602 openfoam.err -rw-rw-r--
1 marcello marcello 82663 Jul 2 1602
openfoam.out marcello_at_infn-ui-01 OpenFOAM.grid
head -n 100 /home/marcello/JobOutput/marcello_-fGr
9naNAp6Phj3P_GacQw/openfoam.out Mon Jul 2
155403 CEST 2007 openfoam pre-processing
file infn-wn-10.ct.pi2s2.it WM_LINK_LANGUAGEc W
M_ARCHlinux WM_JAVAC_OPTIONOpt WM_PROJECT_VERSIO
N1.4 WM_COMPILER_LIB_ARCH WM_PROJECT_INST_DIR/o
pt/exp_soft/cometa/OpenFOAM WM_PROJECT_DIR/opt/ex
p_soft/cometa/OpenFOAM/OpenFOAM-1.4 WM_COMPILER_AR
CH-64
shared installation space
64-bit architecture
12
Submit (4/10)
WM_PROJECTOpenFOAM WM_COMPILERGcc4 WM_MPLIBMPIC
H WM_COMPILER_DIR/opt/exp_soft/cometa/OpenFOAM/li
nux/gcc-4.1.2-64 WM_COMPILE_OPTIONOpt WM_SHELLba
sh WM_DIR/opt/exp_soft/cometa/OpenFOAM/OpenFOAM-1
.4/wmake WM_PROJECT_USER_DIR/home/cometa005/globu
s-tmp.infn-wn-10.14261.0/.mpi/https_3a_2f_2finfn-r
b-01.ct.pi2s2.it_3a9000_2f-fGr9naNAp6Phj3P_5fGacQw
/user-1.4 WM_OPTIONSlinuxGcc4DPOpt WM_PROJECT_LAN
GUAGEc WM_PRECISION_OPTIONDP
native gcc compiler
double precision
13
Submit (5/10)
content of /home/cometa005/globus-tmp.infn-wn-10.
14261.0/.mpi/https_3a_2f_2finfn-rb-01.ct.pi2s2.it_
3a9000_2f-fGr9naNAp6Phj3P_5fGacQw total
5536 lrwxrwxrwx 1 cometa005 cometa 130 Jul
2 1554 cometa005-1.4 -gt /home/cometa005/globus-tm
p.infn-wn-10.14261.0/.mpi/https_3a_2f_2finfn-rb-01
.ct.pi2s2.it_3a9000_2f-fGr9naNAp6Phj3P_5fGacQw/use
r-1.4 -rw-r--r-- 1 cometa005 cometa 0 Jul
2 1553 host14808 -rw-r--r-- 1 cometa005 cometa
64 Jul 2 1553 mpi.post.sh -rw-r--r-- 1
cometa005 cometa 547 Jul 2 1553
mpi.pre.sh -rw-r--r-- 1 cometa005 cometa 0
Jul 2 1554 openfoam.err -rw-r--r-- 1 cometa005
cometa 878 Jul 2 1554 openfoam.out drwxr-xr-
x 5 cometa005 cometa 4096 Jun 18 1315
user-1.4 -rw-r--r-- 1 cometa005 cometa 5632674
Jul 2 1553 user-1.4.tar content of
/opt/exp_soft/cometa/OpenFOAM total 12 -rw-rw-rw-
1 cometa001 cometa 382 Jun 20 1009
env_config drwxrwxrwx 3 cometa001 cometa 4096
Jun 6 0848 linux drwxrwxrwx 10 cometa001
cometa 4096 Jun 8 1823 OpenFOAM-1.4
home dir on the master WN
pre and post processing scripts
user dir
14
Submit (6/10)

• content of /home/cometa005/globus-tmp.infn-wn-10.1
  4261.0/.mpi/https_3a_2f_2finfn-rb01.ct.pi2s2.it_3a
  9000_2f-fGr9naNAp6Phj3P_5fGacQw/user-1.4/run/tutor
  ials
• total 16
• -rwxr-xr-x 1 cometa005 cometa 1308 Jun 19 1631
  controlDict
• -rwxr-xr-x 1 cometa005 cometa 1363 Jun 29 1353
  decomposeParDict
• -rwxr-xr-x 1 cometa005 cometa 1479 Jun 19 1631
  fvSchemes
• -rwxr-xr-x 1 cometa005 cometa 1305 Jun 19 1631
  fvSolution
• /------------------------------------------------
  ---------------------------\
• 
  
• \\ / F ield OpenFOAM The Open
  Source CFD Toolbox
• \\ / O peration Version 1.4
  
• \\ / A nd Web
  http//www.openfoam.org
• \\/ M anipulation
  
• \------------------------------------------------
  ---------------------------/

mesh decomposition
15
Submit (7/10)
1 Date Jul 02 2007 1 Time
155422 1 Host infn-wn-10.ct.pi2s2.it 1
PID 29680 4 Date Jul 02 2007 4 Time
155422 4 Host infn-wn-06.ct.pi2s2.it 4
PID 25510 3 Date Jul 02 2007 3
Time 155422 3 Host infn-wn-10.ct.pi2s2
.it MPI Pstream initialized
with floatTransfer 1 nProcsSimpleSum
0 scheduledTransfer 0
16
Submit (8/10)

• Exec /opt/exp_soft/cometa/OpenFOAM/OpenFOAM-1.
  4/applications/bin/linuxGcc4DPOpt/icoFoam
  /home/cometa005/globus-tmp.infn-wn-10.14261.0/.mpi
  /https_3a_2f_2finfn-rb-01.ct.pi2s2.it_3a9000_2f-fG
  r9naNAp6Phj3P_5fGacQw/cometa005-1.4/run/tutorials/
  icoFoam cavity parallel
• 0 7
• 0 (
• 0 infn-wn-10.ct.pi2s2.it.29680
• 0 infn-wn-10.ct.pi2s2.it.30265
• 0 infn-wn-10.ct.pi2s2.it.30851
• 0 infn-wn-06.ct.pi2s2.it.25510
• 0 infn-wn-06.ct.pi2s2.it.26093
• 0 infn-wn-06.ct.pi2s2.it.26676
• 0 infn-wn-06.ct.pi2s2.it.27259
• 0 )
• 0
• Create time

Process IDs
17
Submit (9/10)

• Create mesh for time 0
• 7 Date Jul 02 2007
• 7 Time 155422
• 7 Host infn-wn-06.ct.pi2s2.it
• 7 PID 27259
• 7 Root /home/cometa005/globus-tmp.infn-wn-10
  .14261.0/.mpi/https_3a_2f_2finfn-rb-01.ct.pi2s2.it
  _3a9000_2f-fGr9naNAp6Phj3P_5fGacQw/cometa005-1.4/r
  un/tutorials/icoFoam
• 7 Case cavity
• 7 Nprocs 8
• Reading transportProperties
• Reading field p
• Reading field U
• Reading/calculating face flux field phi
• Starting time loop

18
Submit (10/10)

• Time 0.005
• Courant Number mean 0 max 0
• DILUPBiCG Solving for Ux, Initial residual 1,
  Final residual 5.45205e-06, No Iterations 11
• DILUPBiCG Solving for Uy, Initial residual 0,
  Final residual 0, No Iterations 0
• DICPCG Solving for p, Initial residual 1,
  Final residual 9.40953e-07, No Iterations 47
• time step continuity errors sum local
  6.69012e-09, global -1.1589e-10, cumulative
  -1.1589e-10
• DICPCG Solving for p, Initial residual
  0.523592, Final residual 9.91174e-07, No
  Iterations 46
• time step continuity errors sum local
  1.10779e-08, global 1.19468e-10, cumulative
  3.57845e-12
• ExecutionTime 0.06 s ClockTime 0 s

19
Developed Tools(1/2)

• Recursive Up/Download from Data Catalog
• middleware extension (bulk operation)
• recursively descends the directorys tree
• on the local file system to upload files on a
  Data Catalog
• on the Data Catalog to download files on the
  local file systems

!/bin/bashVOcometaAPPopenfoamSEinfn-se-01.c
t.pi2s2.itlfc-rm -r "/grid/VO/APP/1" 2gt
filelistFILELISTsed 's/space/_/g'
filelistREVERSELIST""for item in FILELIST
do REVERSELISTitem" "REVERSELISTdoneecho
REVERSELISTROOT_DIR"1"
Download
20
Developed Tools(2/2)
if "ROOT_DIR" ! "" then mkdir
PWD/ROOT_DIR for FILE in REVERSELIST
do FILEFILE1_ echo "FILE"FILE LFC_D
IRecho "FILE" grep 'Directory_not_empty' i
f "LFC_DIR" ! "" then LFC_DIRecho
LFC_DIR'_Directory_not_empty' echo
"Dir LFC_DIR" LOC_DIRecho
LFC_DIR"/grid/VO/APP/" echo
"LOC_DIRLOC_DIR" DIR_TESTls PWD/LOC_DIR
2gtgt/dev/null if "DIR_TEST" ! ""
then echo "PWD/LFC_DIR is an existing
directory" else mkdir "PWD/LOC_DIR" e
cho "created dir PWD/LOC_DIR" fi else
FILEecho FILE'_File_exists' echo
"File FILE" LOC_FILEecho
FILE"/grid/VO/APP/" lcg-cp --vo VO
lfn"FILE" file"PWD/LOC_FILE" echo
"created file PWD/LOC_FILE" fi donefi
Download
21
Future Developments

• and then?
• pre and post - processing
• environment exportation in mpi.pre.sh
• requires middleware modification
• may be useful for other applications
• mesh decomposition as a job
• required only for a few complex cases
• data recollection from slaves
• already developed the required tools
• integration needed
• full 64 bit implementation (almost complete)
• project as a whole
• GUI integration/development
• (general purpose) data storage

22
Fluent generality

• Fluent is a commercial SW for computational fluid
  dynamics (CFD)
• it is a parallel, multi stage application
• several solvers, C-based libraries
• PORTING FLUENT to the Grid
• a lucky experience
• MPI wrapper supplied
• MPICH compliant
• InfiniBand net supported
• already runs on a MVAPICH cluster
• static installation
• actual submission in mpi.pre.sh
• dummy invocation of the regular MPI wrapper

23
Fluent submission procedure

• canonic procedure would be different
• recompile SW with pgcc
• use the Grid built-in MPI wrapper (MPICH/2 over
  IB)
• non sense if a SW package already runs at
  (almost) first shot
• but .
• the reasons for a unique MPI wrapper are
  nonetheless valid
• standardization heterogeneous (proprietary)
  solutions could be difficult to manage in the
  future (with many more users)
• security user should not be allowed to use
  resources by-passing the usual submission
  mechanism (CE unaware)
• so .
• we are looking for a trade-off solution

24
Fluent license

• Fluent is a licensed SW
• a license server
• application contacts the license server
• the accounting system is under development
• the submission procedure for Fluent (sequential
  or parallel) jobs is available at
• https//grid.ct.infn.it/twiki/bin/view/PI2S2/Submi
  tFLUENT

25
Any Questions ?
Thank you very much for your kind attention!