Virtual Augmentation and Simulation for Surgery and Therapy VASST SUPERCOMPUTER www'imaging'robarts'

1
Virtual Augmentation and Simulation for Surgery
and TherapyVASST SUPERCOMPUTERwww.imaging.roba
rts.ca/vasst

• Mark Wachowiak, Ph.D.

2
Outline

• vasst research program and vasst computer system.
• Parallel computing.
• Very brief introduction to shared memory
  programming with OpenMP.
• Summary and pointers to additional information.

3
vasst objective

• Enhance RD of minimally invasive treatments.
• Encourage experimentation in a realistic context
  without humans or animals.
• Planning, experimentation, hypothesis testing,
  monitoring, evaluating, and training.
• Development of new surgical tools.
• Links human-machine interaction with image
  acquistion, procedure modeling, ultimately
  translated into clinical practice.

4
vasst supercomputer

• Enhanced computation facilities.
• Visualization.
• File system and storage.
• Cornerstone of other parallel systems in the lab
  (virtual clusters, dogfish).

More information www.imaging.robarts.ca/vasst
5
vasst configuration

• SGI Altix 3700.
• 20 Itanium II 1.3 GHz CPUs.
• 4 GB shared memory.
• 921 MB swap space.
• Storage
• 2 TB for home directories.
• 7 TB for data storage.
• Linux operating system.

6
Applications to date

• Organ modeling and simulation.
• Linear and nonlinear registration.
• Many other parallel applications are possible.

7
Itanium II CPU

• 64-bit addressing
• ?1019 bytes (theoretical) of addressable RAM.
• Very little limitation on amount of data residing
  in memory.
• Explicitly Parallel Instruction Computing (EPIC)
• Instruction bundles 3 machine code
  instructions.
• Itanium schedules 2 bundles/clock cycle.
• ?6 instructions/clock cycle.

8
Intel compiler

• icc (C/C).
• ifort (FORTRAN).
• Compilers explicitly take advantage of features
  in the Itanium II hardware.
• To get maximum performance, some effort is
  required (the manual is 438 pp.).

9
What vasst is NOT useful for

• Performing extremely fast serial computations
  (use your desktop).
• Running many serial jobs simultaneously (run on a
  cluster or use SHARCNet).
• FOR NOW programs that require heavy
  visualization capabilities.
• Non-optimized software.
• Utilities (Matlab, email, etc.).

10
What vasst IS useful for

• Inherently or easily-parallelizable programs.
• Shared memory applications.
• Running different concurrent programs that
  require and update the same data.
• Large (sized) problems.
• Applications requiring highly-optimized numerical
  libraries.
• Parallel visualization.
• GPU programming.

11
II. Parallel computing
12
Parallel programming

• Distributed memory
• Beowulf clusters, SHARCNet, sometimes vasst.
• MPI (Message passing interface).
• PVM (Parallel virtual machine).
• Shared memory
• vasst.
• OpenMP.
• POSIX threads.

13
Distributed memory - Pros

• Each processor has its own memory.
• CPUs communicate by sending and receiving
  messages.
• A large number of CPUs can be used (1000s).
• Off-the-shelf, homemade clusters of inexpensive
  machines.

14
Distributed memory - Cons

• High learning curve, difficult to program.
• Often requires new algorithms or modifications in
  existing code.
• Frequently, a high communications overhead.
• Debugging is difficult.
• Hardware failures.

15
Shared memory/OpenMP Pros

• Easy to learn.
• Loop parallelism.
• Sectional parallelism.
• Usually does not require major modifications to
  existing programs.
• Used in applications where communication times
  are prohibitive.

16
Shared memory/OpenMP Cons

• Very expensive.
• Memory can be an issue for very large datasets.
• Limited number of processors (order of 100s).

17
Distributed and/or shared memory

• Application dependent.
• Depends on available hardware.
• Hybrid distributed/shared (e.g. MPI/OpenMP)
  applications are possible.

18
III. Shared memory programming with OpenMP
19
OpenMP

• Open specification for multi-processing.
• Not a programming language.
• Not a third-party open source library or add-on.

20
OpenMP (cont.)

• An application programming interface (API) for
  explicitly implementing multi-threaded, shared
  memory parallelism.
• Compiler provides OpenMP support through compiler
  directives.
• OpenMP support available on many UNIX-flavored
  and Windows compilers, but NOT gcc.

21
Compiling a parallel program on vasst with OpenMP
icc o outputname file1.C file2.C filen.C
-fno-alias fast -openmp
22
Compiling a parallel program on vasst with OpenMP
icc o outputname file1.C file2.C filen.C
-fno-alias fast -openmp
Compilers are generally conservative its first
responsibility is to get the right results. If
there is no aliasing of variables in your
program, use this option so that the compiler can
perform optimizations without worrying about
aliasing issues.
23
Compiling a parallel program on vasst with OpenMP
icc o outputname file1.C file2.C filen.C
-fno-alias fast -openmp
The fast flag turns on these optimization
options
-O3 aggressive optimization for speed
-ipo inter-procedural optimization
-static prevents shared library linking
24
Compiling a parallel program on vasst with OpenMP
icc o outputname file1.C file2.C filen.C
-fno-alias fast -openmp
Enables generation of multi-threaded code based
on OpenMP compiler directives.
25
Example Parallel for loop

• Parallel computation of p using Wallis formula

26
IV. Summary
27
Getting the most from vasst

• Write parallel code in OpenMP or POSIX threads.
• Use the Intel icc or ifort (not gcc).
• Use the profiling and optimization tools, or seek
  assistance.
• Try to find the best set of compiler options for
  your specific code.

28
Current issues

• Visualization .
• Queuing and scheduling system.
• Limited number of CPUs.
• Compiler problems.
• Discrepancies in code when using OpenMP.
• Tuned for performance?

29
Parallel programming resources

• www.openmp.org (OpenMP)
• http//www.llnl.gov/computing/tutorials/workshops/
  workshop/pthreads/MAIN.html (POSIX threads)
• http//www.llnl.gov/computing/tutorials/workshops/
  workshop/openMP/MAIN.html
• http//www.intel.com/software/products/compilers/
  clin/docs/manuals.htm
• Parallel Programming in OpenMP, R. Chandra et
  al., Morgan Kaufman, 2001.

30
Acknowledgements

• Chris Wedlake
• Jimmy Scott (SGI)
• Baolai Ge (SHARCNet)
• Dr. Hualiang Zhong