Using the PETSc Parallel Software library in Developing MPP Software for Calculating Exact Cumulative Reaction Probabilities for Large Systems (M. Minkoff and A. Wagner) ANL (MCS/CHM)

1
Using the PETSc Parallel Software library in
Developing MPP Software for Calculating Exact
Cumulative Reaction Probabilities for Large
Systems (M. Minkoff and A. Wagner)ANL (MCS/CHM)

• Introduction
• Problem Description
• MPP Software Tools
• Computations
• Future Direction

2
Parallelization of Cumulative Reaction
Probabilities (CRP) with PETScM. Minkoff
(ANL/MCS) and A. Wagner(ANL/CHM)

• Calculation of gas phase rate constants
• Develop a highly scalable and efficient parallel
  algorithm for calculating the Cumulative Reaction
  Probability, P.
• Use parallel subroutine libraries for higher
  generation parallel machines to develop parallel
  CRP simulation software.
• Implementing Miller and Manthe (1994) method for
  time-independent solution of P in parallel.
• P is determined for an eigenvalue problem with an
  operator involving two Greens functions. The
  eigenvalues are obtained using a Lanczos method.
  The Greens functions are evaluated via a GMRES
  iteration with diagonal preconditioner.

3
Benefits of using PETSc

• Sparsity PETSc allows arbitrarily sparse data
  structures
• GMRES PETSc has GMRES as an option for linear
  solves
• Present tests involve problems in dimensions 3 to
  6. Testing is underway using an SGI Power
  Challenge (ANL), and SGI/CRAY T3E (NERSC).
  (Portability is provided via MPI and PETSc, so
  higher dimensional systems are planned for future
  work).

4
Chemical Dynamics Theory3 angles, 3 stretches6
degrees of freedom
5
Chemical Dynamics Theory

• How fast do chemicals react?
• Rate constant k determines it
• dX / dt -k1XY k2ZY
• many rates at work in devices
• rates express interactions in the chemistry
• individual rates are measurable and calculable
• rates depend on T, P.

6
Chemical Dynamics TheoryN(E) TrP(E)

• Rates are related to
• Cumulative Reaction Probability (CRP), N(E)
• N(E) 4 Tr

7
Chemical Dynamics Theory

• Probability Operator and Its Inverse
• Using probability method calculates a few large
  eigenvalues via iterative methods. The iterative
  evaluation involves the action of two Greens
  function.
• Using inverse probability method involves a
  direct calculation each iteration to obtain a few
  smallest eigenvalues. At each iteration the
  action of a vector by the Greens function is
  required. This leads to solving linear systems
  involving the Hamiltonian.

8
Chemical Dynamics Theory

• The Greens functions have the form
• G(E) (E ie - H)-1
• and so we need to solve two linear systems (at
  each iteration) of the form
• (E ie - H) y x
• where x is known.
• This system is solved via GMRES with
  preconditioning methods (initially diagonal
  scaling).

9
PETSc Portable, Extensible Toolkit for
Scientific Computing
Satish Balay, William Gropp, Lois McInnes, and
Barry Smith MCS Division, Argonne National
Laboratory

• Focus data structures and routines for the
  scalable solution of PDE-based applications
• Object-oriented design using mathematical
  abstractions
• Freely available and supported research code
• Available via http//www.mcs.anl.gov/petsc
• Usable in C, C, and Fortran77/90 (with minor
  limitations in Fortran 77/90 due to their syntax)
• Users manual, hyperlinked manual pages for all
  routines
• Many tutorial-style examples
• Support via email petsc-maint_at_mcs.anl.gov

10
Application Codes Using PETSc
Applications can interface to whatever
abstraction level is most appropriate.
11
PETSc Numerical Components
12
Sample Scalable Performance
600 MHz T3E, 2.8M vertices

• 3D incompressible Euler
• Tetrahedral grid
• Up to 11 million unknowns
• Based on a legacy NASA code, FUN3d, developed
  by W. K. Anderson
• Fully implicit steady-state

• Newton-Krylov-Schwarz algorithm with
  pseudo-transient continuation
• Results courtesy of Dinesh Kaushik and David
  Keyes, Old Dominion University

13
Computations via MPI and PETSc
14
5D/T3E Results for Varying Eigenvalue and G-S
Method
15
Parallel Speedup5D/6D ANL/SGI and NERSC/T3E
16
Storage Required for Higher Dimensions
17
Results and Future Work

• Achieved parallelization with less effort
• Suboptimal but perhaps 2X Optimal Performance
• Testing for 6D and 7D underway.
• MPP CPU and Memory can provide necessary
  resources
• Many degrees of freedom can be approximated, so
  maximum dimension needed is 10.
• Develop block structured preconditioning methods