Scaling of First Principles Electronic Structure Methods on Future Architectures

1
Scaling of First Principles Electronic Structure
Methods on Future Architectures

• W.A. Shelton
• Oak Ridge National Laboratory

2
Collaborators

• Locally Self-consistent Multiple Scattering
  Method (Real Space)
• Oak Ridge National Laboratory
• N.Y. Moghadam, D.M.C. Nicholson, G.M. Stocks,
  X.-G. Zhang, andB. Ujfalussy
• Pittsburgh Supercomputer Center
• Y. Wang
• National Energy Research Supercomputer Center
• A. Canning
• Screened Methods (Tight-binding like methods)
• Oak Ridge National Laboratory
• A. Smirnov
• University of Illinois (Urbana-Champaign)
• D.D. Johnson

3
Acknowledgement of Sponsors

• Department of Energy/Office of Science
• Office of Advanced Scientific Computing Research
• Mathematics, Information and Computer Science
• Applied Mathematical Sciences Program
• Oak Ridge National Laboratory
• Laboratory Directors Research and Development
  Program
• Computing Resources at the Center of
  Computational Sciences located at Oak Ridge
  National Laboratory
• Pittsburgh Supercomputing Center
• National Energy Research Supercomputing Center
  located at Lawrence Berkeley National Laboratory

4
Motivation
The introduction of new architectures

• Rethink the mathematical model
• Design new algorithms that renders a numerical
  solution
• Open new possibilities
• Improved scaling
• For solving problems that previously were
  untenable
• Software technologies being used by researchers
  with access to less advanced hardware technologies

5
Motivation

• Nanoscale Science and Engineering Technology
  Initiative
• SOFT MATERIALS
• Synthetic Polymers and Bio-Inspired Materials
• Systems Dominated by Organic-Inorganic
  Interconnections
• Interfacing Nanostructures to Biological Systems
• HYBRID SOFT-HARD MATERIALS
• Carbon-Based Nanostructures
• Characterization of Active Sites in Catalytic
  Materials
• Nanoporous Membranes and Nanomaterials for
  Ultra-Selective Catalysis
• COMPLEX HARD MATERIALS
• Magnetism in Nanostructured Materials
• Nanoscale Manipulation of Collective Behavior
• Nanoscale Interface Science (Nanoparticles and
  Nanograins)
• Electromagnetic Fields in Confined Structures
• THEORY / MODELING / SIMULATION
• Virtual Synthesis and Nanomaterials Design
• Theoretical Nano-Interface Science

6
Multiscale Simulations
Connect Microscopic-level Processes to
Macroscopic Response of Material
7
Scalable and Accurate First Principles Method
Atomistic Methods
8
Quantum Simulation GoalsAccuracy and Predictive
Capabilities
9
Advances In Hardware Alone Are Not Sufficient
10
Linear Scaling Algorithms Will Enable Solutions
to New Problems
The combination of new advanced computing
platforms and new scaling algorithms will open
new areas in quantum-level materials simulations
11
Algorithm Design for future generation
architectures

• More accurate
• Spectral or pseudo-spectral accuracy
• Wider range of applicability
• Sparse representation
• Memory requirements grow linearly
• Each processor can treat thousands of atoms
• Make use of large number of processors
• Message-Passing
• Each atom/node local message-passing is
  independent of the size of the system
• Time consuming step of model
• Sparse linear solver
• Direct or preconditioned iterative approach

12
Density Functional Theory (DFT)

• DFT in principle is an exact method for treating
  the many body quantum mechanical effects of
  electron exchange and correlation
• At the heart of this formulation is the ascertain
  that the ground-state total energy of an electron
  system in the field of the atomic nuclei is a
  unique functional of the electronic charge
  density
• The total energy functional attains it minimum
  value when evaluated with the true ground-state
  electronic charge density. The quantum
  mechanical effects of electron exchange and
  correlation are contained in the non-local
  exchange-correlation potential Vex-corr(r,r).
• Hence, the electronic interactions are explicitly
  accounted for by the fundamental quantity, the
  electronic charge density
• Unfortunately, there are no analytical forms for
  calculating Vex-corr(r,r)

13
LSDA Multiple Scattering Theory (MST)

• Multiple Scattering Theory (MST)
• J. Korringa, Physica 13, 392, (1947)
• W. Kohn, N. Rostoker, PR, 94, 1111,(1954)
• MST Green function methods
• B. Gyorffy, and M. J. Stott, Band Structure
  Spectroscopy of Metals and Alloys, Ed. D.J.
  Fabian and L. M. Watson (Academic 1972)
• S.J. Faulkner and G.M. Stocks, PR B 21, 3222,
  (1980)

14
Complex Energy Plane
ef is the highest occupied electronic state in
energy
Semi-conductors and insulators could work well
since they have no states at ef
The scattering properties at complex energy can
be used to develop highly efficient real-space
and k-space methods
15
Multiple Scattering Theory

• Multiple scattering theory
• Green function
• Scattering path matrix

• Generalization of t-matrix. Converts
• incoming wave at siteinto outgoing wave
• at site in the presence of all the other sites

16
Real Space Algorithm Design

• Linear scaling
• Each node performs a fixed size local calculation
• Thus each node performs the same number of flops
• Message-Passing
• Each atom/node local message-passing is
  independent of the size of the system
• Time consuming step of model
• Reduce to Linear Algebra step
• BLAS level 3

17
Real Space Parallel Implementation

• Greens function

• Scattering path matrix real space

tM-1
Mt-1(e)-G(Rij,e)
t scattering from single site G structure
constant matrix

• Once M is fixed increasing N does not
• affect the local calculation of M-1
• The LSMS naturally scales linearly with
• increasing N

18
Matrix Inversion
A
B
Partition the m(lmax 1)2 matrix, tMxM into
MM1 M2 into four blocks two of size M1 and two
of size M2
C
D
A-1A-B D-1C Note that the LxL diagonal block of
A-1 is the same LxL block that is desired.
Take A and continue to partition until the
desired matrix size (lmax 1)2 of the central
site is reached
19
J(N) Scaling of Real Space Method
1998 Gordon Bell Prize 1.02 TFLOPS on a 1500
node Cray T3E
20
J(N) Scaling of Real Space Method
21
Real Space Accuracy
22
Tight-Binding MST Representation

• Tight Binding Multiple Scattering Theory
• Embed a constant repulsive potential
• Shifts the energy zero allowing for calculations
  at negative energy
• Rapidly decaying interactions
• Free electron singularities are not a problem
• Sparse representation

23
Screened Structure Constants

• Linear solve using m atom cluster that is less
  than the n atom system
• Easy to perform Fourier transform
• K-space method

• Screened Structure Constants Gs on the left
  unscreened on the right
• Screened structure constants rapidly go to zero,
  whereas the free space structure constants have
  hardly changed

24
Screened MST Methods

• Formulation produces a sparse matrix
  representation
• 2-D case has tridiagonal structure with a few
  distant elements due to periodicity
• 3-D case has scattered elements
• Mainly due to mapping 3-D structure to a matrix
  (2-D)
• A few elements due to periodic boundary
  conditions
• Require block diagonals of the inverse of t(e)
  matrix
• Block diagonals represent the site t(e) matrix
  and are needed to calculate the Greens function
  for each atomic site
• Sparse direct and preconditioned iterative
  methods are used to calculate tii(e)
• SuperLU
• Transpose free Quasi-Minimal Residual Method
  (TFQMR)

25
Screened KKR Accuracy
26
Timing and Scaling of Scr-K KKR-CPA
27
Conclusion

• Initial benchmarking of the Screened KKR method
• SuperLU N1.8 for finding the inverse of the upper
  left block of t
• TFQMR with block Jacobi preconditioner N1.06 for
  finding the inverse of the upper left block of t
• Extremely high sparsity (97-99 zeros increases
  with increasing system size)
• Large number of atoms on a single processor
• Real-space/Scr-KKR hybrid may provide the most
  efficient parallel approach for new generation
  architectres
• Single code contains
• LSMS, KKR-CPA, Scr-LSMS and Scr-KKR-CPA