Constructing a Performance Database for LargeScale Quantum Chemistry Packages

1
Constructing a Performance Database for
Large-Scale Quantum Chemistry Packages

• Meng-Shiou Wu1, Hirotoshi Mori2, Jonathan Bentz3,
  
• Theresa Windus2, Heather Netzloff2, Masha
  Sosonkina1, Mark S. Gordon1,2

1Scalable Computing Laboratory, Ames Laboratory,
US DOE 2Department of Chemistry, Iowa State
University 3Cray Inc.
2
Introduction

• Why Quantum Chemistry (QC)??
• Applications to
• Surface science
• Environmental science
• Biological science
• Nanotechnology
• Much more
• Provides a way to treat problems with greater
  accuracy
• With advent of more powerful computational tools,
  QC can be applied to more and more problems

3
IntroductionThree Major QC Software Packages

• All provide various computations to calculate
  physical properties of molecules including
  energies, geometries, reaction pathways, etc.
• Some calculation types are the same, but some are
  unique to the specific package

4
Introduction

• We have used the Common Component Architecture
  (CCA) framework to enable interoperability
  between GAMESS, NWChem and MPQC
• Obtain a palette of components to utilize
  functionalities between each package

5
Insight into CCAExample--Core Interfaces for
Integral Computations
IntegralEvaluatorFactoryInterface
integral evaluator factory
CCA interface
Integral
Integral
Integral
Integral
Evaluator1
Evaluator2
Evaluator3
Evaluator4
component
Interface
Interface
Interface
Interface
integral
integral
integral
integral
class
evaluator1
evaluator2
evaluator3
evaluator4
Chemistry Package
NWChem
GAMESS
MPQC

• IntegralEvaluatorNInterface for N-center integral
  , N 1,2,3, 4.
• IntegralEvaluatorFactoryInterface provides
  references to the integral evaluators from a
  chemistry program

6
Background

• Questions to ponder
• Easy selection of components with the best
  efficiency
• Find compromises between efficiency and accuracy
• Rely on Computational Quality of Service (CQoS)
  to address these issues
• Automatic selection and configuration of
  components to suit a particular computational
  purpose
• Special requirements for scientific components
• High efficiency and parallel scalability
• Functional qualities, such as the level of
  accuracy achieved for a particular algorithm
• Core question in our CQoS research
• Need an approach for the large-scale applications

7
Motivation for CQoS

• Three basic steps for performance analysis
• code instrumentation
• performance evaluation
• data analysis
• This is straightforward IF applied to smaller
  packages
• Must be augmented to deal with large scale
  applications
• Need mechanisms to handle performance data

8
Issues to be Addressed

• Large number of quantum chemistry computations
  available in each QC packages
• Individual chemists may not know all of the
  available computations and the algorithmic
  details of these computations
• Construction of the database must allow/encourage
  participation of many chemists to try to obtain a
  big picture view
• The construction process should be as automated
  as possible
• No need for chemists to select the right
  performance tools.
• Chemists should not need to manually handle the
  large amounts of performance data
• Management of performance data and the database
  should be transparent to chemists.

9
Stages in the Development of CQoS Tuning Rules
Stage 1
Stage 2
Stage 3
Stage 4
Develop/Collect Capable Implementations
Performance Evaluation
Data Analysis
Tuning Rules Development
Interoperating mechanisms
Source code instrumentation
Performance modeling
Develop Analysis Procedures
Data collecting
Heuristic tuning rules
Exploring important parameters
Minimizing overhead
Performance database
Mechanisms to incorporate tuning rules
Analyze the relationships between parameters
10
Stage 1 Collect/Develop/Incorporate
Implementations

• Goal
• Develop, collect and integrate capable
  methods/implementations
• For packages integrated with CCA, the goal is to
  develop interoperating mechanisms.
• Overhead introduced by the interoperating
  mechanisms must be minimized

Interoperating Mechanisms
Minimizing Overhead
11
Stage 2 Performance Evaluation

• Goal
• Generate useful performance data and explore
  methods to easily manipulate large amounts of
  data
• This is our current area of focus

Source code Instrumentation
Data Collection
Performance Database
12
Stage 3 Performance Analysis

• Goal
• Use the collected performance data and the
  constructed database to conduct analysis.
• Very complex process for quantum chemistry
  computations
• Example to follow

Develop Analysis Procedures
Exploring important parameters
Analyze the relationships between parameters
13
Stage 4 Tuning Rules Development

• Goal
• Develop mechanisms to select one method among a
  pool of methods.
• Depends on the results of performance analysis.
• Two common approaches
• develop performance models
• use heuristic tuning rules.
• Tuning rules must be integrated into the original
  interoperating mechanisms to be useful.

Performance Modeling
Heuristic Tuning rules
Mechanisms to Incorporate Tuning rules
14
Performance Evaluation Source Code
Instrumentation

• Insert performance evaluation functions into
  application source codes.
• Straightforward with automatic instrumentation
  capability provided by existing performance
  tools.
• Limitations exist when you need only part of the
  overall performance data
• Expected performance data may not be generated

15
Source Code Instrumentation(CQoS with GAMESS)

• Example
• In many cases, manual instrumentation is
  unavoidable in GAMESS performance evaluation.

A whole computation
Computation 1
Computation 2
Computation 3
communications
16
GAMESS Performance Tools Integrator (GPERFI)

• Built on top of existing performance tools such
  as TAU (Tuning and Analysis Utilities) or PAPI
• Allows more flexible instrumentation mechanisms
  specific to GAMESS
• Provides mechanisms to extract I/O,
  communication, or partial profiling data from the
  overall wall clock time

17
Performance Evaluation Data Collection
Discard, remove its related data from database.
Post processing TAU performance data.
Genetate application metadata from the input
file/out log files.
Uploading data to the database through PerfDMF
No
Decide if the input file should be tested on
the other platforms
Run the testing on the experimental cluster
Conduct performance analysis with PerfExplorer
Yes
Format verification
Output performance analysis results.
Input files uploading / results viewing interface
Test the input file on different platforms.
Provide input files for experiments
View results and provide comments.
18
Performance Evaluation Performance Database

• Simply conducting many performance evaluations
  and putting the performance data into the
  database is not going to work!!!
• Need to properly collect metadata that are
  related to performance data.
• Need to build relationships between metadata.
• Need participation of chemists!!!

19
Performance Evaluation Example of Construction
Sequence

• Chemist defines molecule similarity between a
  set of molecules.
• Conduct performance evaluation for the base
  molecule.
• Use the performance data collected from the base
  molecule to conduct performance analysis and
  prediction of the other molecules with similar
  structure.

Benzen (bz)
Hydroxyl-bz
Methyl-bz
Amino
20
Performance Evaluation Performance Database

• Use PerfDMF (Performance Data Management
  Framework), included in the TAU toolkit, for data
  manipulation
• Use MySQL as the database engine underneath
  PerfDMF
• Pros Data manipulation is simplified.
• Cons Data manipulation is restricted by the
  schema defined by PerfDMF

Performance Database
PerfDMF
MySQL
21
Metadata Mapping
Trial
Application
Experiment
Experimental runs with different system settings
GAMESS
Computations
Energy
Experiment set 1
Metadata (Platform 1, CPU, cache.., etc.)
Metadata (conv-SCF, .., etc)
Experiment set 2
Application characteristics
Metadata (Platform 2, CPU, cache.., etc.)
Experiment set 3
Metadata (Platform 3, CPU, cache.., etc.)
System characteristics

Energy
Experiment set 1
Metadata (directSCF, .., etc)
Metadata (Platform 1, CPU, cache.., etc.)
Experiment set 2
Metadata (Platform 2, CPU, cache.., etc.)
Experiment set 3
Metadata (Platform 3, CPU, cache.., etc.)


22
Hierarchy of Quantum Chemical Calculation in
GAMESS
Molecule
1. Molecule
data

• of Atoms electrons
• Structure (Linearity/Planarity of the Molecule)

Energy
Gradient
Hessian
2. Property
3. Level of Theory
HF post-HF
DFT
contrl

• BLYP
• PBE
• B3LYP etc.

HF
CI
CC
MPn
4. Basis Sets

• of basis functions
• Angular momentum

basis
5. Integral Algorithm

• Conventional (stored on disk)
• Direct (calculated on the fly)

scf
23
Detail of the Performance Evaluation Stage
Performance Analysis
Tuning Rules Development
Performance Evaluation
Source code Instrumentation (GPERFI for
GAMESS, TAU/PDT for NWChem)
Exploring important parameters
Performance Modeling
Data Collecting (Python programs)
Heuristic Tuning Rules
Analyze the relationship between parameters
Application Metadata
Performance Data
System Metadata
Performance Database
PerfDMF
PerfExplorer
MySQL
24
Complexity in Performance Analysis A Case
Study(Conventional Integral Calculation)
Runtime (sec)
Number of nodes/Number of Proc per node
25
What Needs to be Analyzed?

• Why does the cost of parallel write (PWRT) jump
  dramatically when using 16 processors per node?
• When does global sum (GSUM) become the
  performance bottleneck?
• Investigation of many subroutines to study the
  relationships among
• CPU speed
• I/O bandwidth/ latency
• data size
• communication bandwidth/latency
• number of processors used per node

26
Conclusions

• Initial work to construct a performance database
  that can ease the burden in manipulating
  performance data for quantum chemists
• Data manipulation is mostly transparent to
  chemists. They need only to provide their
  knowledge of chemistry.
• The infrastructure can be used on the other
  applications
• We are working on incorporating NWChem

27
Acknowledgements

• Funding
• US DOE SciDAC initiative
• Resources
• Lawrence Berkeley National Lab-NERSC

28
Questions/Comments??