Allen D' Malony, Sameer S' Shende, Robert Bell Kai Li, Li Li, Kevin Huck, Nick Trebon

1
TAU Parallel Performance System

• Allen D. Malony, Sameer S. Shende, Robert
  BellKai Li, Li Li, Kevin Huck, Nick Trebon
• malony,sameer,bertie,likai,lili,khuck,ntrebon_at_c
  s.uoregon.edu
• Department of Computer and Information Science
• Performance Research Laboratory
• University of Oregon

2
Outline

• Parallel performance problem solving
• TAU architecture and toolkit
• Instrumentation
• Measurement
• Analysis
• Example applications
• Current projects
• Conclusion

3
Motivation

• Parallel performance problem solving
• Characterize performance
• Identify performance problems
• Investigate and correct the problems
• Many factors to consider in problem solving
  process
• May depend upon goals of performance problem
  solving
• Certainly depends on target of interest
• Requirements and objectives guide the process
• How does this translate to performance tools?
• Tools should support the problem solving process
• Validation comes from effective use in the process

4
Parallel Performance Problem Solving

• Performance optimization on real codes and
  systems
• General process based on empirical analysis

• Performance technology requirements guides tools

5
Difficulties for Tool Developers

• Diverse performance observability requirements
• Multiple levels of software and hardware
• Different types and detail of performance data
• Alternative performance problem solving methods
• Multiple targets of software and system
  application
• Demands more robust performance technology
• Broad scope of performance observation
• Flexible and configurable mechanisms
• Technology integration and extension
• Cross-platform portability
• Open, layered, and modular framework architecture

6
Complexity Challenges for Performance Tools

• Computing system environment complexity
• Observation integration and optimization
• Access, accuracy, granularity, and scalability
  constraints
• Diverse/specialized observation
  capabilities/technology
• Restricted modes limit performance problem
  solving
• Sophisticated software development environments
• Programming paradigms and performance models
• Performance data mapping to software abstractions
• Uniformity of performance abstraction across
  platforms
• Rich observation capabilities and flexible
  configuration
• Common performance problem solving methods

7
General Problems (Performance Technology)

• How do we create robust and ubiquitous
  performance technology for the analysis and
  tuning of parallel and distributed software and
  systems in the presence of (evolving) complexity
  challenges?
• How do we apply performance technology
  effectively for the variety and diversity of
  performance problems that arise in the context of
  complex parallel and distributed computer systems?

?
8
TAU Performance System

• Tuning and Analysis Utilities (11 year project
  effort)
• Performance system framework for scalable
  parallel and distributed high-performance
  computing
• Targets a general complex system computation
  model
• Entities nodes / contexts / threads
• Multi-level system / software / parallelism
• Measurement and analysis abstraction
• Integrated toolkit for performance
  instrumentation, measurement, analysis, and
  visualization
• Portable performance profiling and tracing
  facility
• Open software approach with technology
  integration
• University of Oregon , Forschungszentrum Jülich,
  LANL

9
TAU Performance Systems Goals

• Multi-level performance instrumentation
• Multi-language automatic source instrumentation
• Flexible and configurable performance measurement
• Widely-ported parallel performance profiling
  system
• Computer system architectures and operating
  systems
• Different programming languages and compilers
• Support for multiple parallel programming
  paradigms
• Multi-threading, message passing, mixed-mode,
  hybrid
• Support for performance mapping
• Support for object-oriented and generic
  programming
• Integration in complex software systems and
  applications

10
Instrumentation

• Events link execution to observation
• What are the events of interest?
• Routines, class methods, modules, blocks,
• New events can be created by the user
• Event have semantics and may be related
• When do events occur?
• Execution flow, interrupts, predicates,
• Instrumentation allows events to be generated
• Event occurrence allows for observation
• Are the events of interest able to be observed?
• Accurately, reliably, within intrusion
  requirements

11
TAU Instrumentation

• Support for standard program events
• Routines
• Classes and templates
• Statement-level blocks
• Support for user-defined events
• Begin/End events (user-defined timers)
• Atomic events
• Selection of event statistics
• Support definition of semantic entities for
  mapping
• Support for event groups
• Instrumentation optimization

12
TAU Instrumentation

• Flexible instrumentation mechanisms at multiple
  levels
• Source code
• manual
• automatic
• C, C, F77/90/95 (Program Database Toolkit
  (PDT))
• OpenMP (directive rewriting (Opari), POMP spec)
• Object code
• pre-instrumented libraries (e.g., MPI using PMPI)
• statically-linked and dynamically-loaded (e.g.,
  Python)
• Executable code
• dynamic instrumentation (pre-execution)
  (DynInstAPI)
• virtual machine instrumentation (e.g., Java using
  JVMPI)

13
TAU Source Instrumentation

• Automatic source instrumentation (TAUinstr)
• Routine entry/exit and class method entry/exit
• Block entry/exit and statement level (to be
  added)
• Uses an instrumentation specification file
• Include/exclude list for events and files
• Uses command line options for group selection
• Instrumentation event selection (TAUselect)
• Automatic generation of instrumentation
  specification file
• Instrumentation language to describe event
  constraints
• Event identity and location
• Event performance properties (e.g., overhead
  analysis)
• Create TAUselect scripts for performance
  experiments

14
Program Database Toolkit (PDT)
Application / Library
C / C parser
Fortran parser F77/90/95
Program documentation
PDBhtml
Application component glue
IL
IL
SILOON
C / C IL analyzer
Fortran IL analyzer
C / F90/95 interoperability
CHASM
Program Database Files
Automatic source instrumentation
TAU_instr
DUCTAPE
15
PDT 3.0 Functionality

• C statement-level information implementation
• for, while loops, declarations, initialization,
  assignment
• PDB records defined for most constructs
• DUCTAPE
• Processes PDB 1.x, 2.x, 3.x uniformly
• PDT applications
• XMLgen
• PDB to XML converter (Sottile)
• Used for CHASM and CCA tools
• PDBstmt
• Statement callgraph display tool

16
PDT 3.0 Functionality (continued)

• Cleanscape Flint parser fully integrated for
  F90/95
• Flint parser is very robust
• Produces PDB records for TAU instrumentation
  (stage 1)
• Linux x86, HP Tru64, IBM AIX
• Tested on SAGE, POP, ESMF, PET benchmarking codes
• Full PDB 2.0 specification (stage 2) Q1 04
• Statement level support (stage 3) Q3 04
• PDT 3.0 release at SC2003

17
TAU Performance Measurement

• TAU supports profiling and tracing measurement
• Robust timing and hardware performance support
• Support for online performance monitoring
• Profile and trace performance data export to file
  system
• Selective exporting of events and data
• Extension of TAU measurement for multiple
  counters
• Creation of user-defined TAU counters
• Access to system-level metrics
• Support for callpath measurement
• Integration with system-level performance data
• Linux MAGNET/MUSE (Wu Feng, LANL)

18
TAU Measurement with Multiple Counters

• Extend event measurement to capture multiple
  metrics
• Begin/end (interval) events
• User-defined (atomic) events
• Multiple performance data sources can be queried
• Associate counter function list to event
• Defined statically or dynamically
• Different counter sources
• Timers and hardware counters
• User-defined counters (application specified)
• System-level counters
• Monotonically increasing required for begin/end
  events
• Extend user-defined counters to system-level
  counter

19
Performance Analysis and Visualization

• Analysis of parallel profile and trace
  measurement
• Parallel profile analysis
• ParaProf
• ParaVis
• Profile generation from segmented trace data
• Performance database framework (PerfDBF)
• Parallel trace analysis
• Translation to VTF 3.0 and EPILOG (in progress)
• Integration with VNG (Technical University of
  Dresden)
• Online parallel analysis and visualization

20
ParaProf Framework Architecture

• Portable, extensible, and scalable tool for
  profile analysis
• Try to offer best of breed capabilities to
  analysts
• Build as profile analysis framework for
  extensibility

21
Profile Manager Window
22
Case Study SAMRAI (LLNL)

• Structured Adaptive Mesh Refinement Application
  Infrastructure (SAMRAI)
• Programming
• C and MPI
• SPMD
• Instrumentation
• PDT for automatic instrumentation of routines
• MPI interposition wrappers
• SAMRAI timers for interesting code segments
• timers classified in groups (apps, mesh, )
• timer groups are managed by TAU groups

23
Full Profile Window (Exclusive Time)
512 processes
24
Node / Context / Thread Profile Window
25
Derived Metrics
26
Full Profile Window (Metric-specific)
512 processes
27
Terrain Visualization (Full profile)
F
Uintah
28
Scatterplot Visualization

• Each pointcoordinatedeterminedby threevalues
• MPI_Reduce
• MPI_Recv
• MPI_Waitsome
• Min/Maxvalue range
• Effective forclusteranalysis

Uintah
29
ParaProf Enhancements

• Readers completely separated from the GUI
• Access to performance profile database
  
  
• Profile translators
• dynaprof, mpiP, vprof, papiprof
• Callgraph display
• prof/gprof style with hyperlinks
• Integration of 3D performance plotting library
• Scalable profile analysis
• Statistical histograms, cluster analysis,
• Generalized programmable analysis engine
• Cross-experiment analysis

30
TAU Performance Database Framework

• profile data only
• XML representation
• project / experiment / trial

31
PerfDBF Browser
32
PerfDBF Cross-Trial Analysis
33
TAU Performance System Status

• Computing platforms (selected)
• IBM SP / pSeries, SGI Origin 2K/3K, Cray T3E /
  SV-1 / X1, HP (Compaq) SC (Tru64), Sun, Hitachi
  SR8000, NEC SX-5/6, Linux clusters (IA-32/64,
  Alpha, PPC, PA-RISC, Power, Opteron), Apple
  (G4/5, OS X), Windows
• Programming languages
• C, C, Fortran 77/90/95, HPF, Java, OpenMP,
  Python
• Thread libraries
• pthreads, SGI sproc, Java,Windows, OpenMP
• Compilers (selected)
• Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun,
  Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq,
  NEC, Intel

34
Selected Applications of TAU

• Center for Simulation of Accidental Fires and
  Explosion
• University of Utah, ASCI ASAP Center, C-SAFE
• Uintah Computational Framework (UCF) (C)
• Center for Simulation of Dynamic Response of
  Materials
• California Institute of Technology, ASCI ASAP
  Center
• Virtual Testshock Facility (VTF) (Python, Fortran
  90)
• Los Alamos National Lab
• Monte Carlo transport (MCNP) (Susan Post)
• Full code automatic instrumentation and profiling
• ASCI Q validation and scaling
• SAICs Adaptive Grid Eulerian (SAGE) (Jack
  Horner)
• Fortran 90 automatic instrumentation and profiling

35
Selected Applications of TAU (continued)

• Lawrence Livermore National Lab
• Overture
• Radiation diffusion (KULL)
• C automatic instrumentation, callpath profiling
• Sandia National Lab
• DOE CCTTSS SciDAC project
• Common component architecture (CCA) integration
• Combustion code (C, Fortran 90, GrACE, MPI)
• Center for Astrophysical Thermonuclear Flashes
• University of Chicago / Argonne, ASCI ASAP Center
• FLASH code (C, Fortran 90, MPI)

36
Current Projects

• Performance analysis of component software
• CCA Forum and CCA projects
• Performance database and data mining
• Model-based performance diagnosis
• Performance regression testing
• Event-specific instrumentation and measurement
• Dynamic performance measurement and adaptation
• Online performance analysis and visualization
• Integration with OMIS and OCM
• Extension to OCM-G

37
Performance-Engineered Component Software

• Intra- and Inter-component performance
  engineering
• Four general parts
• Performance observation
• integrated measurement and analysis
• Performance query and monitoring
• runtime access to performance information
• Performance control
• mechanisms to alter performance observation
• Performance knowledge
• characterization and modeling
• Consistent with component architecture /
  implementation

38
Main Idea Extend Component Design

• Extend the programming and execution environment
  to be performance observable and performance aware

repository service ports
performance observation ports
performance knowledge ports
componentports



PerformanceKnowledge
PerformanceObservation
Component Core

Component Performance Repository
variants
? measurement ? analysis
? empirical ? analytical
39
Performance Analysis and Modeling

• Component performance
• Individual component's performance depends on
  context (problem being solved, input parameters,
  interactions with other components, etc.)
• Global performance
• Compose individual component performance models
  to create a global model
• Using global models, an optimal subset of
  components could be selected to solve a given
  problem
• Apply to CCA application development and execution

40
Performance Measurement

• Proxy
• Notifies MasterMind of all method invocations of
  a given component, along with performance
  dependent inputs
• MasterMind
• compiles and stores all measurement data
• TAU
• makes all performance measurements

Before
C2
C1
After
Proxy for C2
C2
C1
MasterMind
TAU
41
Simulation Case Study

• Simulation of the interaction of a shock wave
  with an interface between two gases
• Performance measured via the proxy infrastructure

42
CCA Simulation Wiring Diagram
43
Case Study Performance
44
Concluding Remarks

• Complex parallel systems and software pose
  challenging performance analysis problems that
  require robust methodologies and tools
• To build more sophisticated performance tools,
  existing proven performance technology must be
  utilized
• Performance tools must be integrated with
  software and systems models and technology
• Need to understand performance in respect to
  computational model and related performance
  properties
• TAU performance system offers robust performance
  technology that can be broadly integrated

45
Acknowledgements

• Department of Energy (DOE)
• MICS office
• DOE 2000 ACTS contract
• Performance Technology for Tera-class Parallel
  Computer Systems Evolution of the TAU
  Performance System
• Performance Analysis of Parallel Component
  Software
• University of Utah, DOE ASCI Level 1 sub-contract
• DOE ASCI Level 3 (LANL, LLNL)
• NSF National Young Investigator (NYI) award
• Research Centre Juelich
• John von Neumann Institute for Computing
• Dr. Bernd Mohr
• Los Alamos National Laboratory