The Vampir Performance Assistant: Concepts and First Results from the INTONE Project

1
The Vampir Performance AssistantConcepts and
First Results from the INTONE Project
IST-1999-20252

• Wolfgang E. Nagel, M. Winkler, A. Knüpfer
• Center for High Performance Computing (ZHR)
• Dresden University of Technology
• D-01062 Dresden, Germany
• Tel. (49) 351 - 463 35450
• e-mail nagel_at_zhr.tu-dresden.de

2
VAMPIR

• VAMPIR development
• has started more than ten years ago at Research
  Center Jülich
• just from the beginning User driven
• commercialized with Pallas GmbH already in 1995
• successful collaboration with Pallas marketing
  and development
• Projects at TU Dresden
• Scalability vendor support
• OpenMP support, scalability and usability Path
  Forward Program (collab. with Pallas, Intel
  (KAI), US Nat. Labs)
• INTONE OpenMP support and Performance Assistant
• UNICORE Plus First support for Grid

3
(No Transcript)
4
Support for Hardware Registers FMC Application
5
FMC Application
6
Density Functional Theory
7
New Feature Performance Assistant

• Cornerstone of the INTONE performance analysis
  tools
• guides the enduser through the process of
  generating a trace database with the right amount
  of information (Experiment Manager)
• scans the trace database for signs of performance
  problems (Problem Identifier)
• presents identified performance bottlenecks to
  the user (Solution Proposer), highlighting the
  location in the event trace using Vampir
  displays, gives full information about the
  bottleneck on request

8
Duties of the Three Components

• The Experiment Manager uses STF access routines
  to read statistics information from an existing
  trace database, and writes configuration files
  for the instrumentation library
• The Problem Identifier uses STF access routines
  to read both statistics and events from the trace
  database, and reads rules that define performance
  bottlenecks from a Problem Definition database
• The Solution Proposer accesses the set of
  identified problems from the Problem Identifier,
  reads details about the performance bottleneck
  from the trace database using STF access
  routines, and controls the Vampir tool using a
  control interface

9
(No Transcript)
10
Performance Anomalies

• Executions of subroutine and/or OpenMP regions
  that show irregular performance metrics, e.g.
  sporadic very long execution times or very high
  cache misses
• Load balance irregularities for subroutine and/or
  OpenMP regions across process/threads
• Message transmissions that exhibit irregular
  behaviour, e.g. transient occurrences of extreme
  latencies this analysis will be done for any
  pair of processes
• Variations of message transmission metrics across
  processes/threads
• Excessive variation of blocking time inside
  global communication routines signalling load
  balancing and/or synchronization problems.

11
Requirements for the Problem Identifier

• The tool must be up to the task of sifting
  through Tens/Hundreds of Megabyte of data
• The tool must be implemented in a conventional
  programming language (C, C) and be portable to
  most Unix platforms
• The tool must be easily programmable with a
  simple pattern language
• License conditions must allow the use in a
  commercial tool.
• As a result, C5.0 from RuleQuest Research
  (http//www.rulequest.com/see5-info.html) was
  found to be a good candidate for supplying the
  pattern match functionality needed by the Problem
  Identifier. A previous version (C4.5) had been
  available in the public domain, and papers like
  JV2000 have demonstrated the applicability of
  C5.0 to the problem at hand.

12
Solution Proposer

• Solution Proposer will read the problem List
  output by the Problem Identifier
• go through each problem in succession
• For each one, it will pull complete information
  out of the trace database (stack, source
  location, ...) and instruct Vampir using the
  control interface to open and position its
  displays in a way that highlights where the
  assumed performance problem is located
• On demand, a window with the explanation and
  solution texts will be displayed, with specifics
  of the problem under analysis being filled in

13
Short Summary

• The Performance Assistant
• interacts with Vampir/OpenMP
• helps the user with instrumentation and tracing
  configuration
• scans the trace for performance problems
• highlights and presents identified problems
• proposes solutions to the user (final version!)
• uses an extensible rule base for problem
  definitions
• uses an extensible rule base to generate
  solutions
• should be able to learn (userdriven)

14
Trace Example Message Volume
15
Identify Slow Messages
16
Identify High Activity Rates
17
Colorcoded Timeline High Activity Rates
18
Activity Questionnaire
19
Duration Timeline
20
Identify High Message Rates
21
Notifikation

• Separate program ./vampir_notify ltfilenamegt
• Looks for running Vampir
• Starts new Vampir if not yet executing
• Via main menu entry of Vampir "Performance
  Problems"
• Open file interface
• Navigation inside Vampir
• Different tools outside which identify
  performance properties (first implementation
  C4.5/C5, learning system)

22
Interface to C4.5 Analysis Tool
23
Problem Identifier Interface (Example)

• ProblemId 0
• ProblemClass 1
• ProblemName "messages.rate bad"
• ProblemText "Message Transfer Rate in
  MB/s according to Message length too slow"
• TraceFilePath "/home/wen/perf_ass/test/trc/
  qt"
• TraceFileName "mpi.all.vpt"
• StartTimeStamp 0.306977 s
• StopTimeStamp 0.598334 s
• WinParam MainTime Message 0.331257s
  0.574055s 1 0
• ProblemId 1
• ProblemClass 1
• ProblemName "Message Transfer Rate in
  MB/s according to Message length too slow"
• ProblemText "description ..."
• TraceFilePath "/home/wen/perf_ass/test/trc/
  qt"
• TraceFileName "mpi.all.vpt"
• StartTimeStamp 1.275479 s
• StopTimeStamp 1.452643 s
• WinParam MainTime Message 1.290243s
  1.437879s 1 0

24
Problem Identifier Interface (Formal) I

• ProblemId
• Number gt 0
• Unique number within the file
• First line after paragraph delimiter (before
  other descriptive lines)
• Number will not be shown to the user, and,
  therefore, semantical context can be included
• Problems will not necessarily be shown in
  numerical order of ProblemIds
• ProblemClass
• Number gt 0
• Purpose To sort and mark certain kinds of
  problems (not yet implemented)
• ProblemName
• Text within quotations
• This text is shown as only information in folded
  level. All other available information can only
  be reached in unfolded level

25
Problem Identifier Interface II

• ProblemText
• Text within quotations
• To see this text a new display will be opened.
  This can be reached via context menu (Show
  Problem Description).
• Thread (optional)
• List of numbers gt 0
• Separated by space
• Numbers not necessarily in a certain order
• If a thread number does not exist, no error
  message will be printed (i.e. nothing will be
  done) only existing thread numbers are taken
  into consideration when loading a tracefile.
• StartTimeStamp (optional)
• Timestamp (american style) followed by space and
  then unit
• If no value is given, global start of tracefile
  will be used

26
Problem Identifier Interface III

• StopTimeStamp (optional)
• Timestamp (american style) followed by space and
  then unit
• If no value is given, global end of tracefile
  will be used
• TraceFilePath
• Text within quotations
• Absolute Path
• TraceFileName
• Text within quotations
• Name of the tracefile which has to be opened when
  showing this problem. Usually but not necessarily
  all tracefiles in one problem description file
  are the same
• Path and Tracefilename describe a tracefile
  explicitly
• Valid timestamp units are d, h, min, s, ms, us,
  ns, ps, fs, as, zs, ys or empty (means
  clockticks)

27
Problem Identifier Interface IV

• TraceFileFrame (optional)
• List of numbers gt 0
• Separated by space
• Numbers not necessarily in a certain order
• If only one or several out of all frames shell be
  shown (if there are frames at all)
• WinParam (optional)
• Text within quotations
• Not yet completely specified
• Describe which displays are influenced by the
  problem, similar to ShowWin.
• Consists of the display name and an optional
  parameter list, which is important to start
  showing the display with e.g. correct zoom range.
  
• ShowWin (optional)
• Text within quotations
• If this window is one of the WinParam windows, it
  will be shown on the top of all open windows.

28
New Display Performance Problem

• Text and /- signs to show the fold status
• Short problem description when unfolded
• Only problem name when folded
• Highlight one problem

29
Performance Problem Dialog
30
Context Sensitive Windows Message Rate
31
Short Messages
32
Identify High Message Rates
33
High Activity Rates
34
Example with Frames, Selective Processors
35
Final Remarks

• There is a strong need for automatic performance
  analysis
• There will be no single solution to identify all
  kinds of performance properties
• We have designed an open interface which allows
  to present the results of such an automatic
  process easily
• There is a strong connection to real world
  performance analysis
• It is our first implementation step into this
  world, please be patient!
• Comments and recommendations are very welcome

36
The End!