PVM: Parallel Virtual Machine A Users' Guide and Tutorial for Networked Parallel Computing

1
?????

• PVM Parallel Virtual Machine A Users' Guide and
  Tutorial for Networked Parallel Computing
• Basic Programming Techniques
• ????? ???

2
Contents

• Common parallel programming paradigms
• Workload allocation
• Porting existing application to PVM

3
Introduction

• The basic techniques are in terms of
• Task management
• Initialization phases prior to actual computation
• Granularity choices
• Heterogeneity

4
Common Parallel Programming Paradigms

• Crowd computing
• A collection of closely related processes
• Executing the same code
• Perform computations on different portions of the
  workload
• Involving the periodic exchange of intermediate
  results

5
Common Parallel Programming Paradigms

• This paradigm can be further subdivided into two
  categories
• The master-slave model
• Master spawning, initialization, collection and
  display of results, and timing of functions
• Slave perform the actual computation

6
Common Parallel Programming Paradigms

• The node-only model
• Multiple instances of a single program execute
• One process (typically the one initiated
  manually) takes over the noncomputational
  responsibilities

7
Common Parallel Programming Paradigms

• The second model supported by PVM is termed a
  tree computation
• The third model, which we term hybrid, can be
  thought of a combination of the tree model and
  crowd model

8
Crowd Computation

• Three phases
• Initialization of the process group
• Computation
• Collection results and display of output
• The problem can be partitioned into completely
  independent portions

9
Crowd Computation

• Example
• Matrix multiply using Cannon's algorithm
• Use row-wise multicast of matrix A subblocks in
  conjunction with column-wise shifts of matrix B
  subblocks

10
Figure General crowd computation
11
Tree Computations

• Example
• A parallel sorting algorithm
• One process (the manually started process in PVM)
  possesses (inputs or generates) the list to be
  sorted
• It then spawns a second process and sends it half
  the list
• This continues until a tree of appropriate depth
  is constructed
• Each process then independently sorts its portion
  of the list
• sorted sublists are transmitted upwards along the
  tree edges

12
Figure Tree-computation example
13
Workload Allocation

• Distributed-memory parallel computing
• Data decomposition or partitioning
• Function decomposition
• PVM computing model supports both decompositions

14
Data Decomposition

• Consider the addition of two vectors, A1..N and
  B1..N, to produce the result vector, C1..N
• Allocation of N/P elements of each vector to each
  process
• Compute the corresponding N/P elements of the
  resulting vector

15
Data Decomposition

• Statically
• Each process knows a priori
• It shares the workload
• Dynamically
• A control process(e.g., the master process)
  allocates subunits of the workload to processes
  when they become free

16
Data Decomposition

• Input and output
• Individual processes generate their own data
  internally
• Individual process independently input their data
  subsets from external devices
• Parallel I/O facilities are supported
• A controlling process sends individual data
  subsets to each process
• The most common scenario

17
Function Decomposition

• Achieve by partitioning the overall workload in
  terms of different operations
• The three stages of typical program execution
• Input
• Processing
• Result output

18
Function Decomposition

• Parallelism is obtained by currently executing
  the three programs and by establishing a
  pipeline between them

Figure Function decomposition example
19
Porting Existing Applications to PVM

• Two major categories
• Structure and efficiency
• The parallelization of applications may be
• From existing sequential versions
• From parallel versions

20
Porting Existing Applications to PVM

• General guideline
• Decompose loops
• Beginning with outermost loops and working inward
• Detect dependencies

21
Reference

• A Beginners Guide to PVM Virtual Machine,
  available through http//www.itec.uni-klu.ac.at/h
  arald/PVM/pvm_guide.html