Parallel Programming models I

1
Parallel Programming models I
2
Parallel Machines an abstract introduction

• Our main focus will be on three kinds of machines
• Bus-based shared memory machines
• Scalable shared memory machines
• Cache coherent
• Hardware support for remote memory access
• Distributed memory machines

3
Bus based machines
Mem0
Mem1
Memk
PE0
PE1
PE N-1
4
Bus based machines

• Any processor can access any memory location
• Read and write
• Bus bandwidth is a limiting factor
• Also, how do you deal with 2 processors changing
  the same data?
• Locks (more on this later)

5
Scalable shared memory m/cs
PE0
PE0
PE0
6
Scalable shared memory m/cs
PE0
PE0
PE0
Interconnection Network with support for remote
memory access
Mem0
Mem0
Mem0
7
Distributed memory m/cs
PE0
PE0
PE0
Interconnection Network
Mem0
Mem0
Mem0
8
Writing parallel programs

• Programming model
• How should a programmer view the parallel
  machine?
• Sequential programming von Neumann model
• Parallel programming models
• Shared memory (Shared address space) model
• Message passing model
• Shared Objects model

9
Shared Address Space Model

• All memory is accessible to all processes
• Processes are mapped to processors, typically by
  a symmetric OS
• Coordination among processes
• by sharing variables
• Avoid stepping on toes
• using locks and barriers

10
Matrix multiplication
for (i0 iltM i) for (j0 jltN j) for
(k0 kltN k) CIj AikBkj
In a shared memory style, this program is trivial
to parallelize Just have each processor deal
with a different range of I (or J?) (or
Both?)