Title: CS 267: Applications of Parallel Computers Lecture 4: Introduction to Parallel Computers and Paralle
1CS 267 Applications of Parallel
ComputersLecture 4Introduction to Parallel
Computers and Parallel Programming Methodologies
- Horst D. Simon
- http//www.cs.berkeley.edu/strive/cs267
2Outline
- Overview of parallel machines and programming
models - Shared memory
- Shared address space
- Message passing
- Data parallel
- Clusters of SMPs
- Trends in real machines
3A generic parallel architecture
P
P
P
P
M
M
M
M
Interconnection Network
Memory
- Where is the memory physically located?
4Parallel Programming Models
- Control
- How is parallelism created?
- What orderings exist between operations?
- How do different threads of control synchronize?
- Data
- What data is private vs. shared?
- How is logically shared data accessed or
communicated? - Operations
- What are the atomic operations?
- Cost
- How do we account for the cost of each of the
above?
5Simple Example
- Consider a sum of an array function
- Parallel Decomposition
- Each evaluation and each partial sum is a task.
- Assign n/p numbers to each of p procs
- Each computes independent private results and
partial sum. - One (or all) collects the p partial sums and
computes the global sum. - Two Classes of Data
- Logically Shared
- The original n numbers, the global sum.
- Logically Private
- The individual function evaluations.
- What about the individual partial sums?
6Programming Model 1 Shared Memory
- Program is a collection of threads of control.
- Many languages allow threads to be created
dynamically, I.e., mid-execution. - Each thread has a set of private variables, e.g.
local variables on the stack. - Collectively with a set of shared variables,
e.g., static variables, shared common blocks,
global heap. - Threads communicate implicitly by writing and
reading shared variables. - Threads coordinate using synchronization
operations on shared variables
Address
x ...
Shared
y ..x ...
Private
i res s
7Machine Model 1a Shared Memory
- Processors all connected to a large shared
memory. - Typically called Symmetric Multiprocessors (SMPs)
- Sun, HP, Intel, IBM SMPs (nodes of Millennium,
SP) - Local memory is not (usually) part of the
hardware. - Cost much cheaper to access data in cache than
in main memory. - Difficulty scaling to large numbers of processors
- lt32 processors typical
- Advantage uniform memory access (UMA)
8Example Problem in Scaling Shared Memory
From Pat Worley, ORNL
9Example Problem in Scaling Shared Memory
From Pat Worley, ORNL
10Machine Model 1b Distributed Shared Memory
- Memory is logically shared, but physically
distributed - Any processor can access any address in memory
- Cache lines (or pages) are passed around machine
- SGI Origin is canonical example ( research
machines) - Scales to 100s
- Limitation is cache consistency protocols need
to keep cached copies of the same address
consistent
P1
Pn
network
memory
memory
memory
11Shared Memory Code for Computing a Sum
static int s 0
Thread 1 local_s1 0 for i 0, n/2-1
local_s1 local_s1 f(Ai) s s
local_s1
Thread 2 local_s2 0 for i n/2, n-1
local_s2 local_s2 f(Ai) s s
local_s2
What is the problem?
- A race condition or data race occurs when
- two processors (or two threads) access the same
variable, and at least one does a write. - The accesses are concurrent (not synchronized)
12Pitfalls and Solution via Synchronization
- Pitfall in computing a global sum s s
local_si
Thread 1 (initially s0) load s from mem to
reg s slocal_s1 local_s1, in reg
store s from reg to mem
Thread 2 (initially s0) load s from
mem to reg initially 0 s slocal_s2
local_s2, in reg store s from reg to mem
Time
- Instructions from different threads can be
interleaved arbitrarily. - One of the additions may be lost
- Possible solution mutual exclusion with locks
Thread 1 lock load s s slocal_s1
store s unlock
Thread 2 lock load s s slocal_s2
store s unlock
13Programming Model 2 Message Passing
- Program consists of a collection of named
processes. - Usually fixed at program startup time
- Thread of control plus local address space -- NO
shared data. - Logically shared data is partitioned over local
processes. - Processes communicate by explicit send/receive
pairs - Coordination is implicit in every communication
event. - MPI is the most common example
A
A
n
0
14Computing s x(1)x(2) on each processor
Processor 2 receive xremote, proc1 send
xlocal, proc1 xlocal x(2) s
xlocal xremote
Processor 1 send xlocal, proc2
xlocal x(1) receive xremote, proc2 s
xlocal xremote
- Second possible solution -- what could go wrong?
Processor 1 send xlocal, proc2
xlocal x(1) receive xremote, proc2 s
xlocal xremote
Processor 2 send xlocal, proc1
xlocal x(2) receive xremote, proc1 s
xlocal xremote
- What if send/receive acts like the telephone
system? The post office?
15MPI the de facto standard
- In 2002 MPI has become the de facto standard for
parallel computing - The software challenge overcoming the MPI
barrier - MPI created finally a standard for applications
development in the HPC community - Standards are always a barrier to further
development - The MPI standard is a least common denominator
building on mid-80s technology - Programming Model reflects hardware!
I am not sure how I will program a Petaflops
computer, but I am sure that I will need MPI
somewhere HDS 2001
16Machine Model 2 Distributed Memory
- Cray T3E, IBM SP, Millennium.
- Each processor is connected to its own memory and
cache but cannot directly access another
processors memory. - Each node has a network interface (NI) for all
communication and synchronization.
17PC Clusters Contributions of Beowulf
- An experiment in parallel computing systems
- Established vision of low cost, high end
computing - Demonstrated effectiveness of PC clusters for
some (not all) classes of applications - Provided networking software
- Conveyed findings to broad community (great PR)
- Tutorials and book
- Design standard to rally
- community!
- Standards beget
- books, trained people,
- software virtuous cycle
Adapted from Gordon Bell, presentation at
Salishan 2000
18Linuss Law Linux Everywhere
- Software is or should be free (Stallman)
- All source code is open
- Everyone is a tester
- Everything proceeds a lot faster when everyone
works on one code (HPC nothing gets done if
resources are scattered) - Anyone can support and market the code for any
price - Zero cost software attracts users!
- All the developers write lots of code
- Prevents community from losing HPC software (CM5,
T3E)
19Commercially Integrated Tflop/s Clusters Are
Happening today
- Shell largest engineering/scientific cluster
- NCSA 1024 processor cluster (IA64)
- Univ. Heidelberg cluster
- PNNL announced 8 Tflops (peak) IA64 cluster from
HP with Quadrics interconnect - DTF in US announced 4 clusters for a total of 13
Teraflops (peak)
But make no mistake Itanium and McKinley are
not a commodity product
20Internet Computing- SETI_at_home
- Running on 500,000 PCs, 1000 CPU Years per Day
- 485,821 CPU Years so far
- Sophisticated Data Signal Processing Analysis
- Distributes Datasets from Arecibo Radio Telescope
Next Step- Allen Telescope Array
21Programming Model 2b Global Addr Space
- Program consists of a collection of named
processes. - Usually fixed at program startup time
- Local and shared data, as in shared memory model
- But, shared data is partitioned over local
processes - Remote data stays remote on distributed memory
machines - Processes communicate by writes to shared
variables - Explicit synchronization needed to coordinate
- UPC, Titanium, Split-C are some examples
- Global Address Space programming is an
intermediate point between message passing and
shared memory - Most common on a the Cray T3E, which had some
hardware support for remote reads/writes
22Programming Model 3 Data Parallel
- Single thread of control consisting of parallel
operations. - Parallel operations applied to all (or a defined
subset) of a data structure, usually an array - Communication is implicit in parallel operators
- Elegant and easy to understand and reason about
- Coordination is implicit statements executed
synchronousl - Drawbacks
- Not all problems fit this model
- Difficult to map onto coarse-grained machines
A array of all data fA f(A) s sum(fA)
s
23Machine Model 3a SIMD System
- A large number of (usually) small processors.
- A single control processor issues each
instruction. - Each processor executes the same instruction.
- Some processors may be turned off on some
instructions. - Machines are not popular (CM2), but programming
model is.
control processor
. . .
interconnect
- Implemented by mapping n-fold parallelism to p
processors. - Mostly done in the compilers (e.g., HPF).
24Model 3b Vector Machines
- Vector architectures are based on a single
processor - Multiple functional units
- All performing the same operation
- Instructions may specific large amounts of
parallelism (e.g., 64-way) but hardware executes
only a subset in parallel - Historically important
- Overtaken by MPPs in the 90s
- Still visible as a processor architecture within
an SMP
25Earth Simulator Architecture Optimizing for the
full range of tasks
- Parallel Vector Architecture
- High speed (vector) processors
- High memory bandwidth (vector architecture)
- Fast network (new crossbar switch)
Rearranging commodity parts cant match this
performance
26Earth Simulator Configuration of a General
Purpose Supercomputer
- 640 nodes
- 8 vector processors of 8 GFLOPS and 16GB shared
memories per node. - Total of 5,120 processors
- Total 40 Tflop/s peak performance
- Main memory 10 TB
- High bandwidth (32 GB/s), low latency network
connecting nodes. - Disk
- 450 TB for systems operations
- 250 TB for users.
- Mass Storage system 12 Automatic Cartridge
Systems (U.S. made STK PowderHorn9310) total
storage capacity is approximately 1.6 PB.
27Cray SV2 Parallel Vector Architecture
- 12.8 Gflop/s Vector processors
- 4 processor nodes sharing up to 64 GB of memory
- Single System Image to 4096 Processors
- 64 CPUs/800 GFLOPS in LC cabinet
28Machine Model 4 Clusters of SMPs
- SMPs are the fastest commodity machine, so use
them as a building block for a larger machine
with a network - Common names
- CLUMP Cluster of SMPs
- Hierarchical machines, constellations
- Most modern machines look like this
- Millennium, IBM SPs, (not the t3e)...
- What is an appropriate programming model 4 ???
- Treat machine as flat, always use message
passing, even within SMP (simple, but ignores an
important part of memory hierarchy). - Shared memory within one SMP, but message passing
outside of an SMP.
29Cluster of SMP Approach
- A supercomputer is a stretched high-end server
- Parallel system is built by assembling nodes that
are modest size, commercial, SMP servers just
put more of them together
Image from LLNL
30NERSC-3 Vital Statistics
- 5 Teraflop/s Peak Performance 3.05 Teraflop/s
with Linpack - 208 nodes, 16 CPUs per node at 1.5 Gflop/s per
CPU - Worst case Sustained System Performance measure
.358 Tflop/s (7.2) - Best Case Gordon Bell submission 2.46 on 134
nodes (77) - 4.5 TB of main memory
- 140 nodes with 16 GB each, 64 nodes with 32 GBs,
and 4 nodes with 64 GBs. - 40 TB total disk space
- 20 TB formatted shared, global, parallel, file
space 15 TB local disk for system usage - Unique 512 way Double/Single switch configuration
31Outline
- Overview of parallel machines and programming
models - Shared memory
- Shared address space
- Message passing
- Data parallel
- Clusters of SMPs
- Trends in real machines
32TOP500
- Listing of the 500 most powerful
Computers in the World - Yardstick Rmax from
Linpack Axb, dense problem - Updated twice
a year ISCxy in Germany, June xy SCxy in
USA, November xy - All data available from
www.top500.org
TPP performance
Rate
Size
33TOP500 list - Data shown
- Manufacturer Manufacturer or vendor
- Computer Type indicated by manufacturer or
vendor - Installation Site Customer
- Location Location and country
- Year Year of installation/last major update
- Customer Segment Academic,Research,Industry,Vendor
,Class. - Processors Number of processors
- Rmax Maxmimal LINPACK performance
achieved - Rpeak Theoretical peak performance
- Nmax Problemsize for achieving Rmax
- N1/2 Problemsize for achieving half of Rmax
- Nworld Position within the TOP500 ranking
34TOP10
35TOP500 - Performance
36Analysis of TOP500 Data
- Annual performance growth about a factor of 1.82
- Two factors contribute almost equally to the
annual total performance growth - Processor number grows per year on the average by
a factor of 1.30 and the - Processor performance grows by 1.40 compared to
1.58 of Moore's Law - Strohmaier, Dongarra, Meuer, and Simon, Parallel
Computing 25, 1999, pp 1517-1544.
37Manufacturers
38Processor Type
39Chip Technology
40Chip Technology
41Architectures
42NOW - Cluster
43Summary
- Historically, each parallel machine was unique,
along with its programming model and programming
language. - It was necessary to throw away software and start
over with each new kind of machine. - Now we distinguish the programming model from the
underlying machine, so we can write portably
correct codes that run on many machines. - MPI now the most portable option, but can be
tedious. - Writing portably fast code requires tuning for
the architecture. - Algorithm design challenge is to make this
process easy. - Example picking a blocksize, not rewriting whole
algorithm.