Lecture 2: Part I Parallel Programming Models - PowerPoint PPT Presentation

Loading...

PPT – Lecture 2: Part I Parallel Programming Models PowerPoint presentation | free to download - id: 1e94ae-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Lecture 2: Part I Parallel Programming Models

Description:

A programming model is what the programmer sees and uses when ... Defined by Alan Karp [1987]. 25. An Example of SPMD and MPMD Code. MPMD code: parbegin ... – PowerPoint PPT presentation

Number of Views:113
Avg rating:3.0/5.0
Slides: 43
Provided by: drcl
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Lecture 2: Part I Parallel Programming Models


1
Lecture 2 Part I Parallel Programming Models
  • A programming model is what the programmer sees
    and uses when developing program

2
Sequential Programming Model
Processor
P
M
code
Memory
os
3
Parallel Programming
  • Definition PP is the activity of constructing a
    parallel program from a given algorithm.
  • Interface between algorithms and parallel
    architectures.

4
Why Is It So Difficult ?
  • Potentially more complicated than sequential
    programming
  • Diverse parallel programming models
  • Lack of advanced parallel compiler, debugger,
    profiler.
  • More people doing sequential programming

5
QUESTION ?
  • Can we run sequential C code on parallel machine
    to gain speedup ?

6
Levels of Abstraction
Applications
Algorithmic Paradigm
Machine Independent
Language Supported (Programming Models)
Machine Dependent
Hardware Architecture
NOW
Multicomputer
Multiprocessor
7
Parallel Programming
(Sequential or parallel) Application Algorithm
Parallel language and other tools
User (Programmer)
(sequential or parallel) Source program
Parallel Programming
Compiler (including Preprocessor, Assembler, and
Linker)
Run-Time Support and Other Libraries
Native Parallel Code
Parallel Platform (OSHardware)
8
Native Programming Model
  • The lowest-level, user-visible programming model
    provided by a specific parallel computer
    platform.
  • Example Power C in SGI Power Challenge, shmem in
    T3D
  • Other Programming models data parallel (HPF),
    message passing (MPI) can be implemented on top
    of Power C.

9
Algorithmic Paradigms (Engineering) - Parallel
Track
  • Compute-Interact
  • Work-Pool
  • Divide and Conquer
  • Pipelining (Data Stream)
  • Master-Slave

10
Data vs. Control Parallelism
  • Data parallelism
  • Multiple, complete functional units apply same
    operation simultaneously to different
    elements of data set.
  • E.g., Divide the domain evenly among the PEs and
    each PE performs the same task
  • Hardware SIMD/MIMD machine
  • Data Parallel Programming HPF (High Performance
    Fortran), C

11
Data vs. Control Parallelism
  • Control parallelism
  • Apply distinct operations to data elements
    concurrently.
  • Outputs of operations are fed in as inputs to
    other operations, in an arbitrary way.
  • The flow of data forms an arbitrary graph.
  • A pipeline is a special case of this, where the
    graph is just a single path.

12
Compute-Interact
Synchronous Interaction
C
C
C
Synchronous Interaction
13
Work Pool
Get jobs from pool
14
Divide and Conquer
Load
Load/2
Load/4
Merge results
15
Pipelined (Data Stream)
Edge Detection
Edge Linking
Line Generation
Task 1
Task 2
Task 2
16
Pipelined computation
  • Divide computation into a number of stages.
  • Devote separate functional units to each stage
  • if each completes in the same time, then, once
    pipe is full, the throughput of the pipeline is 1
    result per clock.

17
Master-Slave
Master (assign jobs)
Slave
Slave
Slave
18
Algorithmic Paradigms (science) - Sequential Track
  • Divide-and-Conquer
  • Dynamic Programming
  • Branch-and-Bound
  • Backtracking
  • Greedy
  • Parallel Versions ??

19
Parallel Programming Models
  • Not talking about language !!

20
Programming Models
  • Homogeneity refers to the similarity of
    component processes in a parallel program.

21
Programming Models
  • SPMD Single-Program-Multiple-Data, programs are
    homogeneous
  • MPMD Multiple-Program-Multiple-Data, programs
    are heterogeneous
  • SIMD Single-Instruction-Multiple-Data,
    restricted form of SPMD, all processors execute
    the same instruction at the same time. Restricted
    form of SPMD

22
Programming Models
  • Both SPMD and MPMD are MIMD -- different
    instructions can be executed by different
    processes at the same time.

23
What is SPMD?
  • Single Program, Multiple Data
  • Same program runs everywhere
  • Restriction on the general message-passing model
    (MPMD)
  • Most venders only support SPMD parallel programs

24
What is SPMD?
  • General message-passing model (MPMD) can be
    emulated
  • A data-parallel program refers to an SPMD program
    in general.
  • Defined by Alan Karp 1987.

25
An Example of SPMD and MPMD Code
  • MPMD code
  • parbegin
  • A
  • B
  • C
  • SPMD code
  • Main( )
  • myid getid ( )
  • if (myid0) A
  • elseif (myid1) B
  • else (myid2) C

26
MPMD
Network
A
B
C
node 0
node 1
node 2
27
SPMD
Network
Main( ) myid getid ( ) if (myid0)
A elseif (myid1) B else (myid2) C
Main( ) myid getid ( ) if (myid0)
A elseif (myid1) B else (myid2) C
Main( ) myid getid ( ) if (myid0)
A elseif (myid1) B else (myid2) C
node 0
node 1
node 2
28
SPMD Programming
  • Two major phases
  • (1) data distribution choice determine the
    mapping of data onto nodes
  • (2) parallel program generation translate
    sequential algorithm into the SPMD program (only
    write one program !!).

29
Parallel Programming base on SPMD (4 main tasks)
  • (1) Get node and environmental information
  • How many nodes in the system?
  • Who am I ?
  • (2) Access data convert local-to-global and
    global-to-local indexes
  • (3) Insert message-passing primitives to exchange
    data (implicitly or explicitly)
  • (4) Carry out operations on directly accessible
    data (local operation, e.g., C code)

30
Programming Languages
  • Lots of names !!

31
Programming Languages
  • Implicit Parallel (KAP)
  • Data Parallel (Fortran 90, HPF, CM Fortran,..)
  • Message-Passing (MPI, PVM, CMMD, NX, Active
    Message, Fast Message, P4, MPL, LAM, Express)
  • Shared-Variable (X3H5)
  • Hybrid MPI-HPF, Split-C, MPI-Java (SRG)

32
Data Parallel Language
  • main( )
  • double localN, tmpN, pi, w
  • long i,j, N100000
  • w1.0/N
  • forall (I0iltNI)
  • locali (i-0.5)w
  • tmpi 4.0/(
  • pi sum(tmp)

33
Data Parallel Model
  • Single threading (from users viewpoint) one
    process one thread of control. Just like a
    sequential program.
  • Global naming space all variable reside in a
    single address space.
  • Parallel operations on aggregate data structure
    sum( ).

34
Data Parallel Model
  • Loosely synchronization implicit synchronization
    after every statement. (Compared with tight
    synchronization in an SIMD system -- on every
    instruction)

35
Message-Passing Language
M
M
M
C Code
P
P
P
Comm Subroutine
Communication Network
36
Message-Passing Model
  • Multithreading multiple processes simultaneously
    executing
  • Separate address space local variables are not
    visible to other processes.

37
Message-Passing Model
  • Explicit allocation both workload and data are
    explicitly allocated to the processes by the
    user.
  • Explicit interactions communication,
    synchronization, aggregation,...
  • Asynchronous the processes execute
    asynchronously.

38
Message Passing Programming Model
  • Message-passing programming is more directly
    painful, but it tends to build locality of
    reference from the start.
  • The end result seems to be
  • for scalable, highly parallel, high-performance
    results, locality of reference will always be
    central.

39
Shared variable Model
  • pragma pfor iterate (I0NI)
  • for (I0 IltN I)
  • local (I0.5) w
  • pragma critical
  • pipilocal
  • double local, pi, w
  • long I, taskid
  • long numtask
  • w 1.0/N
  • pragma shared (pi, w)
  • pragma local (I, local)

40
Shared Variable Model
  • Single address space (similar to data parallel
    model)
  • Multithreading and asynchronous (similar to
    message-passing)
  • Communication is done implicitly through shared
    reads and writes of variables.
  • Synchronization is explicit.

41
Shared-Memory Programming Model
  • All data shared and visible by executing threads
  • Shared-memory program starts out looking simpler,
    but memory locality forces one to do some strange
    transformations
  • Shared-Memory programming standards ANSI X3H5
    (1993), POSIX Threads (Pthreads), OpenMP, SGI
    Power C

42
Conclusion
  • Parallel programming has lagged far behind the
    advances of parallel hardware
  • Compared to the sequential counterparts, todays
    parallel system software and application software
    are few in quantity and primitive in
    functionality.
  • Likely to continue !!
About PowerShow.com