Parallel Computing - PowerPoint PPT Presentation

Loading...

PPT – Parallel Computing PowerPoint presentation | free to view - id: 208c33-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Parallel Computing

Description:

Traditionally software has been written for serial computation. ... A Library of Parallel Algorithms, www-2.cs.cmu.edu/~scandal/nesl/algorithms.html ... – PowerPoint PPT presentation

Number of Views:42
Avg rating:3.0/5.0
Slides: 29
Provided by: justinr8
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Parallel Computing


1
Parallel Computing
  • Presented by Justin Reschke
  • 9-14-04

2
Overview
  • Concepts and Terminology
  • Parallel Computer Memory Architectures
  • Parallel Programming Models
  • Designing Parallel Programs
  • Parallel Algorithm Examples
  • Conclusion

3
Concepts and TerminologyWhat is Parallel
Computing?
  • Traditionally software has been written for
    serial computation.
  • Parallel computing is the simultaneous use of
    multiple compute resources to solve a
    computational problem.

4
Concepts and TerminologyWhy Use Parallel
Computing?
  • Saves time wall clock time
  • Cost savings
  • Overcoming memory constraints
  • Its the future of computing

5
Concepts and TerminologyFlynns Classical
Taxonomy
  • Distinguishes multi-processor architecture by
    instruction and data
  • SISD Single Instruction, Single Data
  • SIMD Single Instruction, Multiple Data
  • MISD Multiple Instruction, Single Data
  • MIMD Multiple Instruction, Multiple Data

6
Flynns Classical TaxonomySISD
  • Serial
  • Only one instruction and data stream is acted on
    during any one clock cycle

7
Flynns Classical TaxonomySIMD
  • All processing units execute the same instruction
    at any given clock cycle.
  • Each processing unit operates on a different data
    element.

8
Flynns Classical TaxonomyMISD
  • Different instructions operated on a single data
    element.
  • Very few practical uses for this type of
    classification.
  • Example Multiple cryptography algorithms
    attempting to crack a single coded message.

9
Flynns Classical TaxonomyMIMD
  • Can execute different instructions on different
    data elements.
  • Most common type of parallel computer.

10
Concepts and TerminologyGeneral Terminology
  • Task A logically discrete section of
    computational work
  • Parallel Task Task that can be executed by
    multiple processors safely
  • Communications Data exchange between parallel
    tasks
  • Synchronization The coordination of parallel
    tasks in real time

11
Concepts and TerminologyMore Terminology
  • Granularity The ratio of computation to
    communication
  • Coarse High computation, low communication
  • Fine Low computation, high communication
  • Parallel Overhead
  • Synchronizations
  • Data Communications
  • Overhead imposed by compilers, libraries, tools,
    operating systems, etc.

12
Parallel Computer Memory ArchitecturesShared
Memory Architecture
  • All processors access all memory as a single
    global address space.
  • Data sharing is fast.
  • Lack of scalability between memory and CPUs

13
Parallel Computer Memory ArchitecturesDistribute
d Memory
  • Each processor has its own memory.
  • Is scalable, no overhead for cache coherency.
  • Programmer is responsible for many details of
    communication between processors.

14
Parallel Programming Models
  • Exist as an abstraction above hardware and memory
    architectures
  • Examples
  • Shared Memory
  • Threads
  • Messaging Passing
  • Data Parallel

15
Parallel Programming ModelsShared Memory Model
  • Appears to the user as a single shared memory,
    despite hardware implementations.
  • Locks and semaphores may be used to control
    shared memory access.
  • Program development can be simplified since there
    is no need to explicitly specify communication
    between tasks.

16
Parallel Programming ModelsThreads Model
  • A single process may have multiple, concurrent
    execution paths.
  • Typically used with a shared memory architecture.
  • Programmer is responsible for determining all
    parallelism.

17
Parallel Programming ModelsMessage Passing Model
  • Tasks exchange data by sending and receiving
    messages.
  • Typically used with distributed memory
    architectures.
  • Data transfer requires cooperative operations to
    be performed by each process. Ex.- a send
    operation must have a receive operation.
  • MPI (Message Passing Interface) is the interface
    standard for message passing.

18
Parallel Programming ModelsData Parallel Model
  • Tasks performing the same operations on a set of
    data. Each task working on a separate piece of
    the set.
  • Works well with either shared memory or
    distributed memory architectures.

19
Designing Parallel ProgramsAutomatic
Parallelization
  • Automatic
  • Compiler analyzes code and identifies
    opportunities for parallelism
  • Analysis includes attempting to compute whether
    or not the parallelism actually improves
    performance.
  • Loops are the most frequent target for automatic
    parallelism.

20
Designing Parallel ProgramsManual
Parallelization
  • Understand the problem
  • A Parallelizable Problem
  • Calculate the potential energy for each of
    several thousand independent conformations of a
    molecule. When done find the minimum energy
    conformation.
  • A Non-Parallelizable Problem
  • The Fibonacci Series
  • All calculations are dependent

21
Designing Parallel ProgramsDomain Decomposition
  • Each task handles a portion of the data set.

22
Designing Parallel ProgramsFunctional
Decomposition
  • Each task performs a function of the overall work

23
Parallel Algorithm ExamplesArray Processing
  • Serial Solution
  • Perform a function on a 2D array.
  • Single processor iterates through each element in
    the array
  • Possible Parallel Solution
  • Assign each processor a partition of the array.
  • Each process iterates through its own partition.

24
Parallel Algorithm ExamplesOdd-Even
Transposition Sort
  • Basic idea is bubble sort, but concurrently
    comparing odd indexed elements with an adjacent
    element, then even indexed elements.
  • If there are n elements in an array and there are
    n/2 processors. The algorithm is effectively O(n)!

25
Parallel Algorithm ExamplesOdd Even
Transposition Sort
  • Initial array
  • 6, 5, 4, 3, 2, 1, 0
  • 6, 4, 5, 2, 3, 0, 1
  • 4, 6, 2, 5, 0, 3, 1
  • 4, 2, 6, 0, 5, 1, 3
  • 2, 4, 0, 6, 1, 5, 3
  • 2, 0, 4, 1, 6, 3, 5
  • 0, 2, 1, 4, 3, 6, 5
  • 0, 1, 2, 3, 4, 5, 6
  • Worst case scenario.
  • Phase 1
  • Phase 2
  • Phase 1
  • Phase 2
  • Phase 1
  • Phase 2
  • Phase 1

ONLY 7 PASSES!
26
Other Parallelizable Problems
  • The n-body problem
  • Floyds Algorithm
  • Serial O(n3), Parallel O(n log p)
  • Game Trees
  • Divide and Conquer Algorithms

27
Conclusion
  • Parallel computing is fast.
  • There are many different approaches and models of
    parallel computing.
  • Parallel computing is the future of computing.

28
References
  • A Library of Parallel Algorithms,
    www-2.cs.cmu.edu/scandal/nesl/algorithms.html
  • Internet Parallel Computing Archive,
    wotug.ukc.ac.uk/parallel
  • Introduction to Parallel Computing,
    www.llnl.gov/computing/tutorials/parallel_comp/Wh
    atis
  • Parallel Programming in C with MPI and OpenMP,
    Michael J. Quinn, McGraw Hill Higher Education,
    2003
  • The New Turing Omnibus, A. K. Dewdney, Henry Holt
    and Company, 1993
About PowerShow.com