Title: Efficient Exploitation of Fine-Grained Parallelism using a microHeterogeneous Computing Environment
1Efficient Exploitation of Fine-Grained
Parallelism using a microHeterogeneous Computing
Environment
Presented By William Scheidel September 19th, 2002
- Computer Engineering Department
2Agenda
- Objective
- Heterogeneous Computing Background
- microHeterogeneous Computing Architecture
- microHeterogeneous Framework Implementation
- Simulation Results
- Conclusion
- Future Work
- Acknowledgements
3Objectives
- The main objective of this thesis is to propose a
new computing paradigm, called microHeterogeneous
computing or mHC, which incorporates processing
elements (vector processors, digital signal
processors, etc) into a general purpose machine
using the high-performance I/O buses that are
available - This architecture will then be used in order to
efficiently utilize fine-grained parallelism
4Heterogeneous Computing
- An architecture which provides an assortment of
high performance machines for use by an
application - Arose from the realization that no single machine
is capable of performing all tasks in an optimal
manner - Machines differ in both speed as well as in
capabilities and are connected using high speed,
high bandwidth intelligent interconnects that
handle the intercommunication between each of the
machines.
5Motivation For Heterogeneous Computing
- Hypothetical example of the advantage of using a
heterogeneous suite of machines, where the
heterogeneous suite time includes inter-machine
communication overhead. Not drawn to scale.
6Heterogeneous Computing Driving ApplicationImage
Understanding
- Well suited to a heterogeneous environment due to
its complexity and involvement of different types
of parallelism - Consists of three main levels of processing, each
level containing a different type of parallelism - Three main levels can also be executed in
parallel - Lowest Level
- Consists of pixel-based operators and pixel
subset operators such as edge detection - Highest amount of parallelism
- Best suited to mesh connected SIMD machines
- Intermediate Level
- Grouping and organization of features previously
extracted - Communication is irregular, parallelism decreases
as features are grouped - Best suited to medium-grained MIMD machines
- Highest Level
- Knowledge processing
- Uses the data from the previous levels in order
to infer semantic attributes about an image - Requires coarse-grained loosely coupled MIMD
machines
7Heterogeneous Computing Driving ApplicationImage
Understanding
- The three levels of processing required for image
understanding. - Each of the levels contains large amounts of
varying kinds of parallelism that can be
exploited by heterogeneous computing
8Heterogeneous Computing Broad Issues
- Analytical benchmarking
- Code-Type or task profiling
- Matching and Scheduling
- Programming environments
- Interconnection requirements environment
requirements
9Execution Steps of Heterogeneous Applications
- Analytical Benchmarking - determines the optimal
speedup that a particular machine can achieve on
different types of tasks - Code Profiling - determines the modes of
computation that exist in each program segment as
well as the execution times - Task Scheduling tasks are mapped to a
particular machine using some form of scheduling
heuristic - Task Execution the tasks are executed on the
assigned machine
10HC Issues Analytical Benchmarking
- Measure of how well a given machine is able to
perform on a certain type of code - Required in HSC to determine which types of code
should be mapped to which machines - Benchmarking is an offline process
- Example results
- SIMD machines are well suited for matrix
computations / low level image processing - MIMD machines are best suited for tasks that have
limited intercommunication
11HC Issues Code-type Profiling
- Used to determine the types of parallelism in the
code as well as execution time - Tasks are separated into segments which contain a
homogeneous type of parallelism - These segments can then be matched to a
particular machine that is best suited to execute
them - Code-type profiling is an offline process
12Code Profiling Example
- Example results from the code-profiling of a
task. - The task is broken into S segments, each of which
contains embedded homogeneous parallelism.
13HC Issues Matching and Scheduling
- Goal is to map code-types to the best suited
machine - Costs most be carefully weighed
- Computation Costs execution time of a code
segment is dependent on the machine it is run on
as well as the current workload of the machine - Communication Costs dependent on type of
interconnection used and bandwidth - Interference Costs resource contention occurs
when multiple tasks are assigned to a particular
machine - Problem determined to be NP-hard even for
homogeneous environments - Addition of heterogeneous processing elements
adds to the complexity - A large number of heuristic algorithms have been
designed to schedule tasks to machines on
heterogeneous computing systems. - Most such heuristic algorithms developed are
static and assume the ETC (expected time to
compute) for every task on every machine to be
known from code-type profiling and analytical
benchmarking.
14Example Static Scheduling Heuristics for HC
- Opportunistic Load Balancing (OLB) assigns each
task, in arbitrary order, to the next available
machine. - User-Directed Assignment (UDA) assigns each
task, in arbitrary order, to the machine with the
best expected execution time for the task. - Fast Greedy assigns each task, in arbitrary
order, to the machine with the minimum completion
time for that task. - Min-min the minimum completion time for each
task is computed respect to all machines. The
task with the overall minimum completion time is
selected and assigned to the corresponding
machine. The newly mapped task is removed, and
the process repeats until all tasks are mapped. - Max-min The Max-min heuristic is very similar
to the Min-min algorithm. The set of minimum
completion times is calculated for every task.
The task with overall maximum completion time
from the set is selected and assigned to the
corresponding machine. - Greedy or Duplex The Greedy heuristic is
literally a combination of the Min-min and
Max-min heuristics by using the better solution
15Example Static Scheduling Heuristics for HC
- GA The Genetic algorithm (GA) is used for
searching large solution space. It operates on a
population of chromosomes for a given problem.
The initial population is generated randomly. A
chromosome could be generated by any other
heuristic algorithm. - Simulated Annealing (SA) an iterative technique
that considers only one possible solution for
each meta-task at a time. SA uses a procedure
that probabilistically allows solution to be
accepted to attempt to obtain a better search of
the solution space based on a system temperature. - GSA The Genetic Simulated Annealing (GSA)
heuristic is a combination of the GA and SA
techniques. - Tabu Tabu search is a solution space search
that keeps track of the regions of the solution
space which have already been searched so as not
to repeat a search near these areas . - A A is a tree search beginning at a root node
that is usually a null solution. As the tree
grows, intermediate nodes represent partial
solutions and leaf nodes represent final
solutions. Each node has a cost function, and the
node with the minimum cost function is replaced
by its children. Any time a node is added, the
tree is pruned by deleting the node with the
largest cost function. This process continues
until a complete mapping (a leaf node) is reached.
16Example Static Scheduling Heuristics for HCThe
Segmented Min-Min Algorithm
- Every task has a ETC (expected time to compute)
on a specific machine. If there are t tasks and m
machines, we can obtain a t x m ETC
matrix. ETC(i j) is the estimated execution time
for task i on machine j. - The Segmented min-min algorithm sorts the tasks
according to ETCs. - The tasks can be sorted into an ordered list by
the average ETC, the minimum ETC, or the maximum
ETC. - Then, the task list is partitioned into segments
with the equal size.
17Segmented Min-Min Scheduling Heuristic
18Example Dynamic Scheduling Heuristics for
HCHEFT Scheduling Heuristic
19Heterogeneous Parallel Programming
- Parallel Virtual Machine (PVM)
- Enables a collection of heterogeneous computers
to be used as a coherent and flexible concurrent
computational resource - Unit of parallelism are tasks which are generally
processes - User also has the choice to view these resources
as an attributeless collection of virtual
processing elements or choose to exploit the
capabilities of specific machines in the host
pool - Allows multifaceted virtual machines to be
configured within the same framework and permits
messages containing more than one data type to be
exchanged between machines having different data
representations - Message Passing Interface (MPI)
- Each processor is assigned a rank which then
determines which parts of the application that
processor will execute - Division of tasks among processors is left solely
up to the developer writing the application - Includes routines to do point-to-point
communication between two processing elements,
collective operations to simultaneously
communicate information between all processing
elements, and implicit as well as explicit
synchronization
20HC Issues Interconnection Requirements
- Interconnection medium must support high
bandwidths and low latency communications (LANs
wont cut it) - Complexity in a heterogeneous system increases
since different machines use different protocols
for communication - Must support both shared memory and message-based
communication
21Heterogeneous Computing Limitations
- Task Granularity
- Heterogeneous computing only utilizes
coarse-grained parallelism to increase
performance - Coarse-grained parallelism results in large task
sizes and reduced coupling which allows the
processing elements to work more efficiently - Eequirement is also translated to most
heterogeneous schedulers since they are based on
the scheduling of meta-tasks, i.e. tasks that
have no dependencies - Communication Overhead
- Tasks and their working sets must be transmitted
over some form of network in order to execute
them, latency and bandwidth become crucial
factors - Overhead is also incurred when encoding and
decoding the data for different architectures - Cost
- Machines used in heterogeneous computing
environments can be prohibitively expensive - Expensive high speed, low latency networks are
required to achieve the best performance - Not cost effective for applications where only a
small portion of the code would benefit from such
an environment
22microHeterogeneous Computing
- A new computing paradigm that attempts to most
efficiently exploit the fine-grained parallelism
found in most scientific computing applications - Environment is contained within a workstation and
consists of a host processor and a number of
additional PCI (or other high performance I/O
bus) based processing elements - Elements might be DSP based, vector based, FGPA
based, or even reconfigurable computing elements - In combination with a host processor, these
elements create a small scale heterogeneous
computing environment - An mHC specific API was developed that greatly
simplifies using these types of devices for
parallel applications
23microHeterogeneous Computing Environment
24Comparison Between Heterogeneous Computing and
microHeterogeneous Computing
- Task Granularity
- Heterogeneous environments only support
coarse-grained parallelism, while the mHC
environment instead focuses on fine-grained
parallelism by using a tightly coupled shared
memory environment - Task size is reduced to a single function call in
a mHC environment - Drawbacks
- Processing elements used in mHC are not nearly as
powerful as the machines used in a standard
heterogeneous environment - There is a small and finite number of processing
elements that can be added to a single machine - Communication Overhead
- High performance I/O buses are twice as fast as
the fastest network - Less overhead is incurred when encoding and
decoding data since all processing elements use
the same base architecture - Cost Effectiveness
- Machines used in a heterogeneous environment can
cost tens of thousands of dollars each, and
require the extra expense of the high-speed, low
latency interconnects to achieve acceptable
performance - mHC processing elements cost only hundreds of
dollars
25Comparison Between Heterogeneous Computing and
microHeterogeneous Computing
- Analytical Benchmarking and Profiling
- Analytical benchmarking is used for the same
purpose in both computing environments. The
capabilities of each processing element or
machine must be known before program execution
begins so the scheduling algorithm is able to
determine an efficient mapping of tasks. - Profiling, while necessary in heterogeneous
environments is not required in
microHeterogeneous environments - Scheduling Heuristics
- Scheduling algorithms in heterogeneous
environments generally take place during the
compilation stage instead of during execution - The scheduler for an mHC environment must be
dynamic and map tasks in real-time in order to
provide the best performance.
26microHeterogeneous Computing API
- An API was created for microHeterogeneous
computing to provide a flexible and portable
interface - User applications only need to make simple API
calls, mapping of tasks onto available devices is
performed automatically - There is no need for an application to be
recompiled if the underlying implementation of
microHeterogeneous Computing changes as long as
the API is adhered to - The API supports a subset of the Gnu Scientific
Library (GSL) - GSL is a freely distributable scientific API
written in C - Includes an extensive library of scientific
functions and data types - Directly supports the Basic Linear Algebra
Subprograms (BLAS) - Data structures are compatible with those used by
the Vector, Image, and Signal, Processing Library
that is becoming a standard on embedded devices
27microHeterogeneous Computing API (cont)
The microHeterogeneous Computing API provides
support for the following areas of scientific
computing
- Linear Algebra
- EigenVectors and EigenValues
- Fast Fourier Transforms
- Numerical Integration
- Statistics
- Vector Operations
- Matrix Operations
- Polynomial Solvers
- Permutations
- Combinations
- Sorting
28Suitable mHC Devices
- XP-15
- Developed by Texas Memory Systems
- DSP based accelerator card
- Performs 80 32-bit floating point operations per
second - Contains 256 MB of on board DDR Ram
- Supports over 500 different scientific functions
- Increases FFT performance by 20x - 40x over a 1.4
Gigahertz P4 - Pegasus-2
- Developed by Catalina Research
- Vector Processor based
- Supports FFT, matrix and vector operations,
convolutions, filters and more - Supported functions operate between 5x and 15x
faster then a 1.4 Gigahertz P4
29microHeterogeneous Framework Implementation
- Implemented as a dynamically linked library
written purely in C that user applications
interact with by way of the mHC API - The framework creates tasks from the API function
calls and schedules them to the available
processing elements
30Phases of the microHeterogneous Framework
- Initialization
- The framework must be initialized before it may
be used - Scheduler and scheduler parameters chosen
- Bus and Device Configuration Files read
- Log file specified
- Data structures created
- Helper threads are created that move tasks from a
devices task queue to the device. These threads
are real-time threads that use a round-robin
scheduling policy.
31microHeterogeneous Computing Framework Overview
32Device Configuration File
- Determines what devices are available in the
microHeterogeneous environment - File is XML based which makes it easy for other
programs to generate and parse device
configuration files - The following is configurable for each device
- Unique ID
- Name
- Description
- Bus that the device uses
- A list of API calls that the device supports,
each API call in the list contains - The ID and Name of the API call
- The speedup achieved as compared to the host
processor - The expected time to completion (ETC) of the API
call given in microseconds per byte of input
33Example Device Configuration File
- ltmHCDeviceConfiggt
- ltDevicegt
- ltIDgt0lt/IDgt
- ltNamegtHostlt/Namegt
- ltDescriptiongtA bad host.lt/Descriptiongt
- ltBusNamegtLocallt/BusNamegt
- ltBusIDgt0lt/BusIDgt
- ltAPISupportgt
- ltFunctiongt
- ltIDgt26lt/IDgt
- ltNamegtmhc_combination_nextlt/Namegt
- ltSpeedupgt1lt/Speedupgt
- ltCompletionTimegt.015lt/CompletionTimegt
- lt/Functiongt
- ltFunctiongt
- ltIDgt9lt/IDgt
- ltNamegtmhc_vector_sublt/Namegt
- ltSpeedupgt1lt/Speedupgt
- ltCompletionTimegt.001lt/CompletionTimegt
- ltDevicegt
- ltIDgt1lt/IDgt
- ltNamegtVector1lt/Namegt
- ltDescriptiongtA simple vector
processor.lt/Descriptiongt - ltBusNamegtPCIlt/BusNamegt
- ltBusIDgt1lt/BusIDgt
- ltAPISupportgt
- ltFunctiongt
- ltIDgt9lt/IDgt
- ltNamegtmhc_vector_sublt/Namegt
- ltSpeedupgt10lt/Speedupgt
- ltCompletionTimegt.0001lt/CompletionTimegt
- lt/Functiongt
- lt/APISupportgt
- lt/Devicegt
- lt/mHCDeviceConfiggt
34Bus Configuration File
- Determines the bus characteristics being used by
the devices - File is XML based which makes it easy for other
programs to generate and parse bus configuration
files - The following is configurable for each bus
- Unique ID
- Name
- Description
- Initialization time
- Specified in microseconds
- Taken into account once during the framework
initialization - Overhead Time
- Specified in microseconds
- Taken into account once for ever bus transaction
- Transfer Time
- Specified in microseconds per byte
- Taken into account once for every byte that is
transmitted over the bus
35Example Bus Configuration File
- ltmHCBusConfiggt
- ltBusgt
- ltIDgt0lt/IDgt
- ltNamegtLocallt/Namegt
- ltDescriptiongtUsed by the hostlt/Descriptiongt
- ltInitTimegt0lt/InitTimegt
- ltOverheadgt0lt/Overheadgt
- ltTransferTimegt0lt/TransferTimegt
- lt/Busgt
- ltBusgt
- ltIDgt1lt/IDgt
- ltNamegtPCIlt/Namegt
- ltDescriptiongtPCI buslt/Descriptiongt
- ltInitTimegt50lt/InitTimegt
- ltOverheadgt0.01lt/Overheadgt
- ltTransferTimegt0.002lt/TransferTimegt
- lt/Busgt
- lt/mHCBusConfiggt
36Phases of the microHeterogneous Framework
- Task Creation
- A new task is created for every API call that is
made, except for initialization, finalization,
and join calls - Tasks encapsulate all of the information of a
function call - ID of function to execute
- List of pointers to all of the arguments
- List of pointers to all of the data blocks used
as inputs and their sizes - List of pointers to all of the data blocks used
as outputs and their sizes
37Phases of the microHeterogneous Framework (cont)
- Task Scheduling
- After a task is created, it is passed to the
scheduling algorithm that was selected during
initialization - The scheduler determines which device to assign
the task and places the task in that devices
task queue - Done dynamically in real-time
- Profiling of applications is not required
- As soon as the scheduler has mapped the task to a
device the API call returns and the main user
program is allowed to continue execution
38Fast Greedy Scheduling Heuristic
39Real-Time Min-Min Scheduling Heuristic
40Weighted Real-Time Min-Min Scheduling Heuristic
41Phases of the microHeterogneous Framework (cont)
- Task Execution
- If a task is available,
- The helper thread checks to see if there are any
unresolved dependencies - If there are no dependencies, the task is removed
from the task queue and passed to the device
driver for execution, otherwise it sleeps - All tasks are executed on the host processor by
simulated drivers
42mHC Applications
- Four mHC Application were written in order to
test the performance of both the architecture and
the different scheduling algorithms that were
developed - Matrix Performs basic matrix operations on a set
of fifty 100 x 100 matrices - First twenty-five matrices are summed together
- Last twenty-five matrices are subtracted from one
another - Every fifth matrix is scaled by a constant
- Finally, the inverse of all fifty matrices is
determined - Stats Performs basic statistics on a block of
five million values - Divides a block of five million values into 50
blocks of 100,000 values - Calculates the standard deviation of each block
of values - Determines the blocks of data with the minimum
and maximum deviations - Linalg solves fifty sets of linear equations
each containing one hundred and seventy-five
variables - Random Used to stress test the different
scheduling algorithms - Creates random task graphs consisting of 300
tasks - Tasks created are matrix element multiplications
between a group of 25 matrices
43Simulation Methodology
- Each simulation run used a three step process
- The sequential version of the application was run
- Done by using the -s -1 parameter when
initializing the framework - Used to determine the estimated time to
completion (ETC) for each of the API calls on the
host processor - Output recorded for comparison purposes
- The parallel version of the application was run
- Done by using the appropriate scheduler
parameter, bus configuration, and device
configuration files - Used to compare the parallel output to the
sequential output to make sure that the
parallelized results were correct - The parallel version was run using the timing
mode - Done by specifying the -t parameter along with
the parameters used in Step 2. - Used to get the final timing results for the
simulation - Steps 1 and 2 were run five times, the median run
was used for calculations
44Matrix Simulation Results
2 3 4 5 6 7
Fast Greedy 2.33 2.33 2.31 2.34 2.31 2.31
RTmm 1.43 2.00 2.39 3.68 3.75 5.00
WRTmm 1.97 2.78 3.71 3.97 6.58 5.07
45Stats Simulation Results
2 3 4 5 6 7
Fast Greedy 0.90 0.91 0.90 0.90 0.90 0.90
RTmm 0.97 1.00 1.07 1.08 1.15 1.10
WRTmm 1.03 1.11 1.14 1.14 1.16 1.17
46Linalg Simulation Results
2 3 4 5 6 7
Fast Greedy 3.20 3.08 3.12 3.18 3.02 2.98
RTmm 1.42 3.40 2.29 4.34 3.72 6.40
WRTmm 1.64 3.40 4.49 6.00 6.44 8.28
47Random Simulation Similar Processing Elements
- Simulation used between 1 and 6 additional
processing elements, each having a speedup of 20x
48Random Simulation Different Processing Elements
- Simulation used between 1 and 6 additional
processing elements - Speedups of 20x, 10x, 5x, 2x, 1x, and 0.5x were
used for devices 1 through 6 respectively.
49Random Simulation Different Bus Transfer Times
- Simulation used three additional processing
elements with various bus transfer times - Transfer times range from a 64-bit 33 MHz PCI bus
(smallest), to a 100 Mb/s Ethernet connection
(largest)
50Random Simulation Various Speedups
- Simulation used four additional processing
elements with various speedups
51Conclusion
- Accomplishments
- A new computer architecture, microHeterogeneous
Computing, was presented that successfully
exploits fine-grained parallelism in scientific
based applications using additional processing
elements - An API was created that allows developers to
incorporate mHC into their applications without
being required to address task scheduling, load
balancing, or threading issues - A highly configurable mHC framework was
implemented as a standard library which allows
actual mHC compliant applications to be compiled
and executed using standard techniques - Future Work
- Creation of mHC compliant device drivers so that
an actual mHC environment can be created - While the microHeterogeneous API currently
contains the most common scientific functions, it
needs to be expanded in order to become
complimentary to the GNU Scientific Library. - the concept of mHC clusters needs to be fully
explored in order to determine the applicability
of mHC to this area of computing.
52mHC Cluster Based Computing
53Acknowledgements
- I would like to thank the following people for
making this thesis possible - My primary advisor, Dr. Shaaban for allowing me
to work on such an interesting and worthwhile
project - My committee members, Dr. Savakis and Dr.
Heliotis for working with very tight schedules - My family for supporting me through this whole
process - My sister for putting a roof over my head for the
last month and a half