A Multiple Associative Model to Support Branches in Data Parallel Applications

1
A Multiple Associative Model to Support Branches
in Data Parallel Applications

• Wittaya Chantamas and Johnnie W. Baker
• Department of Computer Science
• Kent State University, Kent, OHIO 44242 USA
• Telephone [phone]
• Fax [phone]
• wchantam_at_cs.kent.edu and jbaker_at_cs.kent.edu

2
Outline

• SIMD and Branches
• Single Instruction Multiple Data (SIMD)
• A data parallel program contains branches
• MASC Computational Model
• Multiple Associative Computing (MASC) Model
• MASC model with manager-worker paradigm
• The power of MASC model
• With variations of MASC
• With other models
• ASC language compiler support for the MASC model
• MASC Algorithm
• Shapes algorithm
• Modified ASC Quick Hull algorithm for the MASC
  model with manager-worker paradigm

3
SIMD and Branches

• Most SIMD computers allow masking of PEs while
  determining whether or not that PE should
  participate in the operation in a parallel
  IF-THEN-ELSE statement
• The THEN and ELSE parts have to be executed
  sequentially

4
SIMD and Branches

• A traditional SIMD computer
• IF ( a parallel condition)
• THEN Statement Block A
• ELSE Statement Block B
• Suppose we have 14 PEs

5
SIMD and Branches

• A traditional SIMD computer
• IF ( a parallel condition)
• THEN Statement Block A
• ELSE Statement Block B
• Suppose we have 14 PEs

6
SIMD and Branches

• A traditional SIMD computer
• IF ( a parallel condition)
• THEN Statement Block A
• ELSE Statement Block B
• Suppose we have 14 PEs

7
SIMD and Branches

• A traditional SIMD computer
• IF ( a parallel condition)
• THEN Statement Block A
• ELSE Statement Block B
• Suppose we have 14 PEs

8
SIMD and Branches

• Question How can we improve the execution of the
  branches if we can have more than one instruction
  stream?
• One probable answer Execute each part of the
  branches simultaneously by using each of the
  instruction streams

9
SIMD and Branches

• The MASC computational model
• IF ( a parallel condition)
• THEN Statement Block A
• ELSE Statement Block B
• Suppose we have 14 PEs

10
SIMD and Branches

• The MASC computational model
• IF ( a parallel condition)
• THEN Statement Block A
• ELSE Statement Block B
• Suppose we have 14 PEs

11
Outline

• SIMD and Branches
• Single Instruction Multiple Data (SIMD)
• A data parallel program contains branches
• MASC Computational Model
• Multiple Associative Computing (MASC) Model
• MASC model with manager-worker paradigm
• The power of MASC model
• With variations of MASC
• With other models
• ASC language compiler support for the MASC model
• MASC Algorithm
• Shapes algorithm
• Modified ASC Quick Hull algorithm for the MASC
  model with manager-worker paradigm

12
MASC Computational Model

• An extension of the associative computing model
  or ASC
• ASC model was created to capture Aspro and Staran
  s style of programming
• The associative properties
• Broadcast data in constant time
• Constant time global reduction of
• Boolean values using AND/OR
• Integer values using MAX/MIN
• Constant time data search
• Provides content addressable data
• Eliminates need for sorting and indexing.
• Pick one responder in constant time
• Supported in hardware with the broadcast and
  reduction network

13
MASC Computational Model

• The basic components of the model
• A set of instruction streams
• An array of cells (PE Memory)
• IS-to-Cell broadcast and reduction networks (one
  for each IS, preferable)
• An IS-to-IS Network
• A simple cell network

14
MASC Using Manager-Worker Paradigm

• A variation of the MASC model
• Two types of ISs
• A manager-IS (ID 0 )
• Managing the work pool of tasks
• Coordinating and assigning subtasks (using a FORK
  operation)
• Combining finished tasks (using a JOIN operation)
  from worker-ISs
• Identical worker-ISs (ID 1 to ID m)
• Executing tasks in an associative (e.g., data
  parallel, SIMD) fashion using the PEs currently
  assigned to it

15
MASC Using Manager-Worker Paradigm

• The IS network
• An IS-to-IS broadcast/reduction network
• The manager-IS can use the network to perform a
  Min/MAX or logical reduction on ISs in constant
  time (pick one idle worker-IS)
• The cell network is optional

16
MASC Using Manager-Worker Paradigm

• A Cell is a simple ALU its local memory
• IS-Selector Register (an (lg m)-bit register if
  there are m ISs)
• Holding the instruction stream ID, to which that
  PE is currently listening
• The register can be set or reset by the
  instruction stream that the PE is listening to or
  by the data tested in that PE
• Task-History Stack (use the memory in each cell)
• Holding the task ID
• The default content is empty
• The top of the stack always shows
• the current task the PE is currently executing,
• the task that has just been finished, or
• a new task that has not yet assigned to a
  worker-IS
• At any point in time, each PE listens to exactly
  one IS

17
MASC Using Manager-Worker Paradigm

• A task is broken down into subtasks
• No interaction between subtasks during their
  executions
• A FORK operation
• Generates one or more subtasks from a branch by
  partitioning PEs into group based on the parallel
  condition
• New task id will be push into the Task-History
  Stack
• Those subtasks will be assigned to worker-ISs to
  be executed concurrently by setting the
  IS-Selector register of PEs in the corresponding
  group to the ID of the worker-IS
• A JOIN operation
• Recombines subtasks into the original parent task
  (i.e., the one existing prior to the fork) after
  they have been successfully executed by popping
  top of the Task-History Stack

18
MASC Using Manager-Worker Paradigm

• A work pool (WP-Q)
• Containing tasks ready to be executed

Worker-IS
Worker-IS
Work Pool
19
Fork Operation
Wittaya Chantamas, 08/24/2004
20
Join Operation
Wittaya Chantamas, 08/24/2004
21
The Power of MASC Model

• Among the variations of the MASC model, the
  original MASC model with a simple cell network
  (1-d, 2-d, or hypercube) has the same power as
• A MASC model without any cell network
• 1-d cell network can be simulated in the MASC
  without any cell network in O(1) with a
  polynomial blow-up in size (PEs and ISs)
• A proof of the 2-d and hypercube network case is
  similar to the case of 1-d cell network
• A MASC model with manager-worker paradigm (We
  believed! Need further proof.)

22
The Power of MASC Model

• Comparing to other models, the MASC model
• has the same power as
• Basic and Segmenting Reconfigurable Multiple Bus
  Machine (RMBM)
• CRCW-PRAM
• A restriction version of RM
• A Mesh with Multiple Broadcasting (MMB)
• is less powerful than
• Fused and Extended RMBM
• Reconfigurable Mesh (RM)
• Linear Mesh (LM)

23
ASC Language Compiler Support for the MASC Model

• The MASC model needs a multiple IS support from
  the ASC
• An extension of the ASC language compiler for the
  MASC model
• A MASC directive
• Concurrent data parallel executions of different
  paths in a branch can be achieved by using the
  directive
• / .masc fork /
• A user has a tight control
• Not all different paths in branches will be
  executed concurrently
• Only those in branches with directives will
• Considered as a comment by the ASC compiler (will
  show in .lst file, not show in .iob file)
• No need for a new ASC compiler in order to run an
  ASC program in MASC system
• Need another extension if wanted to add a
  parallel case statement support

24
A parallel IF-THEN-ELSE statement in the ASC
language

• IF condition expression
• THEN statement block A
• ELSE statement block B
• ENDIF

25
a structure code

• main test
• int parallel b, c, d
• logical parallel BCD
• associate b, c, d with BCD
• read b c d in BCD
• b c 2
• c d - 3
• / will be no fork here /
• if (b .lt. c)
• then
• b c
• d 4
• else
• c b
• b d
• endif
• c d

M100 0000
.MI_BEGIN W1100000 beg_of_stmt 1c00 6 0
beg_read 5a00 SYSOT BCD B,C,D, beg_of_stmt
1c00 20 0 mvpa_ 4812 C D .MI_END
W1100000
M100 0000
W110 0000
W110 0000
M111 0000
M111 0000
26
A parallel IF-THEN-ELSE statement in the ASC
language

• / .MASC fork /
• IF condition expression
• THEN statement block A
• ELSE statement block B
• ENDIF

27
a structure code
M100 0000

• main test
• int parallel b, c, d
• logical parallel BCD
• associate b, c, d with BCD
• read b c d in BCD
• b c 2
• c d - 3
• /.MASC FORK /
• if (b .lt. c)
• then
• b c
• d 4
• else
• c b
• b d
• endif
• c d

M100 0000
W110 0000
.MI_BEGIN W1112000 beg_of_stmt 1c00 16 0
mvpa_ 4812 B C beg_of_stmt 1c00 17 0
mvpa_ 4812 D B .MI_END W1112000
W110 0000
M111 0000
M111 0000
W111 1000
W111 1000
W111 2000
W111 2000
W111 X100
W111 X100
M111 X110
M111 X110
28
Outline

• SIMD and Branches
• Single Instruction Multiple Data (SIMD)
• A data parallel program contains branches
• MASC Computational Model
• Multiple Associative Computing (MASC) Model
• MASC model with manager-worker paradigm
• The power of MASC model
• With variations of MASC
• With other models
• ASC language compiler support for the MASC model
• MASC Algorithm
• Shapes algorithm
• Modified ASC Quick Hull algorithm for the MASC
  model with manager-worker paradigm

29
Shape Problem

• The testing problem
• To compute area of basic shapes in a database
• Can use the MASC model to solve this problem
• Each type of shapes required different equation
  to compute the area
• Areas of each shape types can be compute
  simultaneously by partitioning PEs in to groups
  (triangle, rectangle, or circle) and using one IS
  to compute the areas for each group

30
MASC Quick Hull Algorithm

• The convex hull problem
• The convex hull of a set of points S is the
  smallest convex set containing S. In particular,
  each point of set S is either on the boundary of
  or in the interior of the convex hull
• Modified ASC Quick Hull algorithm for the MASC
  model with a limited number of ISs and using
  manager-worker paradigm with work pool

31
MASC Quick Hull Algorithm

• Algorithm MASC Quick Hull (for the upper hull)
• Input A set of points S given as (x,y)
  coordinates, each PE holds one point in S
• Output vertices of the upper convex hull
• The manager assigns the initialization task
  (i.e., task 0) to a worker IS to find two extreme
  points, X-min point (w) and X-max point (e)
• Two points (w and e) in the convex hull are
  identified
• The manager creates task we and places it in the
  work pool. The PEs associated with this task are
  the ones whose point lies above segment we

32
MASC Quick Hull Algorithm

• The manager assigns each task pq in the work pool
  to a worker IS to find another point in the
  convex hull using the PEs assigned to this task.
• Another point (r) in the convex hull is
  identified
• The manager places task pr and task rq in the
  work pool. The PEs associated with each task are
  the ones whose point lies above corresponding
  line segment
• The manager continues to execute 2 steps above
  until there are no active tasks and no tasks
  remain in the work pool, and then terminates the
  algorithm

33
MASC Quick Hull Algorithm
W-IS Execute Task PR
M-IS Join Task PR
M-IS Fork Task 0
M-IS Join Task 0
M-IS Fork Task WE
M-IS Join Task WE
M-IS Fork Task PR and Task RQ
T pr
J
T 0
T we
F
F
J
J
F
T rq
J
W-IS Execute Task 0
W-IS Execute Task WE
W-IS Execute Task RQ
M-IS Join Task RQ
34
MASC Quick Hull Algorithm

• Timing
• n is the number of points in S and m is the
  number of instruction streams
• Still O(n) in the worst case
• If we assume that on the average O(lg n) is the
  number of convex hull points, the average case
  running time is O((lg lg n)(lg n)/m)
• Producing a constant speedup of approximately m
  over the 1-IS version of the same algorithm for
  the average case

35
Conclusion

• Traditional SIMD executes each part of branches
  of a data parallel program sequentially
• MASC can execute most or all parts of the
  branches simultaneously if there are enough
  instruction streams
• The original MASC model with a simple cell
  network is as powerful as a model without any
  cell network or with manager/worker paradigm
• The MASC model is as powerful as many
  computational models such as PRAM and some
  versions of RMBM
• An extension of the ASC compiler is required to
  take the benefit of having multiple ISs
• Some problems can take the advantage of having
  more than one instruction stream. Some do not.