Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks

1
Parallel Self-Adaptive Parallel Processing Neural
Networks with irregular Nodal Processing Powers
using Hierarchical Partitioning of Artificial
neural Networks

• Rahul Kala,
• Department of Information Technology
• Indian Institute of Information Technology and
  Management Gwalior
• http//students.iiitm.ac.in/ipg_200545/
• rahulkalaiiitm_at_yahoo.co.in, rkala_at_students.iiitm.a
  c.in

Kala, Rahul, Shukla, Anupam Tiwari, Ritu (2009)
Self-Adaptive Parallel Processing Neural Networks
with irregular Nodal Processing Powers using
Hierarchical Partitioning,Neural Network World
Journal, Vol 19, No. 6, pp 657-680
2
Problems with Back Propagation Algorithm
3
The Basic Idea
4
Motivations from Human Brain
5
Basic Partitioning Techniques
6
Dataset Partitioning

• Divide data into PEs
• Each PE trains its set of data
• Weights are exchanged and aggregated

7
Basic Technique
8
Communication
PE 1
PE 2
PE N
PE 3
PE 4
9
Layer Partitioning -1

• Divide layers into PEs
• Each PE performs its part of computation
• Different PEs store different weights

10
Layer Partitioning -II

• Feed forward Basic Equation
• Feed back

11
Basic Technique
PE 2
PE N
PE 1
i1
i2
in
12
Communication
13
Working
14
Node Partitioning - I

• Divide nodes into PEs
• Each PE performs its part of computation
• Different PEs store different weights

15
Node Partitioning -II

• Feed forward Basic Equation

• Feed Back

16
Basic Technique
PE 1
i1
PE 2
i2
PE N
in
17
Communication
18
Working
19
Hierarchical Partitioning

• Mixture of three partitioning at different levels
• Level 1 Data Set
• Level 2 Node or Layer

20
General technique
Node 3
Node 2
Node 1
Dataset 1
Layer 2
Layer 1
Dataset 2
Layer 3
Dataset N
Node 1
Node 2
Dataset 3
Dataset 4
Node 3
Node 1
Layer 1
Layer 2
Node 2
Node 3
Layer 3
21
Self Adaptation

• Balance computational load among PEs as per their
  capability
• Works for data set partitioning
• Computation reallocated with some frequency

22
Self Adaptation Server/Client Model
PE1
Server
PE2
PEN
PE3
PE4
23
Communications
Data Set N
Data Set 1
Data Set 2
Data Set 3
Server
Node1
Node1
Layer1
Layer1
Node2
Node2
Layer2
Layer2
Node3
Node3
Layer3
Layer3
24
RESULTS
25
Speedup v/s No of PEs (Input 1)
26
Speedups for Input 1
PE No of inputs Network Architecture Iterations Server Sync after iterations Time Serial Time Parallel Speedup
2 500 8-15-1 150000 50000 823737 982521 0.838391
4 500 8-15-1 150000 50000 823737 530306 1.553324
6 500 8-15-1 150000 50000 823737 384294 2.143507
8 500 8-15-1 150000 50000 823737 263511 3.126006
10 500 8-15-1 150000 50000 823737 211249 3.899365
20 500 8-15-1 150000 50000 823737 89225 9.232132
27
Speedup v/s No of PEs (Input 1) without self
adaptive approach
28
Speedups for Input 1 without self adaptation
PE No of inputs Network Architecture Iterations Server Sync after iterations Time Serial Time Parallel Speedup
2 500 8-15-1 150000 NA 823737 993483 0.829141
4 500 8-15-1 150000 NA 823737 674358 1.221513
6 500 8-15-1 150000 NA 823737 473899 1.738212
8 500 8-15-1 150000 NA 823737 342174 2.407363
10 500 8-15-1 150000 NA 823737 303214 2.716685
20 500 8-15-1 150000 NA 823737 97180 8.476405
29
Speedup v/s No of PEs (Input 1I)
30
Speedups for Input 1I
PE No of inputs Network Architecture Iterations Server Sync after iterations Time Serial Time Parallel Speedup
2 500 11-25-1 150000 50000 1733690 1829312 0.947728
4 500 11-25-1 150000 50000 1733690 984654 1.76071
6 500 11-25-1 150000 50000 1733690 644070 2.691773
8 500 11-25-1 150000 50000 1733690 486311 3.564982
10 500 11-25-1 150000 50000 1733690 429221 4.039155
20 500 11-25-1 150000 50000 1733690 179378 9.665009
31
Speedup v/s No of PEs (Input 1I) without self
adaptive approach
32
Speedups for Input 1I without self adaptation
PE No of inputs Network Architecture Iterations Server Sync after iterations Time Serial Time Parallel Speedup
2 500 11-25-1 150000 NA 1733690 1911289 0.907079
4 500 11-25-1 150000 NA 1733690 1175223 1.475201
6 500 11-25-1 150000 NA 1733690 683619 2.536047
8 500 11-25-1 150000 NA 1733690 548465 3.160986
10 500 11-25-1 150000 NA 1733690 471293 3.678582
20 500 11-25-1 150000 NA 1733690 196229 8.835035
33
Thank You