Layered Routing in Irregular Interconnection Networks

1
Layered Routing in Irregular Interconnection
Networks

• Olav Lysne, Tor Skeie, Sven-Arne Reinemo and
  Ingebjørg Theiss
• Simula Research Laboratory

2
What is an interconnection network?
3
What is an interconnection network

• High speed.
• Extremely low latency.
• Short distance few tens of meters.
• Safe communication the network will never drop
  information.
• Used within computers, within switches and within
  routers.

4
Interconnection networks in Norway

• Dolphin Interconnect Solutions
• Since approx 1990.
• Ships interconnects based on the SCI standard
• From the ashes of Norsk Data
• Scali supercomputers
• From Kongsberg Information Control
• Ships clusters and support software for
  supercomputers.
• Cooperates tightly with Dolphin
• SUN Microsystems
• Bought staff and licensed technology from Dolphin
• Development center at Skullerud
• Simula Research Laboratory
• Working on these issues from the start
• The second most productive academic research
  group in Europe on Interconnection Networks.

5
This work is

• Olav Lysne, Tor Skeie, Sven-Arne Reinemo and
  Ingebjørg Theiss, Layered Routing in Irregular
  Networks, Submitted to IEEE Transactions on
  Parallel and Distributed Systems.
• Referee-comments have been received and revision
  is in progress.
• A compilation of four published papers
• Olav Lysne and Tor Skeie, Load balancing of
  irregular system area networksthrough multiple
  roots, Proceedings of the International
  Conference on Communication in Computing, CIC
  2001, CSREA Press, pages 165 -171
• Tor Skeie, Ingebjørg Theiss and Olav Lysne,
  Evaluation of Minimal Deterministic Routing in
  Irregular Networks, Proceedings of the SSGRR
  International Conference on Infrastructure for
  e-Business, e-education, e-Science, and
  e-Medicine (SSGRR) 2002
• Tor Skeie, Olav Lysne and Ingebjørg Theiss,
  Layered Shortest Path (LASH) Routing in Irregular
  System Area Networks, Proceedings of
  Communication Architecture for Clusters (CAC02),
  IEEE Computer society 2002.
• Ingebjørg Theiss, Tor Skeie and Olav Lysne,
  Minimal Routing in Irregular Networks,
  Proceedings of the International Conference on
  Communications in Computing (CIC) 2002, IEEE
  Computer Society

6
Link level flow control

• Challenges
• Irregularity introduced by topology-agnostic
  technologies, and System Area Network COTS.
• Deadlock is an issue
• Shortest path routing generally hard in irregular
  networks.

C
D
A
B

• Link level flow control
• Mainly used in interconnects and System area
  networks (Myrinet, Servernet, AutoNet, 3GIO, AS,
  InfiniBand)

7
Network Layers

• Physical channels are divided into a number (N)
  of virtual channels
• Infiniband has up to 16, PCI-AS has up to
  16,interconnect in BlueGene/L has 4,
• These channels have mainly been provided for
  other purposes in most technologies
• Assuming N virtual channels per physical, view
  the network as N virtual networks, that are
  logically disjoint.
• Each of these networks are in the following
  called a layer.

8
Main idea

• Assign each source destination pair to one layer,
  such that
• Each individual layer is deadlock free.
• Each individual layer need not be fully
  connected, but the sum of all layers should be.
• Through this we obtain routing flexibility
  previously not inherent in deadlock prone
  technologies.
• This flexibility is in the paper used to obtain
  higher throughput in the network

9
Two new classes of routing strategies methods

• Load-balancing - MROOTS
• One fully connected routing function in each
  layer (Up\Down)
• Known hot spots (Up\Down root) are distributed
  as far from each other as possible.
• Shortest path LAyered SHortest Path - LASH
• Each layer has a subset of the source destination
  pairs, for which the deadlock problem does not
  appear
• No layer is fully connected
• The union of all layers is fully connected.

10
The method has been evaluated for three classes
of networks

• Deterministically routed networks
• All source destination pairs have but one
  possible path
• InfiniBand, Myrinet, most variants of Ethernet
• Source adaptive networks
• Multiple paths possible between sources and
  destinations, but the path for each
  packet/connection is decided a priori by the
  source (source routing)
• Advanced Switching
• Switch-adaptive networks
• Multiple paths possible between sources and
  destinations, and the path decisions are taken on
  the fly by the switches.
• BlueGene\L

11
Evaluated routing methods
12
Results deterministic routing
13
Results Source Adaptive Routing
14
Results Switch Adaptive Routing
15
Conclusions

• We have developed a new class of deadlock free
  routing methods
• Outperforms all prior art routing-functions in
  the deterministic as well as the source-adaptive
  case
• In the switch-adaptive case, we are not better
  than existing methods.
• With the enhancement we give for deterministic
  routing, source adaptivity is no longer suited
  for performance enhancement
• Source adaptivity is, however, still useful for
  fault tolerance.