Lecture 24: Interconnection Networks

1
Lecture 24 Interconnection Networks

• Topics communication latency, centralized and
• decentralized switches (Sections 8.1 8.5)

2
RAID Summary

• RAID 1-5 can tolerate a single fault mirroring
  (RAID 1)
• has a 100 overhead, while parity (RAID 3, 4,
  5) has
• modest overhead
• Can tolerate multiple faults by having multiple
  check
• functions each additional check can cost an
  additional
• disk (RAID 6)
• RAID 6 and RAID 2 (memory-style ECC) are not
• commercially employed

3
I/O Performance

• Throughput (bandwidth) and response times
  (latency)
• are the key performance metrics for I/O
• The description of the hardware characterizes
  maximum
• throughput and average response time (usually
  with no
• queueing delays)
• The description of the workload characterizes
  the real
• throughput corresponding to this throughput
  is an
• average response time

4
Throughput Vs. Response Time

• As load increases, throughput increases (as
  utilization is
• high) simultaneously, response times also go
  up as the
• probability of having to wait for the service
  goes up
• trade-off between throughput and response time
• In systems involving human interaction, there
  are three
• relevant delays data entry time, system
  response time,
• and think time studies have shown that
  improvements
• in response time result in improvements in
  think time ?
• better response time and much better throughput
• Most benchmark suites try to determine
  throughput while
• placing a restriction on response times

5
Estimating Response Time

• Queueing theory provides results that can
  characterize
• some random processes
• Littles Law Mean number of tasks in system
• Arrival rate x mean response time
• The following two results are true for workloads
  with
• interarrival times that follow a Poisson
  distribution
• P(k tasks arrive in time interval t) e-a ak /
  k!
• Timequeue Timeserver x server
  utilization/(1-server utilization)
• Lengthqueue server utilization2 / (1 server
  utilization)

6
Interconnect Types
WAN
LAN
SAN
Bus
1
10
100
1000
10000
Number of autonomous systems connected
7
Communication

• Applications typically send messages through the
  OS
• Steps to send a message
• copy data into an OS buffer
• OS attaches header and trailer
• OS sends transfer command to network interface
  hw
• Steps to receive a message
• copy data from network interface hw into OS
  buffer
• make sure data is correct, send acknowledgment
• copy data into user space and signal the
  application
• If sender does not see an ack in time, it
  resends the data

8
Latency

• Sender overhead the processing delays to send
  the
• application data to the network interface
  hardware
• Time of flight latency for the first bit to
  travel from sender
• to receiver function of distance and speed of
  light
• Transmission time time between arrival of first
  and last
• bit function of message size and network
  bandwidth
• Transport latency sum of time of flight and
  transmission time
• Receiver overhead processing delays to send
  data from
• hw to application usually longer than sender
  overhead

9
Topologies

• Internet topologies are not very regular they
  grew
• incrementally
• Supercomputers have regular interconnect
  topologies
• and trade off cost for high bandwidth
• Nodes can be connected with
• centralized switch all nodes have input and
  output
• wires going to a centralized chip that
  internally
• handles all routing
• decentralized switch each node is connected to
  a
• switch that routes data to one of a few
  neighbors

10
Centralized Crossbar Switch
P0
Crossbar switch
P1
P2
P3
P4
P5
P6
P7
11
Centralized Crossbar Switch
P0
P1
P2
P3
P4
P5
P6
P7
12
Crossbar Properties

• Assuming each node has one input and one output,
  a
• crossbar can provide maximum bandwidth N
  messages
• can be sent as long as there are N unique
  sources and
• N unique destinations
• Maximum overhead WN2 internal switches, where W
  is
• data width and N is number of nodes
• To reduce overhead, use smaller switches as
  building
• blocks trade off overhead for lower effective
  bandwidth

13
Switch with Omega Network
P0
000
000
P1
001
001
P2
010
010
P3
011
011
P4
100
100
P5
101
101
P6
110
110
P7
111
111
14
Omega Network Properties

• The switch complexity is now O(N log N)
• Contention increases P0 ? P5 and P1 ? P7 cannot
• happen concurrently (this was possible in a
  crossbar)
• To deal with contention, can increase the number
  of
• levels (redundant paths) by mirroring the
  network, we
• can route from P0 to P5 via N intermediate
  nodes, while
• increasing complexity by a factor of 2

15
Tree Network

• Complexity is O(N)
• Can yield low latencies when communicating with
  neighbors
• Can build a fat tree by having multiple incoming
  and outgoing links

P0
P3
P2
P1
P4
P7
P6
P5
16
Bisection Bandwidth

• Split N nodes into two groups of N/2 nodes such
  that the
• bandwidth between these two groups is minimum
  that is
• the bisection bandwidth
• Why is it relevant if traffic is completely
  random, the
• probability of a message going across the two
  halves is
• ½ if all nodes send a message, the bisection
• bandwidth will have to be N/2
• The concept of bisection bandwidth confirms that
  the
• tree network is not suited for random traffic
  patterns, but
• for localized traffic patterns

17
Distributed Switches Ring

• Each node is connected to a 3x3 switch that
  routes
• messages between the node and its two neighbors
• Effectively a repeated bus multiple messages in
  transit
• Disadvantage bisection bandwidth of 2 and N/2
  hops on
• average

18
Distributed Switch Options

• Performance can be increased by throwing more
  hardware
• at the problem fully-connected switches every
  switch is
• connected to every other switch N2 wiring
  complexity,
• N2 /4 bisection bandwidth
• Most commercial designs adopt a point between
  the two
• extremes (ring and fully-connected)
• Grid each node connects with its N, E, W, S
  neighbors
• Torus connections wrap around
• Hypercube links between nodes whose binary
  names
• differ in a single bit

19
Topology Examples
Hypercube
Grid
Torus
20
Title

• Bullet