Bridging the Gap

1
Bridging the Gap
Presented to NORDUnet 2006 Gothenburg,
Sweden September 26, 2006

• Jerry Sobieski
• Director, Research Initiatives
• Mid-Atlantic Crossroads (MAX)

2
Bridging the Gap
Presented to NORDUnet 2006 Gothenburg,
Sweden September 26, 2006

• Jerry Sobieski
• Director, Research Initiatives
• Mid-Atlantic Crossroads (MAX)

3
What Gap?
4
Bridging the Gap

• The Technical Challenge
• Where do we need to be in terms of technology
  and network capabilities in 2016?
• What resources are available to build the bridge
  to that future?
• The Organizational challenge
• Given the technical objectives, how do we
  construct Bridge building organizations to
  achieve those objectives?
• What is the role of the network organization in
  the 2016?

5
Scoping the Problem

• Grid applications will incorporate in excess of
  100,000 processors within 5 years.
• Dr. Larry Smarr, On Vector Workshop, UCSD Feb
  2006
• The Global Information Grid will need to store
  and access exabytes of data on a realtime basis
  by 2010
• Dr. Henry Dardy, Optical Fiber Conference, Los
  Angeles, CA USA, Mar 2006
• Each LHC experiment foresees a recorded raw data
  rate of 1 to several PetaBytes/year
• Dr. Harvey Neuman (Cal Tech)
• US Bancorp backs up 100 TB financial data every
  night now.
• David Grabski (VP Information Tech. US Bancorp),
  Qwest High Performance Networking Summit, Denver,
  CO. USA, June 2006.
• The VLA facility is now able to generate 700
  Gbps of astronomical data and will reach 3.2
  Terabits per second by 2009.
• Dr. Steven Durand, National Radio Astronomy
  Observatory, E-VLBI Workshop, MIT Haystack Obs.,
  Sep 2006.

6
Large Scale Distributed Cluster Computing

• The Smarr example 100,000 Processors
• Using todays dual core technology 50K nodes
• Simple 11 bisection 25,000 point-to-point
  links
• A single Gigabit Ethernet interface per node
• 25 Terabits/second total bisection bandwidth
  unidirectional.
• 1 gigabit/node burst 25 Tbps aggregate
• gt 2500 seconds (40 minutes) funneled thru a
  10 GE
• 1 MB/node 25K nodes 25 GB burst 200 Gbits
  20 seconds over 10GE
• A single 9000B packet/node 225MB 1.8 Gbits
  180 ms on a 10GE
• 10x slower than disk
• Real applications employ local storage, but they
  typically have much more complex inter-processor
  communications patterns and requirements

7
Large Scale Distributed Cluster Computing

• Another example 10K processor compute cluster
• 4 Gbyte/processor memory,
• 1GE network interface per processor,
• Checkpoint / mirror this data to a remote storage
  facility 100 km away.
• Burst capability
• 10K processors 1 Gbps 10 Tbps (!)
• Data transfer time
• Per processor 4GB _at_ 1Gbps 32 seconds
• Aggregate 4GB 10K 40 TeraBytes total data to
  be moved
• 40 TBytes _at_ 40 Gbps ? 2.2 hours
• 40 TBytes _at_ 100 Gbps gt 53 mins

8
Clearly, these issues will be challenging

• Parallel and distributed clusters are
  incorporating more nodes faster than Moores Law
  is reducing their size..
• Coordinating 105 MIMD processors is challenging
  just within a single cluster
• How do you power 100K processors? _at_ 200W / node
  50K200W
• 10 Megawatts ! (2000 homes)
• How big a room would you need? _at_160
  cpus/rack100K/160
• 600 racks ! (just for the cpu bladeswhat
  about disks, comms, etc)
• How do you protect the investment? _at_1000/node
  50K nodes 50,000,000 USD
• Centralized clusters of this magnitude will be
  rareLarge super clusters will be constructed
  from confederations of lesser clusters,
  distributed over a large geographic space and
  across many organizations.
• How will they communicate?
• This is the network challenge facing the
  computational research community...and a noble
  quest for the networking community.
• What if in 10 years we are working with much
  larger collaborating virtual organizations?

9
A Related Question

• Given the proliferation of 10G waves, the recent
  availability of 40G technology, the promise of
  100G on the near horizon, and networks with huge
  raw (low level) capacity on the ground now
• Why are we constrained to 10Gbps flows today?
• Why do users/applications need to go to so much
  trouble to get effective use of a single 1GE
  interface (never mind 10G or more)?
• These reflect some fundamental design
  decisions/assumptions of the existing (original)
  internet architecture that may not be applicable
  today
• Its time to look again at how we would like to
  build the Network for future (NOT how would
  you fix the Internet?wrong question)
• The GENI initiative (NSF) hopes to construct a
  Global Environment for Network Innovation
• Envisioned as a 10 yr program, 200M to 400M
  USDstill being formulated

10
Other IETs(Impending Explosive Technologies)

• Near Term
• Hybrid Networking
• Sensor Networks and GRID Integration
• Security and Privacy
• Disaster Resistant Architectures and Continuance
  of Operation
• Driven by 9/11 and Katrina in the US
• Wireless Access and Mobility
• In the future, any product that costs over 10
  will be network addressable.
• Satellite Communications
• Global reach at high speed, space based
  applications (e.g. E-VLBI)
• In the Way Out
• Nano-engineering and self organizing
  communicating motes

11
E-Science ApplicationsThe Dynamic Virtual Global
Collaboratory

• Consider the emerging e-science paradigm
• Global science
• E.g. astrophysics, astronomy, earth sciences,
  climate modeling, etc.
• Global shared resources
• Large Hadron Collider, radio telescopes, polar
  research stations, computational resources, etc.
• Global collaborating science teams
• E-VLBI, HEP, Genomic Research, etc
• These affinity groups combine resources and
  people into a globally distributed virtual
  collaborating organizations to pursue a common
  discipline or objective.

12
Application Specific CollaboratoriesThe E-VLBI
poster child example
Mark 5
Correlator/Compute Cluster
Global RE Hybrid Infrastructure
Mark 5
Visualization station
13
IGrid 2005
14
E-VLBI Application Specific Network
Mark 5
Correlator/Compute Cluster
Global RE Hybrid Infrastructure
Mark 5
15
E-VLBI Application Specific Network
16
The Very Large Array (VLA)

• 27 Antennae
• 120 Gbps each 3.2 Terabits/sec

17
The Technology Bridge

• To use the network transport and switching
  technologies as an example
• Clearly (at least IMHO ?) 40 Gbps or even 100
  Gbps transmission technologies will not be
  adequate to meet the needs 4 or 5 years from now

bigger.

• We need to think

18
Can our networks go faster?

• What if we could move data at Terabits per second
  rather than Gigabits per second?
• The Smarr Example reduces to
• 25K nodes burst 1 Gbit 25 Tbit burst
• ? 25 seconds _at_ 1 Tbps
• -gt 6 seconds _at_ 4 Tbps
• -gt 1 second _at_ 25 Tbps
• Now were talkin!

19
Are Multi-Terabit Link Speeds Possible?

• Dr. Toshio Morioka (NICT) has generated 1000
  waves in the C band on 6 GHz spacing, each wave
  modulated at 2.5 Ghz. -gt 2.5 Tbps.
• Dr. Keren Bergman (Columbia University) has
  demonstrated a photonic switch that can forward
  packets at 160 Gbps - per port.
• The architecture can (in theory) scale to
  hundreds of Tbps/port.
• Many other groups are working on similar OPS
  technologies
• 320 Gbps per wave is possible
• Other approaches to photonic data switching and
  all photonic wavelength switching and translation
  are being explored
• There remains much to be done yet to mature these
  technologiesbut these examples show that these
  technologies are indeed possible and are
  potential bridge material.

20
The Bridge Organization
21
Why do we need a new Bridge?
22
Why Do We Need a New Bridge?

• As the Keepers of the Network Infrastructure, we
  need to expand our perception of our role in the
  future
• The network itself is becoming an anachronistic
  concept
• The purity of the 7 layer model has been broken
• The term Demarc limits our thinking
• The production mindset of stability and
  availability as the prime directive is an old
  school commercial concept
• Past about two 9s, other network issues tend to
  dominate (e.g Thruput, access, security, etc.)
• A rabid need to drive costs down will squeeze
  operating margins
• If our members leave simply because someone
  else offered cheaper servicewe have real
  problems. We are not providing the right
  value.
• If cost is the prime driver, we will end up
  competing with the commercial ISPs and we will
  lose.
• We need to retain the financial capabilities to
  adapt new technologies before there is a clear
  financial ROIi.e. we are not driven by
  profitablity and shareholder equity.
• Advanced capabilities are the product that
  advanced networks deliver This is what
  production should mean for RE networks

23
Next issue What type of Bridge?

• My perspective
• Our role is to help build the overall Cyber
  Infrastructure for tomorrow,
• Our focus is the network components
• But network architecture, design, engineering and
  operations are only part of the whole
• We need to routinely develop and deploy new
  network centric applications and service
  capabilities,
• We need to seamlessly integrate other cyber
  components (Computational, storage, viz, sensors,
  etc.)
• We need to have this broader organization
  interacting and collaborating with the research
  community
• The Bridge to the future will be collaborative
  organizations and initiatives that can perceive
  the entire global cyber infrastructure as the
  Objective not just the network, or the
  supercomputers, or the raid arrays,

24
The Bridge Organization

• The Bridge Organization is different from
  most existing large network organizations
• It has a culture that treats the foundational
  service concepts of network stability,
  reliability, and availability as only one measure
  of success, and maybe not even in the traditional
  fashion. It sees adaptive and integrated network
  resources, and a vibrant collaborative
  future-facing cyber-infrastructure organization
  as the more important mission
• It has a culture that allows, encourages, and
  even requires direct and intimate involvement in
  the development and deployment of new ideas and
  applications, i.e. it works above and below the
  network layer with as much expertise, energy, and
  focus as it does at the network layer.
• It collaborates (not just supports) research
  affinity groups who are stretching the
  capabilities and concepts of networking.
  Experimental deployments of new technologies and
  capabilities are part of the standrad and routine
  service provided to the community.

25
The Bridge to the Future
Users and Applications
Ridiculous, Unbelievable, Unfounded Requirements
of the Lunatic Fringe Future
Regional Networking Organizations
Existing Services off-the-shelf Technologies
Network Telecom Research
26
Summary

• The network of 10 years hence poses tremendous
  (and truly exciting!) technical challenges
• We need to formulate a strategic Technical Bridge
  that allows our brightest technical resources to
  define and guide the future by helping create it.
• We can not wait for these technologies to become
  GA products before we field them
• We need to change our mindset to create Bridge
  Organizations
• Organizations that incorporate a broad spectrum
  of communities into the creation of the regional
  and global cyber-infrastructure and technologies
  for the future

27
What Type of Bridges Shall we Build?
28
Tack så mycket