DOE-UltraScience Net (

1
DOE-UltraScience Net ( network infrastructure)
Update

• JointTechs Meeting
• February 15, 2005
• W. R. Wing

2
Motivation - DOE Needs Extreme Networking

• 160-200 Gbs throughput by 2008
• Probably only achievable by circuit bonding
• 0.1 packet re-order and jitter control
• Probably only achievable at SONET layer
• Line rate, provably-secure, connections
• Only available at SONET or optical layer

3
The Well Known Bandwidth Problem

• DOE needs 160 - 200 Gbs by 2008
• Typical multi-stream throughput limited to 25/-
  Gbs, and programming multi-steam is hard - hero
  efforts yield 30 Gbs
• Typical single-steam throughput to a single
  application 1Gbs, even on a tuned network - hero
  effort yields 5-6 Gbs
• Networks will soon offer 40 Gbs channels
• Firewalls throw it away anyway

4
Even Without Firewalls -We Have a Technology
Problem
DOE needs 160-200 Gbs by 2008 This is
on the blue line This isnt a problem
routers are going to fix - bonded IP
channels ONLY work with VERY large
numbers of streams (as Teragrid has
discovered) Bonding at layer-2 (Ethernet)
is just as bad Solution is circuit
switching and Layer-1 bonding
5
Jitter Control

• Needed to deconstruct data sets, needed for
  remote vis, and for remote instrument control or
  steering
• Routers and switch-routers are path
  deterministic, but not time-deterministic
• Only at layer-1 (or SONET) do you get
  time-determinism

6
Security

• Security requirements will get more and more
  onerous
• Solutions involving IPSec, VPNs, and encryption
  have more and more trouble running at line speed
• Alternate solution is to use optical/SONET
  circuits with provable, known end-points and
  inherent immunity to injected traffic
• Again - only available at layer-1

7
In Summary

• Only at layer-1 do you get
• Zero packet re-ordering
• Zero jitter
• Zero drops due to Congestion
• Known (by definition) paths and end points
• SONET (or below) can do this
• Ethernet switches and switch/routers cant
• MPLS cant
• Layer-1 circuits can
• Bypass firewalls
• Carry non-IP frames (e.g., Fiber-Channel over
  SONET)
• Easily (transparently) support parallel, bonded
  circuits

8
MPLS Jitter - ORNL / Atlanta
9
How do we get there from here?

• It requires a research network we control at
  least down to the SONET layer
• It requires a research network with significant
  span
• It requires a research network with at least two
  lambdas

10
UltraScience Net Research Testbed

• Building an extended-regional lambda-switching
  testbed
• Connect to NLR in Atlanta and Chicago
• Use asset-trading to extend reach (Sunnyvale and
  East Coast)
• Provide an evolving matrix of switching
  capabilities
• Separately fund research projects (e.g.,
  high-performance protocols, control,
  visualization) that will exercise the network and
  directly support applications at the host
  institutions

11
In More Detail -
12
Connections to National Nets
ORNL
ORNL-TVA
13
Research Projects

• A progression of switching approaches
• Study/compare SONET, MPLS(GMPLS), and all optical
• New all-optical technologies coming (e.g., laser
  tuning)
• Fast Local Storage
• Storage Depots
• Transport-optimized storage
• Progression of experimental point-to-point
  transport technologies
• Fiber channel
• Infiniband
• LAN-PHY Ethernet initial transport
• Circuit-Switched backbone, Frame-Switched edge

14
Digression LAN-PHY vs. WAN-PHY

• Conventional Wisdom LAN-PHY will win
• Cheaper, faster, etc. (based on POS experience)
• False cost model (extra processing)
• Ignore SONET advantages
• New Transponders are transparent
• Software selectable OC192 (WANPHY) or LANPHY
• Costs now equal
• Advantage now WAN-PHY
• DCC and OAM non-trivial considerations

15
PoS vs. Ethernet
16
Status and Schedule

• All contracts signed
• Fiber, Co-Lo space and power, equipment and
  installation, smart-hands
• Equipment shipping, built-out started
• Chicago-Sunnyvale paced by NLR cross-connects in
  Chicago (Level(3) to Starlight)
• NLR supplying 10Gig-E (LAN-PHY) initially
• PNNL fiber schedule will pace their connection
• First Chicago-Sunnyvale traffic this month
• Details on following VuGraphs -

17
Phase-1 (February)
18
Phase-2 (February-March)
19
Phase-3 (March - April)
20
Phase-4 (April - May)
21
Thank You
http//www.csm.ornl.gov/ultranet