Dan Magorian

1
Whats MAX Been Up to Lately?Presentation to MAX
membership
Fall 06 Member Meeting

• Dan Magorian
• Director of Engineering Operations

2
MAX Production Net and Research Activities
MAX Activities
Production Net Activities
Research Activities
Internet2 HOPI Test Bed support
DRAGON Experimental Networking
Layer 3 IP Network Services
Layer 1 Optical Services
GIG-EF Research Technologies
NGIX/East, AtlanticWave peer points
Disaster Recovery Colocation
Additional Funded Research
3
MAX Production topology
State of MD pop, Baltimore
BALT
M40E
1. Original Zhone dwdm over Qwest fiber
2. Movaz dwdm over State Md fiber
3. Gige on HHMI dwdm over Abovenet fiber
Ring 4
4. Univ Sys Md MRV dwdm, various fiber
Level3 pop, Mclean VA
CLPK
NGIX
Ring 2
CLPK
MCLN
T640
T640
UMD pop, College Park MD
CLPK
MCLN
Ring 1
Ring 3
DCGW
DCNE
ASHB
ARLG
Equinix pop, Ashburn VA
GWU Qwest DC pops, ISI/E Arlington VA pop
4
What have we gotten done since the Spring?

• At Equinix, transiting IPv6 carrier peerings to
  Internet2, making MAX primary east coast IPv6
  exchange for them.
• Lowered 07 ISP costs 42 32/mb/m for Qwest,
  16/mb/m Cogent.
• Brought up first international 10G peer (GEANT2,
  pan-European network) at NGIX/East, brought up
  Esnet 10G, NASA lambdas.
• Ran out of room at Level3 suite in Mclean adding
  dwdm shelves for high 10G lambda demand to NLR
  and Newnet!
• Baltimore MRV dwdm ring (well, really USM did).
  Will have College Park Baltimore OC48 replaced
  by protected ring in Jan (finally!). Good BERnet
  meetings focusing on 10G member research needs .
• Built new lab across hall at 8400 Baltimore Ave,
  welcome visitors.
• Connected to DCs NCRnet (lambdas), later DC-NET.
  Welcome Chuck Wolfe in new role!
• Bought and used advanced Exfo chromatic
  dispersion/polarization mode dispersion test
  gear, needed as we provision 10G waves over
  contributed unknown fiber. Available to MAX
  community on loan.
• Hired a Senior Optical Engineer (Matt Sinescal.

5
Mid-Atlantic Crossroads
National/International Peering Networks
Regional Network Participants
NGIX-East
DC-NET
Univ System Md
Network Virginia
Institutional Connectors
6
Projects Underway FY07

• 10G L3 backbone upgrade from OC48 finally will
  happen in spring to I2s Newnet. But, being slow
  saved us money. Now will have ability to backup
  connection with neighboring RON.
• Also will have a 10G research connection for
  participant lambdas over Newnet (more details in
  following slides)
• Still trying to get Cienas donation to MAX of
  production and research CoreDirectors (talked
  about at Spring MM, needed for GEANT and Newnet
  lightpaths), hung up in UMD Purchasing.
• Need to evaluate, test, procure, install
  replacement dwdm system for original MAX ring,
  running Luxn/Zhone since 2000. Ironically, still
  adding 10G interfaces due to high demand.
  Looking for technical folks to get involved with
  that project.
• Will get additional test gear for 10G ethernet
  analyzers sonet.
• Continuing measurement and alerting
  infrastructure buildout for redundancy, alerting
  on optical, full info tracking database.
• First production integration of DRAGON
  technology, over AtlanticWave peering switches
  for long distance vlan transit.
• Upgrading Equinix Ashburn lambda (from HHMI) to
  10G.

7
AtlanticWave It finally happened!

• People may remember that weve been talking about
  Awave since 2004, one of the takes forever
  projects.
• Background SURA-funded 10G NLR lambdas tying
  together peers points in NYC, DC, ATL, and MIA
  using p-t-p vlans.
• Finally happened just in time to get extended
  over Florida LambdaRail to SuperComputing 2006 in
  Tampa FL, as well as to Chicago, Ottawa, and on
  to PacificWave.
• And it was badly needed one of the few N-S
  paths for high-performance demonstrations, inc
  DRAGON, Teraflow Testbed, Optical Multicast,
  Nortel-VMT, Caltech-LHC.
• Whole team of engineers at 4 exchange points
  tested and scheduled demos by bandwidth,
  timeslot, vlan, segment.
• Amazingly, considering the 11th hour effort, it
  worked well! Now (slowly) bringing up production
  peerings.
• http//www.atlanticwave.net

8
(No Transcript)
9
What are the implications of this?

• Right now, everyone is talking about NLR and
  Newnet, sustainability/viability of two R E
  backbones, whether competition is good or bad.
• But consider the lashup of STL with CHI with
  WDC that ties P-Wave with Starlight with A-Wave
  creates another free backbone, which I call
  Ad-hoc net. No trustworthy central IU
  operation all access, rules, business plans TBD.
• People might remember that this was the
• original Quilt vision people throwing lines
• to their neighbors, now happening 10 years
  later.
• RON-to-RON peerings they are happening.

10
So what will Ad-hoc net backbones be used
for?lightpath activities meet bean-counters

• Personally, I would like to see production
  peering traffic remain on well-managed
  well-funded backbones, and
• Ad-hoc net (MorphNet, but that name is taken)
  used just for 1) bilateral backup arrangements,
  and 2) optical testbed interoperability and
  research activities that dont need 9s.
• That sounds fine, but there is definite overlap
  with at least some of NLR and Newnets L1 and L2
  offerings. Sure, one 10G pipe isnt that much,
  but pipes are cheap these days. It could grow
  quickly topsy-turvy style with strong demand.
• Especially if it saves people on expensive club
• costs to join. Well see how much cost is a
  driver.

11
Internet2s NewNet transition and NTAC

• People probably know that the old Abilene network
  built on Qwest fiber is being phased out, and
  replaced with NewNet built on Infinera dwdm
  system run by Level3 on their fiber.
• This will not impact either Internet2 or MAX
  participants! Great care is being taken to make
  the transition completely transparent. MAX built
  out to Level3 pop with T640 router in 2005, and
  as one of the top regional optical nets (RONs) is
  totally coordinated with I2 to help them make it
  happen.
• Reusing existing Juniper T640 routers via rolling
  replacement.
• New network also provides sub-10G circuit
  capabilities via Ciena CoreDirectors, and dynamic
  wave capabilities (being built by DRAGON under
  contract to I2) in addition to IP net.
• Dan and Jerry are on Internet2s new Technical
  Advisory Committee (NTAC) Jerry, longtime
  optical WG, Dan now interim chair of new ip
  services WG.
• Huge piece of work under tight deadlines
  (mid-07), NYC-CHI up for I2 MM, great new
  capabilities for the community.

12
The Internet2 Network Architecture
Network Build

• Initially provisioned with ten 10-Gbps
  wavelengths (100 Gbps) with all 10 dropping at
  each core optical node
• Unlimited capacity Will add more wavelengths on
  additional fibers if necessary
• Scalability Potential for 40 100 Gbps
  wavelength support
• Reliability Level3 provides industry standard
  service level agreements (SLAs) for the backbone
  wavelengths
• Flexibility Support for dynamic provisioning and
  wavelength switching

13
The New Internet2 Network
Network Design Infinera optical

• Simple and convenient add/drop technology
• Adding add/drop points doesnt require network
  redesign like traditional optical networking
  technologies
• Simple and convenient wave setup
• Demonstrated high reliability in carrier networks
• 8.5M field operational hours without a single PIC
  failure
• Economics of Infinera system are disruptive in
  the market place
• Every direct connector can access every wave on
  the system if needed because of Infineras
  low-cost transponders

14
The New Internet2 Network
Network Design Ciena CoreDirector

• Consolidates the functionality of Add/Drop
  Multiplexers (ADMs), Multi Service Provisioning
  Platforms (MSPPs), Digital Cross-Connects (DCSs),
  and Ethernet switches in a single, high-capacity
  switching system
• System includes grooming capabilities -
  lightpaths can be built over Ethernet or SONET
• Can take advantage of Next-Gen SONET capabilities
  like GFP, VCAT, and LCAS
• Capable of lightpath provisioning to the campus

15
Typical Optical Node
ESnet
Internet2
Colo Suite
IPcore
RON
CienaCoreDirector
dynamically allocated and routed waves
Direct Optical Connections to RONs
fiber east
fiber west
Level3 Owned and Managed Infinera DTN
fiber north/south
16
Optical Topology
17
Multiservice Switching Topology
18
Layer 3 Topology
19
The Plan from Spring MM MAX-provided customer
colo and disaster recovery sites

• Disaster recovery sites have been a continuing
  request from members at MAX meetings. People
  want affordable colo space to host their own gear
  offsite, yet accessible.
• In response, we have investigated 8 options
  trade deals with Northrup Grumman and
  InterAmerican Development Bank, trade deal with
  UMBC for machine room space in exchange for
  cooling plant upgrade, GWUs VA facility, Lab for
  Telecom Sciences, striking a much better price
  per sq ft and per amp for a lot of space on the
  second floor of Level3 pop in Mclean. Now
  talking to CTIS (new member) and long-time MAX
  supporter HHMIs VA facility.
• Meanwhile, Max unexpectedly acquired a fully
  equipped 1200 sq ft machine room across the hall
  with generator, dual UPS, overhead power
  distribution, underfloor dual AC units. Could
  hold 38 cabinets. Need because MAX test lab out
  of space, but wanted to make available to members
  also.
• Would charge folks upfront for standardized
  cabinets, then lease lambdas back to their
  institution so have private dedicated bandwidth
  and looks like part of their network with no MAX
  L3 in between.

20
Machine-room Space the Final FrontierWhats
wrong with this picture?

• Why has the colo/DR project taken so long?
  Summer has come and gone, when we thought wed
  have an offering.
• Turns out that building networks is easy compared
  to trading for machine room space to enable
  low-cost member colo. Every place has some
  hitch. The core issue power and cooling are gtgt
  expensive ltlt and rack demands are rising from 2
  kw to 6 kw to 20 kw for blade servers, and we
  dont know what the member uptake will be. The
  8400 MAX machine room turned out to be quite
  expensive to renovate power and cooling to handle
  more than 8-10 member racks.
• It would be easy if MAX could go out and spend
  2.5M like Nysernet just did to build a facility
  in Syracuse for 50 racks, but then wed have to
  charge 2000/m/rack, hardly low-cost.
• But were persevering in the search, and also
  Internet2 now has a SIG for disaster
  recovery/colo that may turn into offerings by
  other RONs, as well as research projects into
  distributed backup. Hopefully better news next
  MM.
• What would people like to see? Anyone want to
  help?

21
Recruiting for Engineering and Operations
Technical Advisory Group (TAC)

• Ongoing MAX Strategy and Research Advisory
  Groups.
• Have had Core Engineering Group for years, but
  that needs to be expanded and meet more regularly
  (quarterly).
• Want to offer chance for techies to help us plan
  future MAX engineering projects and directions,
  eg DR/colo and optical.
• Caution this is not for management folks and
  those wearing suits. It will be highly
  technical. We hope that the lead network
  engineers of each institution will be interested.
• The intent is for some sessions to have
  presentations of plans, others to have
  brainstorming sessions on selected topics, and
  periodically to have invited vendor
  presentations.
• Interested? Please mailto//magorian_at_maxgigapop.n
  et.

22
Thanks!