ESnet4 IP Network and Science Data Network Configuration and Roll Out Progress and Projected Schedul

1
ESnet4IP Network and Science Data
NetworkConfiguration and Roll OutProgress and
Projected Schedule as of Nov., 2007For more
information contactWilliam E. Johnston
(wej_at_es.net), ESnet Dept. Head,Joe Burrescia
(joeb_at_es.net), ESnet General Manager, orJim
Gagliardi (jvg_at_es.net), ESnet Operations and
Deployment Manager11/15/2009 1230 AM This
talk is available at www.es.net/ESnet4
2
ESnet4

• Internet2 has partnered with Level 3
  Communications Co. and Infinera Corp. for a
  dedicated optical fiber infrastructure with a
  national footprint and a rich topology - the
  Internet2 Network
• The fiber will be provisioned with Infinera Dense
  Wave Division Multiplexing equipment that uses an
  advanced, integrated optical-electrical design
• Level 3 will maintain the fiber and the DWDM
  equipment
• The DWDM equipment will initially be provisioned
  to provide10 optical circuits (lambdas - ?s)
  across the entire fiber footprint (80 ?s is max.)
• ESnet has partnered with Internet2 to
• Share the optical infrastructure
• Develop new circuit-oriented network services
• Explore mechanisms that could be used for the
  ESnet Network Operations Center (NOC) and the
  Internet2/Indiana University NOC to back each
  other up for disaster recovery purposes

3
ESnet4

• ESnet is building its next generation IP network
  and its new circuit-oriented Science Data Network
  primarily on the Internet2 circuits (?s) that are
  dedicated to ESnet, together with a few National
  Lambda Rail and other circuits
• ESnet will provision and operate its own routing
  and switching hardware that is installed in
  various commercial telecom hubs around the
  country, as it has done for the past 20 years
• ESnets peering relationships with the commercial
  Internet, various US research and education
  networks, and numerous international networks
  will continue and evolve as they have for the
  past 20 years

4
ESnet4

• ESnet4 will also involve an expansion of the
  multi-10Gb/s Metropolitan Area Rings in the
  San Francisco Bay Area, Chicago, Long Island,
  Newport News (VA/Washington, DC area), and
  Atlanta
• provide multiple, independent connections for
  ESnet sites to the ESnet core network
• expandable
• Several 10Gb/s links provided by the Labs that
  will be used to establish multiple, independent
  connections to the ESnet core
• currently PNNL and ORNL

5
Internet2 / Level3 / Infinera Optical
Infrastructure
Seattle 1000 Denny Way Level 3
Pacific Northwest GP 2001 6th Ave Westin Bldg
Albany 316 N Pearl Level 3
Rieth
Cleveland TFN 4000 Chester Level 3
Portland Oregon GP 707 SW Washington Qwest
Boise 435 W McGregor Drive
Portland 1335 NW Northrop Level 3
Cambridge NOX 300 Bent St Level 3
Syracuse
Chicago CIC/MREN MERIT BOREAS Internet2 710 N
Lakeshore Starlight
Rochester
Buffalo
New York 111 8th Level 3
New York NYSERNET 32 Ave of the Americas
Tionesta
Detroit
Chicago 600 W Chicago Level 3 MC
Omaha
Rawlins
Ogden
Philadelphia MAGPI 401 N Broad Level 3
Eureka
Reno
Pittsburgh Pittsburgh GP 143 S 25th Level 3
Edison
Sacramento
Indianapolis 1902 S East St Level 3
Oakland
Washington MAX 1755 Old Meadow Lane McLean,
VA Level 3
Salt Lake Inter-Mountain GP 572 S DeLong Level 3
Denver Front Range GP 1850 Pearl Level 3
Cincinnati
San Francisco
Louisville 848 S 8th St Level 3
St. Louis
Sunnyvale CENIC 1380 Kifer Level 3
Kansas City GPN 1100 Walnut Level 3
Tulsa OneNet 18 W Archer Level 3
Raleigh NCREN 5301 Departure Dr Level 3
San Luis Obispo
Charlotte
Nashville Tennessee GP 2990 Sidco Dr Level 3
Albuquerque New Mexico GP 104 Gold Ave SE Level 3
Raton
Los Angeles 818 W 7th Level 3
Santa Barbara
Atlanta, SLR 345 Courtland
Atlanta 180 Peachtree St NE Level 3 MC
Los Angeles CENIC 600 W 7th Equinix
Phoenix
Rancho De La Fe (tentative)
Tucson
San Diego 8929 Aero Drive Kearny Mesa, CA
Birmingham
Jacksonville FLR 4814 Phillips Hwy Level 3
Dallas
Mobile
El Paso 501 W Overland Level 3
Valentine
Tallahassee
Austin
Sanderson
Infinera equipment sites regen site Internet2
Core Node Internet2 networkDWDM node
(common) DWDM node(ESnet only)
New Orleans
Orlando
Baton Rouge LONI 9987 Burbank Level 3
Tampa
San Antonio
Houston LEARN 1201 N I-45 Level 3
other Level 3 node Level 3 Fiber extension from
Internet2 network to a RON/Core connector node
Miami South Florida GP 45 NW 5th Level 3
6
ESnet IP SDN Final Configuration for 2007

• ESnet equipment installed in Nashville, Aug. 31,
  2007
• ATL-NAS-CHI circuits tested and accepted, Sept.
  15, 2007.

Seattle
(28)
Portland
(8)
Boise
(29)
Boston
(9)
Chicago
(7)
Clev.
(10)
(11)
NYC
(32)
(25)
(13)
Denver
Sunnyvale
(12)
Philadelphia
KC
Salt Lake City
(14)
(15)
(26)
Pitts.
(16)
Wash DC
Indianapolis
(27)
(21)
(0)
(23)
(22)
(30)
Raleigh
Tulsa
LA
Nashville
(24)
Albuq.
OC48
UCSD
(1(3))
(3)
(4)
San Diego
(1)
Atlanta
(2)
(20)
(19)
Jacksonville
El Paso
(17)
(6)
BatonRouge
(5)
Houston
Corrected Map
7
ESnet4 Metro Area Rings Configurations, 2007
San FranciscoBay Area MAN
Seattle
Portland
Boise
Boston
Clev.
NYC
Denver
Sunnyvale
Philadelphia
KC
Salt Lake City
Pitts.
Wash DC
Indianapolis
Raleigh
Tulsa
Nashville
Albuq.
OC48
Atlanta
ESnet IP router hubs
Jacksonville
El Paso
BatonRouge
ESnet SDN OSCARS/MPLS switch hubs
Houston
No. New Mexico Ring
8
ESnet Provides Global High-Speed Internet
Connectivity for DOE Facilities and Collaborators
- 2007
Japan (SINet) Australia (AARNet) Canada
(CAnet4 Taiwan (TANet2) Singaren
KAREN/REANNZ ODN Japan Telecom America NLR-Packetn
et Abilene/I2
CAnet4 France GLORIAD (Russia, China)Korea
(Kreonet2
MREN StarTapTaiwan (TANet2, ASCC)
SEA
AU
PNNL
CHI-SL
LIGO
NEWY
MIT
INL
Lab DC Offices
Salt Lake
FNAL
LVK
JGI
LLNL
ANL
SNLL
LBNL
DOE
NETL
NERSC
MAE-E
AMES
SLAC
PAIX-PA Equinix, etc.
SDSC
AU
NOAA
45 end user sites
Office Of Science Sponsored (22)
NNSA Sponsored (13)
Joint Sponsored (3)
Other Sponsored (NSF LIGO, NOAA)
Laboratory Sponsored (6)
Specific RE network peers Other RE peering
points
commercial peering points

ESnet core hubs
high-speed peering points with Internet2/Abilene
9
ESnet4 IP SDN, 2008 Configuration (Estimated)

• Contract for 2008-2011 completed by mid-FY2007.
• Last three IP circuits (Kansas City-Houston,
  Chicago-Nashville-Atlanta, and Cleveland-Washingto
  n) and associated ESnet equipment, are installed,
  tested, and accepted.
• First SDN ring is completed (Sunnyvale-Atlanta)
  and associated ESnet equipment is installed,
  tested, and accepted.
• Second SDN northern tier circuits installed,
  tested, and accepted.

Seattle
(28)
(? ?)
Portland
(8)
Boise
(29)
Boston
StarLight
(9)
Chicago
(7)
Clev.
(11)
(10)
NYC
Pitts.
(25)
(32)
(13)
Denver
Sunnyvale
(12)
Philadelphia
KC
Salt Lake City
(14)
(15)
(26)
(16)
Wash. DC
(21)
(27)
Indianapolis
(23)
(22)
(30)
(0)
Raleigh
Tulsa
LA
Nashville
Albuq.
OC48
UCSD
(24)
(4)
(3)
San Diego
(1)
Atlanta
(2)
(20)
(19)
Jacksonville
El Paso
(17)
(6)
BatonRouge
(5)
Houston
Corrected Map
10
ESnet4 Metro Area Rings, 2008 Configurations
(Est.)
Long Island MAN
All circuits are 10Gb/s, or multiples thereof, as
indicated. (E.g. 2 ? 20 Gb/s., etc.)
West Chicago MAN
Seattle
(28)
(? ?)
Portland
(8)
Boise
(29)
Boston
(9)
Chicago
(7)
Clev.
(10)
(11)
NYC
(13)
(25)
(32)
(13)
Denver
Sunnyvale
(12)
Philadelphia
(14)
KC
Salt Lake City
(15)
(26)
Pitts.
(16)
Wash DC
(27)
San FranciscoBay Area MAN
Indianapolis
(21)
(21)
(23)
(22)
(30)
Raleigh
Tulsa
LA
Nashville
Albuq.
OC48
(1 ?)
(3)
(24)
(2?)
(4)
San Diego
Newport News - Elite
(1)
Atlanta
(2)
(20)
(19)
Jacksonville
El Paso
(17)
(6)
BatonRouge
(5)
Houston
Uncorrected Map
11
ESnet4 IP SDN, 2009 Configuration (Est.)
Seattle
(28)
(? ?)
Portland
(8)
Boise
(29)
Boston
StarLight
Chicago
(9)
(7)
Clev.
(10)
(11)
NYC
Pitts.
(25)
(32)
(13)
Denver
Sunnyvale
(12)
Philadelphia
(14)
KC
Salt Lake City
(15)
(26)
(16)
Wash. DC
(21)
Indianapolis
(27)
(23)
(22)
(30)
(0)
Raleigh
Tulsa
LA
Nashville
Albuq.
OC48
UCSD
(24)
(4)
(3)
San Diego
(1)
Atlanta
(2)
(20)
(19)
Jacksonville
El Paso
(17)
(6)
BatonRouge
(5)
Houston
Corrected Map
12
ESnet4 IP SDN, 2009 Configuration (Est.)
Long Island MAN
West Chicago MAN
Seattle
(28)
(? ?)
Portland
(8)
3?
Boise
(29)
Boston
(9)
3?
Chicago
(7)
Clev.
2?
3?
(10)
(11)
NYC
Pitts.
3?
(25)
(32)
(13)
Denver
3?
Sunnyvale
(12)
Philadelphia
(14)
KC
Salt Lake City
(15)
3?
3?
(26)
(16)
2?
Wash. DC
(21)
San FranciscoBay Area MAN
2?
3?
Indianapolis
(27)
2?
(23)
2?
(22)
(30)
(0)
Raleigh
3?
Tulsa
LA
Nashville
2?
Albuq.
OC48
2?
(24)
2?
(4)
2?
(3)
San Diego
Newport News - Elite
1?
(1)
Atlanta
(2)
(20)
(19)
Jacksonville
2?
El Paso
2?
(17)
(6)
BatonRouge
(5)
Houston
ESnet SDN switch hubs
Uncorrected Map
13
ESnet4 IP SDN, 2010 Configuration (Est.)
Seattle
(gt1 ?)
Portland
Boise
Boston
StarLight
Chicago
Clev.
NYC
Pitts.
(32)
Denver
Sunnyvale
Philadelphia
KC
Salt Lake City
Wash. DC
(21)
Indianapolis
Raleigh
Tulsa
LA
Nashville
Albuq.
OC48
UCSD
(24)
San Diego
Atlanta
Jacksonville
El Paso
(6)
BatonRouge
(5)
Houston
Corrected Map
14
ESnet4 IP SDN, 2011 Configuration (Est.)
Seattle
(gt1 ?)
Portland
Boise
Boston
StarLight
Chicago
Clev.
NYC
Pitts.
Denver
Sunnyvale
Philadelphia
KC
Salt Lake City
(26)
Wash. DC
Indianapolis
Raleigh
Tulsa
LA
Nashville
Albuq.
OC48
UCSD
(24)
(3)
San Diego
Atlanta
Jacksonville
El Paso
BatonRouge
Houston
Corrected Map
15
ESnet4
Core networks 50-60 Gbps by 2009-2010 (10Gb/s
circuits),500-600 Gbps by 2011-2012 (100 Gb/s
circuits)
Canada (CANARIE)
Europe (GEANT)
CERN (30 Gbps)
Canada (CANARIE)
Asia-Pacific
Asia Pacific
CERN (30 Gbps)
GLORIAD (Russia and China)
USLHCNet
Europe (GEANT)
Asia-Pacific
Science Data Network Core
Seattle
Boston
Chicago
IP Core
Boise
Australia
New York
Kansas City
Cleveland
Denver
Washington DC
Sunnyvale
Atlanta
Tulsa
Albuquerque
LA
Australia
South America (AMPATH)
San Diego
Houston
South America (AMPATH)
IP core hubs
Jacksonville
SDN hubs
Core network fiber path is 14,000 miles /
24,000 km