ESnet4: Networking for the Future of DOE Science December 5, 2006 - PowerPoint PPT Presentation

Loading...

PPT – ESnet4: Networking for the Future of DOE Science December 5, 2006 PowerPoint presentation | free to view - id: 6793d-ZWYyN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

ESnet4: Networking for the Future of DOE Science December 5, 2006

Description:

DOE Office of Science and ESnet the ESnet Mission ' ... The primary DOE network provider ... Based on DOE Lab populations, it is estimated that between 50, ... – PowerPoint PPT presentation

Number of Views:44
Avg rating:3.0/5.0
Slides: 72
Provided by: es97
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: ESnet4: Networking for the Future of DOE Science December 5, 2006


1
ESnet4 Networking for the Future of DOE
ScienceDecember 5, 2006
  • William E. Johnston ESnet Department Head and
    Senior ScientistLawrence Berkeley National
    Laboratory
  • wej_at_es.net, www.es.net

2
DOE Office of Science and ESnet the ESnet
Mission
  • The Office of Science is the single largest
    supporter of basic research in the physical
    sciences in the United States, providing more
    than 40 percent of total funding for the
    Nations research programs in high-energy
    physics, nuclear physics, and fusion energy
    sciences. (http//www.science.doe.gov)
  • This funding supports some 15,000 graduate
    students and post docs.
  • ESnets primary mission is to enable the
    large-scale science that is the mission of the
    Office of Science (SC)
  • Sharing of massive amounts of data
  • Supporting thousands of collaborators world-wide
  • Distributed data processing
  • Distributed data management
  • Distributed simulation, visualization, and
    computational steering
  • ESnet provides network and collaboration services
    to Office of Science laboratories and many other
    DOE programs

3
What ESnet Is
  • A large-scale IP network built on a national
    circuit infrastructure with high-speed
    connections to all major US and international
    research and education (RE) networks
  • An organization of 30 professionals structured
    for the service
  • An operating entity with an FY06 budget of 26.6M
  • A tier 1 ISP (direct peerings will all major
    networks)
  • The primary DOE network provider
  • Provides production Internet service to all of
    the major DOE Labs and most other DOE sites
  • Based on DOE Lab populations, it is estimated
    that between 50,000 -100,000 users depend on
    ESnet for global Internet access
  • additionally, each year more than 18,000 non-DOE
    researchers from universities, other government
    agencies, and private industry use Office of
    Science facilities

PNNL supplements its ESnet service with
commercial service
4
Office of Science US CommunityDrives ESnet
Design for Domestic Connectivity
Pacific Northwest National Laboratory
Idaho National Laboratory
Ames Laboratory
Argonne National Laboratory
Brookhaven National Laboratory
Fermi National Accelerator Laboratory
Lawrence Berkeley National Laboratory
Stanford Linear Accelerator Center
Princeton Plasma Physics Laboratory
Lawrence Livermore National Laboratory
Thomas Jefferson National Accelerator Facility
General Atomics
Oak Ridge National Laboratory
Los Alamos National Laboratory
Sandia National Laboratories
National Renewable Energy Laboratory
5
Footprint of Largest SC Data Sharing
Collaborators Drives the International Footprint
that ESnet Must Support
  • Top 100 data flows generate 50 of all ESnet
    traffic (ESnet handles about 3x109 flows/mo.)
  • 91 of the top 100 flows are from the Labs to
    other institutions (shown) (CY2005 data)

6
What Does ESnet Provide? - 1
  • An architecture tailored to accommodate DOEs
    large-scale science
  • Move huge amounts of data between a small number
    of sites that are scattered all over the world
  • Comprehensive connectivity
  • High bandwidth access to DOE sites and DOEs
    primary science collaborators Research and
    Education institutions in the US, Europe, Asia
    Pacific, and elsewhere
  • Full access to the global Internet for DOE Labs
  • ESnet is a tier 1 ISP managing a full complement
    of Internet routes for global access
  • Highly reliable transit networking
  • Fundamental goal is to deliver every packet that
    is received to the target site

7
What Does ESnet Provide? - 2
  • A full suite of network services
  • IPv4 and IPv6 routing and address space
    management
  • IPv4 multicast (and soon IPv6 multicast)
  • Primary DNS services
  • Circuit services (layer 2 e.g. Ethernet VLANs),
    MPLS overlay networks (e.g. SecureNet when it was
    ATM based)
  • Scavenger service so that certain types of bulk
    traffic can use all available bandwidth, but will
    give priority to any other traffic when it shows
    up
  • Prototype guaranteed bandwidth and virtual
    circuit services

8
What Does ESnet Provide? - 3
  • New network services
  • Guaranteed bandwidth services
  • Via a combination of QoS, MPLS overlay, and layer
    2 VLANS
  • Collaboration services and Grid middleware
    supporting collaborative science
  • Federated trust services / PKI Certification
    Authorities with science oriented policy
  • Audio-video-data teleconferencing
  • Highly reliable and secure operation
  • Extensive disaster recovery infrastructure
  • Comprehensive internal security
  • Cyberdefense for the WAN

9
What Does ESnet Provide? - 4
  • Comprehensive user support, including owning
    all trouble tickets involving ESnet users
    (including problems at the far end of an ESnet
    connection) until they are resolved 24x7x365
    coverage
  • ESnets mission is to enable the network based
    aspects of OSC science, and that includes
    troubleshooting network problems wherever they
    occur
  • A highly collaborative and interactive
    relationship with the DOE Labs and scientists for
    planning, configuration, and operation of the
    network
  • ESnet and its services evolve continuously in
    direct response to OSC science needs
  • Engineering services for special requirements

10
ESnet History
transitionin progress
11
ESnet3 Today Provides Global High-Speed Internet
Connectivity for DOE Facilities and Collaborators
(Fall, 2006)
Japan (SINet) Australia (AARNet) Canada
(CAnet4 Taiwan (TANet2) Singaren
ESnet Science Data Network (SDN) core
CAnet4 France GLORIAD (Russia, China)Korea
(Kreonet2
MREN Netherlands StarTapTaiwan (TANet2, ASCC)
PNWGPoP/PAcificWave
AU
NYC
ESnet IP core Packet over SONET Optical Ring and
Hubs
MAE-E
SNV
Equinix
PAIX-PA Equinix, etc.
AU
ALB
42 end user sites
ELP
Office Of Science Sponsored (22)
International (high speed) 10 Gb/s SDN core 10G/s
IP core 2.5 Gb/s IP core MAN rings ( 10 G/s) Lab
supplied links OC12 ATM (622 Mb/s) OC12 /
GigEthernet OC3 (155 Mb/s) 45 Mb/s and less
NNSA Sponsored (12)
Joint Sponsored (3)
Other Sponsored (NSF LIGO, NOAA)
Laboratory Sponsored (6)
Specific RE network peers Other RE peering
points
commercial peering points

ESnet core hubs
IP
Abilene
high-speed peering points with Internet2/Abilene
12
ESnets Place in U. S. and International Science
  • ESnet, Internet2/Abilene, and National Lambda
    Rail (NLR) provide most of the nations transit
    networking for basic science
  • Abilene provides national transit networking for
    most of the US universities by interconnecting
    the regional networks (mostly via the GigaPoPs)
  • ESnet provides national transit networking and
    ISP service for the DOE Labs
  • NLR provides various science-specific and network
    RD circuits
  • GÉANT plays a role in Europe similar to Abilene
    and ESnet in the US it interconnects the
    European National Research and Education Networks
    (NRENs), to which the European RE sites connect
  • A GÉANT operated, NSF funded like currently
    carries all non-LHC ESnet traffic to Europe, and
    this is a significant fraction of all ESnet
    traffic

13
ESnet is a Highly Reliable Infrastructure
5 nines (gt99.995)
4 nines (gt99.95)
3 nines
Dually connected sites
14
ESnet is An Organization Structured for the
Service
30.7 FTE (full-time staff) total
15
ESnet FY06 Budget is Approximately 26.6M
Approximate Budget Categories
Target carryover 1.0M
Special projects (Chicago and LI MANs) 1.2M
SC RD 0.5M
Carryover 1M
Management and compliance 0.7M
SC Special Projects 1.2M
Other DOE 3.8M
Collaboration services 1.6
Internal infrastructure, security, disaster
recovery 3.4M
Circuits hubs 12.7M
SC operating20.1M
Operations 1.1M
Engineering research 2.9M
WAN equipment 2.0M
Total expenses 26.6M
Total funds 26.6M
16
Strategy for Conduct of Business for the Last Few
Years
  • Increasing Openness
  • Making network data available to the community
    and the Internet
  • Outage Calendar and outage reports
  • web-based GUI for traffic measurements
    (netinfo.es.net)
  • Flow stats
  • Increasing Instrumentation
  • Performance testers (various levels of access to
    test circuits)
  • OWAMP servers (one-way testers - equisitally
    sensitive. Note The OWAMP system is pretty much
    down. It is migrating to perfSONAR and new, more
    relevant, RE sites will be selected for
    continuous monitoring
  • Establish the goal of network performance
    between ESnet sites and Internet2 sites served by
    Abilene is equivalent to network performance
    across one of the networks or the other

17
Strategy for Conduct of Business for the Last Few
Years
  • Increasing involvement with national and
    international collaborations and research
    activities
  • perfSONAR - standards based monitoring platform
  • OSCARS - community developed (ESnet leadership)
    virtual circuit management
  • Increasing partnership with the RE community
  • Joint ESnet/I2/Abileene meetings at Joint Techs
  • LHC network operations working group
    participation
  • DICE meetings
  • Joint Techs meetings participation
  • attendance, talks, program committee
  • All leading up to partnership with Internet2 for
    building ESnet4

18
A Changing Science Environment is the Key Driver
of the Next Generation ESnet
  • Large-scale collaborative science big
    facilities, massive data, thousands of
    collaborators is now a significant aspect of
    the Office of Science (SC) program
  • SC science community is almost equally split
    between Labs and universities
  • SC facilities have users worldwide
  • Very large international (non-US) facilities
    (e.g. LHC and ITER) and international
    collaborators are now a key element of SC science
  • Distributed systems for data analysis,
    simulations, instrument operation, etc., are
    essential and are now common (in fact dominate
    data analysis that now generates 50 of all ESnet
    traffic)

19
Planning the Future Network - ESnet4
  • There are many stakeholders for ESnet
  • SC programs
  • Advanced Scientific Computing Research
  • Basic Energy Sciences
  • Biological and Environmental Research
  • Fusion Energy Sciences
  • High Energy Physics
  • Nuclear Physics
  • Office of Nuclear Energy
  • Major scientific facilities
  • At DOE sites large experiments, supercomputer
    centers, etc.
  • Not at DOE sites LHC, ITER
  • SC supported scientists not at the Labs
    (mostly at US RE institutions)
  • Other collaborating institutions (mostly
    US, European, and AP RE)
  • Other RE networking organizations that support
    major collaborators
  • Mostly US, European, and Asia Pacific networks
  • Lab operations and general population
  • Lab networking organizations

These accountfor 85 of allESnet traffic
20
Planning the Future Network - ESnet4
  • Requirements of the ESnet stakeholders are
    primarily determined by
  • 1) Data characteristics of instruments and
    facilities that will be connected to ESnet
  • What data will be generated by instruments coming
    on-line over the next 5-10 years?
  • How and where will it be analyzed and used?
  • 2) Examining the future process of science
  • How will the processing of doing science change
    over 5-10 years?
  • How do these changes drive demand for new network
    services?
  • 3) Studying the evolution of ESnet traffic
    patterns
  • What are the trends based on the use of the
    network in the past 2-5 years?
  • How must the network change to accommodate the
    future traffic patterns implied by the trends?

21
(1) Requirements from Instruments and Facilities
DOE SC Facilities that are, or will be, the top
network users
  • Advanced Scientific Computing Research
  • National Energy Research Scientific Computing
    Center (NERSC) (LBNL)
  • National Leadership Computing Facility (NLCF)
    (ORNL)
  • Argonne Leadership Class Facility (ALCF) (ANL)
  • Basic Energy Sciences
  • National Synchrotron Light Source (NSLS) (BNL)
  • Stanford Synchrotron Radiation Laboratory (SSRL)
    (SLAC)
  • Advanced Light Source (ALS) (LBNL)
  • Advanced Photon Source (APS) (ANL)
  • Spallation Neutron Source (ORNL)
  • National Center for Electron Microscopy (NCEM)
    (LBNL)
  • Combustion Research Facility (CRF) (SNLL)
  • Biological and Environmental Research
  • William R. Wiley Environmental Molecular Sciences
    Laboratory (EMSL) (PNNL)
  • Joint Genome Institute (JGI)
  • Structural Biology Center (SBC) (ANL)
  • Fusion Energy Sciences
  • DIII-D Tokamak Facility (GA)
  • Alcator C-Mod (MIT)
  • National Spherical Torus Experiment (NSTX)
    (PPPL)
  • ITER
  • High Energy Physics
  • Tevatron Collider (FNAL)
  • B-Factory (SLAC)
  • Large Hadron Collider (LHC, ATLAS, CMS) (BNL,
    FNAL)
  • Nuclear Physics
  • Relativistic Heavy Ion Collider (RHIC) (BNL)
  • Continuous Electron Beam Accelerator Facility
    (CEBAF) (JLab)

14 of 22 are characterized by current case
studies
22
The Largest Facility Large Hadron Collider at
CERN
LHC CMS detector 15m X 15m X 22m,12,500 tons,
700M
human (for scale)
23
(2) Requirements from Examiningthe Future
Process of Science
  • In a major workshop 1, and in subsequent
    updates 2, requirements were generated by
    asking the science community how their process of
    doing science will / must change over the next 5
    and next 10 years in order to accomplish their
    scientific goals
  • Computer science and networking experts then
    assisted the science community in
  • analyzing the future environments
  • deriving middleware and networking requirements
    needed to enable these environments
  • These were complied as case studies that provide
    specific 5 10 year network requirements for
    bandwidth, footprint, and new services

24
Science Networking Requirements Aggregation
Summary
25
Science Network Requirements Aggregation Summary
Immediate Requirements and Drivers
26
3) These Trends are Seen in Observed Evolution of
Historical ESnet Traffic Patterns
top 100 sites to siteworkflows
Terabytes / month
  • ESnet Monthly Accepted Traffic, January, 2000
    June, 2006
  • ESnet is currently transporting more than1
    petabyte (1000 terabytes) per month
  • More than 50 of the traffic is now generated by
    the top 100 sites large-scale science dominates
    all ESnet traffic

27
ESnet Traffic has Increased by10X Every 47
Months, on Average, Since 1990
Apr., 2006 1 PBy/mo.
Nov., 2001 100 TBy/mo.
53 months
Jul., 1998 10 TBy/mo.
40 months
Oct., 1993 1 TBy/mo.
57 months
Terabytes / month
Aug., 1990 100 MBy/mo.
38 months
Log Plot of ESnet Monthly Accepted Traffic,
January, 1990 June, 2006
28
Requirements from Network Utilization Observation
  • In 4 years, we can expect a 10x increase in
    traffic over current levels without the addition
    of production LHC traffic
  • Nominal average load on busiest backbone links is
    1.5 Gbps today
  • In 4 years that figure will be 15 Gbps based on
    current trends
  • Measurements of this type are science-agnostic
  • It doesnt matter who the users are, the traffic
    load is increasing exponentially
  • Predictions based on this sort of forward
    projection tend to be conservative estimates of
    future requirements because they cannot predict
    new uses
  • Bandwidth trends drive requirement for a new
    network architecture
  • New architecture/approach must be scalable in a
    cost-effective way

29
Large-Scale Flow Trends, June 2006Subtitle
Onslaught of the LHC)
Traffic Volume of the Top 30 AS-AS Flows, June
2006(AS-AS mostly Lab to RE site, a few Lab
to RE network, a few other)
Terabytes
FNAL -gt CERN traffic is comparable to BNL -gt
CERNbut on layer 2 flows that are not yet
monitored for traffic soon)
30
Traffic Patterns are Changing Dramatically
total traffic,TBy
total traffic, TBy
1/05
6/06
2 TB/month
2 TB/month
7/05
  • While the total traffic is increasing
    exponentially
  • Peak flow that is system-to-system bandwidth
    is decreasing
  • The number of large flows is increasing

2 TB/month
1/06
2 TB/month
31
The Onslaught of Grids
Question Why is peak flow bandwidth decreasing
while total traffic is increasing?
plateaus indicate the emergence of parallel
transfer systems (a lot of systems transferring
the same amount of data at the same time)
  • Answer Most large data transfers are now done by
    parallel / Grid data movers
  • In June, 2006 72 of the hosts generating the top
    1000 flows were involved in parallel data movers
    (Grid applications)
  • This is the most significant traffic pattern
    change in the history of ESnet
  • This has implications for the network
    architecture that favor path multiplicity and
    route diversity

32
Network Observation Circuit-like Behavior
Look at Top 20 Traffic Generators Historical
Flow Patterns Over 1 year, the work flow /
circuit duration is about 3 months
Gigabytes/day
(no data)
LIGO CalTech (host to host)
33
Network Observation Circuit-like Behavior (2)
Look at Top 20 Traffic Generators Historical
Flow Patterns Over 1 year, work flow / circuit
duration is about 1 day to 1 week
Gigabytes/day
(no data)
SLAC - IN2P3, France (host to host)
34
What is the High-Level View of ESnet Traffic
Patterns?
ESnet Inter-Sector Traffic Summary, Mar. 2006
7
Commercial
48
5
ESnet
Inter-Labtraffic
10
12
RE (mostlyuniversities)
DOE sites
3
Peering Points
58
23
International(almost entirelyRE sites)
43
Traffic coming into ESnet Green Traffic leaving
ESnet Blue Traffic between ESnet sites of
total ingress or egress traffic
  • Traffic notes
  • more than 90 of all traffic Office of Science
  • less that 10 is inter-Lab

35
Requirements from Traffic Flow Observations
  • Most of ESnet science traffic has a source or
    sink outside of ESnet
  • Drives requirement for high-bandwidth peering
  • Reliability and bandwidth requirements demand
    that peering be redundant
  • Multiple 10 Gbps peerings today, must be able to
    add more bandwidth flexibly and cost-effectively
  • Bandwidth and service guarantees must traverse
    RE peerings
  • Collaboration with other RE networks on a common
    framework is critical
  • Seamless fabric
  • Large-scale science is now the dominant user of
    the network
  • Satisfying the demands of large-scale science
    traffic into the future will require a
    purpose-built, scalable architecture
  • Traffic patterns are different than commodity
    Internet

36
Changing Science Environment ? New Demands on
Network
  • Requirements Summary
  • Increased capacity
  • Needed to accommodate a large and steadily
    increasing amount of data that must traverse the
    network
  • High network reliability
  • Essential when interconnecting components of
    distributed large-scale science
  • High-speed, highly reliable connectivity between
    Labs and US and international RE institutions
  • To support the inherently collaborative, global
    nature of large-scale science
  • New network services to provide bandwidth
    guarantees
  • Provide for data transfer deadlines for
  • remote data analysis, real-time interaction with
    instruments, coupled computational simulations,
    etc.

37
ESnet4 - The Response to the Requirements
  • I) A new network architecture and implementation
    strategy
  • Rich and diverse network topology for flexible
    management and high reliability
  • Dual connectivity at every level for all
    large-scale science sources and sinks
  • A partnership with the US research and education
    community to build a shared, large-scale, RE
    managed optical infrastructure
  • a scalable approach to adding bandwidth to the
    network
  • dynamic allocation and management of optical
    circuits
  • II) Development and deployment of a virtual
    circuit service
  • Develop the service cooperatively with the
    networks that are intermediate between DOE Labs
    and major collaborators to ensure and-to-end
    interoperability

38
Next Generation ESnet I) Architecture and
Configuration
  • Main architectural elements and the rationale for
    each element
  • 1) A High-reliability IP core (e.g. the current
    ESnet core) to address
  • General science requirements
  • Lab operational requirements
  • Backup for the SDN core
  • Vehicle for science services
  • Full service IP routers
  • 2) Metropolitan Area Network (MAN) rings to
    provide
  • Dual site connectivity for reliability
  • Much higher site-to-core bandwidth
  • Support for both production IP and circuit-based
    traffic
  • Multiply connecting the SDN and IP cores
  • 2a) Loops off of the backbone rings to provide
  • For dual site connections where MANs are not
    practical
  • 3) A Science Data Network (SDN) core for
  • Provisioned, guaranteed bandwidth circuits to
    support large, high-speed science data flows
  • Very high total bandwidth
  • Multiply connecting MAN rings for protection
    against hub failure
  • Alternate path for production IP traffic

39
ESnet Target Architecture IP CoreScience Data
Network CoreMetro Area Rings
international connections
international connections
international connections
Loop off Backbone
Seattle
SDN Core
Cleveland
Chicago
New York
Denver
IP Core
Sunnyvale
Washington DC
MetropolitanArea Rings
Atlanta
LA
international connections
Albuquerque
international connections
San Diego
10-50 Gbps circuits Production IP core Science
Data Network core Metropolitan Area Networksor
backbone loops for Lab access International
connections
international connections
40
ESnet4
  • Internet2 has partnered with Level 3
    Communications Co. and Infinera Corp. for a
    dedicated optical fiber infrastructure with a
    national footprint and a rich topology - the
    Internet2 Network
  • The fiber will be provisioned with Infinera Dense
    Wave Division Multiplexing equipment that uses an
    advanced, integrated optical-electrical design
  • Level 3 will maintain the fiber and the DWDM
    equipment
  • The DWDM equipment will initially be provisioned
    to provide10 optical circuits (lambdas - ?s)
    across the entire fiber footprint (80 ?s is max.)
  • ESnet has partnered with Internet2 to
  • Share the optical infrastructure
  • Develop new circuit-oriented network services
  • Explore mechanisms that could be used for the
    ESnet Network Operations Center (NOC) and the
    Internet2/Indiana University NOC to back each
    other up for disaster recovery purposes

41
ESnet4
  • ESnet will build its next generation IP network
    and its new circuit-oriented Science Data Network
    primarily on the Internet2 circuits (?s) that are
    dedicated to ESnet, together with a few National
    Lambda Rail and other circuits
  • ESnet will provision and operate its own routing
    and switching hardware that is installed in
    various commercial telecom hubs around the
    country, as it has done for the past 20 years
  • ESnets peering relationships with the commercial
    Internet, various US research and education
    networks, and numerous international networks
    will continue and evolve as they have for the
    past 20 years

42
ESnet4
  • ESnet4 will also involve an expansion of the
    multi-10Gb/s Metropolitan Area Rings in the
    San Francisco Bay Area, Chicago, Long Island,
    Newport News (VA/Washington, DC area), and
    Atlanta
  • provide multiple, independent connections for
    ESnet sites to the ESnet core network
  • expandable
  • Several 10Gb/s links provided by the Labs that
    will be used to establish multiple, independent
    connections to the ESnet core
  • currently PNNL and ORNL

43
ESnet Metropolitan Area Network Ring Architecture
for High Reliability Sites
SDNcoreeast
ESnet production IP core hub
IP coreeast
IP corewest
IP core router
SDN corewest
ESnetIP core hub
ESnet SDNcore hub
MAN fiber ring 2-4 x 10 Gbps channels
provisioned initially,with expansion capacity to
16-64
ESnet managedvirtual circuit services tunneled
through the IP backbone
Large Science Site
ESnet production IP service
ESnet managed? / circuit services
ESnet MANswitch
Independentport card supportingmultiple 10 Gb/s
line interfaces
Site
ESnet switch
Virtual Circuits to Site
Virtual Circuit to Site
Siterouter
Site gateway router
SDN circuitsto site systems
Site LAN
Site edge router
44
Internet2 / Level3 / Infinera Optical
Infrastructure
Seattle 1000 Denny Way Level 3
Pacific Northwest GP 2001 6th Ave Westin Bldg
Albany 316 N Pearl Level 3
Rieth
Cleveland TFN 4000 Chester Level 3
Portland Oregon GP 707 SW Washington Qwest
Boise 435 W McGregor Drive
Portland 1335 NW Northrop Level 3
Cambridge NOX 300 Bent St Level 3
Syracuse
Chicago CIC/MREN MERIT BOREAS Internet2 710 N
Lakeshore Starlight
Rochester
Buffalo
New York 111 8th Level 3
New York NYSERNET 32 Ave of the Americas
Tionesta
Detroit
Chicago 600 W Chicago Level 3 MC
Omaha
Rawlins
Ogden
Philadelphia MAGPI 401 N Broad Level 3
Eureka
Reno
Pittsburgh Pittsburgh GP 143 S 25th Level 3
Edison
Sacramento
Indianapolis 1902 S East St Level 3
Oakland
Washington MAX 1755 Old Meadow Lane McLean,
VA Level 3
Salt Lake Inter-Mountain GP 572 S DeLong Level 3
Denver Front Range GP 1850 Pearl Level 3
Cincinnati
San Francisco
Louisville 848 S 8th St Level 3
St. Louis
Sunnyvale CENIC 1380 Kifer Level 3
Kansas City GPN 1100 Walnut Level 3
Tulsa OneNet 18 W Archer Level 3
Raleigh NCREN 5301 Departure Dr Level 3
San Luis Obispo
Charlotte
Nashville Tennessee GP 2990 Sidco Dr Level 3
Albuquerque New Mexico GP 104 Gold Ave SE Level 3
Raton
Los Angeles 818 W 7th Level 3
Santa Barbara
Atlanta, SLR 345 Courtland
Atlanta 180 Peachtree St NE Level 3 MC
Los Angeles CENIC 600 W 7th Equinix
Phoenix
Rancho De La Fe (tentative)
Tucson
San Diego 8929 Aero Drive Kearny Mesa, CA
Birmingham
Jacksonville FLR 4814 Phillips Hwy Level 3
Dallas
Mobile
El Paso 501 W Overland Level 3
Valentine
Tallahassee
Austin
Sanderson
Infinera equipment sites regen site Internet2
Core Node Internet2 networkDWDM node
(common) DWDM node(ESnet only)
New Orleans
Orlando
Baton Rouge LONI 9987 Burbank Level 3
Tampa
San Antonio
Houston LEARN 1201 N I-45 Level 3
other Level 3 node Level 3 Fiber extension from
Internet2 network to a RON/Core connector node
Miami South Florida GP 45 NW 5th Level 3
45
ESnet4 2009 Configuration(Some of the circuits
may be allocated dynamically from shared a pool.)
Seattle
(28)
(? ?)
Portland
(8)
3?
Boise
(29)
Boston
(9)
3?
Chicago
(7)
Clev.
2?
3?
(10)
(11)
3?
NYC
Pitts.
3?
(25)
(32)
(13)
Denver
3?
Sunnyvale
(12)
Philadelphia
(14)
KC
Salt Lake City
(15)
3?
3?
(26)
2?
(16)
Wash. DC
(21)
2?
3?
Indianapolis
(27)
2?
2?
(23)
(22)
(30)
(0)
Raleigh
3?
Tulsa
LA
Nashville
2?
Albuq.
OC48
2?
(24)
2?
(4)
2?
(3)
San Diego
1?
(1)
Atlanta
(2)
(20)
(19)
Jacksonville
2?
El Paso
2?
(17)
(6)
BatonRouge
(5)
Houston
ESnet SDN switch hubs
46
ESnet4 2009 Configuration(Some of the circuits
may be allocated dynamically from shared a pool.)
Long Island MAN
West Chicago MAN
600 W. Chicago
USLHCNet
Seattle
Starlight
32 AoA, NYC
(28)
(? ?)
Portland
BNL
(8)
3?
Boise
(29)
Boston
USLHCNet
111-8th
(9)
3?
Chicago
(7)
Clev.
2?
3?
(10)
(11)
NYC
FNAL
ANL
Pitts.
3?
(25)
(32)
(13)
Denver
3?
Sunnyvale
(12)
Philadelphia
(14)
KC
Salt Lake City
(15)
3?
3?
(26)
(16)
2?
Wash. DC
(21)
San FranciscoBay Area MAN
2?
3?
Indianapolis
(27)
2?
(23)
2?
(22)
(30)
(0)
Raleigh
3?
Tulsa
LA
Nashville
2?
JGI
Albuq.
OC48
2?
(24)
2?
(4)
2?
(3)
LBNL
San Diego
Newport News - Elite
1?
(1)
Atlanta
SLAC
(2)
(20)
NERSC
(19)
Wash., DC
Jacksonville
2?
El Paso
2?
Atlanta
(17)
(6)
MATP
LLNL
BatonRouge
(5)
Houston
Nashville
SNLL
JLab
Jacksonville
180 Peachtree
ESnet SDN switch hubs
ELITE
Atlanta
ODU
56 Marietta
47
Internet2 and ESnet Optical Node
ESnet
Internet2
IPcore
ESnetmetro-areanetworks
groomingdevice
CienaCoreDirector
dynamically allocated and routed waves (future)
  • support devices
  • measurement
  • out-of-band access
  • monitoring
  • security
  • support devices
  • measurement
  • out-of-band access
  • monitoring
  • .

Network Testbeds
Future access to control plane
fiber east
fiber west
Internet2/Level3National Optical Infrastructure
Infinera DTN
fiber north/south
48
Typical ESnet4 Hub
49
The Evolution of ESnet Architecture
ESnet IP core
ESnet IP core
ESnet Science Data Network (SDN) core
  • ESnet to 2005
  • A routed IP network with sites singly attached
    to a national core ring
  • ESnet from 2006-07
  • A routed IP network with sites dually connected
    on metro area rings or dually connected directly
    to core ring
  • A switched network providing virtual circuit
    services for data-intensive science
  • Rich topology offsets the lack of dual,
    independent national cores

ESnet sites
ESnet hubs / core network connection points
Metro area rings (MANs)
Circuit connections to other science networks
(e.g. USLHCNet)
50
ESnet4 Planed Configuration
Core networks 40-50 Gbps in 2009-2010, 160-400
Gbps in 2011-2012
CERN (30 Gbps)
Canada (CANARIE)
Europe (GEANT)
Canada (CANARIE)
Asia-Pacific
CERN (30 Gbps)
Asia Pacific
GLORIAD (Russia and China)
Europe (GEANT)
Asia-Pacific
Science Data Network Core
Seattle
Cleveland
Boston
Chicago
IP Core
Boise
Australia
New York
Kansas City
Denver
Washington DC
Sunnyvale
Atlanta
Tulsa
Albuquerque
LA
Australia
South America (AMPATH)
San Diego
Houston
South America (AMPATH)
Jacksonville
Core network fiber path is 14,000 miles /
24,000 km
51
Next Generation ESnet II) Virtual Circuits
  • Traffic isolation and traffic engineering
  • Provides for high-performance, non-standard
    transport mechanisms that cannot co-exist with
    commodity TCP-based transport
  • Enables the engineering of explicit paths to meet
    specific requirements
  • e.g. bypass congested links, using lower
    bandwidth, lower latency paths
  • Guaranteed bandwidth (Quality of Service (QoS))
  • User specified bandwidth
  • Addresses deadline scheduling
  • Where fixed amounts of data have to reach sites
    on a fixed schedule, so that the processing does
    not fall far enough behind that it could never
    catch up very important for experiment data
    analysis
  • Reduces cost of handling high bandwidth data
    flows
  • Highly capable routers are not necessary when
    every packet goes to the same place
  • Use lower cost (factor of 5x) switches to
    relatively route the packets
  • Secure
  • The circuits are secure to the edges of the
    network (the site boundary) because they are
    managed by the control plane of the network which
    is isolated from the general traffic
  • Provides end-to-end connections between Labs and
    collaborator institutions

52
Virtual Circuit Service Functional Requirements
  • Support user/application VC reservation requests
  • Source and destination of the VC
  • Bandwidth, start time, and duration of the VC
  • Traffic characteristics (e.g. flow specs) to
    identify traffic designated for the VC
  • Manage allocations of scarce, shared resources
  • Authentication to prevent unauthorized access to
    this service
  • Authorization to enforce policy on
    reservation/provisioning
  • Gathering of usage data for accounting
  • Provide circuit setup and teardown mechanisms and
    security
  • Widely adopted and standard protocols (such as
    MPLS and GMPLS) are well understood within a
    single domain
  • Cross domain interoperability is the subject of
    ongoing, collaborative development
  • secure and-to-end connection setup is provided by
    the network control plane
  • Enable the claiming of reservations
  • Traffic destined for the VC must be
    differentiated from regular traffic
  • Enforce usage limits
  • Per VC admission control polices usage, which in
    turn facilitates guaranteed bandwidth
  • Consistent per-hop QoS throughout the network for
    transport predictability

53
ESnet Virtual Circuit Service OSCARS(On-demand
Secured Circuits and Advanced Reservation System)
  • Software Architecture (see Ref. 9)
  • Web-Based User Interface (WBUI) will prompt the
    user for a username/password and forward it to
    the AAAS.
  • Authentication, Authorization, and Auditing
    Subsystem (AAAS) will handle access, enforce
    policy, and generate usage records.
  • Bandwidth Scheduler Subsystem (BSS) will track
    reservations and map the state of the network
    (present and future).
  • Path Setup Subsystem (PSS) will setup and
    teardown the on-demand paths (LSPs).

User request via WBUI
Reservation Manager
Web-Based User Interface
Path Setup Subsystem
Instructions to routers and switches
to setup/teardown LSPs
User
HumanUser
User feedback
Authentication, Authorization, And
Auditing Subsystem
User Application
Bandwidth Scheduler Subsystem
User app request via AAAS
54
The Mechanisms Underlying OSCARS
Based on Source and Sink IP addresses, route of
LSP between ESnet border routers is determined
using topology information from OSPF-TE. Path of
LSP can be explicitly directed to take SDN
network. On the SDN Ethernet switches all traffic
is MPLS switched (layer 2.5), which stitches
together VLANs
VLAN 1
VLAN 3
VLAN 2
On ingress to ESnet, packets matching reservation
profile are filtered out (i.e. policy based
routing), policed to reserved bandwidth, and
injected into a LSP.
SDN
SDN
SDN
SDN Link
SDN Link
RSVP, MPLS enabled on internal interfaces
Sink
Label Switched Path
IP Link
Source
IP
IP
IP
IP Link
high-priority queue
standard,best-effortqueue
MPLS labels are attached onto packets from Source
and placed in separate queue to ensure guaranteed
bandwidth.
Regular production traffic queue.
Interface queues
55
Environment of Science is Inherently Multi-Domain
  • End points will be at independent institutions
    campuses or research institutes - that are served
    by ESnet, Abilene, GÉANT, and their regional
    networks
  • Complex inter-domain issues typical circuit
    will involve five or more domains - of necessity
    this involves collaboration with other networks
  • For example, a connection between FNAL and DESY
    involves five domains, traverses four countries,
    and crosses seven time zones

FNAL (AS3152) US
GEANT (AS20965) Europe
DESY (AS1754) Germany
ESnet (AS293) US
DFN (AS680) Germany
56
OSCARS Guaranteed Bandwidth VC Service For SC
Science
  • To ensure compatibility, the design and
    implementation is done in collaboration with the
    other major science RE networks and end sites
  • Internet2 Bandwidth Reservation for User Work
    (BRUW)
  • Development of common code base
  • GEANT Bandwidth on Demand (GN2-JRA3),
    Performance and Allocated Capacity for End-users
    (SA3-PACE) and Advance Multi-domain Provisioning
    System (AMPS) extends to NRENs
  • BNL TeraPaths - A QoS Enabled Collaborative Data
    Sharing Infrastructure for Peta-scale Computing
    Research
  • GA Network Quality of Service for Magnetic
    Fusion Research
  • SLAC Internet End-to-end Performance Monitoring
    (IEPM)
  • USN Experimental Ultra-Scale Network Testbed for
    Large-Scale Science
  • In its current phase this effort is being funded
    as a research project by the Office of Science,
    Mathematical, Information, and Computational
    Sciences (MICS) Network RD Program
  • A prototype service has been deployed as a proof
    of concept
  • To date more then 20 accounts have been created
    for beta users, collaborators, and developers
  • More then 100 reservation requests have been
    processed

57
ESnet Virtual Circuit Service Roadmap
  • Dedicated virtual circuits
  • Dynamic virtual circuit allocation
  • Generalized MPLS (GMPLS)

Initial production service

Full production service
2005 2006 2007
2008
  • Interoperability between GMPLS circuits, VLANs,
    and MPLS circuits (layer 1-3)
  • Interoperability between VLANs and MPLS
    circuits(layer 2 3)
  • Dynamic provisioning of Multi-Protocol Label
    Switching (MPLS) circuits in IP nets (layer 3)
    and in VLANs for Ethernets (layer 2)

58
Federated Trust Services Support for
Large-Scale Collaboration
  • Remote, multi-institutional, identity
    authentication is critical for distributed,
    collaborative science in order to permit sharing
    widely distributed computing and data resources,
    and other Grid services
  • Public Key Infrastructure (PKI) is used to
    formalize the existing web of trust within
    science collaborations and to extend that trust
    into cyber space
  • The function, form, and policy of the ESnet trust
    services are driven entirely by the requirements
    of the science community and by direct input from
    the science community
  • International scope trust agreements that
    encompass many organizations are crucial for
    large-scale collaborations
  • ESnet has lead in negotiating and managing the
    cross-site, cross-organization, and international
    trust relationships to provide policies that are
    tailored for collaborative science
  • This service, together with the associated ESnet
    PKI service, is the basis of the routine sharing
    of HEP Grid-based computing resources between US
    and Europe

59
DOEGrids CA (one of several CAs) Usage Statistics
Report as of Dec 1, 2006
60
DOEGrids CA Usage - Virtual Organization Breakdown
Argonne Nat. Lab. Earth System Grid ESnet Fusion
Grid International Very Large Data Grid Lawrence
Berkeley Lab. Nat. Energy Research Supercomputer
Center Oak Ridge Nat. Lab. Pacific Northwest Nat.
Lab. Particle Physics Data Grid Fermi Nat.
Accelerator Lab. LHC Computing GRid Open Science
Grid
OSG Includes (BNL, CDF, CMS, DES, DOSAR,
DZero, Fermilab, fMRI, GADU, geant4, GLOW, GRASE,
GridEx, GROW, i2u2, iVDGL, JLAB, LIGO, mariachi,
MIS, nanoHUB, NWICG, OSG, OSGEDU, SDSS, SLAC,
STAR USATLAS)
DOE-NSF collab. Auto renewals
61
DOEGrids CA (Active Certificates) Usage Statistics
Report as of Dec 1, 2006
62
Summary
  • ESnet is currently satisfying its mission by
    enabling SC science that is dependant on
    networking and distributed, large-scale
    collaboration
  • The performance of ESnet over the past year
    has been excellent, with only minimal unscheduled
    down time. The reliability of the core
    infrastructure is excellent. Availability for
    users is also excellent - DOE 2005 annual review
    of LBL
  • ESnet has put considerable effort into gathering
    requirements from the DOE science community, and
    has a forward-looking plan and expertise to meet
    the five-year SC requirements
  • A Lehman review of ESnet (Feb, 2006) has strongly
    endorsed the plan presented here

63
References
  • High Performance Network Planning Workshop,
    August 2002
  • http//www.doecollaboratory.org/meetings/hpnpw
  • Science Case Studies Update, 2006 (contact
    eli_at_es.net)
  • DOE Science Networking Roadmap Meeting, June 2003
  • http//www.es.net/hypertext/welcome/pr/Roadmap/ind
    ex.html
  • DOE Workshop on Ultra High-Speed Transport
    Protocols and Network Provisioning for
    Large-Scale Science Applications, April 2003
  • http//www.csm.ornl.gov/ghpn/wk2003
  • Science Case for Large Scale Simulation, June
    2003
  • http//www.pnl.gov/scales/
  • Workshop on the Road Map for the Revitalization
    of High End Computing, June 2003
  • http//www.cra.org/Activities/workshops/nitrd
  • http//www.sc.doe.gov/ascr/20040510_hecrtf.pdf
    (public report)
  • ASCR Strategic Planning Workshop, July 2003
  • http//www.fp-mcs.anl.gov/ascr-july03spw
  • Planning Workshops-Office of Science
    Data-Management Strategy, March May 2004
  • http//www-conf.slac.stanford.edu/dmw2004
  • For more information contact Chin Guok
    (chin_at_es.net). Also see
  • - http//www.es.net/oscars

64
  • Additional Information

65
LHC Tier 0, 1, and 2 Connectivity Requirements
Summary
Vancouver
CERN-1
CANARIE
USLHCNet
Seattle
Toronto
Abilene / Gigapop Footprint
CERN-2
Virtual Circuits
ESnet SDN
Boise
CERN-3
Chicago
New York
Denver
Sunnyvale
KC
GÉANT-1
ESnet IP Core
Wash DC
LA
Albuq.
GÉANT-2
San Diego
GÉANT
Atlanta
Dallas
Jacksonville
USLHC nodes
  • Direct connectivity T0-T1-T2
  • USLHCNet to ESnet to Abilene
  • Backup connectivity
  • SDN, GLIF, VCs

Abilene/GigaPoP nodes
ESnet IP core hubs
Tier 1 Centers
ESnet SDN/NLR hubs
Cross connects with Internet2/Abilene
Tier 2 Sites
66
Example Case Study Summary Matrix Fusion
  • Considers instrument and facility requirements,
    the process of science drivers and resulting
    network requirements cross cut with timelines

67
The Increasing Dominance of Science Traffic
Traffic Volume of the Top 100 AS-AS Flows by
Month(Mostly Lab to RE site, a few Lab to RE
network all science)
68
Parallel Data Movers now Predominate
Look at the hosts involved in 2006-01-31 the
plateaus in the host-host top 100 flows are all
parallel transfers (thx. to Eli Dart for this
observation)
69
OSCARS Reservations
  • A user submits a request to the RM specifying
    start and end times, bandwidth requirements, the
    source and destination hosts
  • Using the source and destination host information
    submitted by the user, the ingress and egress
    border routers, and circuit path (MPLS LSP) is
    determined
  • This information is stored by the BSS in a
    database, and a script periodically checks to see
    if the PSS needs to be contacted, either to
    create or tear down the circuit
  • At the requested start time, the PSS configures
    the ESnet provider edge (PE) router (at the start
    end of the path) to create an LSP with the
    specified bandwidth
  • Each router along the route receives the path
    setup request via the Reservation Resource
    Protocol (RSVP) and commits bandwidth (if
    available) creating an end-to-end LSP. The RM is
    notified by RSVP if the end-to-end path cannot be
    established.
  • Packets from the source (e.g. experiment) are
    routed through the sites LAN production path to
    ESnets PE router. On entering the PE router,
    these packets are identified and filtered using
    flow specification parameters (e.g.
    source/destination IP address/port numbers) and
    policed at the specified bandwidth. The packets
    are then injected into the LSP and switched
    (using MPLS) through the network to its
    destination (e.g. computing cluster).
  • A notification of the success or failure of LSP
    setup is passed back to the RM so that the user
    can be notified and the event logged for auditing
    purposes
  • At the requested end time, the PSS tears down the
    LSP

70
Inter-domain Reservations Tough Problem
  • Motivation
  • For a virtual circuit service to be successful,
    it must
  • Be end-to-end, potentially crossing several
    administrative domains
  • Have consistent network service guarantees
    throughout the circuit
  • Observation
  • Setting up an intra-domain circuit is easy
    compared with coordinating an inter-domain
    circuit
  • Issues
  • Cross domain authentication and authorization
  • A mechanism to authenticate and authorize a
    bandwidth on-demand (BoD) circuit request must be
    agreed upon in order to automate the process
  • Multi-domain Acceptable Use Policies (AUPs)
  • Domains may have very specific AUPs dictating
    what the BoD circuits can be used for and where
    they can transit/terminate
  • Domain specific service offerings
  • Domains must have way to guarantee a certain
    level of service for BoD circuits
  • Security concerns
  • Are there mechanisms for a domain to protect
    itself (e.g. RSVP filtering)

71
Inter-domain Path Setup
Routed path from Host B to Host A (via ISP X)
2
ISP A
ISP B
Host A
Host B
OSCARS
RM A
1
Routed path from Host A to Host B (via ISP Y)
  • On receiving the request from the user, OSCARS
    computes the virtual circuit path and determines
    the downstream AS (ISP X).
  • The request is then encapsulated in a message
    forwarded across the network (ISP X) towards Host
    A, crossing all intervening reservations systems
    (RM X), until it reaches the last reservation
    system (RM A) that has administrative control
    over the network (ISP A) that Host A is attached
    to.
  • The remote reservation system (RM A) then
    computes the path of the virtual circuit, and
    initiates the bandwidth reservation requests from
    Host A towards Host B (via ISP Y). This can be
    especially complex when the path back (from Host
    B to A) is asymmetric and traverses ASs (e.g.
    ISP Y) that were not traversed on the forward
    path, causing the local OSCARS to see the path
    originating from a different AS than it
    originally sent the request to.
About PowerShow.com