Size Matters: Network Performance on Jumbo Packets Summer 2004 Joint Techs Loki Jorgenson, Chief Scientist, Apparent Networks

1
Size MattersNetwork Performance on Jumbo
Packets Summer 2004 Joint Techs Loki
Jorgenson, Chief Scientist, Apparent Networks

2
9k MTU Project(s)

• test global path MTU on Abilene, CAnet4, CUDI
  and other R E networks, plus create a useful
  researcher mapping tool
• Internet2 ATEAM - Advanced Test Engineering and
  Measurement
• Bill Rutherford (Rutherford Research/GAIT)
• Kevin Walsh, Nathaniel Mendoza (San Diego
  Supercomputing Center/SDSC)
• John Moore (Centaur Internet2 Technology
  Evaluation Center ITEC/NCSU North Carolina State
  University)
• Loki Jorgenson (Apparent Networks/SFU)
• CAnet testing
• Bryan Caron (Network Manager Subatomic Physics,
  University of Alberta)
• Damir Pobric (Network engineer - CANARIE)

3
Why Jumbo?

• Performance Benefits
• High performance data transfers
• Grid networks
• Meteorology / physics / biotech
• Collaborative/interactive multi-media
• Performance Requirements
• End-to-end path
• From NIC to NIC MTU requirement
• End station is typically the bottleneck
• Gig-E to the desktop

4
Steady State TCP

• If TCP window size and network capacity are not
  rate limiting factors then (roughly)
• 0.7 Max Segment Size (MTU)
• e2e throughput lt
• Round Trip Time (latency)
  sqrtloss
• M. Mathis, et.al.
• Double the MTU, double the throughput
• Half the latency, the throughput
• (shortest path matters)
• Half the loss rate, 40 higher throughput

5
Frame Size vs. MTU vs. MSS An Ethernet Example
6
MTU Performance Testing ATEAM

• Objectives
• Measurements using multiple methodologies
• SmartBits
• iPerf
• AppareNet/aNA
• Determine current dependency on transmission
  packet size
• Determine extent of jumbo MTU access

7
About aNA

• AppareNet Network for Academics
• Currently 16 sequencers across CAnet and Abilene
• NIS in Vancouver, Canada
• 10 Gig-E/Jumbo hosts
• 4 hosts in Canada
• BCnet
• Netara Alliance
• CAnet NOC
• ACORN-NS

• Upgrade ANA to v2.5 (mid-August 2004) from
  v1.6.2
• Web access
• On-demand deployment

8
About AppareNet

• Uses light, non-intrusive active probing
• ICMP or UDP packets in various configurations
• Point-and-shoot to most IP addresses
• Performs network path characterization
• Performs expert system diagnostics
• Single-ended ? two-way measures (e.g.
  half-duplex different from full-duplex)
• Samples network to generate same view as best
  effort application (pre-TCP)

9
Example Measurement Paths SDSC to Ottawa
10
Example Measurement Paths SDSC to Halifax
11
Example Measurement Paths SDSC to Mexico
12
Why use Jumbo Frames?
9000 MTU
8192 MTU
7168 MTU
6144 MTU
5120 MTU
4096 MTU
3072 MTU
2048 MTU
512 MTU

• Increasing MTU gives better performance.
• Advanced Test Engineering and Measurement (ATEAM)
  performance measurements taken across the Abilene
  and CAnet4 backbone
• http//www.ncne.nlanr.net/training/techs/2003/0803
  /presentations/0803-moore1_files/v3_document.htm

13
More GigE Test Results
14
Preventing MTU conflicts Network Negotiation
15
GigE Black Hole Hop

• What is happening?
• RFC 1191 and TCP Slow Start are interacting
• Packets are lost
• Retransmission happen, causing performance
  degradation
• Client responds to some packets, keeping
  connection open
• Overall performance appears slow to client

16
MTU handling via RFC 1191 PMTU discovery

• Advantages
• Router is not loaded
• Maximum performance achieved

• Disadvantages
• reliance on ICMP
• easy to misconfigure

• Applications
• almost all modern applications

17
Intel Pro XT 1000 version 6.2.22.1
18
Intel Pro XT 1000 version 7.0.36.0
19
Intel Pro XT 1000 version 7.4.19.0
20
Avoiding GigE MTU problems

• Maintain logical Layer 3 diagrams

• Assign MTUs based on a per-subnet basis

• Be consistent with MTU values used
• Use 1500 bytes for legacy Ethernet (no registry
  hacks)
• Use recommended 9000 bytes MTU for GigE when
  jumbo frames are used (standard for CAnet and
  Abilene networks)
• Dont forget to add 18 bytes when adjusting frame
  size (e.g. set NIC to 9018 bytes frame size to
  maintain a 9000 byte MTU)
• Dont arbitrarily filter out ICMP messages

21
Resources

• Path MTU tools
• ANA pMTU service uses ANA sequencers across I2
  and CAnethttp//pathmtu.apparenet.com8282/ana_at_
  apparenet.comguest42
• NCNE MTU Discovery Service uses service located
  at NCNE http//www.ncne.org/jumbogram/mtu_discover
  y.php
• pMTU Applet - Java-based client for end-user
  station http//sourceforge.net/projects/mtu-calcu
  lator/ http//ana.apparenet.com8282/pMTU/
  Download
• Jumbo MTU Performance whitepaper
• http//www.apparentNetworks.com/wp/

22
Path MTU Mapping Service
23
pMTU Java Client
pMTU applet available from SourceForge
24
pMTU Database and Map Visualization
pMTU Database and MTU map
25
pMTU Project Status

• Initial prototypes available
• Open source (SourceForge) projects

• pMTU Applet
• Java/Swing client
• Webstart
• Native code driver
• Windows only
• Linux-ready

• pMTU DB/Map
• mySQL/Servlet
• accepts connections
• limited map capability
• saves Applet results

26
Current Work Application Performance

• End-user Performance Benefits
• End-to-end requirement
• Very limited jumbo-access from campus
• Influence of real-world application dynamics
• Identify classes of need that require jumbo
• Identify classes of implementation susceptible to
  jumbo

27
Target Implementations

• Distributed File Systems
• Simple, commonly useful utility
• Challenged by latency, loss
• Often data-intensive
• Persistent use (not one-time)
• Non-denominational

• Visualization Server
• Visual/subjective benefit
• Very demanding
• Resolution dependent load
• Specific to particular fields/users

28
Resources

• On-Line Path MTU tools
• ANA pMTU service uses ANA sequencers across I2
  and CAnethttp//pathmtu.apparenet.com8282/ana_at_
  apparenet.comguest42
• NCNE MTU Discovery Service uses service located
  at NCNE http//www.ncne.org/jumbogram/mtu_discover
  y.php
• pMTU Applet - Java-based client for end-user
  station http//sourceforge.net/projects/mtu-calcu
  lator/ http//ana.apparenet.com8282/Download
• Jumbo MTU Performance whitepaper
• http//www.apparentNetworks.com/wp/

29
Fin

• End of Presentation
• aNA.apparenet.com

30
Fin

• Backup Slides

31
GigE Black Hole Hop
32
MTU handling via fragmentation
33
MTU handling via RFC 1191 PMTU discovery