TCP Offload Through Connection Handoff

1
TCP Offload Through Connection Handoff

• Hyong-youb Kim and Scott Rixner
• Rice University
• April 20, 2006

2
Full TCP Offloading

• Move all TCP/IP processing to the network
  interface
• Computation
• Saves processing resources on the host
• NIC can be customized for TCP/IP processing
• Memory
• Reduces host memory references
• Network interface can exploit small, fast, local
  memory
• Problems
• Network interface can become a performance
  bottleneck
• Limited computation on NIC
• Limited memory capacity on NIC
• Complicates global resource management in the
  stack

3
Solution Connection Handoff

• Only handoff established connections to NIC
• Operating system controls division of work
• Only TCP send and receive on the NIC
• OS performs connection establishment, routing,
• No changes to sockets API
• SPECweb99 performance
• 17 and 32 reduction in cycles per packet
• 15 and 27 improved throughput

4
Unmodified Network Stack

• 3100 instructions per packet
• 50 of all operations are memory references

User Application
5
Network Stack with Connection Handoff
User Application
Socket
Bypass
TCP
IP
Host OS
Ethernet
Driver
Socket
TCP
NIC
IP
Ethernet
Lookup
Transmit/Receive
6
Handoff Interface

• Extend driver/OS API
• Move connections
• Handoff (OS) move connection from OS to NIC
• Restore (OS, NIC) move connection from NIC to OS
• Relay socket operations between OS and NIC
• Send (OS) insert send data into NIC's socket
• Acknowledge (NIC) remove ack'ed data from OS's
  socket
• Receive (NIC) insert received data into OS's
  socket
• Received (OS) remove received data from NIC's
  socket
• Control (OS, NIC) change socket states, etc.
• Misc.
• Forward (OS), Post (OS), Resource (NIC)

7
Example Use
Accept connection, receive request, send
response, close connection
NIC
Host OS
Handoff Command
8
Real Prototype

• Modified FreeBSD 4.7
• AMD Athlon XP 2800 CPU
• Alteon programmable Gigabit Ethernet NIC
• 1MB memory
• Limited to 256 connections
• Actual socket buffer data only in main memory
• 88MHz processor
• Limits maximum throughput

9
TCP Send
0
System
Total
TCP
Bypass
IP
Ethernet
Driver
Call
10
Simulated Machine

• Prototype NIC is too slow
• Simics full-system simulator
• Boots unmodified operating systems
• Use same software as real prototype
• Simulated processor
• 1GHz functional x86 processor
• Timed memory to mimic Athlon XP 2800
• Simulated NIC
• 450MHz functional processor
• Timed 1Gb/s Ethernet wire

11
SPECweb99, 1024 Connections
14000
15 increase in HTTP throughput (Mb/s)
12000
10000
27 increase in HTTP throughput (Mb/s)
8000
Cycles per packet
6000
4000
2000
0
System Call
TCP
IP
Ethernet
Driver
Bypass
Total
12
Summary

• Memory behavior limits TCP performance
• Connection state accesses cause cache pressure
• Offload can help, but full offload is problematic
• Connection handoff offloading made practical
• OS in charge of division of work
• Host network stack largely unaffected
• Ongoing work OS handoff policies