Design of a Diversified Router: Line Card

1
Design of aDiversified Router Line Card
John DeHartjdd_at_arl.wustl.edu http//www.arl.wust
l.edu/arl
2
Revision History

• 5/22/06 (JDD)
• Updates to data passing from Block to Block.
• Buffer descriptor stuff probably needs updating.
• 5/31/06 (JDD)
• Clean up packet formats going from MR ? LC_Egress
• Add MN Shim to
• 6/2/06 (JDD)
• Removed old slides that showed Table lookup stuff
• Updated internal frame formats
• Still need to revisit buffer descriptor stuff and
  block to block data formats.
• 6/11/06 (JDD)
• Begin updating data going between blocks
• Still more to do
• 6/15/06 (JDD)
• Removed the CRC from the Rx ? Key Extract data.
• MSF does not pass us a CRC like we thought so we
  will skip the CRC checking
• 6/19/06 (JDD)
• Updating data formats that go between blocks.
• 6/29/06 (JDD)

3
Design Implementation

• We will now look at how our model might be
  implemented.
• Again, the primary focus will be on wired
  Ethernet as the access technology.
• We will look at RX and TX separately.
• First we will focus on the Substrate Router
  functionality in the LC.
• As we do we will also show the packet formats as
  they traverse other parts of the Substrate
  Router.
• Some of this will apply only to the Shared NP
  case.
• For the sake of simpler diagrams we will reduce
  the IXP PE functional blocks as shown below
• Then we will look at the implementation of a
  common router framework in the IXP PE.

4
Design Implementation
Packet arriving On Port N
Packet leaving On Port M
LC
Txsubstrate
MR
Rxsubstrate
Switch
Switch

IXP PE (Shared)
RX
TX
5
Substrate Link Configuration

• RX Rules for determining type of Frame/Substrate
  Link
• P2P-DC is pre-configured on an interface by
  interface basis.
• ((EtherType Substrate) AND (VLANVLAN0)) ?
  P2P-VLAN0 SL
• VLAN0 is a predefined VLAN Id for use by the
  Substrate Network to connect peer substrate
  routers on a Multi-Access network
• SL id Senders Ethernet Address
• Lookup MLI in SL Specific Table ? MRMI
• ((EtherType Substrate) (VLAN ? VLAN0)) OR
• ((EtherType ARP) and (ProtoType Substrate))
  ? Multi-Access
• MLI is unique across a Multi-Access Substrate
  Link
• MLI Lookup in Multi-Access-SL Table ? MRMI
• ((EtherType ARP) and (ProtoType ? Substrate))
  OR
• ((EtherType ? Substrate) AND
• (EtherType ? ARP)) ? Tunnel or Legacy
• Substrate Link in a Tunnel (e.g. IPv4)
• ((EtherType IP) AND (IP_Proto Substrate)) ?
  Substrate Tunnel in IPv4
• Legacy (e.g. IPv4, ARP)
• ((EtherType IP) AND (IP_Proto TCP)) ? Legacy
  IPv4
• ((EtherType ARP) AND (ProtoType ? Substrate)) ?
  Legacy ARP
• Etc.

6
Substrate Link Configuration

• TX
• Substrate Link uniquely identified by MRMI tuple
• MRMI ? SL
• MetaLink uniquely identified by MRMI tuple
• MRMI ? MLI
• Hence
• MRMI ? SLMLI
• Different Header formats may be defined per SL
• i.e. IPv4 Tunnel header vs. P2P-VLAN0 header
• Located with lookup based on MLI
• P2P-VLAN0
• One or more MLI per MetaNet on the Multi-Access
  network
• Multi-Access
• One MLI per MetaNet on the Multi-Access network
• Meta-Destination Address, to get Ethernet Address
  use ARP
• Legacy
• Substrate Link in a Tunnel
• IPv4
• Gateway

7
Ingress LC Input Frame All SL Types
VLAN-tagged formats

• Blue Shading Determine SL Type
• Black Outline Key Fields from pkt

DstAddr (6B)
SrcAddr (6B)
Type802.1Q (2B)
TCI ? VLAN0 (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
ProtocolSubstrate (1B)
Hdr Cksum (2B)
Src Addr (4B)
Dst Addr (4B)
MLI (2B)
LEN (2B)
Meta Frame
PAD (nB)
CRC (4B)
P2P-DC (Configured)
P2P-Tunnel
8
Ingress LC Input Frame All SL Types
Non VLAN-tagged formats

• Blue Shading Determine SL Type
• Black Outline Key Fields from pkt

DstAddr (6B)
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
TypeIP (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
DstAddr (6B)
DstAddr (6B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
SrcAddr (6B)
SrcAddr (6B)
ProtocolSubstrate (1B)
Protocol (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
TypeSubstrate (2B)
TypeSubstrate (2B)
Src Addr (4B)
Src Addr (4B)
MLI (2B)
MLI (2B)
Dst Addr (4B)
Dst Addr (4B)
LEN (2B)
LEN (2B)
Meta Frame
MLI (2B)
Meta Frame
IP Payload
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
CRC (4B)
CRC (4B)
P2P-DC (Configured)
P2P-Tunnel
Legacy
Multi-Access
9
Ingress LC Input Frame All SL Types
Optional Extension GRE formats

• Blue Shading Determine SL Type
• Black Outline Key Fields from pkt

DstAddr (6B)
SrcAddr (6B)
DstAddr (6B)
Type802.1Q (2B)
SrcAddr (6B)
TCI ? VLAN0 (2B)
TypeIP (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
ProtocolGRE (1B)
ProtocolGRE (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)

Dst Addr (4B)
Dst Addr (4B)
Flags/recur/Ver (2B)
Flags/recur/Ver (2B)
TypeSubstrate (2B)

TypeSubstrate (2B)

Optional Fields (nB)
Optional Fields (nB)
MLI (2B)
MLI (2B)

LEN (2B)
LEN (2B)
Meta Frame
MLI (2B)
Meta Frame
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
VLAN GRE
GRE
10
Egress LC Input Frame

• VLAN Identifies MR
• TxMI With VLAN will identify SL and MLI
• Src MPE
• LEN Length in bytes of Meta Frame
• Shim Flags define what type of Shim Data is
  being given
• or perhaps what type is needed (maybe Substrate
  provides it for broadcast)
• Type (2b)
• NhMnAddr (01b)
• We may need to use ARP to translate to
  MAC/Ethernet Address
• MAC Address(10b)
• Could be used for Broadcast/Multicast
• MN Shim(11b)
• NhAddr field is used as MN Shim for PE to PE info
• Things like
• why this frame is being sent on slow path
• This type should NEVER show up at a LC but should
  only be for PE to PE. One of the PEs could be the
  control processor.
• NULL (00b) No Next Hop Address given or needed.
• Size (6b) Length of Shim Data field in Bytes
• Shim Data See flags above

MR
Substrate
Switch

TxMI (2B)
Src MPE (2B)
LEN (2B)
Shim Flags(2B)
Shim Data (nB)
Meta Frame
11
Substrate/MetaNet Model

• To the Substrate, some Meta Links Pass Through
• Pass through a Substrate Router without visiting
  a Meta Router

MetaLink 1
Substrate Link
MetaLink 2
MetaLink 3
MR_A
MR_A
MR_A
MI
MI
MI
MI
MI
MI
MR_B
MR_B
Pass Through Meta Link
MI
MI
MI
MI
MR_C
MR_C
MR_C
MI
MI
MI
MI
MI
MI
Substrate Router X
Substrate Router Z
Substrate Router Y
12
Pass-Through MetaLink

• For Pass-Through ML, there is no support or need
  for NhAddr for the case where the SL leaving LC Y
  might be multi-access
• Shim Flags 0, Shim Data Length 0
• Allocate special set of MR/MI for use by
  Substrate when handling Pass-Through MetaLinks
• TxMI RxMI

Packet arriving On Port N
Packet arriving At LC Y
Packet arriving On Port N Of LC X
Ethernet Switch
LC Y
LC X

13
LC Functional Blocks
Lookup (2 ME)
Switch Tx (2 ME)
QM/Schd (2 ME)
Hdr Format (1 ME)
S W I T C H
Phy Int Rx (2 ME)
Key Extract (2 ME)
Lookup (2 ME)
Phy Int Tx (2 ME)
QM/Schd (2 ME)
Hdr Format (1 ME)
Key Extract (1 ME)
Switch Rx (2 ME)
Rate Monitor (1 ME)

• ME counts are my first guesses for number needed.
• For each block Ill show
• Function
• What this block does.
• Memory Accesses
• SRAM
• DRAM
• Buffer Descriptor Accesses
• Which fields in Buffer Descriptor the block needs
  to read and/or write.
• Notes
• Additional thoughts about this block.
• Next
• Analyze the SRAM Accesses and try to map them on
  to the available SRAM Channels.
• Analyze the SRAM and DRAM accesses and calculate
  packet processing rates.

14
LC Functional Blocks

• Ingress (Physical Interface ? Switch)
• PhyInt Rx
• KeyExtractor
• Lookup
• Hdr Format
• QM/Scheduler
• Switch Tx
• Egress (Switch ? Physical Interface)
• Switch Rx
• KeyExtractor
• This is only extracting the VLAN and the MI.
  Combine with previous or following block?
• But it involves a DRAM Read so we probably want
  to leave it separate and use all 8 threads.
• RateMonitor
• Before or after the Lookup?
• Is this different than what QM/Scheduler
  will/could do?
• Lookup
• Hdr Format
• QM/Scheduler
• PhyInt Tx

15
LC Buffer Descriptor

• Hopefully we can use the same buffer descriptor
  for the LC and the CRF Processing Engine.
• There might be some fields that are used on one
  and not on the other but thats ok (MR_ID, TxMI,
  VLAN not needed on LC)
• PE Buffer Descriptor
• LW0 buffer_next 32 bits Next Buffer Pointer
  (in a chain of buffers)
• LW1 offset 16 bits Offset to start of data
  in bytes
• LW1 BufferSize 16 bits Length of data in the
  current buffer in bytes
• LW2 reserved 8 bits reserved/unused
• LW2 reserved 4 bits reserved/unused
• LW2 free_list 4 bits Freelist ID
• LW2 packet_size 16 bits (Total packet size
  across multiple buffers)
• LW3 MR_ID 16 bits Meta Router ID
• LW3 TxMI 16 bits Transmit Meta Interface
• LW4 VLAN 16 bits VLAN
• LW4 reserved 16 bits reserved/unused
• LW5 reserved 32 bits reserved/unused
• LW6 reserved 32 bits reserved/unused
• LW7 packet_next 32 bits pointer to next packet
  (unused in cell mode)
• Leave multi-buffer fields there as a template for
  the dedicated blade implementation of a
  jumbo-frame MR.

16
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
RBUF

• Rx
• Function
• Coordinate transfer of packets from RBUF to DRAM
• Memory Accesses
• SRAM
• Write Buffer Descriptor
• Free List?
• DRAM Transfer from RBUF
• Buffer Descriptor Accesses
• Write/Initialize Buffer_Next, Buffer_Size,
  Offset, Free_List, Packet_Size
• Monitoring
• Per Physical Interface
• Pkt Counter
• Byte Counter
• Notes
• Buffer Handle
• contains the SRAM address of the buffer
  descriptor.
• from the SRAM address of the descriptor we can
  calculate the DRAM address of the buffer data.
• Offset of where packet starts should be a
  constant.

17
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
Buf Handle(32b)
Eth Frame Length (12b)
MN Offset (8b)
Lookup Key 71-64 (8b)
Rsv (4b)
Port (8b)
Reserved (8b)
Eth. Frame Len (12b)
Rsv (4b)
Lookup Key63-32 (32b)
Lookup Key 31-0 (32b)

• Key_Extract (2 Microengines)
• Function
• Extracts lookup key based on type of frame that
  was received.
• Peel ARP packets off and send to XScale
• Compare CRC from Rx to CRC at end of Frame.
• Drop if fail.
• Memory Accesses
• DRAM
• Read as much of header as is necessary to extract
  key
• May vary depending on type of Substrate Link
• SRAM None
• Buffer Descriptor Accesses None
• Monitoring
• ARP Pkt Counter
• Notes
• Calculates DRAM Address based on SRAM descriptor
  address in buffer handle and the Offset passed to
  it by RX.
• Frame offset in buffer is a constant and does not
  need to be read from Buffer Descriptor

18
LC Ingress Functional Blocks

• Substrate Link Type
• Will be used as Database ID by Lookup Block
• Lookup Keys

SL Type (SL Type ID) (size)
MLI(16b)
P2P-DC(0x0) (24 bits)
SL(4b) 0000
Port (4b)
MLI (16b)
IP SAddr (32b)
EtherType (16b) 0x0800
P2P-Tunnel - IPv4(0x1) (72 bits)
SL(4b) 0001
Port (4b)
MLI (16b)
Ethernet SAddr (48b)
P2P-VLAN0(0x2) (72 bits)
SL(4b) 0011
Port (4b)
MLI(16b)
MA(0x3) (24 bits)
SL(4b) 0100
Port (4b)
EtherType (16b) 0x0800
Legacy IPv4(0x4) (24 bits)
SL(4b) 0010
Port (4b)
Physical Interface
Substrate Link Type
19
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
Eth Frame Length (12b)
MN Offset (8b)
Lookup Key 71-64 (8b)
Rsv (4b)
Lookup Key63-32 (32b)
Lookup Key 31-0 (32b)

• Lookup
• Function
• Performs Lookup and passes result on to Hdr
  Format.
• Memory Accesses
• DRAM None
• SRAM
• TCAM Lookup
• Write Lookup command
• Read Lookup Result (1-5 words from TCAM SRAM
  Controller or other SRAM Controller)
• Buffer Descriptor Accesses None
• Monitoring
• Notes
• Lookup does no processing on the lookup result.
• Need to decide how lookup result will be stored
  and retrieved.
• See notes on TCAM for information about the
  issues involved.

20
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
Eth Frame Length (12b)
DAddr (8b)
MN Offset (8b)
Port (4b)
RxMI (16b)
Stats Index (16b)
VLAN (MR Id) (12b)
QID (20b)

• Hdr Format
• Function
• From lookup result
• re-writes headers in DRAM to make frame ready to
  transmit.
• Extract QID to pass on to QM/Scheduler
• Memory Accesses
• DRAM
• SRAM
• Read Descriptor and Re-Write Descriptor
• OR
• Atomic Increment/Decrement some fields in
  Descriptor
• Buffer Descriptor Accesses
• Update Size and Offset fields
• Monitoring
• Notes
• Pass Size on to QM/Scheduler so it does not have
  to read buffer descriptor for Enqueue to update Q
  Length.
• Buffer Offset should be a constant and should not
  need to be read from Buffer Descriptor

21
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
V Valid Bit

• QM/Scheduler (See Saileshs slides for more
  details)
• Function
• Enqueue and Dequeue from queues
• Scheduling algorithm
• Drop Policy
• Memory Accesses
• DRAM None
• SRAM
• Q-Array Reads and Writes
• Scheduling Data Structure Reads and Writes
• QLength Data Structure Reads and Writes
• Dequeue Read Buffer Descriptor to retrieve
  Packet Size
• Buffer Descriptor Accesses Read packet size
• Monitoring
• Queue Lengths
• Drops
• Notes

22
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
TBUF
V Valid Bit

• Switch TX
• Function
• Coordinate transfer of packets from DRAM to TBUF
• Memory Accesses
• SRAM Read Buffer Descriptor
• DRAM Transfer to TBUF
• Buffer Descriptor Accesses
• Read Size and Offset
• Monitoring
• Per Physical Interface
• Pkt Counter
• Byte Counter
• Notes
• Calculate DRAM address based on SRAM Descriptor
  address in buffer handle

23
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor
RBUF

• Rx
• Function
• Coordinate transfer of packets from RBUF to DRAM
• Memory Accesses
• SRAM Write Buffer Descriptor
• DRAM Transfer from RBUF
• Buffer Descriptor Accesses
• Write/Initialize Buffer_Next, Buffer_Size,
  Offset, Free_List, Packet_Size
• Notes
• Buffer Handle
• contains the SRAM address of the buffer
  descriptor.
• from the SRAM address of the descriptor we can
  calculate the DRAM address of the buffer data.
• Passing the offset of where the packet starts in
  memory will save the next block from having to
  read the buffer descriptor.
• Perhaps we should just pass the actual DRAM
  Buffer Pointer?

24
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor

• Key_Extract
• Function
• Extracts lookup key based on type of frame that
  was received.
• Memory Accesses
• DRAM
• Read VLAN and TxMI from Frame
• SRAM None
• Buffer Descriptor Accesses None
• Notes
• Calculates DRAM Address based on SRAM descriptor
  address in buffer handle and the Offset passed to
  it by RX.

25
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor

• Rate Monitor
• Function
• Ensures that MR/MIs behave according to their
  Rate Specs.
• Does this need to be a separate function from the
  QM/Scheduler?
• Memory Accesses Unknown at this point
• DRAM
• SRAM
• Buffer Descriptor Accesses Unknown at this point
• Notes

26
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor

• Lookup
• Function
• Performs Lookup and passes result on to Hdr
  Format.
• Memory Accesses
• DRAM None
• SRAM
• TCAM Lookup
• Write Lookup command
• Read Lookup Result (1-5 words from TCAM SRAM
  Controller or other SRAM Controller)
• Buffer Descriptor Accesses None
• Notes
• Lookup does no processing on the lookup result.
• Need to decide how lookup result will be stored
  and retrieved.
• See notes on TCAM for information about the
  issues involved.

27
LC Egress Lookup Result Data Formats
P2P-DC
P2P-Tunnel_IPv4 (w/o VLAN)
MA (w/o VLAN)
MA (with VLAN)
P2P-Tunnel_IPv4 (with VLAN)
MLI(16b)
ETYpe(16b)
Rsv (4b)
Port (4b)
SL (4b)
QID(20b)
Rsv (16b)
VLAN (16b)
P2P-VLAN0
Legacy IPv4 (w/o VLAN)
Legacy IPv4 (with VLAN)
28
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor

• Hdr Format
• Function
• From lookup result
• re-writes headers in DRAM to make frame ready to
  transmit.
• Extract QID to pass on to QM/Scheduler
• Memory Accesses
• DRAM
• SRAM
• Read Descriptor and Re-Write Descriptor
• OR
• Atomic Increment/Decrement some fields in
  Descriptor
• Buffer Descriptor Accesses
• Update Size and Offset fields
• Notes
• Pass Size on to QM/Scheduler so it does not have
  to read buffer descriptor for Enqueue to update Q
  Length.
• Use a special VLAN value for determining if we
  need to do 802.1Q or not?

29
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor
V Valid Bit

• QM/Scheduler (See Saileshs slides for more
  details)
• Function
• Enqueue and Dequeue from queues
• Scheduling algorithm
• Drop Policy
• Memory Accesses
• DRAM None
• SRAM
• Q-Array Reads and Writes
• Scheduling Data Structure Reads and Writes
• QLength Data Structure Reads and Writes
• Dequeue Read Buffer Descriptor to retrieve
  Packet Size
• Buffer Descriptor Accesses Read packet size
• Notes

30
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Rate Monitor
TBUF
V Valid Bit

• Switch TX
• Function
• Coordinate transfer of packets from DRAM to TBUF
• Memory Accesses
• SRAM Read Buffer Descriptor
• DRAM Transfer to TBUF
• Buffer Descriptor Accesses
• Read Size and Offset
• Notes
• Calculate DRAM address based on SRAM Descriptor
  address in buffer handle

31
ML Loopback Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Loopback Hdr Re-Format
Lookup
Phy Int Tx
QM/Schd
Hdr Format
Key Extract
Switch Rx
Rate Monitor

• Loopback path should be able to re-use some of
  the blocks implemented for the LC
• Loopback Hdr Re-Format
• Needs to be able to strip off the previous MRs
  Header.
• For Plain-IP ? IPv4 MR ? MR Y
• When frame arrives at Loopback (between IPv4 MR
  and MR Y) it will still have IP Header which
  should be stripped off before frame is sent to MR
  Y.
• Lookup Result should probably include a length
  field or a Buffer offset field that indicates
  where new Meta Frame should start.

32
LC Notes on TCAM Lookups

• See techX_Design_TCAM_Usage.ppt slides for notes
  on how the TCAM will be used by the Lookup Block

33
Extra

• The next set of slides are for templates or extra
  information if needed

34
Text Slide Template
35
Image Slide Template