Title: Design of a Diversified Router: Dedicated CRF for IPv4 Metarouter
1Design of aDiversified RouterDedicated CRFfor
IPv4 Metarouter
John DeHart, Brandon Heller jdd_at_arl.wustl.edu,
bdh4_at_cec.wustl.edu http//arl.wustl.edu/projects/t
echX/
2Revision History
- 5/22/06 (JDD)
- Created
- Buffer descriptor stuff probably needs updating.
- 6/1/06 (JDD)
- Updating data going between blocks, still in
progress. - 6/2/06 (JDD)
- More cleanup of data going between blocks.
- Buffer descriptor details still need updating.
- 6/5/06 (JDD)
- Slight change to format for Lookup Key and
defining what goes in each word in the NN ring. - Add IP Pkt Length to data Demux passes to Parse
- 6/6/06 (JDD)
- Reorganized the Lookup Result given to Hdr Format
to distinguish between MR portion and Substrate
portion. - Clean up labeling of data to Parse (MN vs. IP
Pkt) - Output from Parse is still IP Pkt Offset and
Length. - Data from Parse to Lookup needs update to reflect
case where lookup is just for Substrate mapping
of MI to LC. - 6/7/06 (JDD)
- Updated notes about Parse blocks input/output
and functionality - 6/15/06 (JDD)
3Revision History
- 6/19/06 (BDH)
- Split Header Format into MR Header Format and
Substrate Encap - Demux is now Substrate Decap
- Reorganization of all slides into logical and
physical formats, coloring scheme - IPv4 MR now has own section, integrated JLs
internal format slides - 6/21/06 (BDH)
- H Flags nuked
- MN Pkt Length into Lookup is now
substrate-defined - Logical communication added from Lookup to
Substrate Encap - Port fields are all 4 bits now
- 6/26/06 (BDH)
- Substrate Decap to Parse format changed
- Changed block diagram to better show that
substrate encloses the MR-specific portions - Added details on Substrate Decap
- Moved IPv4 slides to techX\bdh4\techx\IPv4_MR_shar
ed. These slides should be done by mid-July. - 6/29/06 (BDH)
- Updated format of Tx input
- 6/30/06 (BDH)
- Updated format of Hdr Format to Substrate Encap
data, only handles IPv4 NH_MN_ADDRs now
4Dedicated CRF Slide Organization
Block
Substrate
Input Data
Output Data
Metarouter
- In the at-a-glance format, all blocks are
logical - Logical inputs and outputs
- High-level overview of processing
- Each logical block is like an Intel microblock,
not necessarily an ME - In the detailed format, all blocks are physical
- Physical inputs and outputs
- Specific functionality and implementation notes
- Color scheme
- Blue Substrate, should not change!
- Green Metarouter, different for each MR
5Logical Formats
6Receive
Rx
Buffer Handle
RBUF
Ethernet Frame Len
Port
- Coordinate transfer of packets from RBUF to DRAM
7Substrate Decap
Buffer Handle
Substr Decap
Buffer Handle
Destination MPE
Ethernet Frame Len
Source ID
Port
MN Frame Length
MN Frame Offset
- Read and validate Ethernet header from DRAM
- Read and validate substrate header from DRAM
- Extract Source ID
- Calculate MN frame length and offset
8Parse
Buffer Handle
Buffer Handle
Parse
Destination MPE
Lookup Flags
to Lookup
Source ID
Lookup Key
MN Frame Length
Source ID
MN Frame Offset
MN Pkt Length
to MR Hdr Format
MR Data
- Substrate matches the destination MPE
- Read and align MN header (includes IPv4 Hdr) from
DRAM - MR-specific
- Consume internal header (if packet from other MPE
of MR) - Header validation
- Header modification
- Exception checks
- Extract lookup key and set lookup flags
- Write aligned modified IPv4 header back to DRAM
9Lookup
Buffer Handle
Lookup
Buffer Handle
to MR Header Format
Lookup Input Flags
Lookup Result Flags
Lookup Key
MR Lookup Result
Source ID
MN Pkt Length
Dest Addr
to Substrate Encap
Output Port
QID
- Perform lookup in TCAM
- Increment counters based on Stats Index
- Priority resolution of results from multiple
databases, if needed
10Header Format
Buffer Handle
MR Hdr Format
Buffer Handle
MN Frame Length
from Lookup
Lookup Result Flags
MN Frame Offset
MR Lookup Result
Substrate Type
Substr. Type-dep. Data
MR Data
from MR Parse
- Process Lookup result
- For exceptions, generate internal header
- Decide substrate type
11Substrate Encap
Buffer Handle
Buffer Handle
MN Frame Length
Substr Encap
MN Frame Offset
Output Port
from MR Header Format
Substrate Type
QID
Substr. Type-dep. Data
MN Frame Length
Dest Addr
Output Port
from Lookup
QID
- Write substrate and ethernet headers
12Queue Manager
Buffer Handle
QM
Output Port
QID
MN Frame Length
- CRF queue management for Meta Interface queues
- WRR?
- Details
13Transmit
Tx
TBUF
- Coordinate transfer of packets from DRAM to TBUFs
- Recycle buffer handle
14Physical Formats
15Receive
RBUF
will be
currently
- RBUF format details here
- Buf Handle details here
- Notes
- Well pass the Buffer Handle which contains the
SRAM address of the buffer descriptor. - From the SRAM address of the descriptor we can
calculate the DRAM address of the buffer data.
16Substrate Decap
will be
currently
- SourceID
- specifies RxMI or MPE (each 15-bit)
17Substrate Decap Functions
- Read Ethernet VLAN and Substrate header from DRAM
- Validate Ethernet VLAN packet
- Valid Length?
- Known protocol (VLAN)?
- Broadcast/Multicast source?
- Multicast destination?
- Broadcast destination?
- Local Dest?
- Validate Substrate header
- Known substrate header type (Internal or
Ingress)? - Substrate-reported MN frm len Enet-deduced MN
frm len? - Fill NN ring fields
18Substrate Decap Implementation
- 8 threads, ordered thread execution
- 121 cycles per thread per packet, common case
- 670 cycles of latency, within 1360 cycle limit
for 8 threads - Resource use
- SRAM refs
- 1 per counter to increment (disabled currently)
- DRAM refs
- 3 8B reads Enet and Substrate header
- 2 8B reads Enet checksum
- Optimizations could reduce cycle count further
- projected 80-100 cycles
- combined initial error-check to remove branch
mispredicts - remove/combine DRAM read signals
- remove volatile keywords
- single-critical-section ordered threading
19Parse
Buf Handle(32b)
MR Data (28b)
L Flags (4b)
MR Data (16b)
MN Pkt Len (16b)
Lookup Key143-112 MR/MI (32b)
Lookup Key111-80 (32b)
Lookup Key 79-48 (32b)
Lookup Key 47-16 (32b)
Lookup Key 15- 0 (16b)
Reserved (16b)
- Can Parse adjust the buffer/packet size and
offset? - Can Parse do something like, terminate a tunnel
and strip off an outer header?
20Lookup
Buf Handle(32b)
Buf Handle(32b)
MR Data (28b)
Rsv (4b)
MR Data (28b)
L Flags (4b)
MR Data (16b)
MN Pkt Len (16b)
MR Data (16b)
MN Pkt Len (16b)
Lookup Key143-112 MR/MI (32b)
MR Lookup Result (32b)
Lookup Key111-80 (32b)
MR Lookup Result (32b)
Lookup Key 79-48 (32b)
Port (4b)
QID(20b)
DA(8b)
Lookup Key 47-16 (32b)
Lookup Key 15- 0 (16b)
Reserved (16b)
- L Flags
- bit 0 0 Normal, 1 Substrate Lookup
- bit 1 0 Normal, 1 NH MN Address present in Key
Word1 - Key Word0 MR/MI
- Bit 1 should never be set without bit 0 also
being set.
21Header Format
Buf Handle(32b)
MR Data (28b)
Rsv (4b)
MR Data (16b)
MN Pkt Len (16b)
MR Lookup Result (32b)
MR Lookup Result (32b)
Port (4b)
QID(20b)
DA(8b)
- Egress Simple and Internal formats use just the
dest ID, source ID, ad SH type - MAC fields used for MAC_ADDR Egress format
- NH MN Addr field used for NH_MN_Addr format
22Substrate Encapsulation
Buffer Handle(32b)
Port (4b)
QID(20b)
DA(8b)
MN Pkt Length (16b)
Reserved (16b)
- Substrate header types/formats here?
23Queue Manager
Buffer Handle(32b)
Port (4b)
QID(20b)
DA(8b)
MN Pkt Length (16b)
Reserved (16b)
24Transmit
TBUF
25IPv4 Metarouter
- Look at
- techx\bdh4\techx\IPv4_MR_shared
- for Metarouter-specific IPv4 slides
26Extra
- The next set of slides are for templates or extra
information if needed
27Text Slide Template
28Image Slide Template
29At-a-glance Block Template
Block
Buffer Handle
RBUF
Ethernet Frame Len
Port
30Detailed Block Template
RBUF
31QM/Scheduler on Multiple MEs
QM/Schd (1 ME)
Input Hlpr (1 ME)
HeaderFormat
Tx
QM/Schd (1 ME)
Tx
NN/Scratch Rings
NN Ring
- QID(32b)
- Reserved (8b)
- QM ID (3b)
- QID(17b) 1M queues per QM
- Input Hlpr would use QM ID to select Scratch ring
on which to put request. - QM/Sched then sends on its output NN/scratch ring
to its associated Tx - With 64 entries in Q-Array and 16 entries in CAM,
max number of QM/Schds is probably 4 (2 bits). - Well set aside 3 bits to give us flexibility in
the future.
32Packet Buffer Descriptor Tradeoffs
- Why use a Buffer Descriptor at all?
- QM needs something to link packets/buffers in
queues - ME-to-ME communications costs vs. SRAM access
costs
33Packet Buffer Descriptor def
- Meta Data structure of Packet Buffers (LSB to
MSB) - buffer_next 32 bits Next Buffer Pointer (in a
chain of buffers) - offset 16 bits Offset to start of data in
bytes - BufferSize 16 bits Length of data in the
current buffer in bytes - header_type 8 bits type of header at offset
bytes in to the buffer - rx_stat 4 bits Receive status flags
- free_list 4 bits Freelist ID
- packet_size 16 bits (Total packet size across
multiple buffers) - output_port 16 bits Output Port on the egress
processor - input_port 16 bits Input Port on the ingress
processor - nhid_type 4 bits Nexthop ID type.
- reserved 4 bits Reserved
- fabric_port 8 bits Output port for fabric
indicating blade ID. - nexthop_id 16 bits NextHop IP ID
- color 8 bits Qos Color
- flow_id 24 bits QOS flow ID or MPLS label/flow
id - reserved 16 bits Reserved
- class_id 16 bits Class ID
- packet_next 32 bits pointer to next packet
(unused in cell mode)
34Packet Buffer Descriptor Gets
- buffer_next tx
- Offset rx, tx, fwd
- BufferSize tx, fwd
- header_type tx, fwd
- rx_stat NONE
- free_listpacket_size NONE
- output_port qm(?), tx
- input_port rx, fwd
- nhid_type NONE
- fabric_port qm(?), tx
- nexthop_id
- color
- flow_id
- class_id
- packet_next
35Meta Data Caching
- Meta Data can be cached in one of three places
- SRAM Xfer Registers
- DRAM Xfer Registers
- GPR Registers
- Size of Meta Data Cache is controlled by define
META_CACHE_SIZE - Macro dl_meta_load_cache loads meta data cache
- buffer_handle buffer handle for which meta data
is to be fetched - dl_meta read transfer register prefix
- Xbuf_alloc should be used to allocate the
needed registers - signal_number
- START_LW starting long word for fetch
- NUM_LW number of long words to fetch
- Each microengine (microblock?) can use Meta Data
Caching differently.
36Meta Data Caching
- In the ipv4_v6_forwarder sample app,
- dl_meta_load_cache() used in
- Egress
- ethernet_arp.uc
- pkt_tx_16p.uc
- statistics_util.uc
- tx_helper.uc
- Ingress
- ethernet_arp.uc
- pkt_tx_16p.uc
- statistics_util.uc
- tx_helper.uc
- dl_meta_get_ used in
- Egress
- ethernet_arp.uc
- pkt_tx_16p.uc
- tx_helper.uc
- Ingress
- Ether.uc
37Buffer Handle
38Buffer Descriptor Usage
- Is there a different Buffer Descriptor defn for
LC and PE? - Will we support Multi-Buffer Packets?
- If not, we do not need buffer_next(32b) or
buffer_size(16b) - QM uses packet_next for its packet chaining in
qarray. - Output Port and Input Port probably translate to
TxMI and RxMI - Next Hop fields (nhid_type(4b) and
nexthop_id(16b)) probably can go away. - QOS fields (color(8b) and flow_id(24b)) probably
can go away. - Two reserved fields 4b and 16b can go away.
- class_id(16b) (virtual queue id?) can probably go
away. - fabric_port can probably go away.
39Buffer Descriptor Usage
- PE Buffer Descriptor
- MR_ID (16b)
- TxMI (16b)
- VLAN (16b)
- buffer_next 32 bits Next Buffer Pointer (in a
chain of buffers) - offset 16 bits Offset to start of data in
bytes - BufferSize 16 bits Length of data in the
current buffer in bytes - header_type 8 bits type of header at offset
bytes in to the buffer - rx_stat 4 bits Receive status flags
- free_list 4 bits Freelist ID
- packet_size 16 bits (Total packet size across
multiple buffers) - output_port 16 bits Output Port on the egress
processor - input_port 16 bits Input Port on the ingress
processor - nhid_type 4 bits Nexthop ID type.
- reserved 4 bits Reserved
- fabric_port 8 bits Output port for fabric
indicating blade ID. - nexthop_id 16 bits NextHop IP ID
- color 8 bits Qos Color
40Buffer Descriptor Usage
- PE Buffer Descriptor
- LW0 buffer_next 32 bits Next Buffer Pointer
(in a chain of buffers) - LW1 offset 16 bits Offset to start of data
in bytes - LW1 BufferSize 16 bits Length of data in the
current buffer in bytes - LW2 reserved 8 bits reserved/unused
- LW2 reserved 4 bits reserved/unused
- LW2 free_list 4 bits Freelist ID
- LW2 packet_size 16 bits (Total packet size
across multiple buffers) - LW3 MR_ID 16 bits Meta Router ID
- LW3 TxMI 16 bits Transmit Meta Interface
- LW4 VLAN 16 bits VLAN
- LW4 reserved 16 bits reserved/unused
- LW5 reserved 32 bits reserved/unused
- LW6 reserved 32 bits reserved/unused
- LW7 packet_next 32 bits pointer to next packet
(unused in cell mode) - Leave multi-buffer fields there as a template for
the dedicated blade implementation of a
jumbo-frame MR. - Also reduces changes to Rx, Tx, and QM and
reduces potential problems.
41Multicast Alternatives
- At least Three Options
- Force MRs that need Multicast to be Dedicated
Blade MRs and do their own Multicast - For our short term goals this is probably
sufficient and the best course. - Perhaps longer term we can look at adding it to
the CRF - Treat as exception and send to Xscale
- Provide support in CRF for Multicast
- Use Multi-Hit Lookup capability of the TCAM
- MI Bit mask defined in Lookup Result
- Will put a bound on the number of MIs that can be
supported on an MR because of the size of the
lookup result. - Has issues of mapping bits in the bit mask to
actual MIs. - Lookup Result contains an index into a table
containing MI bit masks - Allow but do not force MRs to provide code to
interpret Lookup Result. - This would also allow other possible extensions
on an MR-specific basis - This carries with it the problem of bounding the
execution time of the MR-specific code in the
Lookup block. For general multicast, this could
be a serious issue. - There are also issues with generating a QID based
on an MI when the QID is not included in the
Lookup Result. - Other options?
42CRF Support for Multicast
Default/Unicast path
MR Interp
HeaderFormat
Parse
MR-Specific Path
Post Process
Lookup
MR-1
. . .
MR-n
43CRF Support for Multicast
Default path
MR Interp
MR-Specific Path
Post Process
Lookup
DRAM Buf Ptr
MR Id
MR Lookup Key
MR Ctrl Blk Ptr
MR Mem Ptr
- We will need some kind of copy count or multicast
bit and last copy bit to let TX know when it can
release the DRAM buffer that holds the packet.
44CRF Support for Multicast
Default path
MR Interp
MR-Specific Path
Post Process
Lookup
DRAM Buf Ptr
DRAM Buf Ptr
MR Id
MR Lookup Key
MR Lookup Key
MR Specific Lookup Result
MR Ctrl Blk Ptr
MR Ctrl Blk Ptr
MR Mem Ptr
MR Mem Ptr
- We will need some kind of copy count or multicast
bit and last copy bit to let TX know when it can
release the DRAM buffer that holds the packet.
45OLD
- The rest of these are old slides that should be
deleted at some point.
46Common Router Framework (CRF) Functional Blocks
Parse
HeaderFormat
Lookup
Tx
DeMux
Rx
MR-1
. . .
MR-n
RBUF
Buf Handle(32b)
- Rx
- Function
- Coordinate transfer of packets from RBUF to DRAM
- Notes
- Well pass the Buffer Handle which contains the
SRAM address of the buffer descriptor. - From the SRAM address of the descriptor we can
calculate the DRAM address of the buffer data.
47Common Router Framework (CRF) Functional Blocks
Parse
HeaderFormat
Lookup
Tx
DeMux
Rx
MR-1
. . .
MR-n
Buf Handle(32b)
- DeMux
- Function
- Read Pkt Header from DRAM
- Use VLAN from Ethernet header to determine
destination MR in order to locate - MR Parse code
- MR specific memory pointers
- Write MR Id to Buffer Descriptor
- Write VLAN to Buffer Descriptor
48Common Router Framework (CRF) Functional Blocks
Parse
HeaderFormat
Lookup
Tx
DeMux
Rx
MR-1
. . .
MR-n
Buf Handle(32b)
DRAM Buf Ptr(32b)
Buffer Offset(16b)
MR Id(16b)
Input MI(16b)
MR Mem Ptr(32b)
MR Lookup Key(16B)
- Parse
- Function
- MR-specific header processing
- Generate MR-specific lookup key (16 Bytes) from
packet - Need CRF functionality to managed multiple MRs in
shared PE. - Notes
- Can Parse adjust the buffer/packet size and
offset? - Can Parse do something like, terminate a tunnel
and strip off an outer header?
49CRF Wrapper Around Parse
MR Selector
50Common Router Framework (CRF) Functional Blocks
Parse
HeaderFormat
Lookup
Tx
DeMux
Rx
MR-1
. . .
MR-n
- Lookup
- Function
- Perform lookup in TCAM based on MR Id and lookup
key - Result
- Output MI
- QID
- Stats index
- MR-specific Lookup Result (flags, etc. ?)
- How wide can/should this be?
51Common Router Framework (CRF) Functional Blocks
Parse
HeaderFormat
Lookup
Tx
DeMux
Rx
MR-1
. . .
MR-n
Buffer Handle(32b)
DRAM Buf Ptr(32b)
- Header Format
- Function
- MR specific packet header formatting
- MR specific Lookup Result processing
- Drop and Miss bits
- Need CRF functionality to managed multiple MRs in
shared PE. - Pulls out QID, Length and Port from MR Result,
etc. - Checks for Drop and Miss bits and deals with
those actions.
Buffer Offset(16b)
MR Id(16b)
MR Mem Ptr(32b)
Lookup Result(Nb)
Includes drop and miss bits
52CRF Wrapper Around Header Format
MR Selector
Buffer Handle
DRAM Buf Ptr(32b)
Buffer Offset
MR Id
MR Mem Ptr
Buffer Offset
Gets written to Buffer Descriptor May also cause
size(s) in Descriptor to be updated. (what about
trimming data, What if it is a buffers
worth Which would change the chaining, Can they
add/trim at either end?
Lookup Result
53Common Router Framework (CRF) Functional Blocks
Parse
HeaderFormat
Lookup
Tx
DeMux
Rx
MR-1
. . .
MR-n
Buffer Handle(32b)
Buf Handle(32b)
- QM
- Function
- CRF queue management for Meta Interface queues
- For performance reasons, QM may actually be
implemented as multiple instances - Each instance on a separate ME would support a
separate set of Meta Interfaces. - See next slide for more details
QID(16b)
Size (16b)
Port(8b)
54QM/Scheduler on Multiple MEs
Output Hlpr (1 ME)
QM/Schd (1 ME)
Input Hlpr (1 ME)
HeaderFormat
Tx
. . .
QM/Schd (1 ME)
Buf Handle(32b)
Scratch Rings
Size (16b)
NN Ring
NN Ring
Port(8b)
- QID(32b)
- Reserved (8b)
- QM ID (4b)
- QID(20b) 1M queues per QM
- Input Hlpr would use QM ID to select Scratch ring
on which to put request. - Output Hlpr would process all Scratch rings
coming from QM/Schd MEs and multiplex onto one NN
ring to TX - With 64 entries in Q-Array and 16 entries in CAM,
max number of QM/Schds is probably 4 (2 bits). - Well set aside 4 bits to give us flexibility in
the future.
55Common Router Framework (CRF) Functional Blocks
Parse
HeaderFormat
Lookup
Tx
DeMux
Rx
MR-1
. . .
MR-n
TBUF
Buffer Handle(32b)
- Tx
- Function
- Coordinate transfer of packets from DRAM to TBUF
56Old Template
Lookup
Tx
Header Format
DeMux
Rx
Parse
TBUF
- Tx
- Function
- Coordinate transfer of packets from DRAM to TBUF
57Old Rejected Overly Busy Slide
Block
- ? Logical interface
- Data passing between layers
- Notes here
- ? Physical format
- Actual Format of data
- Shows type of communication
- Scratch Ring
- NN Ring
- SRAM Rings
- ? Buf Descriptor
- shows fields read/written
Input Data
Output Data
Input Data
Output Data