Multicast Support in the NSP (the

1
Multicast Support in the NSP(the low-level
details)

• NSP Meeting
• 28 October 2002
• Alex Chandra, Yuhua Chen, Sarang Dharmapurikar,
• John Lockwood, Todd Sproull, Wen-Jing Tang,
• David Taylor, Jon Turner

2
Multicast Tree Support

• Support binary copy multicast without pruning
• Extra complexity needed to avoid re-ordering
• Example If leaf E is removed from tree (a),
  then the tree will be maintained as shown in (b).
• Note that recycling node B will only forward one
  copy to leaf C
• Upstream copy suppression is possible

3
Multicast Use of InterPort Shim

• InputVIN stamped at packet arrival
• OutputVIN used to forward to next port
• Previous Port Number (PPN) stamped by multicast
  recycling port
• Used to suppress upstream copies
• Added Multicast Tree Position (MTP) field
• Denotes position of filter/packet in multicast
  tree
• Allows multiple filters per port for a single
  multicast session

4
Multicast Tree Position (MTP)

• Initialized by ISAR to 0x7F (0111 1111)
• CARL performs exact match on 5-tuple and MTP
• Returns a single bit (MB) as a branch indicator
• Prior to send, MTP is modified by OSAR
• Replace rightmost (least significant) zero with
  bit returned by CARL/QM
• Change the bit to the right of this bit from 1 to
  0

5
Classification Route Lookup (CARL)
6
EM/FIPL Shared Memory Map

• 38.5k FIPL entrieds
• 20,480 exact match filters
• Forced pre-allocation of ingress/egress filter
  pairs
• Head and list entries managed separately
• All EM buffers must be aligned on 6 word
  boundaries relative to start of buffer
• All EM pointers are entry pointers

7
Exact Match Search Engine
8
Exact Match Filter Entry Format
9
Exact Match Filter Entry Format

• Multicast filters must set Filter Type field to
  001
• MB (Multicast Branch) bit identifies the branch
  of the multicast tree corresponding to the filter
• Identifies which OutputVIN/QID pair to bind to
  packet
• Forwarded to QM and used by OSAR to update MTP
• Default 0
• NB (Number of Branches) bit indicates the number
  of branches exiting node
• Needed for maintaining incomplete (non-pruned)
  trees
• Default 0
• MTP field replaces Rate2 field for multicast
  filters

10
End of Presentation
11
MSR Configuration with FPX

• SPC handles exceptional, optional, and active
  packet processing
• FPX handles SAR, classification, and queueing
• Plain ingress packets
• Route lookup, exact match, general match
  classification ? DQ
• Plain egress packets
• Exact match classification ?DRR
• Supports 1 Gb/s links with 4 Sub-Port Identifiers
  (SPIs) per port
• Dedicated ingress egress comm. channels with
  SPC
• Dedicated CP, SPC, and DQ control channels

12
RAD Logical Block Diagram
13
RAD Physical Block Diagram
14
NID Virtual Circuit Map

• All traffic arriving from SW switched to RAD_SW
  port
• Traffic arriving from SP0-SP3 switched to RAD_LC
  port
• Due to physical partitioning of PSM and ISAR,
  require unique VCs for ingress and egress SPC
  traffic
• Unique control VCIs for CP, SPC, and DQ rate
  distribution

15
InterPort Shim (FPX/SW)

• 5-bit Input and Output VINs (8 ports w/ 4
  sub-ports)
• Previous Port Number (PPN) stamped by multicast
  recycling port
• Used to suppress upstream copies
• Flags not defined for InterPort shim
• Should we compress to 8 bytes? (Answer No)

16
IntraPort Shim (FPX/SPC)

• Same format for full-packet or header-only
  processing
• Queue Identifier (QID) pins packet to a specific
  queue
• Dedicated queue ranges for SPC traffic, LFS
  flows, etc.
• Queue Length of the return queue for associated
  QID
• Return SPC QIDs transmit SPC queue index
• Only holds if the Re-Classify (RC) flag is not set

17
IntraPort Shim Flags

• DP (Drop Packet) This flag is set when a packet
  should be dropped due to failed checksums,
  mismatched length fields, filters, or SPC
  directives.
• RC (Re-Classify packet) This flag is set by the
  SPC when a packet requires re-classification in
  the FPX due to processing in the SPC.
• NM (No Match) This flag is set when no filter or
  lookup entry exists for the packet. The packet is
  sent to the SPC for classification.
• EX (Exception) This flag is set by ISAR when IP
  Options exist in the packet or the packet is not
  IPv4. The packet is sent to SPC for processing.
• HO (Header-Only processing) This flag is set by
  CARL when only the packet header must be sent to
  the SPC for processing and the packet is more
  than 120B.
• HR, FM, and TO flags are FPX-specific state
• Must be retained with the packet (not modified by
  the SPC)

18
Classification Route Lookup (CARL)

• Route lookup, general filter match, exact filter
  match on ingress packets
• 64 priority levels
• General filters have per-filter priority
• Route lookup results have fixed priority
  (register config.)
• Exact filter match results have fixed priority
  (register config.)
• Exact filter match on egress packets
• ISAR checks Protocol field, port numbers zeroed
  for non-TCP/UDP packets
• Setup filter send ICMP packets directly to SPC
  anything else?
• General filter match on 5-tuple returns one
  exclusive and one non-exclusive filter
• 32-bit counter for matching packets
• Exact filter match on 5-tuple returns OVINs and
  QIDs (2 for multicast packets), LFS rates, and
  flags
• 16-bit packet and 32-bit byte counters for
  matching packets
• Route lookup on IPv4 destination address returns
  OVIN
• 16-bit packet counter for matching packets

19
Exact Filter Match
20
Exact Filter Match Entries (in SRAM)
21
Queue Manager (QMGR)

• DQ for switch-bound packets
• DRR for link-bound packets
• Dedicated SPC-bound packet queues
• QID determines destination
• 0-127 to SPC
• 128-255 from SPC
• 256-439 reserved flow queues
• 440-447 switch datagram queues
• 448-511 line card datagram queues

22
Guidelines for SPC Processing

• Must NOT modify the following fields
• Lower 3-bits of flags and TOT Chunks
• Packet and Second Chunk Pointers
• May set the drop flag for any packet
• May opt to drop a full packet locally
• Must have consistent treatment of RC
  (Re-Classify) flag for all packets of a flow for
  uniform queuing treatment
• If RC is not set, the QID currently in the shim
  will be used
• RC bit must be set for LFS packets returned to
  FPX
• Currently, ALL packets containing options bypass
  CARL and are sent to the SPC for processing

23
Guidelines for Header-Only Processing

• FPX transmits first 120B of packet in three ATM
  cells
• If filter specifies header-only processing, but
  the packet is less than 120B the entire packet
  will be sent to the SPC
• Must ALWAYS return one copy of header to FPX
• Must NOT modify the length of header/data
• May decide to set the drop flag
• Must NOT modify the following fields
• Lower 3-bits of flags and TTL Chunks
• Packet and Second Chunk Pointers
• Refer to previous slide regarding RC flag

24
Control Functions

• NID VCXT configuration
• Register configuration (pass register number,
  operation, and value)
• Read, write, reset all to default values
• System counters and flags (pass counter number)
• Read and reset all
• Per-VCI input and output packet counters
• For SPC traffic VCIs, full-packet and header-only
  counters
• Per-VCI input and output control cell counters
• Drop counters
• Cell drops due to congestion at ISAR (no SDRAM,
  etc.)
• Packet drops due to AAL5 checksum failures
• Flags for IP Header checksum failure and IP and
  AAL5 length mismatch
• Queue drop counters (SPC, LC, SW)

25
Control Functions (2)

• CARL SRAM updates
• Maintain route database and exact match filter
  entries
• Pass command, address, data
• Read/write 32- or 36-bit words
• Exact Filter Match hash table updates
• Pass command, address (hash key), and data
  (pointer)
• Maintain correct order of operations for
  adds/deletes
• Adds write entry in SRAM, update hash table
  entry
• Deletes update hash table entry, deallocate SRAM
  entry
• General Filter Match
• Pass filter table and priority resolution table
  entries
• Consult document for details and data formats

26
Control Registers
27
References

• FPX Support for the MSR, Version 1.0
• Detailed technical report documenting the
  hardware design
• All interfaces, formats, control cells, OpCodes,
  etc.
• Team members
• Design still flexible in certain areas, however
• Coding has begun

28
External Shim Flag Definitions

• AF (Active Flow) This flag is set when a lookup
  result returns a non-zero Stream Identifier
  (SID). The packet is sent to the SPC for active
  processing.
• NR (No Route in table) This flag is set when no
  filter or lookup entry exists for the packet. The
  packet is sent to the SPC for classification.
• OP (IP Options present) This flag is set when IP
  Options (other than Lightweight Flow Setup) exist
  in the packet header. The packet is sent to the
  SPC for options processing.
• UK (Unknown packet type) This flag is set when a
  packet arrives that is not IPv4. The packet is
  sent to the SPC for processing.
• RC (Re-Classify packet) This flag is set by the
  SPC when a packet requires re-classification in
  the FPX due to processing in the SPC.
• HO (Header-Only processing) This flag is set
  when only the packet header must be sent to the
  SPC for processing.
• HL (Header Lenth) This flag specifies how much
  packet data to send to the SPC for header only
  processing (0 first 56 bytes of packet 1
  first 116 bytes of packet). This flag is valid
  when only the packet header must be sent to the
  SPC for processing (HO 1).
• DP (Drop Packet) This flag is set when a packet
  should be dropped due to failed checksums,
  mismatched length fields, or filter directives.

29
Internal Shim Flag Definitions

• IE (Ingress/Egress packet) This flag provides
  for implementation efficiency by allowing
  internal components to identify ingress and
  egress packets via a single bit compare. 0
  Ingress, 1 Egress.
• LP (LFS option present) Flag set to 1 when LFS
  is present.
• LR (LFS rate request) Flag denotes requested
  rate type (0 soft/transient, 1
  firm/constant). Flag is only valid when LP 1.
• SB (SPC-bound packet or header) Flag set to 1
  when packet or header is to be sent to SPC.

30
Input Segmentation and Reassembly

• Process IP traffic on 4 input sub-port VCs, 8
  output port VCs, and 2 SPC VCs
• Other traffic arriving from SW passes through
  control path
• Reassemble IP packet
• Check ATM HEC, IP Header checksum, and AAL5
  checksum
• Drop corrupt packets identified by checksum
  mismatches gt set drop flag in shim
• Compare IP Total Length and AAL5 length
• Drop corrupt packets identified by length
  mismatches gt set drop flag in shim
• Check for IP Options
• If not LFS Option, set option flag in shim for
  forwarding to SPC
• Store payload in SDRAM
• Maintain 14 contexts using local Packet
  Identifiers (PID) generated by Output
  Segmentation and Reassembly (OSAR)
• Using BlockRAM as on-chip buffers, pass 60 byte
  chunks of packet to PSM
• Packet Storage Manger (PSM) returns a packet
  pointer after first chunk of packet is
  transferred
• Ingress traffic insert 8-byte shim prior to IP
  header in first chunk
• Write Input VIN based on packet arrival from SP0
  SP3
• Forward header fields to CARL after entire packet
  is received
• De-multiplex control cells to CCP

31
Output Segmentation and Reassembly

• Dequeue packets based on packet pointers passed
  from Queue Manager
• Select ingress/egress PSM based on shim flag
• Create AAL5 frame for dequeued packets
• Decrement IP TTL for egress packets
• Update IP Header Checksum using incremental
  update algorithm
• Generate AAL5 checksum
• Ingress traffic clear shim internal flags and
  SID to create InterPort Shim
• Egress traffic remove shim
• Update rate field(s) for LFS option packets
• Check for multicast count
• Translate VCI of cells of outgoing AAL5 frames
  based on the shim fields and register set
• Case 1 (Ingress traffic) Output VIN PN / Local
  PN
• Switch traffic VCI Output VIN PN Obase_VCI
• SPC traffic (flags set) VCI SPC VCI
• Case 2 (Egress traffic) Output VIN PN Local PN
• Link traffic VCI Output VIN SPI Ibase_VCI
• SPC traffic (flags set) VCI SPC VCI
• Multiplex control cell responses from CCP onto SW
  port

32
CARL Block Diagram
33
CARL Performance

• Line rate packet classification (3 Gb/s)
• Assuming single-chunk (64B) packet I/O gt 5.86M
  lookups/s
• Route lookup concatenate destination and source
  addresses (64-bits), perform longest-prefix match
• Tree Bitmap w/ 4-bit stride gt 16 nodes deep for
  exact match
• Worst case memory accesses 20
• _at_ 75 MHz w/ 8 FIPL engines gt 3.75M lookups/s
• For typical 24-bit prefix lookups gt 8.33M
  lookups/s
• Returns Output VIN and SID
• For multicast (binary copy), returns two Output
  VINs and one SID
• SID2 SID1 1
• All results have fixed priority

34
CARL Performance (2)

• General filter match linear search over 64
  filters using 5-tuple
• _at_ 75 MHz gt 1.17M table scans/s
• Pipeline of 8 packet headers gt 9.375M lookups/s
• Provides for 37.5 overhead (62.5 efficiency)
• Returns one exclusive filter match and up to
  three non-exclusive filter matches
• Each filter has specific priority
• May be relative to route lookup priority
• Filter table under software control
• Exact Filter Match exact match over 5-tuple
• Algorithm to be selected
• Must share SRAM capacity and bandwidth with Route
  Lookup
• Dynamic filter updates based on implicit flow
  setup and tear-down for LFS support
• Handled by on-chip hardware
• Software filter insertion for reserved flows

35
SPC Triggers

• Packet exception cases
• Not IPv4
• Options other than LFS present
• Filter directives
• Exclusive/non-exclusive general filter or exact
  match filter specifies packet or packet header
  must be sent to SPC
• Network monitoring, active processing, etc.
• No matching filter or route
• Will this ever happen?
• Others?

36
Full-packet SPC Processing

• SPC must have consistent treatment of RC flag for
  all packets of a given flow
• Maintain uniform queuing treatment

37
Header Only SPC Processing

• First cell contains reserved space for chunk
  pointers
• 12 bytes reserved regardless of HL
• Single chunk 56 bytes of packet
• Requires 2 cells
• Double chunk 116 bytes of packet
• Requires 3 cells
• SPC may decide to drop packet by setting DP flag
• SPC must NOT send copies of packet headers back
  to FPX
• SPC must return header on the arrival VC
• SPC_IN_VCI or SPC_EG_VCI

38
Task Assignments

• Overview (David)
• ISAR (Alex)
• OSAR (Yuhua)
• PSM (Sarang)
• CCP (David)
• SRAM Interface (David)
• SDRAM Interface (Sarang)
• CARL (Alex David)
• General Filter Match (Alex)
• Exact Filter Match (Alex/David)
• Route Lookup (David)
• I/O logic priority resolution (Alex/David)
• QM (Yuhua, Wen-Jing, Sarang)
• DQ (Yuhua/Wen-Jing)
• DRR (Yuhua/Wen-Jing/Sarang)
• SPC/PFQ (Yuhua/Wen-Jing)
• Document /project/arl/fpx/papers/msr_design/msr_d
  esign_v1.pdf
• Slides /project/arl/fpx/papers/msr_design/talks/