Fred Kuhns and Alex Chandra

1
FPX/SPC Integration

• Fred Kuhns and Alex Chandra
• Applied Research Laboratory

2
Rule

• if GM filter Exclusive, Ingress and
  Non-multicast
• if OVIN is invalid use result of RL
• if QID valid use it, else use dgram QID
• else
• both OVIN and QID must be valid
• In all cases insert resulting OVIN in inter-port
  shim
• if packet from SPC and RC 0, then OVIN and QID
  must be valid. The SPC may change these fields.
• if GM filter Exclusive, Egress and Non-multicast
• Always use OVIN from Shim
• if QID is valid use it, else use dgram QID
• If packet from SPC and RC 0, then OVIN and QID
  in shim must be valid SPC may alter these
  values.

3
Notes mtg 09/16/02 and subsequent discussions

• TTL Processing Note
• Ingress ISAR If TTL 0,1 and packet from
  line-card then send to SPC, bypassing CARL.
  Otherwise process normally (TTL is not changed or
  inspected further).
• Egress OSAR If TTL gt 0 and packet to line-card,
  then decrement. Always send regardless of TTL
  value (TTL is not changed or inspected further).
• ICMP messages We will support sending and
  receiving the following
• Time exceeded for datagram FPX ingress port
  sends packet to SPC if TTL 1/0. The SPC will
  decide whether to deliver locally or generate
  ICMP message.
• Destination unreachable if there is no route
  entry then send to SPC we already do this. SPC
  responds with ICMP msg if it is unable to resolve
  route.
• ICMP echo and reply This will have the routers
  local IP address so we can implement at CP
  need to look into how we do this at the CP.
  Alternatively, we can define a route table entry
  for the local interfaces IP address to send those
  packets to the SPC.
• Filter on IP header Source Address
• Invalid Source addresses
• The high order 3 bits equal 111. Note class D
  (Multicast) has 1110, class E (Reserved) has 1111
  and broadcast is of course all 1s.
• Address is all 0s
• Address high order byte equals 127d (0x7f) i.e.
  the loopback interface
• In an invalid Source Address then silently drop.
• Note while we should drop directed broadcasts
  and filter for IP spoofing we will not do this
  due to the difficulty of implementation.
  Discussion ??

4
Notes (2)

• Filter on IP header Destination Address
• Invalid Destination addresses for forwarding (IP
  lookup)
• -1, -1, 0, 0, prefix, 0, Class E
  (240.0.0.0 247.255.255.255)
• If valid unicast then forward high order 3 bits
  do not equal 111 and address is not all 0s.
• If valid broadcast address then send to CP
• there are two broadcast address of concern
  0xffffffff and local subnet broadcast.
• Can we use the General Match filter for this?
• If multicast address (first 4 bits 1110) then
  ISSUE??
• If class E, all zeros or any other case then
  silently drop.
• IP Fragmentation
• Assume MTU is same on all ports, so will not need
  to fragment
• if receive an IP fragment, then silently drop.
  Rationale describe??
• Path MTU discovery
• support the IP option form describe approach??
• do not support the ICMP based approach describe
  approach??
• Other Packet Processing Errors
• IP header parameter errors other than version
  Silently drop.
• Errors Checksum error, length error, invalid
  source address, Others?.
• Rationale Since the IP header may be corrupted
  or counterfeit we can not and should not rely on
  any of the field values.

5
Notes(3)

• Will support GM filtering on output port
  selectable via a configuration flag. Default case
  does not enable this feature.
• How do we deal with Multicast and Broadcast?
• with IGMP we could use a GM filter to send these
  multicast packet to the CP can/should we do
  this??
• General Match Filters and default routes
• Alex will add an additional bit to the GM filter
  indicating OVIN validity. If OVIN is not valid
  then the results of the FIPL lookup are used. If
  the fipl lookup fails then treat as you would any
  datagram without a valid route.
• ICMP Redirect
• Should we generate an ICMP redirect if the
  resulting OVIN IVIN??

6
System Architecture
ControlProcessor
Switch Fabric
ATM Switch Core
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
IPP
OPP
FPX
FPX
FPX
FPX
FPX
FPX
Port Processors
SPC
SPC
SPC
SPC
SPC
SPC
LC
LC
LC
LC
LC
LC
Line Cards
7
Logical view of SPC/FPX

• Fast Path Core IP processing in FPX
  (classification, packet scheduling, distributed
  queuing etc.)
• Slow Path Exceptional and active processing in
  SPC
• Control Path messages from the CP

8
Ingress
I only care about the resulting behavior. So some
details are missing if they do not directly
support the story
SPC
VC 56 TO 0 ingress
VC 57 TO 1 egress
FPX
ISAR receive packet from Line card or SPC

• Reassemble packet, verify AAL5 checksum.
• If invalid checksum, silently drop packet.
• From LC
• If bad IP parameters or invalid checksum then
  silently discard packet ip hlen lt 5, ip cksum
  invalid, AAL5 cksum invalid, ip tlen shim_len
  ! AAL5 len
• If IP options or not IPv4, Set EX flag and send
  to SPC
• If TTL 0/1, send to SPC
• otherwise send to CARL
• From SPC
• Drop packet if
• Not HO if hlen lt 5, IP/AAL5 cksum invalid, AAL5
  len ! hlen16
• HO if hlen lt 5, IP/AAL5 cksum invalid, AAL5 len
  ! 12016, DP1
• Note for HO0, ISAR expect SPC to drop packet if
  necessary.
• if RC 1, send to CARL for classification
• Fill in Shim values
• InVIN using PN register, what about sub-port?
• Flags
• RC 1 unless SPC returns RC 0, DP 1 or EX
  1.
• NM 0 in all cases (set by CARL)
• EX 1 when IP options, 0 otherwise

40
41
42
50
43
51
44
52
45
53
46
47
9
Ingress
Notes MTP Multicast Tree Position, 8 bit
SPC
VC 56 TO 0 ingress
VC 57 TO 1 egress
FPX
Classify Packet
out ltE priority, HO, DP, QID, OVINgt ltNHO,
QID, OVINgt
CARL
out ltFlags (FT, HO, LR, TO), OVIN0, QID0, Rate0,
OVIN1, OVIN1 Rate1, MTP?gt
in ltFM, 5-tuple, MTPgt
EM (priority)
out ltOVINgt QID SW_DGRAM_BASE (440)
OVIN(42)
fipl (priority)
in ltdestinationgt
40

• in parallel, GM, EM and dgram lookup
• update OVIN QID with highest priority matching
  filter/route
• if packet from SPC and RC 0 and DP 0 then
  assume TO and OVIN are correct (TO assume 0/SW,
  erroneous for it to 1/LC since route lookup
  performed at ingress port)
• if (QID 0) then QID SW_DGRAM_BASE (440)
  OVIN(42)
• else QID assumed valid and not modified by CARL
• QMGR remaps QID QID 128
• Update TO and DP (GM Exclusive may require we
  drop packet)
• Send to OSAR

41
ISAR
42
50
43
51
44
52
45
53
46
47
10
Ingress
SPC
VC 56 TO 0 ingress
VC 57 TO 1 egress
FPX
QM/OSAR/PSM

• QM
• Place packet in correct queue, schedule for
  transmission
• When scheduled, send to OSAR
• OSAR
• encapsulate in AAL5 frame
• To SW
• update shim flags (all zero), IVIN, OVIN, PPN,
  MTP
• encapsulate in AAL5 frame and determine VC from
  OutVIN
• To SPC (ingress and egress same)
• Encapsulate on AAL5 frame, set VC
• Insert Intra-port shim

40
ISAR/ CARL
41
42
50
43
51
44
52
45
53
46
47
11
FPX Queue assignment after classification
Note on QIDs to the SPC 1 .. 7 are reserved for
SPC bound traffic QID Port Description Flag Suppor
ted 1 Ingress No match in CARL NM1 Yes 2 Ingress
Not IPv4 EX1 Yes 3 Ingress TTL
0/1 EX1 Yes 4 Ingress IP Options EX1 Yes 5 Egres
s Output link down EX1 No 6 Not defined 7 Not
defined
SPC
VC 56 TO 0 ingress
VC 57 TO 1 egress
FPX
After CARL classifies packet it is placed into a
queue and sent on a VC based on TO, SR, QID and
OutVIN. Does not show reclassification. Software
assigns all QIDs except 1-7. The range 128 255
is calculated internally.
QID 1-127 SR 0, TO 1/0
QID 128-255 (qid qid 128) SR 1, TO 1/0
OutVIN (port) used to calculate VC
TO 0
TO 1
RC 1
40
CARL
41
42
50
43
51
44
52
45
53
46
47
OutVIN (sub-port) used to calculate VC
TO 0
TO 1
12
Egress
SPC
VC 56 TO 0 ingress
VC 57 TO 1 egress
FPX
ISAR receive packet from switch
40
41
42
50

• Receive packet from switch Set FM 0 (SW)
• IP len 8 ! AAL5 length, then silently drop
• otherwise same as ingress ISAR

43
51
44
52
45
53
46
47
13
Egress
SPC
VC 56 TO 0 ingress
VC 57 TO 1 egress
FPX
Classify Packet
CARL
out ltFlags (FT, HO, LR, TO), OVIN0, QID0, Rate0,
OVIN1, OVIN1 Rate1, MTP?gt
ISAR

• Exact Match only
• out 1 or 2 tupples TO, OVIN, QID, Rate, MB
• From SPC and RC 0 and DP 0 (bypass pkt), then
  TO flag and OVIN are assumed correct
• if TO 1/LC
• if QID 0
• QID LC_DGRAM_BASE(448) OVIN(10)SA(10)DA(1
  0)
• else
• QID not modified by CARL
• QMGR set QID QID 128
• else same as ingress case
• Send to QM include pkt pointer, rates, shim
  fields.

40
41
42
43
50
44
51
45
52
46
53
47
14
Egress
SPC
VC 56 TO 0 ingress
VC 57 TO 1 egress
FPX
QM/OSAR

• QM
• send pkt, rates and shim fields to OSAR
• OSAR
• Sending to LC
• remove shim,
• update LFS option (rate2)
• decrement TTL if gt 0 (always send even if TTL
  0)
• update IP hdr checksum
• encapsulate in AAL5 frame and calculate VC from
  OutVIN
• Sending to SPC
• Encapsulate on AAL5 frame, set VC
• Insert Intra-port shim

ISAR
40
41
42
43
50
44
51
45
52
46
53
47
15
FPX to SPC Intra-Port Shim
Input VIN
Output VIN
PPN
Flags
Queue Identifier (QID)
Port
Sub
Port
Sub
Port
DP
RC
NM
EX
HO
HR
FM
TO
TTL Chunks
Queue Length
Packet Pointer
Second Chunk Pointer

• Flag Values
• DP (Drop Packet) SPC directive to FPX. Will
  always be zero coming into SPC.
• RC (Reclassify Packet) SPC directive to FPX. May
  be 0 or 1 going to SPC. SPC must redefine.
• NM (No Match) FPX can not classify (no route),
  FPX to SPC. Set to 0 before returning to FPX.
• EX (Exception Packet) Requires non-fast path IP
  options, Non IPv4. FPX to SPC. Set to 0 before
  returning to FPX.
• HO (Header Only) FPX to SPC Set if packet size
  gt 120B and only first chunk sent to SPC. SPC must
  not modify. SPC can not drop internally if this
  bit is set.
• HR (Header Only Return) Internal FPX flag. SPC
  must not modify.
• FM (From LC/SW) Set by FPX. SPC can use but not
  modify.
• TO (To LC/SW) Set by FPX. SPC can use but not
  modify.

16
FPX to SPC, Inter-port Shim
Flags
Input VIN
Output VIN
PPN

• Inter-port Shim expect the Input and Output
  VINs to be correct. No other information needed
  in particular the PPN is not used by the SPC.

17
Looking at the Shim Fields (Intra-Port)

• Flags Input to SPC ? indicates required
  processing NM, EX, HO
• NM ? FPX lookup failed, perform lookup
• If SPC can resolve route then
• set Output VIN, QID
• set NM 0,
• return to FPX.
• else
• send ICMP to source and drop packet (internal to
  SPC).
• EX ? Implies IP Options, IP parameter error, or
  Non IPv4.
• If IP option, process options and return to FPX
  with EX 0. Further iff an EM filter has been
  added/removed then set RC 1.
• If Non IPv4 then SPC either consumes or process
  as if NM is set.
• HO ? Only the header (first 120 bytes) of
  packet.
• if need to drop packet, then set DP 1 and
  return to FPX

18
Scenario Overview

• Default system configuration (routes, FPX
  register file)
• FPX, SPC and CP
• Basic datagram forwarding, no GM or EM filters
• valid packets
• various error conditions
• get traffic statistics (counts)
• Forwarding with GM and EM filters
• CP adds default GM and EM filters
• traffic no matching filters
• traffic match GM and EM. Try mix of traffic
  (dgram, reserved) and sending to SPC
• get traffic statistics (counts)
• Control FPX from SPC
• add/remove EM filter
• get traffic statistics (counts)
• Add/remove filters and route entries under load

19
Default Configuration for Scenarios

• Initial, Basic Configuration
• SPC configuration none for datagram forwarding
  test
• Will use NCHARGE where possible.
• NID image stored in PAL on FPX. Must be
  configured (VC translation table) then correct
  RAD image selected and downloaded.
• RAD Configuration
• FPX register values (separate slide)
• Download default FIPL table (separate slide)
• Default general match and exact match filters -
  none for initial datagram forwarding test
• Other FPX specific configuration issues?
• None that I know of

20
RAD Configuration

• CP configuration module will handle basic
  configuration issues
• RAD configuration
• initialize and configure SPCs (indicate if FPX
  present)
• use NCHARGE to download and start RAD.
• configure NID VC translation table
• RAD Initialization see separate slide with
  register file defaults
• Set/verify default VCI values CP Control VCI
• Registers SPC Control VCI, DQ Control VCI, Ibase
  VCI, Obase VCI, SPC IN VCI and SPC EG VCI
• Set port number
• register PN
• Set/verify default route and exact match
  priorities
• registers RL Priority and EM Priority
• Set/verify default link rate, SPC rate and speed
  advantage
• registers QM Speed, QM Link and QM SPC
• Does software set or just read the registers RL
  RootNodePtr and EM Offset?
• Configure CARL for default tests
• add our usual default routes see separate slide
  with default routes
• For classifier tests, download default GM and EM
  filters see separate slide

21
FPX Register File

• Configuration registers
• We need to have test cases where we read, modify
  and verify.
• Default values are correct for testing.

Register Number Register Name Width (bits) Default Value Description
0x00 PN (Port Number) 3 0 Physical port number of the FPX
0x01 CP Control VCI 16 0/35 Control cell VCI CP to/from FPX
0x02 SPC Control VCI 16 1/29 Control cell VCI SPC to/from FPX
0x03 DQ Control VCI 16 1/61 Control cell VCI for DQ
0x04 Ibase VCI 16 0/128 Base VCI for link traffic
0x05 Obase VCI 16 0/64 Base VCI for switch traffic
0x06 SPC IN VCI 16 1/62 VCI for ingress traffic SPC to/from FPX
0x07 SPC EG VCI 16 1/63 VCI for egress traffic SPC to/from FPX
0x08 RL RootNodePtr 18 0 (0x0) Root node pointer for Tree Bitmap
0x09 RL Priority 6 60 (0x3C) Priority of route lookup results
0x0A EM Offset 3 7 (0x7) Address offset of exact match entries
0x0B EM Priority 6 56 (0x38) Priority of exact match results
0x0C QM Speed 8 2 (0x02) Speed advantage (ratio)
0x0D QM Link 8 x.875, y13 (0xED) (983 Mb/s) Link rate in multiples of 64kb/s
0x0E QM SPC 8 x.5, y11 (0x8B) (197 Mb/s) SPC rate in multiples of 64kb/s
22
Default Routes and Interface Addresses

• Download default routing table at initialization.
• SPC represents the VIN differently This needs to
  be fixed.
• At some point we need to have a better (i.e. more
  convenient) way of specifying routes and
  dynamically changing them using a standard
  routing protocol such as OSPF.

PN SP Prefix/Width IP Address SPCVIN (10/6) FPXVIN (3/2)
0 0 192.168.32.0/24 192.168.32.1 0x0000 (0) 0x00
0 1 192.168.33.0/24 192.168.33.1 0x0001 (1) 0x01
0 2 192.168.34.0/24 192.168.34.1 0x0002 (2) 0x02
0 3 192.168. 35.0/24 192.168. 35.1 0x0003 (3) 0x03
1 0 192.168. 36.0/24 192.168. 36.1 0x0040 (64) 0x04
1 1 192.168. 37.0/24 192.168. 37.1 0x0041 (65) 0x05
1 2 192.168. 38.0/24 192.168. 38.1 0x0042 (66) 0x06
1 3 192.168. 39.0/24 192.168. 39.1 0x0043 (67) 0x07
2 0 192.168. 40.0/24 192.168. 40.1 0x0080 (128) 0x08
2 1 192.168. 41.0/24 192.168. 41.1 0x0081 (129) 0x09
2 2 192.168. 42.0/24 192.168. 42.1 0x0082 (130) 0x0a
2 3 192.168. 43.0/24 192.168. 43.1 0x0083 (131) 0x0b
3 0 192.168. 44.0/24 192.168. 44.1 0x00c0 (192) 0x0c
3 1 192.168. 45.0/24 192.168. 45.1 0x00c1 (193) 0x0d
3 2 192.168. 46.0/24 192.168. 46.1 0x00c2 (194) 0x0e
3 3 192.168. 47.0/24 192.168. 47.1 0x00c3 (195) 0x0f
4 0 192.168. 48.0/24 192.168. 48.1 0x0100 (256) 0x10
4 1 192.168. 49.0/24 192.168. 49.1 0x0101 (257) 0x11
4 2 192.168. 50.0/24 192.168. 50.1 0x0102 (258) 0x12
4 3 192.168. 51.0/24 192.168. 51.1 0x0103 (259) 0x13
5 0 192.168. 52.0/24 192.168. 52.1 0x0140 (320) 0x14
5 1 192.168. 53.0/24 192.168. 53.1 0x0141 (321) 0x15
5 2 192.168. 54.0/24 192.168. 54.1 0x0142 (322) 0x16
5 3 192.168. 55.0/24 192.168. 55.1 0x0143 (323) 0x17
6 0 192.168. 56.0/24 192.168. 56.1 0x0180 (384) 0x18
6 1 192.168. 57.0/24 192.168. 57.1 0x0181 (385) 0x19
6 2 192.168. 58.0/24 192.168. 58.1 0x0182 (386) 0x1a
6 3 192.168. 59.0/24 192.168. 59.1 0x0183 (387) 0x1b
7 0 192.168. 60.0/24 192.168. 60.1 0x01c0 (448) 0x1c
7 1 192.168. 61.0/24 192.168. 61.1 0x01c1 (449) 0x1d
7 2 192.168. 62.0/24 192.168. 62.1 0x01c2 (450) 0x1e
7 3 192.168. 63.0/24 192.168. 63.1 0x01c3 (451) 0x1f
23
Scenario 1 Datagram Forwarding, No G/E filters

• Datagram Forwarding, Valid packet, Ingress and
  Egress Ports
• Valid route, valid IP header. Forward packet
• Datagram Forwarding, Error/Exceptional
  processing, Ingress Port
• IP option other than LFS, send to SPC with EX
  flag 1 (see LFS processing below)
• Valid packet, no route, send to SPC with NM flag
  1 and QID 1
• SPC has route, update OutVIN and QID. Send back
  to FPX, NM 0
• SPC does not have route, send ICMP destination
  unreachable to source if HO 0 then drop packet
  else set DP flag 1, return packet to FPX.
• Valid route, TTL 0 or 1. Send to SPC with EX
  flag 1 and QID 3?.
• If packet destination address is local then
  deliver to CP.
• If not local then send ICMP Time Exceeded message
  to source, if HO 0 then drop packet else set DP
  flag 1 and return to FPX.
• Checksum error, invalid length or other parameter
  error, FPX silently drops packet.
• If the IP version is not valid then set EX flag
  1 and send to SPC
• if IP version not supported then if HO 0 drop
  pkt else set DP flag 1 and return packet to FPX
  for dropping. Otherwise SPC processes packet
  though it is not clear what this means.
• Buffer overrun silently drop packet.
• Datagram Forwarding, Error/Exceptional
  processing, Egress Port
• IP options are not checked, Route and GM filters
  are not checked, Assume SHIM correct
• Checksum or other parameter error, drop silently
• If indicated interface or VC not available?, set
  EX flag 1 and send to SPC with QID 4?. SPC
  will send ICMP destination unreachable message to
  source, set DP flag 1 and return to FPX.
• Buffer overrun silently drop packet.

24
Scenario 2 Matching GM or EM Filters

• Install GM and EM filters from CP
• See word document for filter specification
• Dynamically add and remove EM filters
• Dynamically add and remove GM filters
• Dynamically add and remove routes
• Read packet counters for each filter type
• Packet Matches GM repeat for both Ingress and
  Egress ports
• Non-Exclusive filter
• Exclusive filter
• Drop Packet, Send to SPC, Header Only
• Both Exclusive and Non-Exclusive filter
• Exclusive sent to SPC, Non-Exclusive send header
  only to SPC
• Packet Matches EM filter repeat for both
  Ingress and Egress ports
• Reserved flow do not send to SPC but set OVIN
  and QID
• Send to SPC, complete packet
• Send to SPC, Header Only
• Packet Matches EM and GM Filters at Ingress port
• Alter priorities to verify CARL operation (which
  lookup engine wins)
• EM and GM exclusive, EM has higher priority,
  reserved

25
Scenario X LFS Option Processing

• Case 1 new LFS flow
• FPX send packet with IP option to SPC before it
  is classified.
• SPC processes IP option and send control cell to
  FPX to install new EM filter for flow.
• SPC assigns an unused QID
• Does the SPC have to also assign the OutVIN? Is
  so then the SPC has a complete routing table
  available.
• SPC waits for FPX to reply to control message
  before sending packet back with the RC flag 1.
• The FPX classifies the packet and places in
  correct queue.
• Case 2 Existing LFS reservation, no change
  required
• RC flag 1, packet returned to SPC.
• Case 3 Existing LFS reservation, decrease
  allocation
• Increase available count, send control message to
  FPX (updated reservation), wait for reply, RC
  flag 1, send packet to FPX, look for any
  partial allocations, allocate freed bandwidth,
  send any necessary control cells to FPX with
  updates
• Case 4 Existing LFS reservation, increase
  allocation
• if request can be satisfied then send control
  cell to FPX updating entry, wait for reply, set
  RC flag 1 and return to FPX.
• if request can be partially satisfied then add
  entry to Partial Allocation Table, send control
  cell to FPX updating entry, wait for reply, set
  RC flag 1 and return to FPX.
• if request can not be satisfied then add entry to
  Partial Allocation Table, set RC flag 1 and
  return to FPX.
• Case 5 Existing LFS reservation, Release request
• Increase available count, send control message to
  FPX removing EM filter, set the QID and OutVIN in
  shim to be same as the released EM filter, (RC
  0), send packet to FPX.
• Case 6 No existing LFS reservation, Release
  request
• do nothing, send packet to FPX with RC flag 1.