iSER Draft Status

1
iSER Draft Status

• draft-ietf-ips-iser-00
• Mike Ko
• November 8, 2004

2
iSER Flow Control for Control-Type PDU

• As currently defined, the iSER protocol does not
  provide additional flow control beyond that
  provided by the iSCSI layer on control-type PDUs
• From section 8.1 The iSER Layer SHOULD
  provision enough Untagged buffers for handling
  incoming RDMAP Send Message Types to prevent a
  buffer underrun condition
• iWARP Verbs mechanisms such as the Shared Receive
  Queue mechanism can be used to effectively
  address the Send Message flow control question
• Most iSCSI PDUs are paced by the CmdSN window
  mechanism in iSCSI
• Question Should some form of send side flow
  control be established for iSCSI control-type
  PDUs not regulated by the CmdSN window?

3
Flow Control Alternatives for Unexpected PDUs

• Require the receiver to replenish buffers faster
  than the arrival rate of incoming PDUs including
  the unexpected ones
• Imposes a constraint on the implementation that
  may not be achievable in all circumstances
• Require the RNIC to do graceful handling on lack
  of receiver buffer situations
• Mandates the implementation of an optional
  feature
• Declare a limit on the number of unexpected PDUs
  that can be sent
• If architected correctly, can also accommodate
  options 1, 2, and 5

4
Flow Control Alternatives for Unexpected PDUs

• Define an explicit credit based flow control
  mechanism
• Basically a receive side flow control
• Extra overhead even when options 1, 2, and 5 are
  used
• Require the RNIC to use Shared RQ
• Mandates the implementation of an optional
  feature
• Drop the link if there is buffer overrun
• Default solution if no other flow control
  alternatives
• Drastic solution compared with other alternatives

5
Declared Limit on Sender Side on the Number of
Unexpected PDUs

• First discussed on the IPS reflector in
  July/August of 2003 between Mike Ko, Caitlin
  Bestler, David Black, etc.
• Idea revived and refined in October 2004
• A new key, MaxOutstandingUnexpectedPDU, can be
  used by both the initiator and the target to
  declare the maximum number of outstanding
  unexpected control-type PDUs it can receive
• Key has session wide scope
• Default is none
• None works for options 1, 2, and 5
• If not none, is there a good default which is
  not implementation dependent?

6
iSCSI Control-Type PDUs from the Initiator
Regulated by CmdSN Window
If the Initiator sends this PDU It must provision a buffer for this PDU from the Target
SCSI Command SCSI Response or Reject before the task is active
Task Management Function Request Task Management Function Response
Text Request Text Response
Login Request Login Response
NOP-out (request) ITT / 0xffffffff TTT 0xffffffff NOP-in (response) ITT / 0xffffffff TTT 0xffffffff
7
iSCSI Control-Type PDUs from Initiator with Known
Buffering Requirement at Target

• SCSI Data-out PDU
• For unsolicited data, the iSER layer at the
  target can use FirstBurstLength to determine the
  amount of buffering required
• Solicited data is handled using RDMA Read
  Response
• Data cannot be solicited since an R2T is always
  transformed into an RDMA Read Request and is
  never sent as is by the target
• Section 7.3.4 to be updated to remove references
  on sending solicited SCSI Data-out PDUs
• NOP-out PDU with ITT 0xffffffff and TTT /
  0xffffffff (ping echo) which is sent as a
  response to a NOP-in PDU from the target

8
iSCSI Control-Type PDUs from the Initiator
Subject to Declared Limit

• Any PDU with the immediate flag set
• For the PDUs listed on slide 6, the PDU is
  outstanding until the target responds with the
  corresponding response PDU
• For NOP-out PDU with ITT TTT 0xffffffff, the
  PDU is outstanding until the target responds with
  a control-type PDU with ExpCmdSN set to at least
  x 1 where x is the CmdSN of the PDU with the
  immediate flag set
• NOP-out PDU with ITT 0xffffffff and TTT /
  0xffffffff (ping echo) is not subject to the
  declared limit since it is sent as a response to
  a NOP-in PDU from the target
• SNACK PDU
• Not needed in iSER
• If sent, PDU is outstanding until the target
  responds with a SCSI Response PDU for the
  referenced command

9
Potential Problem with Overuse of Unidirectional
NOP-out

• When the target sends a PDU with ExpCmdSN set to
  x1, up to MaxOutstandingUnexpectedPDU
  unidirectional NOP-out PDUs with ITT TTT
  0xffffffff with CmdSN x could be in flight from
  the initiator
• Upon receiving the PDU from the target, the
  initiator can send MaxOutstandingUnexpectedPDU
  unidirectional NOP-out PDUs with CmdSN x1
• To guard against this pathetic case, the target
  needs to provision the number of buffers equal to
  twice MaxOutstandingUnexpectedPDU in this
  situation for the unexpected PDUs
• The number of buffers for unexpected PDUs can be
  reduced to MaxOutstandingUnexpectedPDU when the
  initiator sends a non-immediate PDU with CmdSN
  x1
• Recommend that we add a cautionary note instead
  against the overuse of the unidirectional NOP-out

10
iSCSI Control-Type PDUs from the Target Subject
to Declared Limit

• The following PDUs are outstanding until the
  initiator sends a control-type PDU with ExpStatSN
  set to at least x 1 where x is the StatSN of
  the unexpected PDU
• Asynchronous Message PDU
• Reject PDU sent after the task is active
• NOP-in PDU initiated by the target with ITT TTT
  0xffffffff
• A NOP-in PDU sent as a ping request by the target
  with ITT 0xffffffff and TTT / 0xffffffff is
  outstanding until the initiator responds with a
  NOP-out PDU (ping echo)

11
Potential Problem with Overuse of Unidirectional
NOP-in

• For a NOP-in PDU from the target with ITT TTT
  0xffffffff, StatSN is not advanced
• When the initiator sends a PDU with ExpStatSN set
  to x1 where x is the StatSN of this NOP-in PDU,
  up to MaxOutstandingUnexpectedPDU unidirectional
  NOP-in PDUs with ITT TTT 0xffffffff with
  StatSN x could be in flight from the target
• Upon receiving the PDU from the initiator, the
  target can send MaxOutstandingUnexpectedPDU
  unidirectional NOP-in PDUs with CmdSN x1
• To guard against this pathetic case, the
  initiator needs to provision the number of
  buffers equal to twice MaxOutstandingUnexpectedPDU
  in this situation for the unexpected PDUs
• The number of buffers for unexpected PDUs can be
  reduced to MaxOutstandingUnexpectedPDU when the
  target sends a PDU with StatSN x1
• Recommend that we add a cautionary note instead
  against the overuse of the unidirectional NOP-in