idnits 2.17.1 draft-ietf-storm-iser-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. == There are 2 instances of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. == The 'Obsoletes: ' line in the draft header should list only the _numbers_ of the RFCs which will be obsoleted by this document (if approved); it should not include the word 'RFC' in the list. -- The abstract seems to indicate that this document obsoletes RFC5046, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: Note that RFC 3720 requires that when a target receives a NOP-Out request with a valid Initiator Task Tag, it responds with a NOP-In with the same Initiator Task Tag that was provided in the NOP-Out request. Furthermore, it returns the first MaxRecvDataSegmentLength bytes of the initiator provided Ping Data. Since there is no MaxRecvDataSegmentLength common to the initiator and the target in iSER, the length of the data sent with the NOP-Out request MUST not exceed InitiatorMaxRecvDataSegmentLength. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: The iSCSI Layer MUST not send a Login Request (or a Login Response) PDU during the full feature phase. A Login Request (or a Login Response) PDU, if used, MUST be treated as an iSCSI protocol error. The iSER Layer MAY reject such a PDU from the iSCSI Layer with an appropriate error code. If a Login Request PDU is received by the iSCSI Layer at the target, it MUST respond with a Reject PDU with a reason code of "protocol error". == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: PDUs in the Unregulated and Unexpected category are PDUs with the immediate flag set. The number of PDUs in this category which can be sent by an initiator is controlled by the value of MaxOutstandingUnexpectedPDUs declared by the target. (See section 6.7.) After a PDU in this category is sent by the initiator, it is outstanding until it is retired. At any time, the number of outstanding unexpected PDUs MUST not exceed the value of MaxOutstandingUnexpectedPDUs declared by the target. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: For the control-type PDUs that can be sent by a target and are unexpected by the initiator, the number is controlled by MaxOutstandingUnexpectedPDUs declared by the initiator. (See section 6.7.) After a PDU in this category is sent by a target, it is outstanding until it is retired. At any time, the number of outstanding unexpected PDUs MUST not exceed the value of MaxOutstandingUnexpectedPDUs declared by the initiator. The initiator uses the value of MaxOutstandingUnexpectedPDUs that it declared to determine the amount of buffer resources required for control-type PDUs in this category that can be sent by a target. The following is a list of the PDUs in this category and the conditions for retiring the outstanding PDU: -- The document date (December 2010) is 4874 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC2119' is defined on line 3439, but no explicit reference was found in the text == Unused Reference: 'VERBS' is defined on line 3449, but no explicit reference was found in the text == Unused Reference: 'IPSEC' is defined on line 3453, but no explicit reference was found in the text ** Obsolete normative reference: RFC 5046 (Obsoleted by RFC 7145) ** Obsolete normative reference: RFC 3720 (Obsoleted by RFC 7143) ** Obsolete normative reference: RFC 793 (ref. 'TCP') (Obsoleted by RFC 9293) -- No information found for draft-hilland-iwarp-verbs-v1 - is the name correct? -- Obsolete informational reference (is this intentional?): RFC 2401 (ref. 'IPSEC') (Obsoleted by RFC 4301) Summary: 3 errors (**), 0 flaws (~~), 12 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Storage Maintenance (StorM) Working Group Mike Ko 3 Internet Draft Huawei Symantec 4 Intended status : Proposed Standard June 14, 2010 5 Expires : December 2010 6 Obsoletes: RFC 5046 8 iSCSI Extensions for RDMA Specification 9 draft-ietf-storm-iser-01.txt 11 Status of this Memo 13 This Internet-Draft is submitted to IETF in full conformance with 14 the provisions of BCP 78 and BCP 79. 16 Internet-Drafts are working documents of the Internet Engineering 17 Task Force (IETF), its areas, and its working groups. Note that 18 other groups may also distribute working documents as Internet- 19 Drafts. 21 Internet-Drafts are draft documents valid for a maximum of six 22 months and may be updated, replaced, or obsoleted by other documents 23 at any time. It is inappropriate to use Internet-Drafts as 24 reference material or to cite them other than as "work in progress." 26 The list of current Internet-Drafts can be accessed at 27 http://www.ietf.org/1id-abstracts.html. 29 The list of Internet-Draft Shadow Directories can be accessed at 30 http://www.ietf.org/shadow.html. 32 This Internet-Draft will expire on December, 2010. 34 Abstract 36 iSCSI Extensions for RDMA provides the RDMA data transfer capability 37 to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol such 38 as the iWARP protocol suite. An RDMA-Capable Protocol provides RDMA 39 Read and Write services, which enable data to be transferred 40 directly into SCSI I/O Buffers without intermediate data copies. 41 This document describes the extensions to the iSCSI protocol to 42 support RDMA services as provided by an RDMA-Capable Protocol such 43 as the iWARP protocol suite. 45 This document obsoletes RFC5046 46 Table of Contents 48 1 Definitions and Acronyms....................................6 49 1.1 Definitions.................................................6 50 1.2 Acronyms...................................................12 51 1.3 Conventions................................................14 52 2 Introduction...............................................15 53 2.1 Motivation.................................................15 54 2.2 Architectural Goals........................................16 55 2.3 Protocol Overview..........................................17 56 2.4 RDMA services and iSER.....................................18 57 2.4.1 STag......................................................18 58 2.4.2 Send......................................................19 59 2.4.3 RDMA Write................................................19 60 2.4.4 RDMA Read.................................................19 61 2.5 SCSI Read Overview.........................................20 62 2.6 SCSI Write Overview........................................20 63 2.7 iSCSI/iSER Layering........................................21 64 3 Upper Layer Interface Requirements.........................22 65 3.1 Operational Primitives offered by iSER.....................22 66 3.1.1 Send_Control..............................................23 67 3.1.2 Put_Data..................................................23 68 3.1.3 Get_Data..................................................23 69 3.1.4 Allocate_Connection_Resources.............................24 70 3.1.5 Deallocate_Connection_Resources...........................24 71 3.1.6 Enable_Datamover..........................................24 72 3.1.7 Connection_Terminate......................................25 73 3.1.8 Notice_Key_Values.........................................25 74 3.1.9 Deallocate_Task_Resources.................................25 75 3.2 Operational Primitives used by iSER........................26 76 3.2.1 Control_Notify............................................26 77 3.2.2 Data_Completion_Notify....................................26 78 3.2.3 Data_ACK_Notify...........................................27 79 3.2.4 Connection_Terminate_Notify...............................27 80 3.3 iSCSI Protocol Usage Requirements..........................27 81 4 Lower Layer Interface Requirements.........................29 82 4.1 Interactions with the RCaP Layer...........................29 83 4.2 Interactions with the Transport Layer......................30 84 5 Connection Setup and Termination...........................31 85 5.1 iSCSI/iSER Connection Setup................................31 86 5.1.1 Initiator Behavior........................................32 87 5.1.2 Target Behavior...........................................34 88 5.1.3 iSER Hello Exchange.......................................35 89 5.2 iSCSI/iSER Connection Termination..........................36 90 5.2.1 Normal Connection Termination at the Initiator............36 91 5.2.2 Normal Connection Termination at the Target...............37 92 5.2.3 Termination without Logout Request/Response PDUs..........37 93 6 Login/Text Operational Keys................................39 94 6.1 HeaderDigest and DataDigest................................39 95 6.2 MaxRecvDataSegmentLength...................................39 96 6.3 RDMAExtensions.............................................40 97 6.4 TargetRecvDataSegmentLength................................41 98 6.5 InitiatorRecvDataSegmentLength.............................41 99 6.6 OFMarker and IFMarker......................................42 100 6.7 MaxOutstandingUnexpectedPDUs...............................42 101 6.8 MaxAHSLength...............................................42 102 6.9 WriteAddressForSolicitedDataOnly...........................43 103 7 iSCSI PDU Considerations...................................44 104 7.1 iSCSI Data-Type PDU........................................44 105 7.2 iSCSI Control-Type PDU.....................................45 106 7.3 iSCSI PDUs.................................................45 107 7.3.1 SCSI Command..............................................45 108 7.3.2 SCSI Response.............................................47 109 7.3.3 Task Management Function Request/Response.................48 110 7.3.4 SCSI Data-out.............................................50 111 7.3.5 SCSI Data-in..............................................50 112 7.3.6 Ready To Transfer (R2T)...................................53 113 7.3.7 Asynchronous Message......................................55 114 7.3.8 Text Request & Text Response..............................55 115 7.3.9 Login Request & Login Response............................55 116 7.3.10 Logout Request & Logout Response........................55 117 7.3.11 SNACK Request...........................................56 118 7.3.12 Reject..................................................56 119 7.3.13 NOP-Out & NOP-In........................................56 120 8 Flow Control and STag Management...........................57 121 8.1 Flow Control for RDMA Send Message Types...................57 122 8.1.1 Flow Control for Control-Type PDUs from the Initiator.....57 123 8.1.2 Flow Control for Control-Type PDUs from the Target........60 124 8.2 Flow Control for RDMA Read Resources.......................61 125 8.3 STag Management............................................61 126 8.3.1 Allocation of STags.......................................62 127 8.3.2 Invalidation of STags.....................................62 128 9 iSER Control and Data Transfer.............................64 129 9.1 iSER Header Format.........................................64 130 9.2 iSER Header Format for iSCSI Control-Type PDU..............64 131 9.3 iSER Header Format for iSER Hello Message..................67 132 9.4 iSER Header Format for iSER HelloReply Message.............68 133 9.5 SCSI Data Transfer Operations..............................69 134 9.5.1 SCSI Write Operation......................................69 135 9.5.2 SCSI Read Operation.......................................70 136 9.5.3 Bidirectional Operation...................................70 137 10 iSER Error Handling and Recovery...........................71 138 10.1 Error Handling............................................71 139 10.1.1 Errors in the Transport Layer...........................71 140 10.1.2 Errors in the RCaP Layer................................72 141 10.1.3 Errors in the iSER Layer................................72 142 10.1.4 Errors in the iSCSI Layer...............................74 143 10.2 Error Recovery............................................76 144 10.2.1 PDU Recovery............................................76 145 10.2.2 Connection Recovery.....................................77 146 11 Security Considerations....................................78 147 12 IANA Considerations........................................79 148 13 References.................................................80 149 13.1 Normative References......................................80 150 13.2 Informative References....................................80 151 14 Summary of Changes from RFC 5046...........................82 152 14.1 Proposed Changes To Be Considered.........................83 153 14.2 Changes To Be Done........................................83 154 Appendix A.......................................................84 155 14.3 iWARP Message Format for iSER.............................84 156 14.3.1 iWARP Message Format for iSER Hello Message.............84 157 14.3.2 iWARP Message Format for iSER HelloReply Message........85 158 14.3.3 iWARP Message Format for SCSI Read Command PDU..........86 159 14.3.4 iWARP Message Format for SCSI Read Data.................87 160 14.3.5 iWARP Message Format for SCSI Write Command PDU.........88 161 14.3.6 iWARP Message Format for RDMA Read Request..............89 162 14.3.7 iWARP Message Format for Solicited SCSI Write Data......90 163 14.3.8 iWARP Message Format for SCSI Response PDU..............91 164 15 Appendix B.................................................92 165 15.1 Architectural discussion of iSER over InfiniBand..........92 166 15.2 The Host side of the iSCSI & iSER connections in Infiniband92 167 15.3 The Storage side of iSCSI & iSER mixed network environment93 168 15.4 Discovery processes for an InfiniBand Host................93 169 15.5 IBTA Connection specifications............................94 170 16 Acknowledgments............................................95 171 Table of Figures 173 Figure 1 Example of iSCSI/iSER Layering in Full Feature Phase...21 174 Figure 2 iSER Header Format.....................................64 175 Figure 3 iSER Header Format for iSCSI Control-Type PDU..........65 176 Figure 4 iSER Header Format for iSER Hello Message..............67 177 Figure 5 iSER Header Format for iSER HelloReply Message.........68 178 Figure 6 SendSE Message containing an iSER Hello Message........84 179 Figure 7 SendSE Message containing an iSER HelloReply Message...85 180 Figure 8 SendSE Message containing a SCSI Read Command PDU......86 181 Figure 9 RDMA Write Message containing SCSI Read Data...........87 182 Figure 10 SendSE Message containing a SCSI Write Command PDU....88 183 Figure 11 RDMA Read Request Message.............................89 184 Figure 12 RDMA Read Response Message containing SCSI Write Data.90 185 Figure 13 SendInvSE Message containing SCSI Response PDU........91 186 Figure 14 iSCSI and iSER on IB..................................92 187 Figure 15 Storage Controller with TCP, iWARP, and IB Connections93 189 1 Definitions and Acronyms 191 1.1 Definitions 193 Advertisement (Advertised, Advertise, Advertisements, Advertises) - 194 The act of informing a remote iSER Layer that a local node's 195 buffer is available to it. A Node makes a buffer available for 196 incoming RDMA Read Request Message or incoming RDMA Write 197 Message access by informing the remote iSER Layer of the Tagged 198 Buffer identifiers (STag, TO, and buffer length). Note that 199 this Advertisement of Tagged Buffer information is the 200 responsibility of the iSER Layer on either end and is not 201 defined by the RDMA-Capable Protocol. A typical method would be 202 for the iSER Layer to embed the Tagged Buffer's STag, TO, and 203 buffer length in a Send Message destined for the remote iSER 204 Layer. 206 Completion (Completed, Complete, Completes) - Completion is defined 207 as the process by the RDMA-Capable Protocol layer to inform the 208 iSER Layer, that a particular RDMA Operation has performed all 209 functions specified for the RDMA Operation. 211 Connection - A connection is a logical circuit between the initiator 212 and the target, e.g., a TCP connection. Communication between 213 the initiator and the target occurs over one or more 214 connections. The connections carry control messages, SCSI 215 commands, parameters, and data within iSCSI Protocol Data Units 216 (iSCSI PDUs). 218 Connection Handle - An information element that identifies the 219 particular iSCSI connection and is unique for a given iSCSI-iSER 220 pair. Every invocation of an Operational Primitive is qualified 221 with the Connection Handle. 223 Data Sink - The peer receiving a data payload. Note that the Data 224 Sink can be required to both send and receive RCaP Messages to 225 transfer a data payload. 227 Data Source - The peer sending a data payload. Note that the Data 228 Source can be required to both send and receive RCaP Messages to 229 transfer a data payload. 231 Datamover Interface (DI) - The interface between the iSCSI Layer and 232 the Datamover Layer as described in [DA]. 234 Datamover Layer - A layer that is directly below the iSCSI Layer and 235 above the underlying transport layers. This layer exposes and 236 uses a set of transport independent Operational Primitives for 237 the communication between the iSCSI Layer and itself. The 238 Datamover layer, operating in conjunction with the transport 239 layers, moves the control and data information on the iSCSI 240 connection. In this specification, the iSER Layer is the 241 Datamover layer. 243 Datamover Protocol - A Datamover protocol is the wire-protocol that 244 is defined to realize the Datamover layer functionality. In 245 this specification, the iSER protocol is the Datamover protocol. 247 Event - An indication provided by the RDMA-Capable Protocol layer to 248 the iSER Layer to indicate a Completion or other condition 249 requiring immediate attention. 251 Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming 252 outstanding RDMA Read Requests that the RDMA-Capable Controller 253 can handle on a particular RCaP Stream at the Data Source. For 254 some RDMA-Capable Protocol layers, the term "IRD" may be known 255 by a different name. For example, for InfiniBand, the 256 equivalent for IRD is the Responder Resources. 258 Invalidate STag - A mechanism used to prevent the Remote Peer from 259 reusing a previous explicitly Advertised STag, until the iSER 260 Layer at the local node makes it available through a subsequent 261 explicit Advertisement. 263 I/O Buffer - A buffer that is used in a SCSI Read or Write operation 264 so SCSI data may be sent from or received into that buffer. 266 iSCSI - The iSCSI protocol as defined in [RFC3720] is a mapping of 267 the SCSI Architecture Model of SAM-2 over TCP. 269 iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data- 270 type PDU and also not a SCSI Data-out PDU carrying solicited 271 data is defined as an iSCSI control-type PDU. Specifically, it 272 is to be noted that SCSI Data-out PDUs for unsolicited data are 273 defined as iSCSI control-type PDUs. 275 iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI 276 PDU that causes data transfer, transparent to the remote iSCSI 277 Layer, to take place between the peer iSCSI nodes on a full 278 feature phase iSCSI connection. An iSCSI data-type PDU, when 279 requested for transmission by the sender iSCSI Layer, results in 280 the associated data transfer without the participation of the 281 remote iSCSI Layer, i.e. the PDU itself is not delivered as-is 282 to the remote iSCSI Layer. The following iSCSI PDUs constitute 283 the set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU. 285 iSCSI Layer - A layer in the protocol stack implementation within an 286 end node that implements the iSCSI protocol and interfaces with 287 the iSER Layer via the Datamover Interface. 289 iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI Layer at the 290 initiator and the iSCSI Layer at the target divide their 291 communications into messages. The term "iSCSI protocol data 292 unit" (iSCSI PDU) is used for these messages. 294 iSCSI/iSER Connection - An iSER-assisted iSCSI connection. 296 iSCSI/iSER Session - An iSER-assisted iSCSI session. 298 iSCSI-iSER Pair - The iSCSI Layer and the underlying iSER Layer. 300 iSER - iSCSI Extensions for RDMA, the protocol defined in this 301 document. 303 iSER-assisted - A term generally used to describe the operation of 304 iSCSI when the iSER functionality is also enabled below the 305 iSCSI Layer for the specific iSCSI/iSER connection in question. 307 iSER-IRD - This variable represents the maximum number of incoming 308 outstanding RDMA Read Requests that the iSER Layer at the 309 initiator declares on a particular RCaP Stream. 311 iSER-ORD - This variable represents the maximum number of 312 outstanding RDMA Read Requests that the iSER Layer can initiate 313 on a particular RCaP Stream. This variable is maintained only 314 by the iSER Layer at the target. 316 iSER Layer - The layer that implements the iSCSI Extensions for RDMA 317 (iSER) protocol. 319 iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and 320 [MPA] when layered above [TCP]. [RDMAP] and [DDP] may be 321 layered above SCTP or other transport protocols. 323 Local Mapping - A task state record maintained by the iSER Layer 324 that associates the Initiator Task Tag to the Local STag(s). 325 The specifics of the record structure are implementation 326 dependent. 328 Local Peer - The implementation of the RDMA-Capable Protocol on the 329 local end of the connection. Used to refer to the local entity 330 when describing protocol exchanges or other interactions between 331 two Nodes. 333 Node - A computing device attached to one or more links of a 334 network. A Node in this context does not refer to a specific 335 application or protocol instantiation running on the computer. 336 A Node may consist of one or more RDMA-Capable Controllers 337 installed in a host computer. 339 Operational Primitive - An Operational Primitive is an abstract 340 functional interface procedure that requests another layer to 341 perform a specific action on the requestor's behalf or notifies 342 the other layer of some event. The Datamover Interface between 343 an iSCSI Layer and a Datamover layer within an iSCSI end node 344 uses a set of Operational Primitives to define the functional 345 interface between the two layers. Note that not every 346 invocation of an Operational Primitive may elicit a response 347 from the requested layer. A full discussion of the Operational 348 Primitive types and request-response semantics available to 349 iSCSI and iSER can be found in [DA]. 351 Outbound RDMA Read Queue Depth (ORD) - The maximum number of 352 outstanding RDMA Read Requests that the RDMA-Capable Controller 353 can initiate on a particular RCaP Stream at the Data Sink. For 354 some RDMA-Capable Protocol layer, the term "ORD" may be known by 355 a different name. For example, for InfiniBand, the equivalent 356 for ORD is the Initiator Depth. 358 Phase-Collapse - Refers to the optimization in iSCSI where the SCSI 359 status is transferred along with the final SCSI Data-in PDU from 360 a target. See section 3.2 in [RFC3720]. 362 RCaP Message - One or more packets of the network layer comprising a 363 single RDMA operation or a part of an RDMA Read Operation of the 364 RDMA-Capable Protocol. For iWARP, an RCaP Message is known as 365 an RDMAP Message. 367 RCaP Stream - A single bidirectional association between the peer 368 RDMA-Capable Protocol layers on two Nodes over a single 369 transport-level stream. For iWARP, an RCaP Stream is known as 370 an RDMAP Stream, and the association is created following a 371 successful Login Phase during which iSER support is negotiated. 373 RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that 374 provides a reliable RDMA transport functionality, e.g., iWARP, 375 InfiniBand, etc. 377 RDMA-Capable Controller - A network I/O adapter or embedded 378 controller with RDMA functionality. For example, for iWARP, 379 this could be an RNIC, and for InfiniBand, this could be a HCA 380 (Host Channel Adapter) or TCA (Target Channel Adapter). 382 RDMA-enabled Network Interface Controller (RNIC) - A network I/O 383 adapter or embedded controller with iWARP functionality. 385 RDMA Operation - A sequence of RCaP Messages, including control 386 Messages, to transfer data from a Data Source to a Data Sink. 387 The following RDMA Operations are defined - RDMA Write 388 Operation, RDMA Read Operation, Send Operation, Send with 389 Invalidate Operation, Send with Solicited Event Operation, Send 390 with Solicited Event and Invalidate Operation, and Terminate 391 Operation. 393 RDMA Protocol (RDMAP) - A wire protocol that supports RDMA 394 Operations to transfer ULP data between a Local Peer and the 395 Remote Peer as described in [RDMAP]. 397 RDMA Read Operation - An RDMA Operation used by the Data Sink to 398 transfer the contents of a Data Source buffer from the Remote 399 Peer to a Data Sink buffer at the Local Peer. An RDMA Read 400 operation consists of a single RDMA Read Request Message and a 401 single RDMA Read Response Message. 403 RDMA Read Request - An RCaP Message used by the Data Sink to request 404 the Data Source to transfer the contents of a buffer. The RDMA 405 Read Request Message describes both the Data Source and the Data 406 Sink buffers. 408 RDMA Read Response - An RCaP Message used by the Data Source to 409 transfer the contents of a buffer to the Data Sink, in response 410 to an RDMA Read Request. The RDMA Read Response Message only 411 describes the Data Sink buffer. 413 RDMA Write Operation - An RDMA Operation used by the Data Source to 414 transfer the contents of a Data Source buffer from the Local 415 Peer to a Data Sink buffer at the Remote Peer. The RDMA Write 416 Message only describes the Data Sink buffer. 418 Remote Direct Memory Access (RDMA) - A method of accessing memory on 419 a remote system in which the local system specifies the remote 420 location of the data to be transferred. Employing an RDMA- 421 Capable Controller in the remote system allows the access to take 422 place without interrupting the processing of the CPU(s) on the 423 system. 425 Remote Mapping - A task state record maintained by the iSER Layer 426 that associates the Initiator Task Tag to the Advertised STag(s). 427 The specifics of the record structure are implementation 428 dependent. 430 Remote Peer - The implementation of the RDMA-Capable Protocol on the 431 opposite end of the connection. Used to refer to the remote 432 entity when describing protocol exchanges or other interactions 433 between two Nodes. 435 SCSI Layer - This layer builds/receives SCSI CDBs (Command 436 Descriptor Blocks) and sends/receives them with the remaining 437 command execute [SAM2] parameters to/from the iSCSI Layer. 439 Send - An RDMA Operation that transfers the contents of a Buffer 440 from the Local Peer to a Buffer at the Remote Peer. 442 Send Message Type - A Send Message, Send with Invalidate Message, 443 Send with Solicited Event Message, or Send with Solicited Event 444 and Invalidate Message. 446 SendInvSE Message - A Send with Solicited Event and Invalidate 447 Message. 449 SendSE Message - A Send with Solicited Event Message 451 Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for 452 an R2T PDU. The semantics for both types of sequence numbers 453 are as defined in [RFC3720]. 455 Session, iSCSI Session - The group of Connections that link an 456 initiator SCSI port with a target SCSI port form an iSCSI 457 session (equivalent to a SCSI I-T nexus). Connections can be 458 added to and removed from a session even while the I-T nexus is 459 intact. Across all connections within a session, an initiator 460 sees one and the same target. 462 Solicited Event (SE) - A facility by which an RDMA Operation sender 463 may cause an Event to be generated at the recipient, if the 464 recipient is configured to generate such an Event, when a Send 465 with Solicited Event or Send with Solicited Event and Invalidate 466 Message is received. 468 Steering Tag (STag) - An identifier of a Tagged Buffer on a Node 469 (Local or Remote) as defined in [RDMAP] and [DDP]. For other 470 RDMA-Capable Protocols, the Steering Tag may be known by 471 different names but will be herein referred to as STags. For 472 example, for Infiniband, a Remote STag is known as an R-Key, and 473 a Local STag is known as an L-Key, and both will be considered 474 STags. 476 Tagged Buffer - A buffer that is explicitly Advertised to the iSER 477 Layer at the remote node through the exchange of an STag, Tagged 478 Offset, and length. 480 Tagged Offset (TO) - The offset within a Tagged Buffer. 482 Traditional iSCSI - Refers to the iSCSI protocol as defined in 483 [RFC3720] (i.e. without the iSER enhancements). 485 Untagged Buffer - A buffer that is not explicitly Advertised to the 486 iSER Layer at the remode node. 488 1.2 Acronyms 490 Acronym Definition 492 -------------------------------------------------------------- 494 AHS Additional Header Segment 496 BHS Basic Header Segment 498 CO Connection Only 500 CRC Cyclic Redundancy Check 502 DDP Direct Data Placement Protocol 504 DI Datamover Interface 506 HCA Host Channel Adapter 508 IANA Internet Assigned Numbers Authority 510 IB Infiniband 512 IETF Internet Engineering Task Force 514 I/O Input - Output 515 IO Initialize Only 517 IP Internet Protocol 519 IPoIB IP over Infiniband 521 IPsec Internet Protocol Security 523 iSER iSCSI Extensions for RDMA 525 ITT Initiator Task Tag 527 LO Leading Only 529 MPA Marker PDU Aligned Framing for TCP 531 NOP No Operation 533 NSG Next Stage (during the iSCSI Login Phase) 535 OS Operating System 537 PDU Protocol Data Unit 539 R2T Ready To Transfer 541 R2TSN Ready To Transfer Sequence Number 543 RDMA Remote Direct Memory Access 545 RDMAP Remote Direct Memory Access Protocol 547 RFC Request For Comments 549 RNIC RDMA-enabled Network Interface Controller 551 SAM2 SCSI Architecture Model - 2 553 SCSI Small Computer Systems Interface 555 SNACK Selective Negative Acknowledgment - also 557 Sequence Number Acknowledgement for data 559 STag Steering Tag 561 SW Session Wide 562 TCA Target Channel Adapter 564 TCP Transmission Control Protocol 566 TMF Task Management Function 568 TTT Target Transfer Tag 570 TO Tagged Offset 572 ULP Upper Level Protocol 574 1.3 Conventions 576 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 577 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 578 document are to be interpreted as described in RFC 2119. 580 2 Introduction 582 2.1 Motivation 584 The iSCSI protocol ([RFC3720]) is a mapping of the SCSI Architecture 585 Model (see [SAM2]) over the TCP protocol. SCSI commands are carried 586 by iSCSI requests and SCSI responses and status are carried by iSCSI 587 responses. Other iSCSI protocol exchanges and SCSI Data are also 588 transported in iSCSI PDUs. 590 Out-of-order TCP segments in the Traditional iSCSI model have to be 591 stored and reassembled before the iSCSI protocol layer within an end 592 node can place the data in the iSCSI buffers. This reassembly is 593 required because not every TCP segment is likely to contain an iSCSI 594 header to enable its placement and TCP itself does not have a built- 595 in mechanism for signaling ULP message boundaries to aid placement 596 of out-of-order segments. This TCP reassembly at high network 597 speeds is quite counter-productive for the following reasons: wasted 598 memory bandwidth in data copying, need for reassembly memory, wasted 599 CPU cycles in data copying, and the general store-and-forward 600 latency from an application perspective. TCP reassembly was 601 recognized as a serious issue in [RFC3720], and the notion of a 602 "sync and steering layer" was introduced that is optional to 603 implement and use. One specific sync and steering mechanism, called 604 "markers", was defined in [RFC3720] which provides an application- 605 level way of framing iSCSI PDUs within the TCP data stream even when 606 the TCP segments are not yet reassembled to be in-order. 608 With these defined techniques in [RFC3720], a Network Interface 609 Controller customized for iSCSI (SNIC) could offload the TCP/IP 610 processing and support direct data placement, but most iSCSI 611 implementations do not support iSCSI "markers", making SNIC 612 marker-based direct data placement unusable in practice. 614 The iWARP protocol stack provides direct data placement 615 functionality that is usable in practice, and in addition, there is 616 also interest in using iSCSI with other RDMA protocol stacks that 617 support direct data placement, such as the one provided by 618 InfiniBand. The generic term RDMA-Capable Protocol (RCaP) is used 619 to refer to the RDMA functionality provided by such protocol stacks. 621 With the availability of RDMA-Capable Controllers within a host 622 system, which does not have SNICs, it is appropriate for iSCSI to be 623 able to exploit the direct data placement function of the RDMA- 624 Capable Controller like other applications. 626 iSCSI Extensions for RDMA (iSER) is designed precisely to take 627 advantage of generic RDMA technologies - iSER's goal is to permit 628 iSCSI to employ direct data placement and RDMA capabilities using a 629 generic RDMA-Capable Controller. In summary, iSCSI/iSER protocol 630 stack is designed to enable scaling to high speeds by relying on a 631 generic data placement process and RDMA technologies and products, 632 which enable direct data placement of both in-order and out-of-order 633 data. 635 This document describes iSER as a protocol extension to iSCSI, both 636 for convenience of description and also because it is true in a very 637 strict protocol sense. However, it is to be noted that iSER is in 638 reality extending the connectivity of the iSCSI protocol defined in 639 [RFC3720], and the name iSER reflects this reality. 641 When the iSCSI protocol as defined in [RFC3720] (i.e. without the 642 iSER enhancements) is intended in the rest of the document, the term 643 "Traditional iSCSI" is used to make the intention clear. 645 2.2 Architectural Goals 647 This section summarizes the architectural goals that guided the 648 design of iSER. 650 1. Provide an RDMA data transfer model for iSCSI that enables direct 651 in order or out of order data placement of SCSI data into pre- 652 allocated SCSI buffers while maintaining in order data delivery. 654 2. Not require any major changes to SCSI Architecture Model [SAM2] 655 and SCSI command set standards. 657 3. Utilize existing iSCSI infrastructure (sometimes referred to as 658 "iSCSI ecosystem") including but not limited to MIB, 659 bootstrapping, negotiation, naming & discovery, and security. 661 4. Require a session to operate in the Traditional iSCSI data 662 transfer mode if iSER is not supported by either the initiator or 663 the target (not require iSCSI full feature phase interoperability 664 between an end node operating in Traditional iSCSI mode, and an 665 end node operating in iSER-assisted mode). 667 5. Allow initiator and target implementations to utilize generic 668 RDMA-Capable Controllers such as RNICs, or implement iSCSI and 669 iSER in software (not require iSCSI or iSER specific assists in 670 the RCaP implementation or RDMA-Capable Controller). 672 6. Require full and only generic RCaP functionality at both the 673 initiator and the target. 675 7. Implement a light weight Datamover protocol for iSCSI with minimal 676 state maintenance. 678 2.3 Protocol Overview 680 Consistent with the architectural goals stated in section 2.2, the 681 iSER protocol does not require changes in the iSCSI ecosystem or any 682 related SCSI specifications. iSER protocol defines the mapping of 683 iSCSI PDUs to RCaP Messages in such a way that it is entirely 684 feasible to realize iSCSI/iSER implementations that are based on 685 generic RDMA-Capable Controllers. The iSER protocol layer requires 686 minimal state maintenance to assist an iSCSI full feature phase 687 connection, besides being oblivious to the notion of an iSCSI 688 session. The crucial protocol aspects of iSER may be summarized 689 thus: 691 1. iSER-assisted mode is negotiated during the iSCSI login for each 692 session, and an entire iSCSI session can only operate in one mode 693 (i.e. a connection in a session cannot operate in iSER-assisted 694 mode if a different connection of the same session is already in 695 full feature phase in the Traditional iSCSI mode). 697 2. Once in iSER-assisted mode, all iSCSI interactions on that 698 connection use RCaP Messages. 700 3. A Send Message Type is used for carrying an iSCSI control-type 701 PDU preceded by an iSER header. See section 7.2 for more details 702 on iSCSI control-type PDUs. 704 4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages 705 are used for carrying control and all data information associated 706 with the iSCSI data-type PDUs. See section 7.1 for more details 707 on iSCSI data-type PDUs. 709 5. Target drives all data transfer (with the exception of iSCSI 710 unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA 711 Read Requests and RDMA Writes respectively. 713 6. RCaP is responsible for ensuring data integrity. (For example, 714 iWARP includes a CRC-enhanced framing layer called MPA on top of 715 TCP; and for Infiniband, the CRCs are included in the Reliable 716 Connection mode). For this reason, iSCSI header and data digests 717 are negotiated to "None" for iSCSI/iSER sessions. 719 7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully 720 supported by iSER. (However, see section 7.3.11 on the handling 721 of SNACK Request PDUs.) 723 8. iSER requires no changes to iSCSI authentication, security, and 724 text mode negotiation mechanisms. 726 Note that Traditional iSCSI implementations may have to be adapted 727 to employ iSER. It is expected that the adaptation when required is 728 likely to be centered around the upper layer interface requirements 729 of iSER (section 3). 731 2.4 RDMA services and iSER 733 iSER is designed to work with software and/or hardware protocol 734 stacks providing the protocol services defined in RCaP documents 735 such as [RDMAP], [IB], etc. The following subsections describe the 736 key protocol elements of RCaP services that iSER relies on. 738 2.4.1 STag 740 An STag is the identifier of an I/O Buffer unique to an RDMA-Capable 741 Controller that the iSER Layer Advertises to the remote iSCSI/iSER 742 node in order to complete a SCSI I/O. 744 In iSER, Advertisement is the act of informing the target by the 745 initiator that an I/O Buffer is available at the initiator for RDMA 746 Read or RDMA Write access by the target. The initiator Advertises 747 the I/O Buffer by including the STag in the header of an iSER 748 Message containing the SCSI Command PDU to the target. The base 749 Tagged Offset is not explicitly specified, but the target must 750 always assume it as zero. The buffer length is as specified in the 751 SCSI Command PDU. 753 The iSER Layer at the initiator Advertises the STag for the I/O 754 Buffer of each SCSI I/O to the iSER Layer at the target in the iSER 755 header of the SendSE Message containing the SCSI Command PDU, unless 756 the I/O can be completely satisfied by unsolicited data alone. 758 The iSER Layer at the target provides the STag for the I/O Buffer 759 that is the Data Sink of an RDMA Read Operation (section 2.4.4) to 760 the RCaP layer on the initiator node - i.e. this is completely 761 transparent to the iSER Layer at the initiator. 763 The iSER protocol is defined so that the Advertised STag is 764 automatically invalidated upon a normal completion of the associated 765 task. This automatic invalidation is realized via the SendInvSE 766 Message carrying the SCSI Response PDU. There are two exceptions to 767 this automatic invalidation - bidirectional commands, and abnormal 768 completion of a command. The iSER Layer at the initiator is 769 required to explicitly invalidate the STag in these cases, in 770 addition to sanity checking the automatic invalidation even when 771 that does happen. 773 2.4.2 Send 775 Send is the RDMA Operation that is not addressed to an Advertised 776 buffer by the sending side, and thus uses Untagged buffers on the 777 receiving side. 779 The iSER Layer at the initiator uses the Send Operation to transmit 780 any iSCSI control-type PDU to the target. As an example, the 781 initiator uses Send Operations to transfer iSER Messages containing 782 SCSI Command PDUs to the iSER Layer at the target. 784 An iSER layer at the target uses the Send Operation to transmit any 785 iSCSI control-type PDU to the initiator. As an example, the target 786 uses Send Operations to transfer iSER Messages containing SCSI 787 Response PDUs to the iSER Layer at the initiator. 789 2.4.3 RDMA Write 791 RDMA Write is the RDMA Operation that is used to place data into an 792 Advertised buffer on the receiving side. The sending side addresses 793 the Message using an STag and a Tagged Offset that are valid on the 794 Data Sink. 796 The iSER Layer at the target uses the RDMA Write Operation to 797 transfer the contents of a local I/O Buffer to an Advertised I/O 798 Buffer at the initiator. The iSER Layer at the target uses the RDMA 799 Write to transfer whole or part of the data required to complete a 800 SCSI Read command. 802 The iSER Layer at the initiator does not employ RDMA Writes. 804 2.4.4 RDMA Read 806 RDMA Read is the RDMA Operation that is used to retrieve data from 807 an Advertised buffer on a remote node. The sending side of the RDMA 808 Read Request addresses the Message using an STag and a Tagged Offset 809 that are valid on the Data Source in addition to providing a valid 810 local STag and Tagged Offset that identify the Data Sink. 812 The iSER Layer at the target uses the RDMA Read Operation to 813 transfer the contents of an Advertised I/O Buffer at the initiator 814 to a local I/O Buffer at the target. The iSER Layer at the target 815 uses the RDMA Read to fetch whole or part of the data required to 816 complete a SCSI Write Command. 818 The iSER Layer at the initiator does not employ RDMA Reads. 820 2.5 SCSI Read Overview 822 The iSER Layer at the initiator receives the SCSI Command PDU from 823 the iSCSI Layer. The iSER Layer at the initiator generates an STag 824 for the I/O Buffer of the SCSI Read and Advertises the buffer by 825 including the STag as part of the iSER header for the PDU. The iSER 826 Message is transferred to the target using a SendSE Message. 828 The iSER Layer at the target uses one or more RDMA Writes to 829 transfer the data required to complete the SCSI Read. 831 The iSER Layer at the target uses a SendInvSE Message to transfer 832 the SCSI Response PDU back to the iSER Layer at the initiator. The 833 iSER Layer at the initiator notifies the iSCSI Layer of the 834 availability of the SCSI Response PDU. 836 2.6 SCSI Write Overview 838 The iSER Layer at the initiator receives the SCSI Command PDU from 839 the iSCSI Layer. If solicited data transfer is involved, the iSER 840 Layer at the initiator generates an STag for the I/O Buffer of the 841 SCSI Write and Advertises the buffer by including the STag as part 842 of the iSER header for the PDU. The iSER Message is transferred to 843 the target using a SendSE Message. 845 The iSER Layer at the initiator may optionally send one or more non- 846 immediate unsolicited data PDUs to the target using Send Message 847 Types. 849 If solicited data transfer is involved, the iSER Layer at the target 850 uses one or more RDMA Reads to transfer the data required to 851 complete the SCSI Write. 853 The iSER Layer at the target uses a SendInvSE Message to transfer 854 the SCSI Response PDU back to the iSER Layer at the initiator. The 855 iSER Layer at the initiator notifies the iSCSI Layer of the 856 availability of the SCSI Response PDU. 858 2.7 iSCSI/iSER Layering 860 iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer 861 and the RCaP layer. Note that the RCaP layer may be composed of one 862 or more distinct protocol layers depending on the specifics of the 863 RCaP. Figure 1 shows an example of the relationship between SCSI, 864 iSCSI, iSER, and the different RCaP layers. For TCP, the RCaP is 865 iWARP. For Infiniband, the RCaP is the Reliable Connected Transport 866 Service. Note that the iSCSI layer as described here supports the 867 RDMA Extensions as used in iSER. 869 +-------------------------------------+ 870 | SCSI | 871 +-------------------------------------+ 872 | iSCSI | 873 DI ------> +-------------------------------------+ 874 | iSER | 875 +---------+--------------+------------+ 876 | RDMAP | | | 877 +---------+ Infiniband | | 878 | DDP | Reliable | Other | 879 +---------+ Connected | RDMA- | 880 | MPA | Transport | Capable | 881 +---------+ Service | Protocol | 882 | TCP | | | 883 +---------+--------------+------------+ 884 | | Infiniband | Other | 885 | IP | Network | Network | 886 | | Layer | Layer | 887 +---------+--------------+------------+ 889 Figure 1 Example of iSCSI/iSER Layering in Full Feature Phase 891 3 Upper Layer Interface Requirements 893 This section discusses the upper layer interface requirements in the 894 form of an abstract model of the required interactions between the 895 iSCSI Layer and the iSER Layer. The abstract model used here is 896 derived from the architectural model described in [DA]. [DA] also 897 provides a functional overview of the interactions between the iSCSI 898 Layer and the datamover layer as intended by the Datamover 899 Architecture. 901 The interface requirements are specified by Operational Primitives. 902 An Operational Primitive is an abstract functional interface 903 procedure between the iSCSI Layer and the iSER Layer that requests 904 one layer to perform a specific action on behalf of the other layer 905 or notifies the other layer of some event. Whenever an Operational 906 Primitive in invoked, the Connection_Handle qualifier is used to 907 identify a particular iSCSI connection. For some Operational 908 Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI 909 data buffer associated with the requested or completed operation. 911 The abstract model and the Operational Primitives defined in this 912 section facilitate the description of the iSER protocol. In the 913 rest of the iSER specification, the compliance statements related to 914 the use of these Operational Primitives are only for the purpose of 915 the required interactions between the iSCSI Layer and the iSER 916 Layer. Note that the compliance statements related to the 917 Operational Primitives in the rest of this specification only 918 mandate functional equivalence on implementations, but do not put 919 any requirements on the implementation specifics of the interface 920 between the iSCSI Layer and the iSER Layer. 922 Each Operational Primitive is invoked with a set of qualifiers which 923 specify the information context for performing the specific action 924 being requested of the Operational Primitive. While the qualifiers 925 are required, the method of realizing the qualifiers (e.g., by 926 passing synchronously with invocation, or by retrieving from task 927 context, or by retrieving from shared memory, etc.) is 928 implementation dependent. 930 3.1 Operational Primitives offered by iSER 932 The iSER protocol layer MUST support the following Operational 933 Primitives to be used by the iSCSI protocol layer. 935 3.1.1 Send_Control 937 Input qualifiers: Connection_Handle, BHS and AHS (if any) of 938 the iSCSI PDU, PDU-specific qualifiers 940 Return results: Not specified 942 This is used by the iSCSI Layers at the initiator and the target to 943 request the outbound transfer of an iSCSI control-type PDU (see 944 section 7.2). Qualifiers that only apply for a particular control- 945 type PDU are known as PDU-specific qualifiers, e.g., 946 ImmediateDataSize for a SCSI Write command. For details on PDU- 947 specific qualifiers, see section 7.3. The iSCSI Layer can only 948 invoke the Send_Control Operational Primitive when the connection is 949 in iSER-assisted mode. 951 3.1.2 Put_Data 953 Input qualifiers: Connection_Handle, content of a SCSI Data-in 954 PDU header, Data_Descriptor, Notify_Enable 956 Return results: Not specified 958 This is used by the iSCSI Layer at the target to request the 959 outbound transfer of data for a SCSI Data-in PDU from the buffer 960 identified by the Data_Descriptor qualifier. The iSCSI Layer can 961 only invoke the Put_Data Operational Primitive when the connection 962 is in iSER-assisted mode. 964 The Notify_Enable qualifier is used to indicate to the iSER Layer 965 whether or not it should generate an eventual local completion 966 notification to the iSCSI Layer. See section 3.2.2 on 967 Data_Completion_Notify for details. 969 3.1.3 Get_Data 971 Input qualifiers: Connection_Handle, content of an R2T PDU, 972 Data_Descriptor, Notify_Enable 974 Return results: Not specified 976 This is used by the iSCSI Layer at the target to request the inbound 977 transfer of solicited data requested by an R2T PDU into the buffer 978 identified by the Data_Descriptor qualifier. The iSCSI Layer can 979 only invoke the Get_Data Operational Primitive when the connection 980 is in iSER-assisted mode. 982 The Notify_Enable qualifier is used to indicate to the iSER Layer 983 whether or not it should generate the eventual local completion 984 notification to the iSCSI Layer. See section 3.2.2 on 985 Data_Completion_Notify for details. 987 3.1.4 Allocate_Connection_Resources 989 Input qualifiers: Connection_Handle, Resource_Descriptor 990 (optional) 992 Return results: Status 994 This is used by the iSCSI Layers at the initiator and the target to 995 request the allocation of all connection resources necessary to 996 support RCaP for an operational iSCSI/iSER connection. The iSCSI 997 Layer may optionally specify the implementation-specific resource 998 requirements for the iSCSI connection using the Resource_Descriptor 999 qualifier. 1001 A return result of Status=success means the invocation succeeded, 1002 and a return result of Status=failure means that the invocation 1003 failed. If the invocation is for a Connection_Handle for which an 1004 earlier invocation succeeded, the request will be ignored by the 1005 iSER Layer and the result of Status=success will be returned. Only 1006 one Allocate_Connection_Resources Operational Primitive invocation 1007 can be outstanding for a given Connection_Handle at any time. 1009 3.1.5 Deallocate_Connection_Resources 1011 Input qualifiers: Connection_Handle 1013 Return results: Not specified 1015 This is used by the iSCSI Layers at the initiator and the target to 1016 request the deallocation of all connection resources that were 1017 allocated earlier as a result of a successful invocation of the 1018 Allocate_Connection_Resources Operational Primitive. 1020 3.1.6 Enable_Datamover 1022 Input qualifiers: Connection_Handle, 1023 Transport_Connection_Descriptor, Final Login_Response_PDU 1024 (optional) 1026 Return results: Not specified 1028 This is used by the iSCSI Layers at the initiator and the target to 1029 request that iSER-assisted mode be used for the connection. The 1030 Transport_Connection_Descriptor qualifier is used to identify the 1031 specific connection associated with the Connection_Handle. The 1032 iSCSI layer can only invoke the Enable_Datamover Operational 1033 Primitive when there was a corresponding prior resource allocation. 1035 The Final_Login_Response_PDU input qualifier is applicable only for 1036 a target, and contains the final Login Response PDU that concludes 1037 the iSCSI Login Phase. 1039 3.1.7 Connection_Terminate 1041 Input qualifiers: Connection_Handle 1043 Return results: Not specified 1045 This is used by the iSCSI Layers at the initiator and the target to 1046 request that a specified iSCSI/iSER connection be terminated and all 1047 associated connection and task resources be freed. When this 1048 Operational Primitive invocation returns to the iSCSI layer, the 1049 iSCSI layer may assume full ownership of all iSCSI-level resources, 1050 e.g. I/O Buffers, associated with the connection. 1052 3.1.8 Notice_Key_Values 1054 Input qualifiers: Connection_Handle, number of keys, list of 1055 Key-Value pairs 1057 Return results: Not specified 1059 This is used by the iSCSI Layers at the initiator and the target to 1060 request the iSER Layer to take note of the specified Key-Value pairs 1061 which were negotiated by the iSCSI peers for the connection. 1063 3.1.9 Deallocate_Task_Resources 1065 Input qualifiers: Connection_Handle, ITT 1067 Return results: Not specified 1069 This is used by the iSCSI Layers at the initiator and the target to 1070 request the deallocation of all RCaP-specific resources allocated by 1071 the iSER Layer for the task identified by the ITT qualifier. The 1072 iSER Layer may require a certain number of RCaP-specific resources 1073 associated with the ITT for each new iSCSI task. In the normal 1074 course of execution, these task-level resources in the iSER Layer 1075 are assumed to be transparently allocated on each task initiation 1076 and deallocated on the conclusion of each task as appropriate. In 1077 exception scenarios where the task does not conclude with a SCSI 1078 Response PDU, the iSER Layer needs to be notified of the individual 1079 task terminations to aid its task-level resource management. This 1080 Operational Primitive is used for this purpose, and is not needed 1081 when a SCSI Response PDU normally concludes a task. Note that RCaP- 1082 specific task resources are deallocated by the iSER Layer when a 1083 SCSI Response PDU normally concludes a task, even if the SCSI Status 1084 was not success. 1086 3.2 Operational Primitives used by iSER 1088 The iSER layer MUST use the following Operational Primitives offered 1089 by the iSCSI protocol layer when the connection is in iSER-assisted 1090 mode. 1092 3.2.1 Control_Notify 1094 Input qualifiers: Connection_Handle, an iSCSI control-type PDU 1096 Return results: Not specified 1098 This is used by the iSER Layers at the initiator and the target to 1099 notify the iSCSI Layer of the availability of an inbound iSCSI 1100 control-type PDU. A PDU is described as "available" to the iSCSI 1101 Layer when the iSER Layer notifies the iSCSI Layer of the reception 1102 of that inbound PDU, along with an implementation-specific 1103 indication as to where the received PDU is. 1105 3.2.2 Data_Completion_Notify 1107 Input qualifiers: Connection_Handle, ITT, SN 1109 Return results: Not specified 1111 This is used by the iSER Layer to notify the iSCSI Layer of the 1112 completion of outbound data transfer that was requested by the iSCSI 1113 Layer only if the invocation of the Put_Data Operational Primitive 1114 (see section 3.1.2) was qualified with Notify_Enable set. SN refers 1115 to the DataSN associated with the SCSI Data-In PDU. 1117 This is used by the iSER Layer to notify the iSCSI Layer of the 1118 completion of inbound data transfer that was requested by the iSCSI 1119 Layer only if the invocation of the Get_Data Operational Primitive 1120 (see section 3.1.3) was qualified with Notify_Enable set. SN refers 1121 to the R2TSN associated with the R2T PDU. 1123 3.2.3 Data_ACK_Notify 1125 Input qualifier: Connection_Handle, ITT, DataSN 1127 Return results: Not specified 1129 This is used by the iSER Layer at the target to notify the iSCSI 1130 Layer of the arrival of the data acknowledgement (as defined in 1131 [RFC3720]) requested earlier by the iSCSI Layer for the outbound 1132 data transfer via an invocation of the Put_Data Operational 1133 Primitive where the A-bit in the SCSI Data-in PDU is set to 1. See 1134 section 7.3.5. DataSN refers to the expected DataSN of the next 1135 SCSI Data-in PDU which immediately follows the SCSI Data-in PDU with 1136 the A-bit set to which this notification corresponds, with semantics 1137 as defined in [RFC3720]. 1139 3.2.4 Connection_Terminate_Notify 1141 Input qualifiers: Connection_Handle 1143 Return results: Not specified 1145 This is used by the iSER Layers at the initiator and the target to 1146 notify the iSCSI Layer of the unsolicited termination or failure of 1147 an iSCSI/iSER connection. The iSER Layer MUST deallocate the 1148 connection and task resources associated with the terminated 1149 connection before the invocation of this Operational Primitive. 1150 Note that the Connection_Terminate_Notify Operational Primitive is 1151 not invoked when the termination of the connection was earlier 1152 requested by the local iSCSI Layer. 1154 3.3 iSCSI Protocol Usage Requirements 1156 To operate in an iSER-assisted mode, the iSCSI Layers at both the 1157 initiator and the target MUST negotiate the RDMAExtensions key (see 1158 section 6.3) to "Yes" on the leading connection. If the 1159 RDMAExtensions key is not negotiated to "Yes", then iSER-assisted 1160 mode MUST NOT be used. If the RDMAExtensons key is negotiated to 1161 "Yes" but the invocation of the Allocate_Connection_Resources 1162 Operational Primitive to the iSER layer fails, the iSCSI layer MUST 1163 fail the iSCSI Login process or terminate the connection as 1164 appropriate. See section 10.1.3.1 for details. 1166 If the RDMAExtensions key is negotiated to "Yes", the iSCSI Layer 1167 MUST satisfy the following protocol usage requirements from the iSER 1168 protocol: 1170 1. The iSCSI Layer at the initiator MUST set ExpDataSN to 0 in Task 1171 Management Function Requests for Task Allegiance Reassignment 1172 for read/bidirectional commands, so as to cause the target to 1173 send all unacknowledged read data. 1175 2. The iSCSI Layer at the target MUST always return the SCSI status 1176 in a separate SCSI Response PDU for read commands, i.e., there 1177 MUST NOT be a "phase collapse" in concluding a SCSI Read 1178 Command. 1180 3. The iSCSI Layers at both the initiator and the target MUST 1181 support the keys as defined in section 6 on Login/Text 1182 Operational Keys. If used as specified, these keys MUST NOT be 1183 answered with NotUnderstood and the semantics as defined MUST be 1184 followed for each iSER-assisted connection. 1186 4. The iSCSI Layer at the initiator MUST NOT issue SNACKs for PDUs. 1188 4 Lower Layer Interface Requirements 1190 4.1 Interactions with the RCaP Layer 1192 The iSER protocol layer is layered on top of an RCaP layer (see 1193 Figure 1) and the following are the key features that are assumed to 1194 be supported by any RCaP layer: 1196 * The RCaP layer supports all basic RDMA operations, including RDMA 1197 Write Operation, RDMA Read Operation, Send Operation, Send with 1198 Invalidate Operation, Send with Solicited Event Operation, Send 1199 with Solicited Event & Invalidate Operation, and Terminate 1200 Operation. 1202 * The RCaP layer provides reliable, in-order message delivery and 1203 direct data placement. 1205 * When the iSER Layer initiates an RDMA Read Operation following an 1206 RDMA Write Operation on one RCaP Stream, the RDMA Read Response 1207 Message processing on the remote node will be started only after 1208 the preceding RDMA Write Message payload is placed in the memory 1209 of the remote node. 1211 * The RCaP layer encapsulates a single iSER Message into a single 1212 RCaP Message on the Data Source side. The RCaP layer 1213 decapsulates the iSER Message before delivering it to the iSER 1214 Layer on the Data Sink side. 1216 * When the iSER Layer provides the STag to be remotely invalidated 1217 to the RCaP layer for a SendInvSE Message, the RCaP layer uses 1218 this STag as the STag to be invalidated in the SendInvSE Message. 1220 * The RCaP layer uses the STag and Tagged Offset provided by the 1221 iSER Layer for the RDMA Write and RDMA Read Request Messages. 1223 * When the RCaP layer delivers the content of an RDMA Send Message 1224 Type to the iSER Layer, the RCaP layer provides the length of the 1225 RDMA Send message. This ensures that the iSER Layer does not 1226 have to carry a length field in the iSER header. 1228 * When the RCaP layer delivers the SendSE or SendInvSE Message to 1229 the iSER Layer, it notifies the iSER Layer with the mechanism 1230 provided on that interface. 1232 * When the RCaP layer delivers a SendInvSE Message to the iSER 1233 Layer, it passes the value of the STag that was invalidated. 1235 * The RCaP layer propagates all status and error indications to the 1236 iSER Layer. 1238 * For a transport layer that operates in byte stream mode such as 1239 TCP, the RCaP implementation supports the enabling of the RDMA 1240 mode after Connection establishment and the exchange of Login 1241 parameters in byte stream mode. For a transport layer that 1242 provides message delivery capability such as [IB], the RCaP 1243 implementation supports the use of the messaging capability by 1244 the iSCSI Layer directly for the Login phase after connection 1245 establishment before enabling iSER-assisted mode. 1247 * Whenever the iSER Layer terminates the RCaP Stream, the RCaP 1248 layer terminates the associated Connection. 1250 4.2 Interactions with the Transport Layer 1252 After the iSER connection is established, the RCaP layer and the 1253 underlying transport layer are responsible for maintaining the 1254 Connection and reporting to the iSER Layer any Connection failures. 1256 5 Connection Setup and Termination 1258 5.1 iSCSI/iSER Connection Setup 1260 During connection setup, the iSCSI Layer at the initiator is 1261 responsible for establishing a connection with the target. After 1262 the connection is established, the iSCSI Layers at the initiator and 1263 the target enter the Login Phase using the same rules as outlined in 1264 [RFC3720]. The connection transitions into the iSCSI full feature 1265 phase in iSER-assisted mode following a successful login negotiation 1266 between the initiator and the target in which iSER-assisted mode is 1267 negotiated and the connection resources necessary to support RCaP 1268 have been allocated at both the initiator and the target. The same 1269 connection MUST be used for both the iSCSI Login phase and the 1270 subsequent iSER-assisted full feature phase. 1272 iSER-assisted mode MUST be enabled only if it is negotiated on the 1273 leading connection during the LoginOperationalNegotiation Stage of 1274 the iSCSI Login Phase. iSER-assisted mode is negotiated using the 1275 RDMAExtensions= key. Both the initiator and the 1276 target MUST exchange the RDMAExtensions key with the value set to 1277 "Yes" to enable iSER-assisted mode. If both the initiator and the 1278 target fail to negotiate the RDMAExtensions key set to "Yes", then 1279 the connection MUST continue with the login semantics as defined in 1280 [RFC3720]. If the RDMAExtensions key is not negotiated to Yes, then 1281 for some RCaP implementation (such as [IB]), the connection may need 1282 to be re-established in TCP capable mode. (For InfiniBand this will 1283 require an [IPoIB] type connection.) 1285 iSER-assisted mode is defined for a Normal session only and the 1286 RDMAExtensions key MUST NOT be negotiated for a Discovery session. 1287 Discovery sessions are always conducted using the transport layer as 1288 described in [RFC3720]. 1290 An iSER enabled node is not required to initiate the RDMAExtensions 1291 key exchange if its preference is for the Traditional iSCSI mode. 1292 The RDMAExtensions key, if offered, MUST be sent in the first 1293 available Login Response or Login Request PDU in the 1294 LoginOperationalNegotiation stage. This is due to the fact that the 1295 value of some login parameters might depend on whether iSER-assisted 1296 mode is enabled or not. 1298 iSER-assisted mode is a session-wide attribute. If both the 1299 initiator and the target negotiated RDMAExtensions="Yes" on the 1300 leading connection of a session, then all subsequent connections of 1301 the same session MUST enable iSER-assisted mode without having to 1302 exchange RDMAExtensions key during the iSCSI Login Phase. 1304 Conversely, if both the initiator and the target failed to negotiate 1305 RDMAExtensions to "Yes" on the leading connection of a session, then 1306 the RDMAExtensions key MUST NOT be negotiated further on any 1307 additional subsequent connection of the session. 1309 When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest 1310 and the DataDigest keys MUST be negotiated to "None" on all 1311 iSCSI/iSER connections participating in that iSCSI session. This is 1312 because, for an iSCSI/iSER connection, RCaP is responsible for 1313 providing error detection that is at least as good as a 32-bit CRC 1314 for all iSER Messages. Furthermore, all SCSI Read data are sent 1315 using RDMA Write Messages instead of the SCSI Data-in PDUs, and all 1316 solicited SCSI write data are sent using RDMA Read Response Messages 1317 instead of the SCSI Data-out PDUs. HeaderDigest and DataDigest 1318 which apply to iSCSI PDUs would not be appropriate for RDMA Read and 1319 RDMA Write operations used with iSER. 1321 5.1.1 Initiator Behavior 1323 If the outcome of the iSCSI negotiation is to enable iSER-assisted 1324 mode, then on the initiator side, prior to sending the Login Request 1325 with the T (Transit) bit set to 1 and the NSG (Next Stage) field set 1326 to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to 1327 allocate the connection resources necessary to support RCaP by 1328 invoking the Allocate_Connection_Resources Operational Primitive. 1329 The connection resources required are defined by implementation and 1330 are outside the scope of this specification. The iSCSI Layer may 1331 invoke the Notice_Key_Values Operational Primitive before invoking 1332 the Allocate_Connection_Resources Operational Primitive to request 1333 the iSER Layer to take note of the negotiated values of the iSCSI 1334 keys for the Connection. The specific keys to be passed in as input 1335 qualifiers are implementation dependent. These may include, but not 1336 limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. 1338 To minimize the potential for a denial of service attack, the iSCSI 1339 Layer MUST NOT request the iSER Layer to allocate the connection 1340 resources necessary to support RCaP until the iSCSI layer is 1341 sufficiently far along in the iSCSI Login Phase that it is 1342 reasonably certain that the peer side is not an attacker. In 1343 particular, if the Login Phase includes a SecurityNegotiation stage, 1344 the iSCSI Layer MUST defer the connection resource allocation (i.e. 1345 invoking the Allocate_Connection_Resources Operational Primitive) to 1346 the LoginOperationalNegotiation stage ([RFC3720]) so that the 1347 resource allocation occurs after the authentication phase is 1348 completed. 1350 Among the connection resources allocated at the initiator is the 1351 Inbound RDMA Read Queue Depth (IRD). As described in section 9.5.1, 1352 R2Ts are transformed by the target into RDMA Read operations. IRD 1353 limits the maximum number of simultaneously incoming outstanding 1354 RDMA Read Requests per an RCaP Stream from the target to the 1355 initiator. The required value of IRD is outside the scope of the 1356 iSER specification. The iSER Layer at the initiator MUST set IRD to 1357 1 or higher if R2Ts are to be used in the connection. However, the 1358 iSER Layer at the initiator MAY set IRD to 0 based on implementation 1359 configuration which indicates that no R2Ts will be used on that 1360 connection. Initially, the iSER-IRD value at the initiator SHOULD 1361 be set to the IRD value at the initiator and MUST NOT be more than 1362 the IRD value. 1364 On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be 1365 set to 0 since the iSER Layer at the initiator does not issue RDMA 1366 Read Requests to the target. 1368 Failure to allocate the requested connection resources locally 1369 results in a login failure and its handling is described in section 1370 10.1.3.1. 1372 If the iSER Layer at the initiator is successful in allocating the 1373 connection resources necessary to support RCaP, the following events 1374 MUST occur in the specified sequence: 1376 1. The iSER Layer MUST return a success status to the iSCSI Layer 1377 in response to the Allocate_Connection_Resources Operational 1378 Primitive. 1380 2. After the target returns the Login Response with the T bit set 1381 to 1 and the NSG field set to FullFeaturePhase, and a status 1382 class of 0 (Success), the iSCSI Layer MUST invoke the 1383 Enable_Datamover Operational Primitive with the following 1384 qualifiers. (See section 10.1.4.6 for the case when the status 1385 class is not Success.): 1387 a. Connection_Handle that identifies the iSCSI connection. 1389 b. Transport_Connection_Descriptor which identifies the 1390 specific transport connection associated with the 1391 Connection_Handle. 1393 3. The iSER Layer MUST send the iSER Hello Message as the first 1394 iSER Message. See Section 5.1.3 on iSER Hello Exchange. 1396 5.1.2 Target Behavior 1398 If the outcome of the iSCSI negotiation is to enable iSER-assisted 1399 mode, then on the target side, prior to sending the Login Response 1400 with the T (Transit) bit set to 1 and the NSG (Next Stage) field set 1401 to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to 1402 allocate the resources necessary to support RCaP by invoking the 1403 Allocate_Connection_Resources Operational Primitive. The connection 1404 resources required are defined by implementation and are outside the 1405 scope of this specification. Optionally, the iSCSI Layer may invoke 1406 the Notice_Key_Values Operational Primitive before invoking the 1407 Allocate_Connection_Resources Operational Primitive to request the 1408 iSER Layer to take note of the negotiated values of the iSCSI keys 1409 for the Connection. The specific keys to be passed in as input 1410 qualifiers are implementation dependent. These may include, but not 1411 limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. 1413 To minimize the potential for a denial of service attack, the iSCSI 1414 Layer MUST NOT request the iSER Layer to allocate the connection 1415 resources necessary to support RCaP until the iSCSI layer is 1416 sufficiently far along in the iSCSI Login Phase that it is 1417 reasonably certain that the peer side is not an attacker. In 1418 particular, if the Login Phase includes a SecurityNegotiation stage, 1419 the iSCSI Layer MUST defer the connection resource allocation (i.e. 1420 invoking the Allocate_Connection_Resources Operational Primitive) to 1421 the LoginOperationalNegotiation stage ([RFC3720]) so that the 1422 resource allocation occurs after the authentication phase is 1423 completed. 1425 Among the connection resources allocated at the target is the 1426 Outbound RDMA Read Queue Depth (ORD). As described in section 1427 9.5.1, R2Ts are transformed by the target into RDMA Read operations. 1428 The ORD limits the maximum number of simultaneously outstanding RDMA 1429 Read Requests per RCaP Stream from the target to the initiator. 1430 Initially, the iSER-ORD value at the target SHOULD be set to the ORD 1431 value at the target. 1433 On the other hand, the IRD at the target MAY be set to 0 since the 1434 iSER Layer at the target does not expect RDMA Read Requests to be 1435 issued by the initiator. 1437 Failure to allocate the requested connection resources locally 1438 results in a login failure and its handling is described in section 1439 10.1.3.1. 1441 If the iSER Layer at the target is successful in allocating the 1442 connection resources necessary to support RCaP, the following events 1443 MUST occur in the specified sequence: 1445 1. The iSER Layer MUST return a success status to the iSCSI Layer 1446 in response to the Allocate_Connection_Resources Operational 1447 Primitive. 1449 2. The iSCSI Layer MUST invoke the Enable_Datamover Operational 1450 Primitive with the following qualifiers: 1452 a. Connection_Handle that identifies the iSCSI connection. 1454 b. Transport_Connection_Descriptor which identifies the 1455 specific transport connection associated with the 1456 Connection_Handle. 1458 c. The final transport layer (e.g. TCP) message containing the 1459 Login Response with the T bit set to 1 and the NSG field set 1460 to FullFeaturePhase 1462 3. The iSER Layer MUST send the final Login Response PDU in the 1463 native transport mode to conclude the iSCSI Login Phase. If the 1464 underlying transport is TCP, then the iSER Layer MUST send the 1465 final Login Response PDU in byte stream mode. 1467 4. After receiving the iSER Hello Message from the initiator, the 1468 iSER Layer MUST respond with the iSER HelloReply Message to be 1469 sent as the first iSER Message. See section 5.1.3 on iSER Hello 1470 Exchange for more details. 1472 Note: In the above sequence, the operations as described in bullets 1473 3 and 4 MUST be performed atomically for iWARP connections. Failure 1474 to do this may result in race conditions. 1476 5.1.3 iSER Hello Exchange 1478 The first iSER Message sent by the iSER Layer at the initiator to 1479 the target MUST be the iSER Hello Message. The iSER Hello Message 1480 is used by the iSER Layer at the initiator to declare iSER 1481 parameters to the target. See section 9.3 on iSER Header Format for 1482 iSER Hello Message. 1484 In response to the iSER Hello Message, the iSER Layer at the target 1485 MUST return the iSER HelloReply Message as the first iSER Message 1486 sent by the target. The iSER HelloReply Message is used by the iSER 1487 Layer at the target to declare iSER parameters to the initiator. 1488 See section 9.4 on iSER Header Format for iSER HelloReply Message. 1490 In the iSER Hello Message, the iSER Layer at the initiator declares 1491 the iSER-IRD value to the target. 1493 Upon receiving the iSER Hello Message, the iSER Layer at the target 1494 MUST set the iSER-ORD value to the minimum of the iSER-ORD value at 1495 the target and the iSER-IRD value declared by the initiator. The 1496 iSER Layer at the target MAY adjust (lower) its ORD value to match 1497 the iSER-ORD value if the iSER-ORD value is smaller than the ORD 1498 value at the target in order to free up the unused resources. 1500 In the iSER HelloReply Message, the iSER Layer at the target 1501 declares the iSER-ORD value to the initiator. 1503 Upon receiving the iSER HelloReply Message, the iSER Layer at the 1504 initiator MAY adjust (lower) its IRD value to match the iSER-ORD 1505 value in order to free up the unused resources, if the iSER-ORD 1506 value declared by the target is smaller than the iSER-IRD value 1507 declared by the initiator. 1509 It is an iSER level negotiation failure if the iSER parameters 1510 declared in the iSER Hello Message by the initiator are unacceptable 1511 to the target. This includes the following: 1513 * The initiator-declared iSER-IRD value is greater than 0 and the 1514 target-declared iSER-ORD value is 0. 1516 * The initiator-supported and the target-supported iSER protocol 1517 versions do not overlap. 1519 See section 10.1.3.2 on the handling of the error situation. 1521 5.2 iSCSI/iSER Connection Termination 1523 5.2.1 Normal Connection Termination at the Initiator 1525 The iSCSI Layer at the initiator terminates an iSCSI/iSER connection 1526 normally by invoking the Send_Control Operational Primitive 1527 qualified with the Logout Request PDU. The iSER Layer at the 1528 initiator MUST use a SendSE Message to send the Logout Request PDU 1529 to the target. After the iSER Layer at the initiator receives the 1530 SendSE Message containing the Logout Response PDU from the target, 1531 it MUST notify the iSCSI Layer by invoking the Control_Notify 1532 Operational Primitive qualified with the Logout Response PDU. 1534 After the iSCSI logout process is complete, the iSCSI layer at the 1535 target is responsible for closing the iSCSI/iSER connection as 1536 described in Section 5.2.2. After the RCaP layer at the initiator 1537 reports that the Connection has been closed, the iSER Layer at the 1538 initiator MUST deallocate all connection and task resources (if any) 1539 associated with the connection, invalidate the Local Mapping(s) (if 1540 any) that associate the ITT(s) used on that connection to the local 1541 STag(s) before notifying the iSCSI Layer by invoking the 1542 Connection_Terminate_Notify Operational Primitive. 1544 5.2.2 Normal Connection Termination at the Target 1546 Upon receiving the SendSE Message containing the Logout Request PDU, 1547 the iSER Layer at the target MUST notify the iSCSI Layer at the 1548 target by invoking the Control_Notify Operational Primitive 1549 qualified with the Logout Request PDU. The iSCSI Layer completes 1550 the logout process by invoking the Send_Control Operational 1551 Primitive qualified with the Logout Response PDU. The iSER Layer at 1552 the target MUST use a SendSE Message to send the Logout Response PDU 1553 to the initiator. After the iSCSI logout process is complete, the 1554 iSCSI Layer at the target MUST request the iSER Layer at the target 1555 to terminate the RCaP Stream by invoking the Connection_Terminate 1556 Operational Primitive. 1558 As part of the termination process, the RCaP layer MUST close the 1559 Connection. When the RCaP layer notifies the iSER Layer after the 1560 RCaP Stream and the associated Connection are terminated, the iSER 1561 Layer MUST deallocate all connection and task resources (if any) 1562 associated with the connection, and invalidate the Local and Remote 1563 Mapping(s) (if any) that associate the ITT(s) used on that 1564 connection to the local STag(s) and the Advertised STag(s) 1565 respectively. 1567 5.2.3 Termination without Logout Request/Response PDUs 1569 5.2.3.1 Connection Termination Initiated by the iSCSI Layer 1571 The Connection_Terminate Operational Primitive MAY be invoked by the 1572 iSCSI Layer to request the iSER Layer to terminate the RCaP Stream 1573 without having previously exchanged the Logout Request and Logout 1574 Response PDUs between the two iSCSI/iSER nodes. As part of the 1575 termination process, the RCaP layer will close the Connection. When 1576 the RCaP layer notifies the iSER Layer after the RCaP Stream and the 1577 associated Connection are terminated, the iSER Layer MUST perform 1578 the following actions. 1580 If the Connection_Terminate Operational Primitive is invoked by the 1581 iSCSI Layer at the target, then the iSER Layer at the target MUST 1582 deallocate all connection and task resources (if any) associated 1583 with the connection, and invalidate the Local and Remote Mappings 1584 (if any) that associate the ITT(s) used on the connection to the 1585 local STag(s) and the Advertised STag(s) respectively. 1587 If the Connection_Terminate Operational Primitive is invoked by the 1588 iSCSI Layer at the initiator, then the iSER Layer at the initiator 1589 MUST deallocate all connection and task resources (if any) 1590 associated with the connection, and invalidate the Local Mapping(s) 1591 (if any) that associate the ITT(s) used on the connection to the 1592 local STag(s). 1594 5.2.3.2 Connection Termination Notification to the iSCSI Layer 1596 If the iSCSI/iSER connection is terminated without the invocation of 1597 Connection_Terminate from the iSCSI Layer, the iSER Layer MUST 1598 notify the iSCSI Layer that the iSCSI/iSER connection has been 1599 terminated by invoking the Connection_Terminate_Notify Operational 1600 Primitive. 1602 Prior to invoking Connection_Terminate_Notify, the iSER Layer at the 1603 target MUST deallocate all connection and task resources (if any) 1604 associated with the connection, and invalidate the Local and Remote 1605 Mappings (if any) that associate the ITT(s) used on the connection 1606 to the local STag(s) and the Advertised STag(s) respectively. 1608 Prior to invoking Connection_Terminate_Notify, the iSER Layer at the 1609 initiator MUST deallocate all connection and task resources (if any) 1610 associated with the connection, and invalidate the Local Mappings 1611 (if any) that associate the ITT(s) used on the connection to the 1612 local STag(s). 1614 If the remote iSCSI/iSER node initiated the closing of the 1615 Connection (e.g., by sending a TCP FIN or TCP RST), the iSER Layer 1616 MUST notify the iSCSI Layer after the RCaP layer reports that the 1617 Connection is closed by invoking the Connection_Terminate_Notify 1618 Operational Primitive. 1620 Another example of a Connection termination without a preceding 1621 logout is when the iSCSI Layer at the initiator does an implicit 1622 logout (connection reinstatement). 1624 6 Login/Text Operational Keys 1626 Certain iSCSI login/text operational keys have restricted usage in 1627 iSER, and additional keys are used to support the iSER protocol 1628 functionality. All other keys defined in [RFC3720] and not 1629 discussed in this section may be used on iSCSI/iSER connections with 1630 the same semantics. 1632 6.1 HeaderDigest and DataDigest 1634 Irrelevant when: RDMAExtensions=Yes 1636 Negotiations resulting in RDMAExtensions=Yes for a session implies 1637 HeaderDigest=None and DataDigest=None for all connections in that 1638 session and overrides both the default and an explicit setting. 1640 6.2 MaxRecvDataSegmentLength 1642 For an iSCSI connection belonging to a session in which 1643 RDMAExtensions=Yes was negotiated on the leading connection of the 1644 session, MaxRecvDataSegmentLength need not be declared in the Login 1645 Phase. Instead InitiatorRecvDataSegmentLength (as described in 1646 section 6.5) and TargetRecvDataSegmentLength (as described in 1647 section 6.4) keys are negotiated. The values of the local and 1648 remote MaxRecvDataSegmentLength are derived from the 1649 InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys 1650 even if the MaxRecvDataSegmentLength was declared during the login 1651 phase. 1653 In the full feature phase, the initiator MUST consider the value of 1654 its local MaxRecvDataSegmentLength (that it would have declared to 1655 the target) as having the value of InitiatorRecvDataSegmentLength, 1656 and the value of the remote MaxRecvDataSegmentLength (that would 1657 have been declared by the target) as having the value of 1658 TargetRecvDataSegmentLength. Similarly, the target MUST consider 1659 the value of its local MaxRecvDataSegmentLength (that it would have 1660 declared to the initiator) as having the value of 1661 TargetRecvDataSegmentLength, and the value of the remote 1662 MaxRecvDataSegmentLength (that would have been declared by the 1663 initiator) as having the value of InitiatorRecvDataSegmentLength. 1665 Note that RFC 3720 requires that when a target receives a NOP-Out 1666 request with a valid Initiator Task Tag, it responds with a NOP-In 1667 with the same Initiator Task Tag that was provided in the NOP-Out 1668 request. Furthermore, it returns the first MaxRecvDataSegmentLength 1669 bytes of the initiator provided Ping Data. Since there is no 1670 MaxRecvDataSegmentLength common to the initiator and the target in 1671 iSER, the length of the data sent with the NOP-Out request MUST not 1672 exceed InitiatorMaxRecvDataSegmentLength. 1674 The MaxRecvDataSegmentLength key is applicable only for iSCSI 1675 control-type PDUs. 1677 6.3 RDMAExtensions 1679 Use: LO (leading only) 1681 Senders: Initiator and Target 1683 Scope: SW (session-wide) 1685 RDMAExtensions= 1687 Irrelevant when: SessionType=Discovery 1689 Default is No 1691 Result function is AND 1693 This key is used by the initiator and the target to negotiate the 1694 support for iSER-assisted mode. To enable the use of iSER-assisted 1695 mode, both the initiator and the target MUST exchange 1696 RDMAExtensions=Yes. iSER-assisted mode MUST NOT be used if either 1697 the initiator or the target offers RDMAExtensions=No. 1699 An iSER-enabled node is not required to initiate the RDMAExtensions 1700 key exchange if it prefers to operate in the Traditional iSCSI mode. 1701 However, if the RDMAExtensions key is to be negotiated, an initiator 1702 MUST offer the key in the first Login Request PDU in the 1703 LoginOperationalNegotiation stage of the leading connection, and a 1704 target MUST offer the key in the first Login Response PDU with which 1705 it is allowed to do so (i.e., the first Login Response PDU issued 1706 after the first Login Request PDU with the C bit set to 0) in the 1707 LoginOperationalNegotiation stage of the leading connection. In 1708 response to the offered key=value pair of RDMAExtensions=yes, an 1709 initiator MUST respond in the next Login Request PDU with which it 1710 is allowed to do so, and a target MUST respond in the next Login 1711 Response PDU with which it is allowed to do so. 1713 Negotiating the RDMAExtensions key first enables a node to negotiate 1714 the optimal value for other keys. Certain iSCSI keys such as 1715 MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T, 1716 ImmediateData, etc., may be negotiated differently depending on 1717 whether connection is in Traditional iSCSI mode or iSER-assisted 1718 mode. 1720 6.4 TargetRecvDataSegmentLength 1722 Use: IO (Initialize only) 1724 Senders: Initiator and Target 1726 Scope: CO (connection-only) 1728 Irrelevant when: RDMAExtensions=No 1730 TargetRecvDataSegmentLength= 1732 Default is 8192 bytes 1734 Result function is minimum 1736 This key is relevant only for the iSCSI connection of an iSCSI 1737 session if RDMAExtensions=Yes was negotiated on the leading 1738 connection of the session. It is used by the initiator and the 1739 target to negotiate the maximum size of the data segment that an 1740 initiator may send to the target in an iSCSI control-type PDU in the 1741 full feature phase. For SCSI Command PDUs and SCSI Data-out PDUs 1742 containing non-immediate unsolicited data to be sent by the 1743 initiator, the initiator MUST send all non-Final PDUs with a data 1744 segment size of exactly TargetRecvDataSegmentLength whenever the 1745 PDUs constitute a data sequence whose size is larger than 1746 TargetRecvDataSegmentLength. 1748 6.5 InitiatorRecvDataSegmentLength 1750 Use: IO (Initialize only) 1752 Senders: Initiator and Target 1754 Scope: CO (connection-only) 1756 Irrelevant when: RDMAExtensions=No 1758 InitiatorRecvDataSegmentLength= 1760 Default is 8192 bytes 1762 Result function is minimum 1763 This key is relevant only for the iSCSI connection of an iSCSI 1764 session if RDMAExtensions=Yes was negotiated on the leading 1765 connection of the session. It is used by the initiator and the 1766 target to negotiate the maximum size of the data segment that a 1767 target may send to the initiator in an iSCSI control-type PDU in the 1768 full feature phase. 1770 6.6 OFMarker and IFMarker 1772 Irrelevant when: RDMAExtensions=Yes 1774 Negotiations resulting in RDMAExtensions=Yes for a session implies 1775 OFMarker=No and IFMarker=No for all connections in that session and 1776 overrides both the default and an explicit setting. 1778 6.7 MaxOutstandingUnexpectedPDUs 1780 Use: LO (leading only), Declarative 1782 Senders: Initiator and Target 1784 Scope: SW (session-wide) 1786 Irrelevant when: RDMAExtensions=No 1788 MaxOutstandingUnexpectedPDUs= 1791 Default is 0 1793 This key is used by the initiator and the target to declare the 1794 maximum number of outstanding "unexpected" iSCSI control-type PDUs 1795 that it can receive in the full feature phase. It is intended to 1796 allow the receiving side to determine the amount of buffer resources 1797 needed beyond the normal flow control mechanism available in iSCSI. 1798 An initiator or target should select a value such that it would not 1799 impose an unnecessary constraint on the iSCSI Layer under normal 1800 circumstances. The value of 0 is defined to indicate that the 1801 declarer has no limit on the maximum number of outstanding 1802 "unexpected" iSCSI control-type PDUs that it can receive. See 1803 sections 8.1.1 and 8.1.2 for the usage of this key. Note that iSER 1804 Hello and HelloReply Messages are not iSCSI control-type PDUs and 1805 are not affected by this key. 1807 6.8 MaxAHSLength 1809 Use: LO (leading only), Declarative 1810 Senders: Initiator and Target 1812 Scope: SW (session-wide) 1814 Irrelevant when: RDMAExtensions=No 1816 MaxAHSLength= 1818 Default is 0 1820 This key is used by the intiator and target to declare the maximum 1821 size of AHS in an iSCSI control-type PDU that it can receive in the 1822 full feature phase. It is intended to allow the receiving side to 1823 determine the amount of resources needed for receive buffering. An 1824 initiator or target should select a value such that it would not 1825 impose an unnecessary constraint on the iSCSI Layer under normal 1826 circumstances. The value of 0 is defined to indicate that the 1827 declarer has no limit on the maximum size of AHS in iSCSI control- 1828 type PDUs that it can receive. 1830 6.9 WriteAddressForSolicitedDataOnly 1832 Use: LO (leading only), Declarative 1834 Senders: Initiator 1836 Scope: SW (session-wide) 1838 RDMAExtensions= 1840 Irrelevant when: RDMAExtensions=No 1842 Default is No 1844 This key is used by the intiator to declare to the target the origin 1845 of the Write Virtual Address used in the iSER header of an iSCSI 1846 control-type PDU. When set to No, the Write Virtual Address points 1847 to an I/O buffer that contains all the write data, including both 1848 the unsolicited and solicited data. When set to Yes, the Write 1849 Virtual Address points to an I/O buffer that only contains the 1850 solicited data. 1852 7 iSCSI PDU Considerations 1854 When a connection is in the iSER-assisted mode, two types of message 1855 transfers are allowed between the iSCSI Layer at the initiator and 1856 the iSCSI Layer at the target. These are known as the iSCSI data- 1857 type PDUs and the iSCSI control-type PDUs and these terms are 1858 described in the following sections. 1860 7.1 iSCSI Data-Type PDU 1862 An iSCSI data-type PDU is defined as an iSCSI PDU that causes data 1863 transfer, transparent to the remote iSCSI layer, to take place 1864 between the peer iSCSI nodes in the full feature phase of an 1865 iSCSI/iSER connection. An iSCSI data-type PDU, when requested for 1866 transmission by the iSCSI Layer in the sending node, results in the 1867 data being transferred without the participation of the iSCSI Layers 1868 at the sending and the receiving nodes. This is due to the fact 1869 that the PDU itself is not delivered as-is to the iSCSI Layer in the 1870 receiving node. Instead, the data transfer operations are 1871 transformed into the appropriate RDMA operations which are handled 1872 by the RDMA-Capable Controller. The set of iSCSI data-type PDUs 1873 consists of SCSI Data-in PDUs and R2T PDUs. 1875 If the invocation of the Operational Primitive by the iSCSI Layer to 1876 request the iSER Layer to process an iSCSI data-type PDU is 1877 qualified with Notify_Enable set, then upon completing the RDMA 1878 operation, the iSER Layer at the target MUST notify the iSCSI Layer 1879 at the target by invoking the Data_Completion_Notify Operational 1880 Primitive qualified with ITT and SN. There is no data completion 1881 notification at the initiator since the RDMA operations are 1882 completely handled by the RDMA-Capable Controller at the initiator 1883 and the iSER Layer at the initiator is not involved with the data 1884 transfer associated with iSCSI data-type PDUs. 1886 If the invocation of the Operational Primitive by the iSCSI Layer to 1887 request the iSER Layer to process an iSCSI data-type PDU is 1888 qualified with Notify_Enable cleared, then upon completing the RDMA 1889 operation, the iSER Layer at the target MUST NOT notify the iSCSI 1890 Layer at the target and MUST NOT invoke the Data_Completion_Notify 1891 Operational Primitive. 1893 If an operation associated with an iSCSI data-type PDU fails for any 1894 reason, the contents of the Data Sink buffers associated with the 1895 operation are considered indeterminate. 1897 7.2 iSCSI Control-Type PDU 1899 Any iSCSI PDU that is not an iSCSI data-type PDU and also not a SCSI 1900 Data-out PDU carrying solicited data is defined as an iSCSI control- 1901 type PDU. The iSCSI Layer invokes the Send_Control Operational 1902 Primitive to request the iSER Layer to process an iSCSI control-type 1903 PDU. iSCSI control-type PDUs are transferred using Send Message 1904 Types of RCaP. Specifically, it is to be noted that SCSI Data-Out 1905 PDUs carrying unsolicited data are defined as iSCSI control-type 1906 PDUs. See section 7.3.4 on the treatment of SCSI Data-out PDUs. 1908 When the iSER Layer receives an iSCSI control-type PDU, it MUST 1909 notify the iSCSI Layer by invoking the Control_Notify Operational 1910 Primitive qualified with the iSCSI control-type PDU. 1912 7.3 iSCSI PDUs 1914 This section describes the handling of each of the iSCSI PDU types 1915 by the iSER Layer. The iSCSI Layer requests the iSER Layer to 1916 process the iSCSI PDU by invoking the appropriate Operational 1917 Primitive. A Connection_Handle MUST qualify each of these 1918 invocations. In addition, BHS and the optional AHS of the iSCSI PDU 1919 as defined in [RFC3720] MUST qualify each of the invocations. The 1920 qualifying Connection_Handle, the BHS and the AHS are not explicitly 1921 listed in the subsequent sections. 1923 7.3.1 SCSI Command 1925 Type: control-type PDU 1927 PDU-specific qualifiers (for SCSI Write or bidirectional 1928 command): ImmediateDataSize, UnsolicitedDataSize, 1929 DataDescriptorOut 1931 PDU-specific qualifiers (for SCSI Read or bidirectional 1932 command): DataDescriptorIn 1934 The iSER Layer at the initiator MUST send the SCSI command in a 1935 SendSE Message to the target. 1937 For a SCSI Write or bidirectional command, the iSCSI Layer at the 1938 initiator MUST invoke the Send_Control Operational Primitive as 1939 follows: 1941 * If there is immediate data to be transferred for the SCSI write 1942 or bidirectional command, the qualifier ImmediateDataSize MUST be 1943 used to define the number of bytes of immediate unsolicited data 1944 to be sent with the write or bidirectional command, and the 1945 qualifier DataDescriptorOut MUST be used to define the 1946 initiator's I/O Buffer containing the SCSI Write data. 1948 * If there is unsolicited data to be transferred for the SCSI Write 1949 or bidirectional command, the qualifier UnsolicitedDataSize MUST 1950 be used to define the number of bytes of immediate and non- 1951 immediate unsolicited data for the command. The iSCSI Layer will 1952 issue one or more SCSI Data-out PDUs for the non-immediate 1953 unsolicited data. See Section 7.3.4 on SCSI Data-out. 1955 * If there is solicited data to be transferred for the SCSI Write 1956 or bidirectional command, as indicated by the Expected Data 1957 Transfer Length in the SCSI Command PDU exceeding the value of 1958 UnsolicitedDataSize, the iSER Layer at the initiator MUST do the 1959 following: 1961 a. It MUST allocate a Write STag for the I/O Buffer defined by 1962 the qualifier DataDescriptorOut. DataDescriptorOut 1963 describes the I/O buffer starting with the immediate 1964 unsolicited data (if any), followed by the non-immediate 1965 unsolicited data (if any) and solicited data. This means 1966 that the BufferOffset for the SCSI Data-out for this command 1967 is equal to the TO. This implies zero TO for this STag 1968 points to the beginning of this I/O Buffer. 1970 b. It MUST establish a Local Mapping that associates the 1971 Initiator Task Tag (ITT) to the Write STag. 1973 c. It MUST Advertise the Write STag to the target by sending it 1974 as the Write STag in the iSER header of the iSER Message 1975 (the payload of the SendSE Message of RCaP) containing the 1976 SCSI Write or bidirectional command PDU. See section 9.2 on 1977 iSER Header Format for iSCSI Control-Type PDU. 1979 For a SCSI Read or bidirectional command, the iSCSI Layer at the 1980 initiator MUST invoke the Send_Control Operational Primitive 1981 qualified with DataDescriptorIn which defines the initiator's I/O 1982 Buffer for receiving the SCSI Read data. The iSER Layer at the 1983 initiator MUST do the following: 1985 a. It MUST allocate a Read STag for the I/O Buffer. 1987 b. It MUST establish a Local Mapping that associates the 1988 Initiator Task Tag (ITT) to the Read STag. 1990 c. It MUST Advertise the Read STag to the target by sending it 1991 as the Read STag in the iSER header of the iSER Message (the 1992 payload of the SendSE Message of RCaP) containing the SCSI 1993 Read or bidirectional command PDU. See section 9.2 on iSER 1994 Header Format for iSCSI Control-Type PDU. 1996 If the amount of unsolicited data to be transferred in a SCSI 1997 Command exceeds TargetRecvDataSegmentLength, then the iSCSI Layer at 1998 the initiator MUST segment the data into multiple iSCSI control-type 1999 PDUs, with the data segment length in all PDUs generated except the 2000 last one having exactly the size TargetRecvDataSegmentLength. The 2001 data segment length of the last iSCSI control-type PDU carrying the 2002 unsolicited data can be up to TargetRecvDataSegmentLength. 2004 When the iSER Layer at the target receives the SCSI Command, it MUST 2005 establish a Remote Mapping that associates the ITT to the Advertised 2006 Write STag and the Read STag if present in the iSER header. The 2007 Write STag is used by the iSER Layer at the target in handling the 2008 data transfer associated with the R2T PDU(s) as described in section 2009 7.3.6. The Read STag is used in handling the SCSI Data-in PDU(s) 2010 from the iSCSI Layer at the target as described in section 7.3.5. 2012 7.3.2 SCSI Response 2014 Type: control-type PDU 2016 PDU-specific qualifiers: DataDescriptorStatus 2018 The iSCSI Layer at the target MUST invoke the Send_Control 2019 Operational Primitive qualified with DataDescriptorStatus which 2020 defines the buffer containing the sense and response information. 2021 The iSCSI Layer at the target MUST always return the SCSI status for 2022 a SCSI command in a separate SCSI Response PDU. "Phase collapse" 2023 for transferring SCSI status in a SCSI Data-in PDU MUST NOT be used. 2024 The iSER Layer at the target sends the SCSI Response PDU according 2025 to the following rules: 2027 * If no STags were Advertised by the initiator in the iSER Message 2028 containing the SCSI command PDU, then the iSER Layer at the 2029 target MUST send a SendSE Message containing the SCSI Response 2030 PDU. 2032 * If the initiator Advertised a Read STag in the iSER Message 2033 containing the SCSI Command PDU, then the iSER Layer at the 2034 target MUST send a SendInvSE Message containing the SCSI Response 2035 PDU. The header of the SendInvSE Message MUST carry the Read 2036 STag to be invalidated at the initiator. 2038 * If the initiator Advertised only the Write STag in the iSER 2039 Message containing the SCSI command PDU, then the iSER Layer at 2040 the target MUST send a SendInvSE Message containing the SCSI 2041 Response PDU. The header of the SendInvSE Message MUST carry the 2042 Write STag to be invalidated at the initiator. 2044 When the iSCSI Layer at the target invokes the Send_Control 2045 Operational Primitive to send the SCSI Response PDU, the iSER Layer 2046 at the target MUST invalidate the Remote Mapping that associates the 2047 ITT to the Advertised STag(s) before transferring the SCSI Response 2048 PDU to the initiator. 2050 Upon receiving the SendInvSE Message containing the SCSI Response 2051 PDU from the target, the RCaP layer at the initiator will invalidate 2052 the STag specified in the header. The iSER Layer at the initiator 2053 MUST ensure that the correct STag is invalidated. If both the Read 2054 and the Write STags were Advertised earlier by the initiator, then 2055 the iSER Layer at the initiator MUST explicitly invalidate the Write 2056 STag upon receiving the SendInvSE Message because the header of the 2057 SendInvSE Message can only carry one STag (in this case the Read 2058 STag) to be invalidated. 2060 The iSER Layer at the initiator MUST ensure the invalidation of the 2061 STag(s) used in a command before notifying the iSCSI Layer at the 2062 initiator by invoking the Control_Notify Operational Primitive 2063 qualified with the SCSI Response. This precludes the possibility of 2064 using the STag(s) after the completion of the command thereby 2065 causing data corruption. 2067 When the iSER Layer at the initiator receives the SendSE or the 2068 SendInvSE Message containing the SCSI Response PDU, it SHOULD 2069 invalidate the Local Mapping that associates the ITT to the local 2070 STag(s). The iSER Layer MUST ensure that all local STag(s) 2071 associated with the ITT are invalidated before notifying the iSCSI 2072 Layer of the SCSI Response PDU by invoking the Control_Notify 2073 Operational Primitive qualified with the SCSI Response PDU. 2075 7.3.3 Task Management Function Request/Response 2077 Type: control-type PDU 2079 PDU-specific qualifiers (for TMF Request): DataDescriptorOut, 2080 DataDescriptorIn 2082 The iSER Layer MUST use a SendSE Message to send the Task Management 2083 Function Request/Response PDU. 2085 For the Task Management Function Request with the TASK REASSIGN 2086 function, the iSER Layer at the initiator MUST do the following: 2088 * It MUST use the ITT as specified in the Referenced Task Tag from 2089 the Task Management Function Request PDU to locate the existing 2090 STag(s), if any, in the Local Mapping(s) that associates the ITT 2091 to the local STag(s). 2093 * It MUST invalidate the existing STag(s), if any, and the Local 2094 Mapping(s) that associates the ITT to the local STag(s). 2096 * It MUST allocate a Read STag for the I/O Buffer as defined by the 2097 qualifier DataDescriptorIn if the Send_Control Operational 2098 Primitive invocation is qualified with DataDescriptorIn. 2100 * It MUST allocate a Write STag for the I/O Buffer as defined by 2101 the qualifier DataDescriptorOut if the Send_Control Operational 2102 Primitive invocation is qualified with DataDescriptorOut. 2104 * If STags are allocated, it MUST establish new Local Mapping(s) 2105 that associate the ITT to the allocated STag(s). 2107 * It MUST Advertise the STags, if allocated, to the target in the 2108 iSER header of the SendSE Message carrying the iSCSI PDU, as 2109 described in section 9.2. 2111 For the Task Management Function Request with the TASK REASSIGN 2112 function for a SCSI Read or bidirectional command, the iSCSI Layer 2113 at the initiator MUST set ExpDataSN to 0 since the data transfer and 2114 acknowledgements happen transparently to the iSCSI Layer at the 2115 initiator. This provides the flexibility to the iSCSI Layer at the 2116 target to request transmission of only the unacknowledged data as 2117 specified in [RFC3720]. 2119 When the iSER Layer at the target receives the Task Management 2120 Function Request with the TASK REASSIGN function, it MUST do the 2121 following: 2123 * It MUST use the ITT as specified in the Referenced Task Tag from 2124 the Task Management Function Request PDU to locate the mappings 2125 that associate the ITT to the Advertised STag(s) and the local 2126 STag(s), if any. 2128 * It MUST invalidate the local STaq(s), if any, associated with the 2129 ITT. 2131 * It MUST replace the Advertised STag(s) in the Remote Mapping that 2132 associates the ITT to the Advertised STag(s) with the Write STag 2133 and the Read STag if present in the iSER header. The Write STag 2134 is used in the handling of the R2T PDU(s) from the iSCSI Layer at 2135 the target as described in section 7.3.6. The Read STag is used 2136 in the handling of the SCSI Data-in PDU(s) from the iSCSI Layer 2137 at the target as described in section 7.3.5. 2139 7.3.4 SCSI Data-out 2141 Type: control-type PDU 2143 PDU-specific qualifiers: DataDescriptorOut 2145 The iSCSI Layer at the initiator MUST invoke the Send_Control 2146 Operational Primitive qualified with DataDescriptorOut which defines 2147 the initiator's I/O Buffer containing unsolicited SCSI Write data. 2149 If the amount of unsolicited data to be transferred as SCSI Data-out 2150 exceeds TargetRecvDataSegmentLength, then the iSCSI Layer at the 2151 initiator MUST segment the data into multiple iSCSI control-type 2152 PDUs, with the DataSegmentLength having the value of 2153 TargetRecvDataSegmentLength in all PDUs generated except the last 2154 one. The DataSegmentLength of the last iSCSI control-type PDU 2155 carrying the unsolicited data can be up to 2156 TargetRecvDataSegmentLength. The iSCSI Layer at the target MUST 2157 perform the reassembly function for the unsolicited data. 2159 For unsolicited data, if the F bit is set to 0 in a SCSI Data-out 2160 PDU, the iSER Layer at the initiator MUST use a Send Message to send 2161 the SCSI Data-out PDU. If the F bit is set to 1, the iSER Layer at 2162 the initiator MUST use a SendSE Message to send the SCSI Data-out 2163 PDU. 2165 Note that for solicited data, the SCSI Data-out PDUs are not used 2166 since R2T PDUs are not delivered to the iSCSI layer at the 2167 initiator; instead R2T PDUs are transformed by the iSER layer at the 2168 target into RDMA Read operations. (See section 7.3.6.) 2170 7.3.5 SCSI Data-in 2172 Type: data-type PDU 2174 PDU-specific qualifiers: DataDescriptorIn 2176 When the iSCSI Layer at the target is ready to return the SCSI Read 2177 data to the initiator, it MUST invoke the Put_Data Operational 2178 Primitive qualified with DataDescriptorIn which defines the SCSI 2179 Data-in buffer. See section 7.1 on the general requirement on the 2180 handling of iSCSI data-type PDUs. SCSI Data-in PDU(s) are used in 2181 SCSI Read data transfer as described in section 9.5.2. 2183 The iSER Layer at the target MUST do the following for each 2184 invocation of the Put_Data Operational Primitive: 2186 1. It MUST use the ITT in the SCSI Data-in PDU to locate the remote 2187 Read STag in the Remote Mapping that associates the ITT to 2188 Advertised STag(s). The Remote Mapping was established earlier 2189 by the iSER Layer at the target when the SCSI Read Command was 2190 received from the initiator. 2192 2. It MUST generate and send an RDMA Write Message containing the 2193 read data to the initiator. 2195 a. It MUST use the remote Read STag as the Data Sink STag of 2196 the RDMA Write Message. 2198 b. It MUST use the Buffer Offset from the SCSI Data-in PDU as 2199 the Data Sink Tagged Offset of the RDMA Write Message. 2201 c. It MUST use DataSegmentLength from the SCSI Data-in PDU to 2202 determine the amount of data to be sent in the RDMA Write 2203 Message. 2205 3. It MUST associate DataSN and ITT from the SCSI Data-in PDU with 2206 the RDMA Write operation. If the Put_Data Operational Primitive 2207 invocation was qualified with Notify_Enable set, then when the 2208 iSER Layer at the target receives a completion from the RCaP 2209 layer for the RDMA Write Message, the iSER Layer at the target 2210 MUST notify the iSCSI Layer by invoking the 2211 Data_Completion_Notify Operational Primitive qualified with 2212 DataSN and ITT. Conversely, if the Put_Data Operational 2213 Primitive invocation was qualified with Notify_Enable cleared, 2214 then the iSER Layer at the target MUST NOT notify the iSCSI 2215 Layer on completion and MUST NOT invoke the 2216 Data_Completion_Notify Operational Primitive. 2218 When the A-bit is set to 1 in the SCSI Data-in PDU, the iSER Layer 2219 at the target MUST notify the iSCSI Layer at the target when the 2220 data transfer is complete at the initiator. To perform this 2221 additional function, the iSER Layer at the target can take advantage 2222 of the operational ErrorRecoveryLevel if previously disclosed by the 2223 iSCSI Layer via an earlier invocation of the Notice_Key_Values 2224 Operational Primitive. There are two approaches that can be taken: 2226 1. If the iSER Layer at the target knows that the operational 2227 ErrorRecoveryLevel is 2, or if the iSER Layer at the target does 2228 not know the operational ErrorRecoveryLevel, then the iSER Layer 2229 at the target MUST issue a zero-length RDMA Read Request Message 2230 following the RDMA Write Message. When the iSER Layer at the 2231 target receives a completion for the RDMA Read Request Message 2232 from the RCaP layer, implying that the RDMA-Capable Controller 2233 at the initiator has completed processing the RDMA Write Message 2234 due to the completion ordering semantics of RCaP, the iSER Layer 2235 at the target MUST notify the iSCSI Layer at the target by 2236 invoking the Data_Ack_Notify Operational Primitive qualified 2237 with ITT and DataSN (see section 3.2.3). 2239 2. If the iSER Layer at the target knows that the operational 2240 ErrorRecoveryLevel is 1, then the iSER Layer at the target MUST 2241 do one of the following: 2243 a. It MUST notify the iSCSI Layer at the target by invoking the 2244 Data_Ack_Notify Operational Primitive qualified with ITT and 2245 DataSN (see section 3.2.3) when it receives the local 2246 completion from the RCaP layer for the RDMA Write Message. 2247 This is allowed since digest errors do not occur in iSER 2248 (see section 10.1.4.2) and a CRC error will cause the 2249 connection to be terminated and the task to be terminated 2250 anyway. The local RDMA Write completion from the RCaP layer 2251 guarantees that the RCaP layer will not access the I/O 2252 Buffer again to transfer the data associated with that RDMA 2253 Write operation. 2255 b. Alternatively, it MUST use the same procedure for handling 2256 the data transfer completion at the initiator as for 2257 ErrorRecoveryLevel 2. 2259 It should be noted that the iSCSI Layer at the target cannot set the 2260 A-bit to 1 if the ErrorRecoveryLevel=0. 2262 SCSI status MUST always be returned in a separate SCSI Response PDU. 2263 The S bit in the SCSI Data-in PDU MUST always be set to 0. There 2264 MUST NOT be a "phase collapse" in the SCSI Data-in PDU. 2266 Since the RDMA Write Message only transfers the data portion of the 2267 SCSI Data-in PDU but not the control information in the header, such 2268 as ExpCmdSN, if timely updates of such information is crucial, the 2269 iSCSI Layer at the initiator MAY issue NOP-Out PDUs to request the 2270 iSCSI Layer at the target to respond with the information using NOP- 2271 In PDUs. 2273 7.3.6 Ready To Transfer (R2T) 2275 Type: data-type PDU 2277 PDU-specific qualifiers: DataDescriptorOut 2279 In order to send an R2T PDU, the iSCSI Layer at the target MUST 2280 invoke the Get_Data Operational Primitive qualified with 2281 DataDescriptorOut which defines the I/O Buffer for receiving the 2282 SCSI Write data from the initiator. See section 7.1 on the general 2283 requirements on the handling of iSCSI data-type PDUs. 2285 The iSER Layer at the target MUST do the following for each 2286 invocation of the Get_Data Operational Primitive: 2288 1. It MUST ensure a valid local STag for the I/O Buffer and a valid 2289 Local Mapping that associates the Initiator Task Tag (ITT) to 2290 the local STag. This may involve allocating a valid local STag 2291 and establishing a Local Mapping. 2293 2. It MUST use the ITT in the R2T to locate the remote Write STag 2294 in the Remote Mapping that associates the ITT to Advertised 2295 STag(s). The Remote Mapping was established earlier by the iSER 2296 Layer at the target when the iSER Message containing the 2297 Advertised Write STag and the SCSI Command PDU for a SCSI Write 2298 or bidirectional command was received from the initiator. 2300 3. If the iSER-ORD value at the target is set to 0, the iSER Layer 2301 at the target MUST terminate the connection and free up the 2302 resources associated with the connection (as described in 5.2.3) 2303 if it received the R2T PDU from the iSCSI Layer at the target. 2304 Upon termination of the connection, the iSER Layer at the target 2305 MUST notify the iSCSI Layer at the target by invoking the 2306 Connection_Terminate_Notify Operational Primitive. 2308 4. If the iSER-ORD value at the target is set to greater than 0, 2309 the iSER Layer at the target MUST transform the R2T PDU into an 2310 RDMA Read Request Message. While transforming the R2T PDU, the 2311 iSER Layer at the target MUST ensure that the number of 2312 outstanding RDMA Read Request Messages does not exceed iSER-ORD 2313 value. To transform the R2T PDU, the iSER Layer at the target: 2315 a. MUST derive the local STag and local Tagged Offset from the 2316 DataDescriptorOut that qualified the Get_Data invocation. 2318 b. MUST use the local STag as the Data Sink STag of the RDMA 2319 Read Request Message. 2321 c. MUST use the local Tagged Offset as the Data Sink Tagged 2322 Offset of the RDMA Read Request Message. 2324 d. MUST use the Desired Data Transfer Length from the R2T PDU 2325 as the RDMA Read Message Size of the RDMA Read Request 2326 Message. 2328 e. MUST use the remote Write STag as the Data Source STag of 2329 the RDMA Read Request Message. 2331 f. MUST use the Buffer Offset from the R2T PDU as the Data 2332 Source Tagged Offset of the RDMA Read Request Message. 2334 5. It MUST associate R2TSN and ITT from the R2T PDU with the RDMA 2335 Read operation. If the Get_Data Operational Primitive 2336 invocation was qualified with Notify_Enable set, then when the 2337 iSER Layer at the target receives a completion from the RCaP 2338 layer for the RDMA Read operation, the iSER Layer at the target 2339 MUST notify the iSCSI Layer by invoking the 2340 Data_Completion_Notify Operational Primitive qualified with 2341 R2TSN and ITT. Conversely, if the Get_Data Operational 2342 Primitive invocation was qualified with Notify_Enable cleared, 2343 then the iSER Layer at the target MUST NOT notify the iSCSI 2344 Layer on completion and MUST NOT invoke the 2345 Data_Completion_Notify Operational Primitive. 2347 When the RCaP layer at the initiator receives a valid RDMA Read 2348 Request Message, it will return an RDMA Read Response Message 2349 containing the solicited write data to the target. When the RCaP 2350 layer at target receives the RDMA Read Response Message from the 2351 initiator, it will place the solicited data in the I/O Buffer 2352 referenced by the Data Sink STag in the RDMA Read Response Message. 2354 Since the RDMA Read Request Message from the target does not 2355 transfer the control information in the R2T PDU such as ExpCmdSN, if 2356 timely updates of such information is crucial, the iSCSI Layer at 2357 the initiator MAY issue NOP-Out PDUs to request the iSCSI Layer at 2358 the target to respond with the information using NOP-In PDUs. 2360 Similarly, since the RDMA Read Response Message from the initiator 2361 only transfers the data but not the control information normally 2362 found in the SCSI Data-out PDU, such as ExpStatSN, if timely updates 2363 of such information is crucial, the iSCSI Layer at the target MAY 2364 issue NOP-In PDUs to request the iSCSI Layer at the initiator to 2365 respond with the information using NOP-Out PDUs. 2367 7.3.7 Asynchronous Message 2369 Type: control-type PDU 2371 PDU-specific qualifiers: DataDescriptorSense 2373 The iSCSI Layer MUST invoke the Send_Control Operational Primitive 2374 qualified with DataDescriptorSense which defines the buffer 2375 containing the sense and iSCSI Event information. The iSER Layer 2376 MUST use a SendSE Message to send the Asynchronous Message PDU. 2378 7.3.8 Text Request & Text Response 2380 Type: control-type PDU 2382 PDU-specific qualifiers: DataDescriptorTextOut (for Text 2383 Request), DataDescriptorIn (for Text Response) 2385 The iSCSI Layer MUST invoke the Send_Control Operational Primitive 2386 qualified with DataDescriptorTextOut (or DataDescriptorIn) which 2387 defines the Text Request (or Text Response) buffer. The iSER Layer 2388 MUST use SendSE Messages to send the Text Request (or Text Response 2389 PDUs). 2391 7.3.9 Login Request & Login Response 2393 During the login negotiation, the iSCSI Layer interacts with the 2394 transport layer directly and the iSER Layer is not involved. See 2395 section 5.1 on iSCSI/iSER Connection Setup. If the underlying 2396 transport is TCP, the Login Request PDUs and the Login Response PDUs 2397 are exchanged when the connection between the initiator and the 2398 target is still in the byte stream mode. 2400 The iSCSI Layer MUST not send a Login Request (or a Login Response) 2401 PDU during the full feature phase. A Login Request (or a Login 2402 Response) PDU, if used, MUST be treated as an iSCSI protocol error. 2403 The iSER Layer MAY reject such a PDU from the iSCSI Layer with an 2404 appropriate error code. If a Login Request PDU is received by the 2405 iSCSI Layer at the target, it MUST respond with a Reject PDU with a 2406 reason code of "protocol error". 2408 7.3.10 Logout Request & Logout Response 2410 Type: control-type PDU 2412 PDU-specific qualifiers: None 2414 The iSER Layer MUST use a SendSE Message to send the Logout Request 2415 or Logout Response PDU. Section 5.2.1 and 5.2.2 describe the 2416 handling of the Logout Request and the Logout Response at the 2417 initiator and the target and the interactions between the initiator 2418 and the target to terminate a connection. 2420 7.3.11 SNACK Request 2422 Since HeaderDigest and DataDigest must be negotiated to "None", 2423 there are no digest errors when the connection is in iSER-assisted 2424 mode. Also since RCaP delivers all messages in the order they were 2425 sent, there are no sequence errors when the connection is in iSER- 2426 assisted mode. Therefore the iSCSI Layer MUST NOT send SNACK 2427 Request PDUs. A SNCAK Request PDU, if used, MUST be treated as an 2428 iSCSI protocol error. The iSER Layer MAY reject such a PDU from the 2429 iSCSI Layer with an appropriate error code. If a SNACK Request PDU 2430 is received by the iSCSI Layer at the target, it MUST respond with a 2431 Reject PDU with a reason code of "protocol error". 2433 7.3.12 Reject 2435 Type: control-type PDU 2437 PDU-specific qualifiers: DataDescriptorReject 2439 The iSCSI Layer MUST invoke the Send_Control Operational Primitive 2440 qualified with DataDescriptorReject which defines the Reject buffer. 2441 The iSER Layer MUST use a SendSE Message to send the Reject PDU. 2443 7.3.13 NOP-Out & NOP-In 2445 Type: control-type PDU 2447 PDU-specific qualifiers: DataDescriptorNOPOut (for NOP-Out), 2448 DataDescriptorNOPIn (for NOP-In) 2450 The iSCSI Layer MUST invoke the Send_Control Operational Primitive 2451 qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn) which 2452 defines the Ping (or Return Ping) data buffer. The iSER Layer MUST 2453 use SendSE Messages to send the NOP-Out (or NOP-In) PDU. 2455 8 Flow Control and STag Management 2457 8.1 Flow Control for RDMA Send Message Types 2459 Send Message Types in RCaP are used by the iSER Layer to transfer 2460 iSCSI control-type PDUs. Each Send Message Type in RCaP consumes an 2461 Untagged Buffer at the Data Sink. However, neither the RCaP layer 2462 nor the iSER Layer provides an explicit flow control mechanism for 2463 the Send Message Types. Therefore, the iSER Layer SHOULD provision 2464 enough Untagged buffers for handling incoming Send Message Types to 2465 prevent buffer exhaustion at the RCaP layer. If buffer exhaustion 2466 occurs, it may result in the termination of the connection. 2468 An implementation may choose to satisfy the buffer requirement by 2469 using a common buffer pool shared across multiple connections, with 2470 usage limits on a per connection basis and usage limits on the 2471 buffer pool itself. In such an implementation, exceeding the buffer 2472 usage limit for a connection or the buffer pool itself may trigger 2473 interventions from the iSER Layer to replenish the buffer pool 2474 and/or to isolate the connection causing the problem. 2476 iSER also provides the MaxOutstandingUnexpectedPDUs key to be used 2477 by the initiator and the target to declare the maximum number of 2478 outstanding "unexpected" control-type PDUs that it can receive. It 2479 is intended to allow the receiving side to determine the amount of 2480 buffer resources needed beyond the normal flow control mechanism 2481 available in iSCSI. 2483 The buffer resources required at both the initiator and the target 2484 as a result of control-type PDUs sent by the initiator is described 2485 in section 8.1.1. The buffer resources required at both the 2486 initiator and target as a result of control-type PDUs sent by the 2487 target is described in section 8.1.2. 2489 8.1.1 Flow Control for Control-Type PDUs from the Initiator 2491 The control-type PDUs that can be sent by an initiator to a target 2492 can be grouped into the following categories: 2494 1. Regulated: Control-type PDUs in this category are regulated by 2495 the iSCSI CmdSN window mechanism and the immediate flag is not 2496 set. 2498 2. Unregulated but Expected: Control-type PDUs in this category 2499 are not regulated by the iSCSI CmdSN window mechanism but are 2500 expected by the target. 2502 3. Unregulated and Unexpected: Control-type PDUs in this category 2503 are not regulated by the iSCSI CmdSN window mechanism and are 2504 "unexpected" by the target. 2506 8.1.1.1 Control-Type PDUs from the Initiator in the Regulated Category 2508 Control-type PDUs that can be sent by the initiator in this category 2509 are regulated by the iSCSI CmdSN window mechanism and the immediate 2510 flag is not set. 2512 The queuing capacity required of the iSCSI layer at the target is 2513 described in section 3.2.2.1 of [RFC3720]. For each of the control- 2514 type PDUs that can be sent by the initiator in this category, the 2515 initiator MUST provision for the buffer resources required for the 2516 corresponding control-type PDU sent as a response from the target. 2517 The following is a list of the PDUs that can be sent by the 2518 initiator and the PDUs that are sent by the target in response: 2520 a. When an initiator sends a SCSI Command PDU, it expects a 2521 SCSI Response PDU from the target. 2523 b. When the initiator sends a Task Management Function Request 2524 PDU, it expects a Task Management Function Response PDU from 2525 the target. 2527 c. When the initiator sends a Text Request PDU, it expects a 2528 Text Response PDU from the target. 2530 d. When the initiator sends a Logout Request PDU, it expects a 2531 Logout Response PDU from the target. 2533 e. When the initiator sends a NOP-Out PDU as a ping request 2534 with ITT != 0xffffffff and TTT = 0xffffffff, it expects a 2535 NOP-In PDU from the target with the same ITT and TTT as in 2536 the ping request. 2538 The response from the target for any of the PDUs enumerated here may 2539 alternatively be in the form of a Reject PDU sent instead before the 2540 task is active, as described in section 6.3 of [RFC3720]. 2542 8.1.1.2 Control-Type PDUs from the Initiator in the Unregulated but 2543 Expected Category 2545 For the control-type PDUs in the Unregulated but Expected category, 2546 the amount of buffering resources required at the target can be 2547 predetermined. The following is a list of the PDUs in this 2548 category: 2550 a. SCSI Data-out PDUs are used by the initiator to send 2551 unsolicited data. The amount of buffer resources required 2552 by the target can be determined using FirstBurstLength. 2553 Note that SCSI Data-out PDUs are not used for solicited 2554 data since the R2T PDU which is used for solicitation is 2555 transformed into RDMA Read operations by the iSER layer at 2556 the target. See section 7.3.4. 2558 b. A NOP-Out PDU with TTT != 0xffffffff is sent as a ping 2559 response by the initiator to the NOP-In PDU sent as a ping 2560 request by the target. 2562 8.1.1.3 Control-Type PDUs from the Initiator in the Unregulated and 2563 Unexpected Category 2565 PDUs in the Unregulated and Unexpected category are PDUs with the 2566 immediate flag set. The number of PDUs in this category which can 2567 be sent by an initiator is controlled by the value of 2568 MaxOutstandingUnexpectedPDUs declared by the target. (See section 2569 6.7.) After a PDU in this category is sent by the initiator, it is 2570 outstanding until it is retired. At any time, the number of 2571 outstanding unexpected PDUs MUST not exceed the value of 2572 MaxOutstandingUnexpectedPDUs declared by the target. 2574 The target uses the value of MaxOutstandingUnexpectedPDUs that it 2575 declared to determine the amount of buffer resources required for 2576 control-type PDUs in this category that can be sent by an initiator. 2577 For the initiator, for each of the control-type PDUs that can be 2578 sent in this category, the initiator MUST provision for the buffer 2579 resources if required for the corresponding control-type PDU that 2580 can be sent as a response from the target. 2582 An outstanding PDU in this category is retired as follows. If the 2583 CmdSN of the PDU sent by the initiator in this category is x, the 2584 PDU is outstanding until the initiator sends a non-immediate 2585 control-type PDU on the same connection with CmdSN = y (where y is 2586 at least x) and the target responds with a control-type PDU on any 2587 connection where ExpCmdSN is at least y+1. 2589 When the number of outstanding unexpected control-type PDUs equals 2590 MaxOutstandingUnexpectedPDUs, the iSCSI Layer at the initiator MUST 2591 NOT generate any unexpected PDUs which otherwise it would have 2592 generated, even if it is intended for immediate delivery. 2594 8.1.2 Flow Control for Control-Type PDUs from the Target 2596 Control-type PDUs that can be sent by a target and are expected by 2597 the initiator are listed in the Regulated category. (See section 2598 8.1.1.1.) 2600 For the control-type PDUs that can be sent by a target and are 2601 unexpected by the initiator, the number is controlled by 2602 MaxOutstandingUnexpectedPDUs declared by the initiator. (See 2603 section 6.7.) After a PDU in this category is sent by a target, it 2604 is outstanding until it is retired. At any time, the number of 2605 outstanding unexpected PDUs MUST not exceed the value of 2606 MaxOutstandingUnexpectedPDUs declared by the initiator. The 2607 initiator uses the value of MaxOutstandingUnexpectedPDUs that it 2608 declared to determine the amount of buffer resources required for 2609 control-type PDUs in this category that can be sent by a target. 2610 The following is a list of the PDUs in this category and the 2611 conditions for retiring the outstanding PDU: 2613 a. For an Asynchronous Message PDU with StatSN = x, the PDU is 2614 outstanding until the initiator sends a control-type PDU 2615 with ExpStatSN set to at least x+1. 2617 b. For a Reject PDU with StatSN = x which is sent after a task 2618 is active, the PDU is outstanding until the initiator sends 2619 a control-type PDU with ExpStatSN set to at least x+1. 2621 c. For a NOP-In PDU with ITT = 0xffffffff and StatSN = x, the 2622 PDU is outstanding until the initiator responds with a 2623 control-type PDU on the same connection where ExpStatSN is 2624 at least x+1. But if the NOP-In PDU is sent as a ping 2625 request with TTT != 0xffffffff, the PDU can also be retired 2626 when the initiator sends a NOP-Out PDU with the same ITT and 2627 TTT as in the ping request. Note that when a target sends a 2628 NOP-In PDU as a ping request, it must provision a buffer for 2629 the NOP-Out PDU sent as a ping response from the initiator. 2631 When the number of outstanding unexpected control-type PDUs equals 2632 MaxOutstandingUnexpectedPDUs, the iSCSI Layer at the target MUST NOT 2633 generate any unexpected PDUs which otherwise it would have 2634 generated, even if its intent is to indicate an iSCSI error 2635 condition (e.g., Asynchronous Message, Reject). Task timeouts as in 2636 the initiator waiting for a command completion or other connection 2637 and session level exceptions will ensure that correct operational 2638 behavior will result in these cases despite not generating the PDU. 2639 This rule overrides any other requirements elsewhere which require 2640 that a Reject PDU MUST be sent. 2642 (Implementation note: SCSI task timeout and recovery can be a 2643 lengthy process and hence SHOULD be avoided by proper provisioning 2644 of resources.) 2646 (Implementation note: To ensure that the initiator has a means to 2647 inform the target that outstanding PDUs have been retired, the 2648 target should reserve the last unexpected control-type PDU allowable 2649 by the value of MaxOutstandingUnexpectedPDUs declared by the 2650 initiator for sending a NOP-In ping request with TTT != 0xffffffff 2651 to allow the initiator to return the NOP-Out ping response with the 2652 current ExpStatSN.) 2654 8.2 Flow Control for RDMA Read Resources 2656 The total number of RDMA Read operations that can be active 2657 simultaneously on an iSCSI/iSER connection depends on the amount of 2658 resources allocated as declared in the iSER Hello exchange described 2659 in section 5.1.3. Exceeding the number of RDMA Read operations 2660 allowed on a connection will result in the connection being 2661 terminated by the RCaP layer. The iSER Layer at the target 2662 maintains the iSER-ORD to keep track of the maximum number of RDMA 2663 Read Requests that can be issued by the iSER Layer on a particular 2664 RCaP Stream. 2666 During connection setup (see section 5.1), iSER-IRD is known at the 2667 initiator and iSER-ORD is known at the target after the iSER Layers 2668 at the initiator and the target have respectively allocated the 2669 connection resources necessary to support RCaP, as directed by the 2670 Allocate_Connection_Resources Operational Primitive from the iSCSI 2671 Layer before the end of the iSCSI Login Phase. In the full feature 2672 phase, the first message sent by the initiator is the iSER Hello 2673 Message (see section 9.3) which contains the value of iSER-IRD. In 2674 response to the iSER Hello Message, the target sends the iSER 2675 HelloReply Message (see section 9.4) which contains the value of 2676 iSER-ORD. The iSER Layer at both the initiator and the target MAY 2677 adjust (lower) the resources associated with iSER-IRD and iSER-ORD 2678 respectively to match the iSER-ORD value declared in the HelloReply 2679 Message. The iSER Layer at the target MUST flow control the RDMA 2680 Read Request Messages to not exceed the iSER-ORD value at the 2681 target. 2683 8.3 STag Management 2685 An STag, as defined in [RDMAP], is an identifier of a Tagged Buffer 2686 used in an RDMA operation. The allocation and the subsequent 2687 invalidation of the STags are specified in this document if the 2688 STags are exposed on the wire by being Advertised in the iSER header 2689 or declared in the header of an RCaP Message. 2691 8.3.1 Allocation of STags 2693 When the iSCSI Layer at the initiator invokes the Send_Control 2694 Operational Primitive to request the iSER Layer at the initiator to 2695 process a SCSI Command, zero, one, or two STags may be allocated by 2696 the iSER Layer. See section 7.3.1 for details. The number of STags 2697 allocated depends on whether the command is unidirectional or 2698 bidirectional and whether solicited write data transfer is involved 2699 or not. 2701 When the iSCSI Layer at the initiator invokes the Send_Control 2702 Operational Primitive to request the iSER Layer at the initiator to 2703 process a Task Management Function Request with the TASK REASSIGN 2704 function, besides allocating zero, one, or two STags, the iSER Layer 2705 MUST invalidate the existing STags, if any, associated with the ITT. 2706 See section 7.3.3 for details. 2708 The iSER Layer at the target allocates a local Data Sink STag when 2709 the iSCSI Layer at the target invokes the Get_Data Operational 2710 Primitive to request the iSER Layer to process an R2T PDU. See 2711 section 7.3.6 for details. 2713 8.3.2 Invalidation of STags 2715 The invalidation of the STags at the initiator at the completion of 2716 a unidirectional or bidirectional command when the associated SCSI 2717 Response PDU is sent by the target is described in section 7.3.2. 2719 When a unidirectional or bidirectional command concludes without the 2720 associated SCSI Response PDU being sent by the target, the iSCSI 2721 Layer at the initiator MUST request the iSER Layer at the initiator 2722 to invalidate the STags by invoking the Deallocate_Task_Resources 2723 Operational Primitive qualified with ITT. In response, the iSER 2724 Layer at the initiator MUST locate the STag(s) (if any) in the Local 2725 Mapping that associates the ITT to the local STag(s). The iSER 2726 Layer at the initiator MUST invalidate the STag(s) (if any) and the 2727 Local Mapping. 2729 For an RDMA Read operation used to realize a SCSI Write data 2730 transfer, the iSER Layer at the target SHOULD invalidate the Data 2731 Sink STag at the conclusion of the RDMA Read operation referencing 2732 the Data Sink STag (to permit the immediate reuse of buffer 2733 resources). 2735 For an RDMA Write operation used to realize a SCSI Read data 2736 transfer, the Data Source STag at the target is not declared to the 2737 initiator and is not exposed on the wire. Invalidation of the STag 2738 is thus not specified. 2740 When a unidirectional or bidirectional command concludes without the 2741 associated SCSI Response PDU being sent by the target, the iSCSI 2742 Layer at the target MUST request the iSER Layer at the target to 2743 invalidate the STags by invoking the Deallocate_Task_Resources 2744 Operational Primitive qualified with ITT. In response, the iSER 2745 Layer at the target MUST locate the local STag(s) (if any) in the 2746 Local Mapping that associates the ITT to the local STag(s). The 2747 iSER Layer at the target MUST invalidate the local STag(s) (if any) 2748 and the mapping. 2750 9 iSER Control and Data Transfer 2752 For iSCSI data-type PDUs (see section 7.1), the iSER Layer uses RDMA 2753 Read and RDMA Write operations to transfer the solicited data. For 2754 iSCSI control-type PDUs (see section 7.2), the iSER Layer uses Send 2755 Message Types of RCaP. 2757 9.1 iSER Header Format 2759 An iSER header MUST be present in every Send Message Type of RCaP. 2760 The iSER header is located in the first 12 bytes of the message 2761 payload of the Send Message Type of RCaP, as shown in Figure 2. 2763 0 1 2 3 2764 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2765 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2766 | Opcode| Opcode Specific Fields | 2767 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2768 | Opcode Specific Fields | 2769 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2770 | Opcode Specific Fields | 2771 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2772 Figure 2 iSER Header Format 2774 Opcode - Operation Code: 4 bits 2776 The Opcode field identifies the type of iSER Messages: 2778 0001b = iSCSI control-type PDU 2780 0010b = iSER Hello Message 2782 0011b = iSER HelloReply Message 2784 All other opcodes are reserved. 2786 9.2 iSER Header Format for iSCSI Control-Type PDU 2788 The iSER Layer uses Send Message Types of RCaP to transfer iSCSI 2789 control-type PDUs (see section 7.2). The message payload of each of 2790 the Send Message Types of RCaP used for transferring an iSER Message 2791 contains an iSER Header followed by an iSCSI control-type PDU. 2793 The iSER header in a Send Message Type of RCaP carrying an iSCSI 2794 control-type PDU MUST have the format as described in Figure 3. 2796 0 1 2 3 2797 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2798 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2799 | |W|R| | 2800 | 0001b |S|S| Reserved | 2801 | |V|V| | 2802 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2803 | Write STag | 2804 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2805 | Write Virtual Address High | 2806 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2807 | Write Virtual Address Low | 2808 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2809 | Read STag | 2810 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2811 | Read Virtual Address High | 2812 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2813 | Read Virtual Address Low | 2814 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2815 Figure 3 iSER Header Format for iSCSI Control-Type PDU 2817 WSV - Write STag Valid flag: 1 bit 2819 This flag indicates the validity of the Write STag field of the 2820 iSER Header. If set to one, the Write STag field in this iSER 2821 Header is valid. If set to zero, the Write STag field in this 2822 iSER Header MUST be ignored at the receiver. The Write STag 2823 Valid flag is set to one when there is solicited data to be 2824 transferred for a SCSI Write or bidirectional command, or when 2825 there are non-immediate unsolicited and solicited data to be 2826 transferred for the referenced task specified in a Task 2827 Management Function Request with the TASK REASSIGN function. 2829 RSV - Read STag Valid flag: 1 bit 2831 This flag indicates the validity of the Read STag field of the 2832 iSER Header. If set to one, the Read STag field in this iSER 2833 Header is valid. If set to zero, the Read STag field in this 2834 iSER Header MUST be ignored at the receiver. The Read STag 2835 Valid flag is set to one for a SCSI Read or bidirectional 2836 command, or a Task Management Function Request with the TASK 2837 REASSIGN function. 2839 Write STag - Write Steering Tag: 32 bits 2841 This field contains the Write STag when the Write STag Valid 2842 flag is set to one. For a SCSI Write or bidirectional command, 2843 the Write STag is used to Advertise the initiator's I/O Buffer 2844 containing the solicited data. For a Task Management Function 2845 Request with the TASK REASSIGN function, the Write STag is used 2846 to Advertise the initiator's I/O Buffer containing the non- 2847 immediate unsolicited data and solicited data. This Write STag 2848 is used as the Data Source STag in the resultant RDMA Read 2849 operation(s). When the Write STag Valid flag is set to zero, 2850 this field MUST be set to zero and ignored on receive. 2852 Write Virtual Address High: 32 bits 2854 This field contains the high order bits of the Virtual Address 2855 for the SCSI Write command when the Write STag Valid flag is 2856 set to one. When the Write STag Valid flag is set to zero, 2857 this field MUST be set to zero and ignored on receive. 2859 Write Virtual Address Low: 32 bits 2861 This field contains the low order bits of the Virtual Address 2862 for the SCSI Write command when the Write STag Valid flag is 2863 set to one. When the Write STag Valid flag is set to zero, 2864 this field MUST be set to zero and ignored on receive. 2866 Read STag - Read Steering Tag: 32 bits 2868 This field contains the Read STag when the Read STag Valid flag 2869 is set to one. The Read STag is used to Advertise the 2870 initiator's Read I/O Buffer of a SCSI Read or bidirectional 2871 command, or a Task Management Function Request with the TASK 2872 REASSIGN function. This Read STag is used as the Data Sink 2873 STag in the resultant RDMA Write operation(s). When the Read 2874 STag Valid flag is zero, this field MUST be set to zero and 2875 ignored on receive. 2877 Read Virtual Address High: 32 bits 2879 This field contains the high order bits of the Virtual Address 2880 for the SCSI Read command when the Read STag Valid flag is set 2881 to one. When the Read STag Valid flag is set to zero, this 2882 field MUST be set to zero and ignored on receive. 2884 Read Virtual Address Low: 32 bits 2886 This field contains the low order bits of the Virtual Address 2887 for the SCSI Read command when the Read STag Valid flag is set 2888 to one. When the read STag Valid flag is set to zero, this 2889 field MUST be set to zero and ignored on receive. 2891 Reserved: 2893 Reserved fields MUST be set to zero on transmit and MUST be 2894 ignored on receive. 2896 9.3 iSER Header Format for iSER Hello Message 2898 An iSER Hello Message MUST only contain the iSER header which MUST 2899 have the format as described in Figure 4. iSER Hello Message is the 2900 first iSER Message sent on the RCaP Stream from the iSER Layer at 2901 the initiator to the iSER Layer at the target. 2903 0 1 2 3 2904 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2905 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2906 | | | | | | 2907 | 0010b | Rsvd | MaxVer| MinVer| iSER-IRD | 2908 | | | | | | 2909 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2910 | Reserved | 2911 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2912 | Reserved | 2913 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2914 Figure 4 iSER Header Format for iSER Hello Message 2916 MaxVer - Maximum Version: 4 bits 2918 This field specifies the maximum version of the iSER protocol 2919 supported. It MUST be set to 10 to indicate the version of the 2920 specification described in this document. 2922 MinVer - Minimum Version: 4 bits 2924 This field specifies the minimum version of the iSER protocol 2925 supported. It MUST be set to 10 to indicate the version of the 2926 specification described in this document. 2928 iSER-IRD: 16 bits 2930 This field contains the value of the iSER-IRD at the initiator. 2932 Reserved (Rsvd): 2934 Reserved fields MUST be set to zero on transmit, and MUST be 2935 ignored on receive. 2937 9.4 iSER Header Format for iSER HelloReply Message 2939 An iSER HelloReply Message MUST only contain the iSER header which 2940 MUST have the format as described in Figure 5. The iSER HelloReply 2941 Message is the first iSER Message sent on the RCaP Stream from the 2942 iSER Layer at the target to the iSER Layer at the initiator. 2944 0 1 2 3 2945 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2946 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2947 | | |R| | | | 2948 | 0011b |Rsvd |E| MaxVer| CurVer| iSER-ORD | 2949 | | |J| | | | 2950 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2951 | Reserved | 2952 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2953 | Reserved | 2954 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2955 Figure 5 iSER Header Format for iSER HelloReply Message 2957 REJ - Reject flag: 1 bit 2959 This flag indicates whether the target is rejecting this 2960 connection. If set to one, the target is rejecting the 2961 connection. 2963 MaxVer - Maximum Version: 4 bits 2965 This field specifies the maximum version of the iSER protocol 2966 supported. It MUST be set to 10 to indicate the version of the 2967 specification described in this document. 2969 CurVer - Current Version: 4 bits 2971 This field specifies the current version of the iSER protocol 2972 supported. It MUST be set to 10 to indicate the version of the 2973 specification described in this document. 2975 iSER-ORD: 16 bits 2977 This field contains the value of the iSER-ORD at the target. 2979 Reserved (Rsvd): 2981 Reserved fields MUST be set to zero on transmit, and MUST be 2982 ignored on receive. 2984 9.5 SCSI Data Transfer Operations 2986 The iSER Layer at the initiator and the iSER Layer at the target 2987 handle each SCSI Write, SCSI Read, and bidirectional operation as 2988 described below. 2990 9.5.1 SCSI Write Operation 2992 The iSCSI Layer at the initiator MUST invoke the Send_Control 2993 Operational Primitive to request the iSER Layer at the initiator to 2994 send the SCSI Write Command. The iSER Layer at the initiator MUST 2995 request the RCaP layer to transmit a SendSE Message with the message 2996 payload consisting of the iSER header followed by the SCSI Command 2997 PDU and immediate data (if any). If there is solicited data, the 2998 iSER Layer MUST Advertise the Write STag in the iSER header of the 2999 SendSE Message, as described in section 9.2. Upon receiving the 3000 SendSE Message, the iSER Layer at the target MUST notify the iSCSI 3001 Layer at the target by invoking the Control_Notify Operational 3002 Primitive qualified with the SCSI Command PDU. See section 7.3.1 3003 for details on the handling of the SCSI Write Command. 3005 For the non-immediate unsolicited data, the iSCSI Layer at the 3006 initiator MUST invoke a Send_Control Operational Primitive qualified 3007 with the SCSI Data-out PDU. Upon receiving each Send or SendSE 3008 Message containing the non-immediate unsolicited data, the iSER 3009 Layer at the target MUST notify the iSCSI Layer at the target by 3010 invoking the Control_Notify Operational Primitive qualified with the 3011 SCSI Data-out PDU. See section 7.3.4 for details on the handling of 3012 the SCSI Data-out PDU. 3014 For the solicited data, when the iSCSI Layer at the target has an 3015 I/O Buffer available, it MUST invoke the Get_Data Operational 3016 Primitive qualified with the R2T PDU. See section 7.3.6 for details 3017 on the handling of the R2T PDU. 3019 When the data transfer associated with this SCSI Write operation is 3020 complete, the iSCSI Layer at the target MUST invoke the Send_Control 3021 Operational Primitive when it is ready to send the SCSI Response 3022 PDU. Upon receiving a SendSE or SendInvSE Message containing the 3023 SCSI Response PDU, the iSER Layer at the initiator MUST notify the 3024 iSCSI Layer at the initiator by invoking the Control_Notify 3025 Operational Primitive qualified with the SCSI Response PDU. See 3026 section 7.3.2 for details on the handling of the SCSI Response PDU. 3028 9.5.2 SCSI Read Operation 3030 The iSCSI Layer at the initiator MUST invoke the Send_Control 3031 Operational Primitive to request the iSER Layer at the initiator to 3032 send the SCSI Read Command. The iSER Layer at the initiator MUST 3033 request the RCaP layer to transmit a SendSE Message with the message 3034 payload consisting of the iSER header followed by the SCSI Command 3035 PDU. The iSER Layer at the initiator MUST Advertise the Read STag 3036 in the iSER header of the SendSE Message, as described in section 3037 9.2. Upon receiving the SendSE Message, the iSER Layer at the 3038 target MUST notify the iSCSI Layer at the target by invoking the 3039 Control_Notify Operational Primitive qualified with the SCSI Command 3040 PDU. See section 7.3.1 for details on the handling of the SCSI Read 3041 Command. 3043 When the requested SCSI data is available in the I/O Buffer, the 3044 iSCSI Layer at the target MUST invoke the Put_Data Operational 3045 Primitive qualified with the SCSI Data-in PDU. See section 7.3.5 3046 for details on the handling of the SCSI Data-in PDU. 3048 When the data transfer associated with this SCSI Read operation is 3049 complete, the iSCSI Layer at the target MUST invoke the Send_Control 3050 Operational Primitive when it is ready to send the SCSI Response 3051 PDU. Upon receiving the SendInvSE Message containing the SCSI 3052 Response PDU, the iSER Layer at the initiator MUST notify the iSCSI 3053 Layer at the initiator by invoking the Control_Notify Operational 3054 Primitive qualified with the SCSI Response PDU. See section 7.3.2 3055 for details on the handling of the SCSI Response PDU. 3057 9.5.3 Bidirectional Operation 3059 The initiator and the target handle the SCSI Write and the SCSI Read 3060 portions of this bidirectional operation the same as described in 3061 Section 9.5.1 and Section 9.5.2 respectively. 3063 10 iSER Error Handling and Recovery 3065 RCaP provides the iSER Layer with reliable in-order delivery. 3066 Therefore, the error management needs of an iSER-assisted connection 3067 are somewhat different than those of a Traditional iSCSI connection. 3069 10.1 Error Handling 3071 iSER error handling is described in the following sections, 3072 classified loosely based on the sources of errors: 3074 1. Those originating at the transport layer (e.g., TCP). 3076 2. Those originating at the RCaP layer. 3078 3. Those originating at the iSER Layer. 3080 4. Those originating at the iSCSI Layer. 3082 10.1.1 Errors in the Transport Layer 3084 If the transport layer is TCP, then TCP packets with detected errors 3085 are silently dropped by the TCP layer and result in retransmission 3086 at the TCP layer. This has no impact on the iSER Layer. However, 3087 connection loss (e.g., link failure) and unexpected termination 3088 (e.g., TCP graceful or abnormal close without the iSCSI Logout 3089 exchanges) at the transport layer will cause the iSCSI/iSER 3090 connection to be terminated as well. 3092 10.1.1.1 Failure in the Transport Layer Before RCaP Mode is Enabled 3094 If the Connection is lost or terminated before the iSCSI Layer 3095 invokes the Allocate_Connection_Resources Operational Primitive, the 3096 login process is terminated and no further action is required. 3098 If the Connection is lost or terminated after the iSCSI Layer has 3099 invoked the Allocate_Connection_Resources Operational Primitive, 3100 then the iSCSI Layer MUST request the iSER Layer to deallocate all 3101 connection resources by invoking the Deallocate_Connection_Resources 3102 Operational Primitive. 3104 10.1.1.2 Failure in the Transport Layer After RCaP Mode is Enabled 3106 If the Connection is lost or terminated after the iSCSI Layer has 3107 invoked the Enable_Datamover Operational Primitive, the iSER Layer 3108 MUST notify the iSCSI Layer of the connection loss by invoking the 3109 Connection_Terminate_Notify Operational Primitive. Prior to 3110 invoking the Connection_Terminate_Notify Operational Primitive, the 3111 iSER layer MUST perform the actions described in Section 5.2.3.2. 3113 10.1.2 Errors in the RCaP Layer 3115 The RCaP layer does not have error recovery operations built in. If 3116 errors are detected at the RCaP layer, the RCaP layer will terminate 3117 the RCaP Stream and the associated Connection. 3119 10.1.2.1 Errors Detected in the Local RCaP Layer 3121 If an error is encountered at the local RCaP layer, the RCaP layer 3122 MAY send a Terminate Message to the Remote Peer to report the error 3123 if possible. (For iWARP, see [RDMAP] for the list of errors where a 3124 Terminate Message is sent.) The RCaP layer is responsible for 3125 terminating the Connection. After the RCaP layer notifies the iSER 3126 Layer that the Connection is terminated, the iSER Layer MUST notify 3127 the iSCSI Layer by invoking the Connection_Terminate_Notify 3128 Operational Primitive. Prior to invoking the 3129 Connection_Terminate_Notify Operational Primitive, the iSER layer 3130 MUST perform the actions described in Section 5.2.3.2. 3132 10.1.2.2 Errors Detected in the RCaP Layer at the Remote Peer 3134 If an error is encountered at the RCaP layer at the Remote Peer, the 3135 RCaP layer at the Remote Peer may send a Terminate Message to report 3136 the error if possible. If it is unable to send the Terminate 3137 Message, the Connection is terminated. This is treated the same as 3138 a failure in the transport layer after RDMA is enabled as described 3139 in section 10.1.1.2. 3141 If an error is encountered at the RCaP layer at the Remote Peer and 3142 it is able to send a Terminate Message, the RCaP layer at the Remote 3143 Peer is responsible for terminating the connection. After the local 3144 RCaP layer notifies the iSER Layer that the Connection is 3145 terminated, the iSER Layer MUST notify the iSCSI Layer by invoking 3146 the Connection_Terminate_Notify Operational Primitive. Prior to 3147 invoking the Connection_Terminate_Notify Operational Primitive, the 3148 iSER layer MUST perform the actions described in Section 5.2.3.2. 3150 10.1.3 Errors in the iSER Layer 3152 The error handling due to errors at the iSER Layer is described in 3153 the following sections. 3155 10.1.3.1 Insufficient Connection Resources to Support RCaP at 3156 Connection Setup 3158 After the iSCSI Layer at the initiator invokes the 3159 Allocate_Connection_Resources Operational Primitive during the iSCSI 3160 login negotiation phase, if the iSER Layer at the initiator fails to 3161 allocate the connection resources necessary to support RCaP, it MUST 3162 return a status of failure to the iSCSI Layer at the initiator. The 3163 iSCSI Layer at the initiator MUST terminate the Connection as 3164 described in Section 5.2.3.1. 3166 After the iSCSI Layer at the target invokes the 3167 Allocate_Connection_Resources Operational Primitive during the iSCSI 3168 login negotiation phase, if the iSER Layer at the target fails to 3169 allocate the connection resources necessary to support RCaP, it MUST 3170 return a status of failure to the iSCSI Layer at the target. The 3171 iSCSI Layer at the target MUST send a Login Response with a status 3172 class of 3 (Target Error), and a status code of "0302" (Out of 3173 Resources). The iSCSI Layers at the initiator and the target MUST 3174 terminate the Connection as described in Section 5.2.3.1. 3176 10.1.3.2 iSER Negotiation Failures 3178 If the RCaP or iSER related parameters declared by the initiator in 3179 the iSER Hello Message is unacceptable to the iSER Layer at the 3180 target, the iSER Layer at the target MUST set the Reject (REJ) flag, 3181 as described in section 9.4, in the iSER HelloReply Message. The 3182 following are the cases when the iSER Layer MUST set the REJ flag to 3183 1 in the HelloReply Message: 3185 * The initiator-declared iSER-IRD value is greater than 0 and the 3186 target-declared iSER-ORD value is 0. 3188 * The initiator-supported and the target-supported iSER protocol 3189 versions do not overlap. 3191 After requesting the RCaP layer to send the iSER HelloReply Message, 3192 the handling of the error situation is the same as that for iSER 3193 format errors as described in section 10.1.3.3. 3195 10.1.3.3 iSER Format Errors 3197 The following types of errors in an iSER header are considered 3198 format errors: 3200 * Illegal contents of any iSER header field 3201 * Inconsistent field contents in an iSER header 3203 * Length error for an iSER Hello or HelloReply Message (see section 3204 9.3 and 9.4) 3206 When a format error is detected, the following events MUST occur in 3207 the specified sequence: 3209 1. The iSER Layer MUST request the RCaP layer to terminate the RCaP 3210 Stream. The RCaP layer MUST terminate the associated 3211 Connection. 3213 2. The iSER Layer MUST notify the iSCSI Layer of the connection 3214 termination by invoking the Connection_Terminate_Notify 3215 Operational Primitive. Prior to invoking the 3216 Connection_Terminate_Notify Operational Primitive, the iSER 3217 layer MUST perform the actions described in Section 5.2.3.2. 3219 10.1.3.4 iSER Protocol Errors 3221 The first iSER Message sent by the iSER Layer at the initiator MUST 3222 be the iSER Hello Message (see section 9.3). Likewise, the first 3223 iSER Message sent by the iSER Layer at the target MUST be the iSER 3224 HelloReply Message (see section 9.4). Failure to send the iSER 3225 Hello or HelloReply Message, as indicated by the wrong Opcode in the 3226 iSER header, is a protocol error. The handling of this error 3227 situation is the same as that for iSER format errors as described in 3228 section 10.1.3.3. 3230 If the sending side of an iSER-enabled connection acts in a manner 3231 not permitted by the negotiated or declared login/text operational 3232 key values as described in section 6, this is a protocol error and 3233 the receiving side MAY handle this the same as for iSER format 3234 errors as described in section 10.1.3.3. 3236 10.1.4 Errors in the iSCSI Layer 3238 The error handling due to errors at the iSCSI Layer is described in 3239 the following sections. For error recovery, see section 10.2. 3241 10.1.4.1 iSCSI Format Errors 3243 When an iSCSI format error is detected, the iSCSI Layer MUST request 3244 the iSER Layer to terminate the RCaP Stream by invoking the 3245 Connection_Terminate Operational Primitive. For more details on the 3246 connection termination, see Section 5.2.3.1. 3248 10.1.4.2 iSCSI Digest Errors 3250 In the iSER-assisted mode, the iSCSI Layer will not see any digest 3251 error because both the HeaderDigest and the DataDigest keys are 3252 negotiated to "None". 3254 10.1.4.3 iSCSI Sequence Errors 3256 For Traditional iSCSI, sequence errors are caused by dropped PDUs 3257 due to header or data digest errors. Since digests are not used in 3258 iSER-assisted mode and the RCaP layer will deliver all messages in 3259 the order they were sent, sequence errors will not occur in iSER- 3260 assisted mode. 3262 10.1.4.4 iSCSI Protocol Error 3264 When the iSCSI Layer handles certain protocol errors by dropping the 3265 connection, the error handling is the same as that for iSCSI format 3266 errors as described in section 10.1.4.1. 3268 When the iSCSI Layer uses the iSCSI Reject PDU and response codes to 3269 handle certain other protocol errors, no special handling at the 3270 iSER Layer is required. 3272 10.1.4.5 SCSI Timeouts and Session Errors 3274 This is handled at the iSCSI Layer and no special handling at the 3275 iSER Layer is required. 3277 10.1.4.6 iSCSI Negotiation Failures 3279 For negotiation failures that happen during the Login Phase at the 3280 initiator after the iSCSI Layer has invoked the 3281 Allocate_Connection_Resources Operational Primitive and before the 3282 Enable_Datamover Operational Primitive has been invoked, the iSCSI 3283 Layer MUST request the iSER Layer to deallocate all connection 3284 resources by invoking the Deallocate_Connection_Resources 3285 Operational Primitive. The iSCSI Layer at the initiator MUST 3286 terminate the Connection. 3288 For negotiation failures during the Login Phase at the target, the 3289 iSCSI Layer can use a Login Response with a status class other than 3290 0 (success) to terminate the Login Phase. If the iSCSI Layer has 3291 invoked the Allocate_Connection_Resources Operational Primitive and 3292 before the Enable_Datamover Operational Primitive has been invoked, 3293 the iSCSI Layer at the target MUST request the iSER Layer at the 3294 target to deallocate all connection resources by invoking the 3295 Deallocate_Connection_Resources Operational Primitive. The iSCSI 3296 Layer at both the initiator and the target MUST terminate the 3297 Connection. 3299 During the iSCSI Login Phase, if the iSCSI Layer at the initiator 3300 receives a Login Response from the target with a status class other 3301 than 0 (Success) after the iSCSI Layer at the initiator has invoked 3302 the Allocate_Connection_Resources Operational Primitive, the iSCSI 3303 Layer MUST request the iSER Layer to deallocate all connection 3304 resources by invoking the Deallocate_Connection_Resources 3305 Operational Primitive. The iSCSI Layer MUST terminate the 3306 Connection in this case. 3308 For negotiation failures during the full feature phase, the error 3309 handling is left to the iSCSI Layer and no special handling at the 3310 iSER Layer is required. 3312 10.2 Error Recovery 3314 Error recovery requirements of iSCSI/iSER are the same as that of 3315 Traditional iSCSI. All three ErrorRecoveryLevels as defined in 3316 [RFC3720] are supported in iSCSI/iSER. 3318 * For ErrorRecoveryLevel 0, session recovery is handled by iSCSI 3319 and no special handling by the iSER Layer is required. 3321 * For ErrorRecoveryLevel 1, see section 10.2.1 on PDU Recovery. 3323 * For ErrorRecoveryLevel 2, see section 10.2.2 on Connection 3324 Recovery. 3326 The iSCSI Layer may invoke the Notice_Key_Values Operational 3327 Primitive during connection setup to request the iSER Layer to take 3328 note of the value of the operational ErrorRecoveryLevel, as 3329 described in sections 5.1.1 and 5.1.2. 3331 10.2.1 PDU Recovery 3333 As described in sections 10.1.4.2 and 10.1.4.3, digest and sequence 3334 errors will not occur in the iSER-assisted mode. If the RCaP layer 3335 detects an error, it will close the iSCSI/iSER connection, as 3336 described in section 10.1.2. Therefore, PDU recovery is not useful 3337 in the iSER-assisted mode. 3339 The iSCSI Layer at the initiator SHOULD disable iSCSI timeout-driven 3340 PDU retransmissions. 3342 10.2.2 Connection Recovery 3344 The iSCSI Layer at the initiator MAY reassign connection allegiance 3345 for non-immediate commands which are still in progress and are 3346 associated with the failed connection by using a Task Management 3347 Function Request with the TASK REASSIGN function. See section 7.3.3 3348 for more details. 3350 When the iSCSI Layer at the initiator does a task reassignment for a 3351 SCSI Write command, it MUST qualify the Send_Control Operational 3352 Primitive invocation with DataDescriptorOut which defines the I/O 3353 Buffer for both the non-immediate unsolicited data and the solicited 3354 data. This allows the iSCSI Layer at the target to use recovery 3355 R2Ts to request for data originally sent as unsolicited and 3356 solicited from the initiator. 3358 When the iSCSI Layer at the target accepts a reassignment request 3359 for a SCSI Read command, it MUST request the iSER Layer to process 3360 SCSI Data-in for all unacknowledged data by invoking the Put_Data 3361 Operational Primitive. See section 7.3.5 on the handling of SCSI 3362 Data-in. 3364 When the iSCSI Layer at the target accepts a reassignment request 3365 for a SCSI Write command, it MUST request the iSER Layer to process 3366 a recovery R2T for any non-immediate unsolicited data and any 3367 solicited data sequences that have not been received by invoking the 3368 Get_Data Operational Primitive. See section 7.3.6 on the handling 3369 of Ready To Transfer (R2T). 3371 The iSCSI Layer at the target MUST NOT issue recovery R2Ts on an 3372 iSCSI/iSER connection for a task for which the connection allegiance 3373 was never reassigned. The iSER Layer at the target MAY reject such 3374 a recovery R2T received via the Get_Data Operational Primitive 3375 invocation from the iSCSI Layer at the target, with an appropriate 3376 error code. 3378 The iSER Layer at the target will process the requests invoked by 3379 the Put_Data and Get_Data Operational Primitives for a reassigned 3380 task in the same way as for the original commands. 3382 11 Security Considerations 3384 When iSER is layered on top of an RCaP layer and provides the RDMA 3385 extensions to the iSCSI protocol, the security considerations of 3386 iSER are the same as that of the underlying RCaP layer. For iWARP, 3387 this is described in [RDMAP] and [RDDPSEC]. 3389 Since iSER-assisted iSCSI protocol is still functionally iSCSI from 3390 a security considerations perspective, all of the iSCSI security 3391 requirements as described in [RFC3720] and [RFC3723] apply. If iSER 3392 is layered on top of a non-IP based RCaP layer, all the security 3393 protocol mechanisms applicable to that RCaP layer is also applicable 3394 to an iSCSI/iSER connection. If iSER is layered on top of a non-IP 3395 protocol, the IPsec mechanism as specified in [RFC3720] MUST be 3396 implemented at any point where the iSER protocol enters the IP 3397 network (e.g., via gateways), and the non-IP protocol SHOULD 3398 implement (optional to use) a packet by packet security protocol 3399 equal in strength to the IPsec mechanism specified by [RFC3720]. 3401 To minimize the potential for a denial of service attack, the iSCSI 3402 Layer MUST NOT request the iSER Layer to allocate the connection 3403 resources necessary to support RCaP until the iSCSI layer is 3404 sufficiently far along in the iSCSI Login Phase that it is 3405 reasonably certain that the peer side is not an attacker, as 3406 described in sections 5.1.1 and 5.1.2. 3408 12 IANA Considerations 3410 This document has no actions for IANA. 3412 13 References 3414 13.1 Normative References 3416 [RFC5046] M. Ko et al., "iSCSI Externsions for Remote Direct Memory 3417 Access", RFC 5046, October 2007 3419 [RFC3720] J. Satran et al., "iSCSI", RFC 3720, April 2004 3421 [RFC3723] B. Aboba et al., "Securing Block Storage Protocols over 3422 IP", RFC 3723, April 2004. 3424 [RDMAP] R. Recio et al., "An RDMA Protocol Specification", RFC 5040, 3425 October 2007 3427 [DDP] H. Shah et al., "Direct Data Placement over Reliable 3428 Transports", RFC 5041, October 2007 3430 [MPA] P. Culley et al., "Marker PDU Aligned Framing for TCP 3431 Specification", RFC 5044, October 2007 3433 [RDDPSEC] J. Pinkerton et al., "DDP/RDMAP Security", RFC 5042, 3434 October 2007 3436 [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, 3437 September 1981 3439 [RFC2119] Bradner, S., "Key Words for use in RFCs to Indicate 3440 Requirement Levels", BCP 14, RFC 2119, March 1997 3442 13.2 Informative References 3444 [SAM2] T10/1157D, SCSI Architecture Model - 2 (SAM-2) 3446 [DA] M. Chadalapaka et al., "Datamover Architecture for iSCSI", RFC 3447 5047, October 2007 3449 [VERBS] J. Hilland et al., "RDMA Protocol Verbs Specification", 3450 RDMAC Consortium Draft Specification draft-hilland-iwarp-verbs- 3451 v1.0-RDMAC, April 2003 3453 [IPSEC] S. Kent et al., "Security Architecture for the Internet 3454 Protocol", RFC 2401, November 1998 3456 [IB] InfiniBand Architecture Specification Volume 1 Release 1.2, 3457 October 2004 3459 [IPoIB] H.K. Chu et al, "Transmission of IP over InfiniBand", RFC 3460 4391, March 2006 3462 14 Summary of Changes from RFC 5046 3464 1. Removed the requirement that a connection be opened in "normal" 3465 TCP mode and transitioned to zero-copy mode. The allows the spec 3466 to conform to existing implementation for both Infiniband and 3467 iWARP. Changes were made in sections 2, 3.1.6, 4.2, 5.1, 5.1.1, 3468 5.1.2, 5.1.3, 10.1.3.4, and 11. 3470 2. Added a clause in section 6.2 to clarify that the initiator must 3471 not send more than InitiatorMaxRecvDataSegmentLength worth of 3472 data when a NOP-Out request is sent with a valid Initiator Task 3473 Tag. Since InitiatorMaxRecvDataSegmentLength can be smaller than 3474 TargetMaxRecvDataSegmentLength, returning the original data in 3475 the NOP-Out request in this situation can overflow the receive 3476 buffer unless the length of the data sent with the NOP-Out 3477 request is less than InitiatorMaxRecvDataSegmentLength. 3479 3. Added MaxAHSLength key in section 6.8 to set a limit on the AHS 3480 Length as proposed by Alex Nezhinsky. This is useful when 3481 posting receive buffers in knowing what the maximum possible 3482 message length is in a PDU which contains AHS. 3484 4. Added WriteAddressForSolicitedDataOnly key in section 6.9 to 3485 indicate how the memory region will be used as proposed by Alex 3486 Nezhinsky. Alex argued that WSTAG (RKEY+VA for Infiniband) sent 3487 by the initiator with a WRITE SCSI command already accounts for 3488 the offset associated with the solicited data, and does not 3489 reference the entire data. The rationale is that the initiator 3490 sometimes treats the memory regions intended for unsolicited and 3491 solicited data differently, as they are carried out using 3492 different Infiniband mechanisms (Send from the initiator, RDMA 3493 from the target) that imply different registration modes. In 3494 contrast, the model implied by the iSER spec creators was that 3495 the memory occupied by data is treated as contiguous (or 3496 virtually contiguous, by means of scatter-gather mechanisms) and 3497 homogenous region. Adding a new key will allow coping with 3498 current implementations. 3500 5. Added two 64-bit fields in iSER header in section 9.2 for the 3501 Read Virtual Address and the Write Virtual Address to conform to 3502 the Infiniband format as proposed by Bob Russell. This allows 3503 one implementation such as the OFED stack to be used in both the 3504 Infiniband and the iWARP environment. 3506 14.1 Proposed Changes To Be Considered 3508 6. Alex Nezhinsky stated that iSER Hello and iSER HelloReply 3509 messages are not implemented in practice. He argued that we 3510 should update the spec to remove the requirement to exchange the 3511 iSER Hello and iSER HelloReply messages. 3513 14.2 Changes To Be Done 3515 Update all diagrams to conform with the expanded iSER header format. 3517 Appendix A 3519 14.3 iWARP Message Format for iSER 3521 This section is for information only and is NOT part of the 3522 standard. It simply depicts the iWARP Message format for the 3523 various iSER Messages when the transport layer is TCP. 3525 14.3.1 iWARP Message Format for iSER Hello Message 3527 The following figure depicts an iSER Hello Message encapsulated in 3528 an iWARP SendSE Message. 3530 0 1 2 3 3531 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3532 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3533 | MPA Header | DDP Control | RDMA Control | 3534 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3535 | Reserved | 3536 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3537 | (Send) Queue Number | 3538 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3539 | (Send) Message Sequence Number | 3540 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3541 | (Send) Message Offset | 3542 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3543 | 0010b | Zeros | 0001b | 0001b | iSER-IRD | 3544 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3545 | All Zeros | 3546 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3547 | All Zeros | 3548 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3549 | MPA CRC | 3550 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3551 Figure 6 SendSE Message containing an iSER Hello Message 3553 14.3.2 iWARP Message Format for iSER HelloReply Message 3555 The following figure depicts an iSER HelloReply Message encapsulated 3556 in an iWARP SendSE Message. The Reject (REJ) flag is set to 0. 3558 0 1 2 3 3559 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3560 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3561 | MPA Header | DDP Control | RDMA Control | 3562 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3563 | Reserved | 3564 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3565 | (Send) Queue Number | 3566 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3567 | (Send) Message Sequence Number | 3568 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3569 | (Send) Message Offset | 3570 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3571 | 0011b |Zeros|0| 0001b | 0001b | iSER-ORD | 3572 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3573 | All Zeros | 3574 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3575 | All Zeros | 3576 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3577 | MPA CRC | 3578 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3579 Figure 7 SendSE Message containing an iSER HelloReply Message 3581 14.3.3 iWARP Message Format for SCSI Read Command PDU 3583 The following figure depicts a SCSI Read Command PDU embedded in an 3584 iSER Message encapsulated in an iWARP SendSE Message. For this 3585 particular example, in the iSER header, the Write STag Valid flag is 3586 set to zero, the Read STag Valid flag is set to one, the Write STag 3587 field is set to all zeros, and the Read STag field contains a valid 3588 Read STag. 3590 0 1 2 3 3591 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3592 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3593 | MPA Header | DDP Control | RDMA Control | 3594 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3595 | Reserved | 3596 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3597 | (Send) Queue Number | 3598 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3599 | (Send) Message Sequence Number | 3600 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3601 | (Send) Message Offset | 3602 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3603 | 0001b |0|1| All zeros | 3604 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3605 | All Zeros | 3606 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3607 | Read STag | 3608 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3609 | SCSI Read Command PDU | 3610 // // 3611 | | 3612 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3613 | MPA CRC | 3614 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3615 Figure 8 SendSE Message containing a SCSI Read Command PDU 3617 14.3.4 iWARP Message Format for SCSI Read Data 3619 The following figure depicts an iWARP RDMA Write Message carrying 3620 SCSI Read data in the payload: 3622 0 1 2 3 3623 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3624 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3625 | MPA Header | DDP Control | RDMA Control | 3626 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3627 | Data Sink STag | 3628 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3629 | Data Sink Tagged Offset | 3630 + + 3631 | | 3632 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3633 | SCSI Read data | 3634 // // 3635 | | 3636 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3637 | MPA CRC | 3638 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3639 Figure 9 RDMA Write Message containing SCSI Read Data 3641 14.3.5 iWARP Message Format for SCSI Write Command PDU 3643 The following figure depicts a SCSI Write Command PDU embedded in an 3644 iSER Message encapsulated in an iWARP SendSE Message. For this 3645 particular example, in the iSER header, the Write STag Valid flag is 3646 set to one, the Read STag Valid flag is set to zero, the Write STag 3647 field contains a valid Write STag, and the Read STag field is set to 3648 all zeros since it is not used. 3650 0 1 2 3 3651 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3652 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3653 | MPA Header | DDP Control | RDMA Control | 3654 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3655 | Reserved | 3656 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3657 | (Send) Queue Number | 3658 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3659 | (Send) Message Sequence Number | 3660 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3661 | (Send) Message Offset | 3662 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3663 | 0001b |1|0| All zeros | 3664 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3665 | Write STag | 3666 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3667 | All Zeros | 3668 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3669 | SCSI Write Command PDU | 3670 // // 3671 | | 3672 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3673 | MPA CRC | 3674 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3675 Figure 10 SendSE Message containing a SCSI Write Command PDU 3677 14.3.6 iWARP Message Format for RDMA Read Request 3679 An iSCSI R2T is transformed into an iWARP RDMA Read Request Message. 3680 The following figure depicts an iWARP RDMA Read Request Message: 3682 0 1 2 3 3683 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3684 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3685 | MPA Header | DDP Control | RDMA Control | 3686 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3687 | Reserved (Not Used) | 3688 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3689 | DDP (RDMA Read Request) Queue Number | 3690 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3691 | DDP (RDMA Read Request) Message Sequence Number | 3692 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3693 | DDP (RDMA Read Request) Message Offset | 3694 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3695 | Data Sink STag (SinkSTag) | 3696 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3697 | | 3698 + Data Sink Tagged Offset (SinkTO) + 3699 | | 3700 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3701 | RDMA Read Message Size (RDMARDSZ) | 3702 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3703 | Data Source STag (SrcSTag) | 3704 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3705 | | 3706 + Data Source Tagged Offset (SrcTO) + 3707 | | 3708 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3709 | MPA CRC | 3710 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3711 Figure 11 RDMA Read Request Message 3713 14.3.7 iWARP Message Format for Solicited SCSI Write Data 3715 The following figure depicts an iWARP RDMA Read Response Message 3716 carrying the solicited SCSI Write data in the payload: 3718 0 1 2 3 3719 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3720 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3721 | MPA Header | DDP Control | RDMA Control | 3722 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3723 | Data Sink STag | 3724 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3725 | Data Sink Tagged Offset | 3726 + + 3727 | | 3728 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3729 | SCSI Write Data | 3730 // // 3731 | | 3732 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3733 | MPA CRC | 3734 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3735 Figure 12 RDMA Read Response Message containing SCSI Write Data 3737 14.3.8 iWARP Message Format for SCSI Response PDU 3739 The following figure depicts a SCSI Response PDU embedded in an iSER 3740 Message encapsulated in an iWARP SendInvSE Message: 3742 0 1 2 3 3743 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 3744 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3745 | MPA Header | DDP Control | RDMA Control | 3746 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3747 | Invalidate STag | 3748 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3749 | (Send) Queue Number | 3750 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3751 | (Send) Message Sequence Number | 3752 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3753 | (Send) Message Offset | 3754 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3755 | 0001b |0|0| All Zeros | 3756 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3757 | All Zeros | 3758 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3759 | All Zeros | 3760 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3761 | SCSI Response PDU | 3762 // // 3763 | | 3764 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3765 | MPA CRC | 3766 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3767 Figure 13 SendInvSE Message containing SCSI Response PDU 3769 15 Appendix B 3771 15.1 Architectural discussion of iSER over InfiniBand 3773 This section explains how an InfiniBand network (with Gateways) 3774 would be structured. It is informational only and is intended to 3775 provide insight on how iSER is used in an InfiniBand environment. 3777 15.2 The Host side of the iSCSI & iSER connections in Infiniband 3779 Figure 14 defines the topologies in which iSCSI and iSER will be 3780 able to operate on an InfiniBand Network. 3782 +---------+ +---------+ +---------+ +---------+ +--- -----+ 3783 | Host | | Host | | Host | | Host | | Host | 3784 | | | | | | | | | | 3785 +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ 3786 |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| 3787 +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ 3788 |----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB 3789 IB| IB | IB | IB | IB | SubNet2 SWTCH 3790 +-v-----------v-----------v-----------v-----------v---------+ 3791 | InfiniBand Switch for Subnet1 | 3792 +---+-----+--------+-----+--------+-----+------------v------+ 3793 | TCA | | TCA | | TCA | | 3794 +-----+ +-----+ +-----+ | IB 3795 / IB \ / IB \ / \ +--+--v--+--+ 3796 | iSER | | iSER | | IPoIB | | | TCA | | 3797 | Gateway | | Gateway | | Gateway | | +-----+ | 3798 | to | | to | | to | | Storage | 3799 | iSCSI | | iSER | | IP | | Controller| 3800 | TCP | | iWARP | |Ethernet | +-----+-----+ 3801 +---v-----| +---v-----| +----v----+ 3802 | EN | EN | EN 3803 +--------------+---------------+----> to IP based storage 3804 Ethernet links that carry iSCSI or iWARP 3806 Figure 14 iSCSI and iSER on IB 3808 In Figure 14, the Host systems are connected via the InfiniBand Host 3809 Channel Adapters (HCAs) to the InfiniBand links. With the use of IB 3810 switch(es), the InfiniBand links connect the HCA to InfiniBand 3811 Target Channel Adapters (TCAs) located in gateways or Storage 3812 Controllers. An iSER-capable IB-IP Gateway converts the iSER 3813 Messages encapsulated in IB protocols to either standard iSCSI, or 3814 iSER Messages for iWARP. An [IPoIB] Gateway converts the InfiniBand 3815 [IPoIB] protocol to IP protocol, and in the iSCSI case, permits 3816 iSCSI to be operated on an IB Network between the Hosts and the 3817 [IPoIB] Gateway. 3819 15.3 The Storage side of iSCSI & iSER mixed network environment 3821 Figure 15 shows a storage controller that has three different portal 3822 groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP 3823 or iSCSI (TPG-2), and one supporting iSER/IB (TPG-1). 3825 | | | 3826 | | | 3827 +--+--v--+----------+--v--+----------+--v--+--+ 3828 | | IB | |iWARP| | EN | | 3829 | | | | TCP | | NIC | | 3830 | |(TCA)| | RNIC| | | | 3831 | +-----| +-----+ +-----+ | 3832 | TPG-1 TPG-2 TPG-4 | 3833 | 9.1.3.3 9.1.2.4 9.1.2.6 | 3834 | | 3835 | Storage Controller | 3836 | | 3837 +---------------------------------------------+ 3839 Figure 15 Storage Controller with TCP, iWARP, and IB Connections 3841 The normal iSCSI portal group advertising processes (via SLP, iSNS, 3842 or SendTargets) are available to a Storage Controller. 3844 15.4 Discovery processes for an InfiniBand Host 3846 An InfiniBand Host system can gather portal group IP address from 3847 SLP, iSNS, or the SendTargets discovery processes by using TCP/IP 3848 via [IPoIB]. After obtaining one or more remote portal IP 3849 addresses, the Initiator uses the standard IP mechanisms to resolve 3850 the IP address to a local outgoing interface and the destination 3851 hardware address (Ethernet MAC or IB GID of the target or a gateway 3852 leading to the target). If the resolved interface is an [IPoIB] 3853 network interface, then the target portal can be reached through an 3854 InfiniBand fabric. In this case the Initiator can establish an 3855 iSCSI/TCP or iSCSI/iSER session with the Target over that InfiniBand 3856 interface, using the Hardware Address (InfiniBand GID) obtained 3857 through the standard Address Resolution (ARP) processes. 3859 If more than one IP address are obtained through the discovery 3860 process, the Initiator should select a Target IP address that is on 3861 the same IP subnet as the Initiator if one exists. This will avoid 3862 a potential overhead of going through a gateway when a direct path 3863 exists. 3865 In addition a user can configure manual static IP route entries if a 3866 particular path to the target is preferred. 3868 15.5 IBTA Connection specifications 3870 It is outside the scope of this document, but it is expected that 3871 the InfiniBand Trade Association (IBTA) has or will define: 3873 * The iSER ServiceID 3875 * A Means for permitting a Host to establish a connection with a 3876 peer InfiniBand end-node, and that peer indicating when that 3877 end-node supports iSER, so the Host would be able to fall back 3878 to iSCSI/TCP over [IPoIB]. 3880 * A Means for permitting the Host to establish connections with 3881 IB iSER connections on storage controllers or IB iSER connected 3882 Gateways in preference to [IPoIB] connected Gateways/Bridges or 3883 connections to Target Storage Controllers that also accept 3884 iSCSI via [IPoIB]. 3886 * A Means for combining the IB ServiceID for iSER and the IP port 3887 number such that the IB Host can use normal IB connection 3888 processes, yet ensure that the iSER target peer can actually 3889 connect to the required IP port number. 3891 16 Acknowledgments 3893 Credit goes to the authors of [RFC5046], M. Ko, M. Chadalapaka, J. 3894 Hufferd, U. Elzur, H. Shah, and P. Thaler for coming up with the 3895 first version of the iSER specification. 3897 Author's Address 3899 Mike Ko 3900 Huawei Symantec 3901 20245 Stevens Creek Blvd. 3902 Cupertino, CA 95014, USA 3903 Phone: +1-408-510-7465 3904 Email: michael@huaweisymantec.com 3906 Copyright Notice 3908 Copyright (c) 2009 IETF Trust and the persons identified as the 3909 document authors. All rights reserved. 3911 This document is subject to BCP 78 and the IETF Trust's Legal 3912 Provisions Relating to IETF Documents 3913 (http://trustee.ietf.org/license-info) in effect on the date of 3914 publication of this document. Please review these documents 3915 carefully, as they describe your rights and restrictions with 3916 respect to this document. Code Components extracted from this 3917 document must include Simplified BSD License text as described in 3918 Section 4.e of the Trust Legal Provisions and are provided without 3919 warranty as described in the Simplified BSD License.