Network Working Group J. Satran Request for Comments: 3720 K. Meth Category: Standards Track IBM C. Sapuntzakis Cisco Systems M. Chadalapaka Hewlett-Packard Co. E. Zeidner IBM April 2004 Internet Small Computer Systems Interface (iSCSI) Status of this Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (2003). All Rights Reserved. Abstract This document describes a transport protocol for Internet Small Computer Systems Interface (iSCSI) that works on top of TCP. The iSCSI protocol aims to be fully compliant with the standardized SCSI architecture model. SCSI is a popular family of protocols that enable systems to communicate with I/O devices, especially storage devices. SCSI protocols are request/response application protocols with a common standardized architecture model and basic command set, as well as standardized command sets for different device classes (disks, tapes, media-changers etc.). As system interconnects move from the classical bus structure to a network structure, SCSI has to be mapped to network transport protocols. IP networks now meet the performance requirements of fast system interconnects and as such are good candidates to "carry" SCSI. Satran, et al. Standards Track [Page 1] RFC 3720 iSCSI April 2004 Table of Contents 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 9 2. Definitions and Acronyms. . . . . . . . . . . . . . . . . . . 10 2.1. Definitions. . . . . . . . . . . . . . . . . . . . . . 10 2.2. Acronyms . . . . . . . . . . . . . . . . . . . . . . . 14 2.3. Conventions. . . . . . . . . . . . . . . . . . . . . . 16 2.3.1. Word Rule. . . . . . . . . . . . . . . . . . 16 2.3.2. Half-Word Rule . . . . . . . . . . . . . . . 17 2.3.3. Byte Rule. . . . . . . . . . . . . . . . . . 17 3. Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.1. SCSI Concepts. . . . . . . . . . . . . . . . . . . . . 17 3.2. iSCSI Concepts and Functional Overview . . . . . . . . 18 3.2.1. Layers and Sessions. . . . . . . . . . . . . 19 3.2.2. Ordering and iSCSI Numbering . . . . . . . . 19 3.2.2.1. Command Numbering and Acknowledging . . . . . . . . . . 20 3.2.2.2. Response/Status Numbering and Acknowledging . . . . . . . . . . 23 3.2.2.3. Data Sequencing . . . . . . . . 24 3.2.3. iSCSI Login. . . . . . . . . . . . . . . . . 24 3.2.4. iSCSI Full Feature Phase . . . . . . . . . . 25 3.2.4.1. Command Connection Allegiance . . 26 3.2.4.2. Data Transfer Overview. . . . . . 27 3.2.4.3. Tags and Integrity Checks . . . . 28 3.2.4.4. Task Management . . . . . . . . . 28 3.2.5. iSCSI Connection Termination . . . . . . . . 29 3.2.6. iSCSI Names. . . . . . . . . . . . . . . . . 29 3.2.6.1. iSCSI Name Properties . . . . . . 30 3.2.6.2. iSCSI Name Encoding . . . . . . . 31 3.2.6.3. iSCSI Name Structure. . . . . . . 32 3.2.6.3.1. Type "iqn." (iSCSI Qualified Name) . . . 32 3.2.6.3.2. Type "eui." (IEEE EUI-64 format). . . . 34 3.2.7. Persistent State . . . . . . . . . . . . . . 34 3.2.8. Message Synchronization and Steering . . . . 35 3.2.8.1. Sync/Steering and iSCSI PDU Length . . . . . . . . . . . . . 36 3.3. iSCSI Session Types. . . . . . . . . . . . . . . . . . 36 3.4. SCSI to iSCSI Concepts Mapping Model . . . . . . . . . 37 3.4.1. iSCSI Architecture Model . . . . . . . . . . 37 3.4.2. SCSI Architecture Model. . . . . . . . . . . 39 3.4.3. Consequences of the Model. . . . . . . . . . 41 3.4.3.1. I_T Nexus State . . . . . . . . . 42 3.5. Request/Response Summary . . . . . . . . . . . . . . . 42 3.5.1. Request/Response Types Carrying SCSI Payload 43 3.5.1.1. SCSI-Command . . . . . . . . . . 43 Satran, et al. Standards Track [Page 2] RFC 3720 iSCSI April 2004 3.5.1.2. SCSI-Response . . . . . . . . . 43 3.5.1.3. Task Management Function Request. 44 3.5.1.4. Task Management Function Response 44 3.5.1.5. SCSI Data-Out and SCSI Data-In. . 44 3.5.1.6. Ready To Transfer (R2T) . . . . . 45 3.5.2. Requests/Responses carrying SCSI and iSCSI Payload. . . . . . . . . . . . . . . . . . . 46 3.5.2.1. Asynchronous Message. . . . . . . 46 3.5.3. Requests/Responses Carrying iSCSI Only Payload. . . . . . . . . . . . . . . . . . . 46 3.5.3.1. Text Request and Text Response. . 46 3.5.3.2. Login Request and Login Response. 47 3.5.3.3. Logout Request and Response . . . 47 3.5.3.4. SNACK Request . . . . . . . . . . 48 3.5.3.5. Reject. . . . . . . . . . . . . . 48 3.5.3.6. NOP-Out Request and NOP-In Response . . . . . . . . . . . . 48 4. SCSI Mode Parameters for iSCSI. . . . . . . . . . . . . . . . 48 5. Login and Full Feature Phase Negotiation. . . . . . . . . . . 48 5.1. Text Format. . . . . . . . . . . . . . . . . . . . . . 50 5.2. Text Mode Negotiation. . . . . . . . . . . . . . . . . 53 5.2.1. List negotiations. . . . . . . . . . . . . . 56 5.2.2. Simple-value Negotiations. . . . . . . . . . 56 5.3. Login Phase. . . . . . . . . . . . . . . . . . . . . . 57 5.3.1. Login Phase Start. . . . . . . . . . . . . . 60 5.3.2. iSCSI Security Negotiation . . . . . . . . . 62 5.3.3. Operational Parameter Negotiation During the Login Phase. . . . . . . . . . . . . . . 63 5.3.4. Connection Reinstatement . . . . . . . . . . 64 5.3.5. Session Reinstatement, Closure, and Timeout. 64 5 5.3.5.1. Loss of Nexus Notification. . . . . 65 5.3.6. Session Continuation and Failure . . . . . . 65 5.4. Operational Parameter Negotiation Outside the Login Phase. . . . . . . . . . . . . . . . . . . . . . . . . 66 6. iSCSI Error Handling and Recovery . . . . . . . . . . . . . . 67 6.1. Overview . . . . . . . . . . . . . . . . . . . . . . . 67 6.1.1. Background . . . . . . . . . . . . . . . . . 67 6.1.2. Goals. . . . . . . . . . . . . . . . . . . . 67 6.1.3. Protocol Features and State Expectations . . 68 6.1.4. Recovery Classes . . . . . . . . . . . . . . 69 6.1.4.1. Recovery Within-command . . . . . 69 6.1.4.2. Recovery Within-connection. . . . 70 6.1.4.3. Connection Recovery . . . . . . . 71 6.1.4.4. Session Recovery. . . . . . . . . 72 6.1.5. Error Recovery Hierarchy . . . . . . . . . . . 72 6.2. Retry and Reassign in Recovery . . . . . . . . . . . . 74 6.2.1. Usage of Retry . . . . . . . . . . . . . . . 74 Satran, et al. Standards Track [Page 3] RFC 3720 iSCSI April 2004 6.2.2. Allegiance Reassignment. . . . . . . . . . . 75 6.3. Usage Of Reject PDU in Recovery. . . . . . . . . . . . 76 6.4. Connection Timeout Management. . . . . . . . . . . . . 76 6.4.1. Timeouts on Transport Exception Events . . . 77 6.4.2. Timeouts on Planned Decommissioning. . . . . 77 6.5. Implicit Termination of Tasks. . . . . . . . . . . . . 77 6.6. Format Errors. . . . . . . . . . . . . . . . . . . . . 78 6.7. Digest Errors. . . . . . . . . . . . . . . . . . . . . 78 6.8. Sequence Errors. . . . . . . . . . . . . . . . . . . . 80 6.9. SCSI Timeouts. . . . . . . . . . . . . . . . . . . . . 81 6.10. Negotiation Failures . . . . . . . . . . . . . . . . . 81 6.11. Protocol Errors. . . . . . . . . . . . . . . . . . . . 82 6.12. Connection Failures. . . . . . . . . . . . . . . . . . 82 6.13. Session Errors . . . . . . . . . . . . . . . . . . . . 83 7. State Transitions . . . . . . . . . . . . . . . . . . . . . . 84 7.1. Standard Connection State Diagrams . . . . . . . . . . 84 7.1.1. State Descriptions for Initiators and Targets. . . . . . . . . . . . . . . . . . . 84 7.1.2. State Transition Descriptions for Initiators and Targets. . . . . . . . . . . . . . . . . 85 7.1.3. Standard Connection State Diagram for an Initiator. . . . . . . . . . . . . . . . . . 88 7.1.4. Standard Connection State Diagram for a Target . . . . . . . . . . . . . . . . . . . 90 7.2. Connection Cleanup State Diagram for Initiators and Targets. . . . . . . . . . . . . . . . . . . . . . . . 92 7.2.1. State Descriptions for Initiators and Targets. . . . . . . . . . . . . . . . . . . 94 7.2.2. State Transition Descriptions for Initiators and Targets. . . . . . . . . . . . . . . . . 94 7.3. Session State Diagrams . . . . . . . . . . . . . . . . 95 7.3.1. Session State Diagram for an Initiator . . . 95 7.3.2. Session State Diagram for a Target . . . . . 96 7.3.3. State Descriptions for Initiators and Targets. . . . . . . . . . . . . . . . . . . 97 7.3.4. State Transition Descriptions for Initiators and Targets. . . . . . . . . . . . . . . . . 98 8. Security Considerations . . . . . . . . . . . . . . . . . . . 99 8.1. iSCSI Security Mechanisms. . . . . . . . . . . . . . . 100 8.2. In-band Initiator-Target Authentication. . . . . . . . 100 8.2.1. CHAP Considerations. . . . . . . . . . . . . 101 8.2.2. SRP Considerations . . . . . . . . . . . . . 103 8.3. IPsec. . . . . . . . . . . . . . . . . . . . . . . . . 104 8.3.1. Data Integrity and Authentication. . . . . . 104 8.3.2. Confidentiality. . . . . . . . . . . . . . . 105 8.3.3. Policy, Security Associations, and Cryptographic Key Management . . . . . . . . 105 9. Notes to Implementers . . . . . . . . . . . . . . . . . . . . 106 Satran, et al. Standards Track [Page 4] RFC 3720 iSCSI April 2004 9.1. Multiple Network Adapters. . . . . . . . . . . . . . . 106 9.1.1. Conservative Reuse of ISIDs. . . . . . . . . 107 9.1.2. iSCSI Name, ISID, and TPGT Use . . . . . . . 107 9.2. Autosense and Auto Contingent Allegiance (ACA) . . . . 109 9.3. iSCSI Timeouts . . . . . . . . . . . . . . . . . . . . 109 9.4. Command Retry and Cleaning Old Command Instances . . . 110 9.5. Synch and Steering Layer and Performance . . . . . . . 110 9.6. Considerations for State-dependent Devices and Long-lasting SCSI Operations . . . . . . . . . . . . . 111 9.6.1. Determining the Proper ErrorRecoveryLevel. . 112 10. iSCSI PDU Formats . . . . . . . . . . . . . . . . . . . . . . 112 10.1. iSCSI PDU Length and Padding . . . . . . . . . . . . . 113 10.2. PDU Template, Header, and Opcodes. . . . . . . . . . . 113 10.2.1. Basic Header Segment (BHS) . . . . . . . . . 114 10.2.1.1. I . . . . . . . . . . . . . . . . 115 10.2.1.2. Opcode. . . . . . . . . . . . . . 115 10.2.1.3. Final (F) bit . . . . . . . . . . 116 10.2.1.4. Opcode-specific Fields. . . . . . 116 10.2.1.5. TotalAHSLength. . . . . . . . . . 116 10.2.1.6. DataSegmentLength . . . . . . . . 116 10.2.1.7. LUN . . . . . . . . . . . . . . . 116 10.2.1.8. Initiator Task Tag. . . . . . . . 117 10.2.2. Additional Header Segment (AHS) . . . . . . . 117 10.2.2.1. AHSType . . . . . . . . . . . . . 117 10.2.2.2. AHSLength . . . . . . . . . . . . 117 10.2.2.3. Extended CDB AHS. . . . . . . . . 118 10.2.2.4. Bidirectional Expected Read-Data Length AHS. . . . . . . . . . . . 118 10.2.3. Header Digest and Data Digest. . . . . . . . 118 10.2.4. Data Segment . . . . . . . . . . . . . . . . 119 10.3. SCSI Command . . . . . . . . . . . . . . . . . . . . . 119 10.3.1. Flags and Task Attributes (byte 1) . . . . . 120 10.3.2. CmdSN - Command Sequence Number. . . . . . . 120 10.3.3. ExpStatSN. . . . . . . . . . . . . . . . . . 120 10.3.4. Expected Data Transfer Length. . . . . . . . 121 10.3.5. CDB - SCSI Command Descriptor Block. . . . . 121 10.3.6. Data Segment - Command Data. . . . . . . . . 121 10.4. SCSI Response. . . . . . . . . . . . . . . . . . . . . 122 10.4.1. Flags (byte 1) . . . . . . . . . . . . . . . 123 10.4.2. Status . . . . . . . . . . . . . . . . . . . 123 10.4.3. Response . . . . . . . . . . . . . . . . . . 124 10.4.4. SNACK Tag. . . . . . . . . . . . . . . . . . 125 10.4.5. Residual Count . . . . . . . . . . . . . . . 125 10.4.6. Bidirectional Read Residual Count. . . . . . 125 10.4.7. Data Segment - Sense and Response Data Segment. . . . . . . . . . . . . . . . . . . 125 10.4.7.1. SenseLength . . . . . . . . . . . 126 10.4.7.2. Sense Data. . . . . . . . . . . . 126 Satran, et al. Standards Track [Page 5] RFC 3720 iSCSI April 2004 10.4.8. ExpDataSN. . . . . . . . . . . . . . . . . . 127 10.4.9. StatSN - Status Sequence Number. . . . . . . 127 10.4.10. ExpCmdSN - Next Expected CmdSN from this Initiator. . . . . . . . . . . . . . . . . . 128 10.4.11. MaxCmdSN - Maximum CmdSN from this Initiator 128 10.5. Task Management Function Request . . . . . . . . . . . 129 10.5.1. Function . . . . . . . . . . . . . . . . . . 129 10.5.2. TotalAHSLength and DataSegmentLength . . . . 132 10.5.3. LUN. . . . . . . . . . . . . . . . . . . . . 132 10.5.4. Referenced Task Tag. . . . . . . . . . . . . 132 10.5.5. RefCmdSN . . . . . . . . . . . . . . . . . . 132 10.5.6. ExpDataSN. . . . . . . . . . . . . . . . . . 133 10.6. Task Management Function Response. . . . . . . . . . . 134 10.6.1. Response . . . . . . . . . . . . . . . . . . 134 10.6.2. Task Management Actions on Task Sets . . . . 136 10.6.3. TotalAHSLength and DataSegmentLength . . . . 137 10.7. SCSI Data-Out & SCSI Data-In . . . . . . . . . . . . . 137 10.7.1. F (Final) Bit. . . . . . . . . . . . . . . . 139 10.7.2. A (Acknowledge) Bit. . . . . . . . . . . . . 139 10.7.3. Flags (byte 1) . . . . . . . . . . . . . . . 140 10.7.4. Target Transfer Tag and LUN. . . . . . . . . 140 10.7.5. DataSN . . . . . . . . . . . . . . . . . . . 141 10.7.6. Buffer Offset. . . . . . . . . . . . . . . . 141 10.7.7. DataSegmentLength. . . . . . . . . . . . . . 141 10.8. Ready To Transfer (R2T). . . . . . . . . . . . . . . . 142 10.8.1. TotalAHSLength and DataSegmentLength . . . . 143 10.8.2. R2TSN. . . . . . . . . . . . . . . . . . . . 143 10.8.3. StatSN . . . . . . . . . . . . . . . . . . . 144 10.8.4. Desired Data Transfer Length and Buffer Offset . . . . . . . . . . . . . . . . . . . 144 10.8.5. Target Transfer Tag. . . . . . . . . . . . . 144 10.9. Asynchronous Message . . . . . . . . . . . . . . . . . 145 10.9.1. AsyncEvent . . . . . . . . . . . . . . . . . 146 10.9.2. AsyncVCode . . . . . . . . . . . . . . . . . 147 10.9.3. LUN. . . . . . . . . . . . . . . . . . . . . 147 10.9.4. Sense Data and iSCSI Event Data. . . . . . . 148 10.9.4.1. SenseLength . . . . . . . . . . . 148 10.10. Text Request . . . . . . . . . . . . . . . . . . . . . 149 10.10.1. F (Final) Bit. . . . . . . . . . . . . . . . 150 10.10.2. C (Continue) Bit . . . . . . . . . . . . . . 150 10.10.3. Initiator Task Tag . . . . . . . . . . . . . 150 10.10.4. Target Transfer Tag. . . . . . . . . . . . . 150 10.10.5. Text . . . . . . . . . . . . . . . . . . . . 151 10.11. Text Response. . . . . . . . . . . . . . . . . . . . . 152 10.11.1. F (Final) Bit. . . . . . . . . . . . . . . . 152 10.11.2. C (Continue) Bit . . . . . . . . . . . . . . 153 10.11.3. Initiator Task Tag . . . . . . . . . . . . . 153 10.11.4. Target Transfer Tag. . . . . . . . . . . . . 153 Satran, et al. Standards Track [Page 6] RFC 3720 iSCSI April 2004 10.11.5. StatSN . . . . . . . . . . . . . . . . . . . 154 10.11.6. Text Response Data . . . . . . . . . . . . . 154 10.12. Login Request. . . . . . . . . . . . . . . . . . . . . 154 10.12.1. T (Transit) Bit. . . . . . . . . . . . . . . 155 10.12.2. C (Continue) Bit . . . . . . . . . . . . . . 155 10.12.3. CSG and NSG. . . . . . . . . . . . . . . . . 156 10.12.4. Version. . . . . . . . . . . . . . . . . . . 156 10.12.4.1. Version-max. . . . . . . . . . . 156 10.12.4.2. Version-min. . . . . . . . . . . 156 10.12.5. ISID . . . . . . . . . . . . . . . . . . . . 157 10.12.6. TSIH . . . . . . . . . . . . . . . . . . . . 158 10.12.7. Connection ID - CID. . . . . . . . . . . . . 158 10.12.8. CmdSN. . . . . . . . . . . . . . . . . . . . 159 10.12.9. ExpStatSN. . . . . . . . . . . . . . . . . . 159 10.12.10. Login Parameters . . . . . . . . . . . . . . 159 10.13. Login Response . . . . . . . . . . . . . . . . . . . . 160 10.13.1. Version-max. . . . . . . . . . . . . . . . . 160 10.13.2. Version-active . . . . . . . . . . . . . . . 161 10.13.3. TSIH . . . . . . . . . . . . . . . . . . . . 161 10.13.4. StatSN . . . . . . . . . . . . . . . . . . . 161 10.13.5. Status-Class and Status-Detail . . . . . . . 161 10.13.6. T (Transit) Bit. . . . . . . . . . . . . . . 164 10.13.7. C (Continue) Bit . . . . . . . . . . . . . . 164 10.13.8. Login Parameters . . . . . . . . . . . . . . 164 10.14. Logout Request . . . . . . . . . . . . . . . . . . . . 165 10.14.1. Reason Code. . . . . . . . . . . . . . . . . 167 10.14.2. TotalAHSLength and DataSegmentLength . . . . 168 10.14.3. CID. . . . . . . . . . . . . . . . . . . . . 168 10.14.4. ExpStatSN. . . . . . . . . . . . . . . . . . 168 10.14.5. Implicit termination of tasks. . . . . . . . 168 10.15. Logout Response. . . . . . . . . . . . . . . . . . . . 169 10.15.1. Response . . . . . . . . . . . . . . . . . . 170 10.15.2. TotalAHSLength and DataSegmentLength . . . . 170 10.15.3. Time2Wait. . . . . . . . . . . . . . . . . . 170 10.15.4. Time2Retain. . . . . . . . . . . . . . . . . 170 10.16. SNACK Request. . . . . . . . . . . . . . . . . . . . . 171 10.16.1. Type . . . . . . . . . . . . . . . . . . . . 172 10.16.2. Data Acknowledgement . . . . . . . . . . . . 173 10.16.3. Resegmentation . . . . . . . . . . . . . . . 173 10.16.4. Initiator Task Tag . . . . . . . . . . . . . 174 10.16.5. Target Transfer Tag or SNACK Tag . . . . . . 174 10.16.6. BegRun . . . . . . . . . . . . . . . . . . . 174 10.16.7. RunLength. . . . . . . . . . . . . . . . . . 174 10.17. Reject . . . . . . . . . . . . . . . . . . . . . . . . 175 10.17.1. Reason . . . . . . . . . . . . . . . . . . . 176 10.17.2. DataSN/R2TSN . . . . . . . . . . . . . . . . 177 10.17.3. StatSN, ExpCmdSN and MaxCmdSN. . . . . . . . 177 10.17.4. Complete Header of Bad PDU . . . . . . . . . 177 Satran, et al. Standards Track [Page 7] RFC 3720 iSCSI April 2004 10.18. NOP-Out. . . . . . . . . . . . . . . . . . . . . . . . 178 10.18.1. Initiator Task Tag . . . . . . . . . . . . . 179 10.18.2. Target Transfer Tag. . . . . . . . . . . . . 179 10.18.3. Ping Data. . . . . . . . . . . . . . . . . . 179 10.19. NOP-In . . . . . . . . . . . . . . . . . . . . . . . . 180 10.19.1. Target Transfer Tag. . . . . . . . . . . . . 181 10.19.2. StatSN . . . . . . . . . . . . . . . . . . . 181 10.19.3. LUN. . . . . . . . . . . . . . . . . . . . . 181 11. iSCSI Security Text Keys and Authentication Methods . . . . . 181 11.1. AuthMethod . . . . . . . . . . . . . . . . . . . . . . 182 11.1.1. Kerberos . . . . . . . . . . . . . . . . . . 184 11.1.2. Simple Public-Key Mechanism (SPKM) . . . . . 184 11.1.3. Secure Remote Password (SRP) . . . . . . . . 185 11.1.4. Challenge Handshake Authentication Protocol (CHAP) . . . . . . . . . . . . . . . . . . . 186 12. Login/Text Operational Text Keys. . . . . . . . . . . . . . . 187 12.1. HeaderDigest and DataDigest. . . . . . . . . . . . . . 188 12.2. MaxConnections . . . . . . . . . . . . . . . . . . . . 190 12.3. SendTargets. . . . . . . . . . . . . . . . . . . . . . 191 12.4. TargetName . . . . . . . . . . . . . . . . . . . . . . 191 12.5. InitiatorName. . . . . . . . . . . . . . . . . . . . . 192 12.6. TargetAlias. . . . . . . . . . . . . . . . . . . . . . 192 12.7. InitiatorAlias . . . . . . . . . . . . . . . . . . . . 193 12.8. TargetAddress. . . . . . . . . . . . . . . . . . . . . 193 12.9. TargetPortalGroupTag . . . . . . . . . . . . . . . . . 194 12.10. InitialR2T . . . . . . . . . . . . . . . . . . . . . . 194 12.11. ImmediateData. . . . . . . . . . . . . . . . . . . . . 195 12.12. MaxRecvDataSegmentLength . . . . . . . . . . . . . . . 196 12.13. MaxBurstLength . . . . . . . . . . . . . . . . . . . . 196 12.14. FirstBurstLength . . . . . . . . . . . . . . . . . . . 197 12.15. DefaultTime2Wait . . . . . . . . . . . . . . . . . . . 197 12.16. DefaultTime2Retain . . . . . . . . . . . . . . . . . . 198 12.17. MaxOutstandingR2T. . . . . . . . . . . . . . . . . . . 198 12.18. DataPDUInOrder . . . . . . . . . . . . . . . . . . . . 198 12.19. DataSequenceInOrder. . . . . . . . . . . . . . . . . . 199 12.20. ErrorRecoveryLevel . . . . . . . . . . . . . . . . . . 199 12.21. SessionType. . . . . . . . . . . . . . . . . . . . . . 200 12.22. The Private or Public Extension Key Format . . . . . . 200 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 201 13.1. Naming Requirements. . . . . . . . . . . . . . . . . . 203 13.2. Mechanism Specification Requirements . . . . . . . . . 203 13.3. Publication Requirements . . . . . . . . . . . . . . . 203 13.4. Security Requirements. . . . . . . . . . . . . . . . . 203 13.5. Registration Procedure . . . . . . . . . . . . . . . . 204 13.5.1. Present the iSCSI extension item to the Community. . . . . . . . . . . . . . . . . . 204 13.5.2. iSCSI extension item review and IESG approval . . . . . . . . . . . . . . . . . . 204 Satran, et al. Standards Track [Page 8] RFC 3720 iSCSI April 2004 13.5.3. IANA Registration. . . . . . . . . . . . . . 204 13.5.4. Standard iSCSI extension item-label format . 204 13.6. IANA Procedures for Registering iSCSI extension items. 205 References. . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 Appendix A. Sync and Steering with Fixed Interval Markers . . . . 209 A.1. Markers At Fixed Intervals . . . . . . . . . . . . . . 209 A.2. Initial Marker-less Interval . . . . . . . . . . . . . 210 A.3. Negotiation. . . . . . . . . . . . . . . . . . . . . . 210 A.3.1. OFMarker, IFMarker . . . . . . . . . . . . . 210 A.3.2. OFMarkInt, IFMarkInt . . . . . . . . . . . . 211 Appendix B. Examples . . . . . . . . . . . . . . . . . . . . . . 212 B.1. Read Operation Example . . . . . . . . . . . . . . . . 212 B.2. Write Operation Example. . . . . . . . . . . . . . . . 213 B.3. R2TSN/DataSN Use Examples. . . . . . . . . . . . . . . 214 B.4. CRC Examples . . . . . . . . . . . . . . . . . . . . . 217 Appendix C. Login Phase Examples . . . . . . . . . . . . . . . . 219 Appendix D. SendTargets Operation. . . . . . . . . . . . . . . . 229 Appendix E. Algorithmic Presentation of Error Recovery Classes . 233 E.1. General Data Structure and Procedure Description . . . 233 E.2. Within-command Error Recovery Algorithms . . . . . . . 234 E.2.1. Procedure Descriptions . . . . . . . . . . . 234 E.2.2. Initiator Algorithms . . . . . . . . . . . . 235 E.2.3. Target Algorithms. . . . . . . . . . . . . . 237 E.3. Within-connection Recovery Algorithms. . . . . . . . . 240 E.3.1. Procedure Descriptions . . . . . . . . . . . 240 E.3.2. Initiator Algorithms . . . . . . . . . . . . 241 E.3.3. Target Algorithms. . . . . . . . . . . . . . 243 E.4. Connection Recovery Algorithms . . . . . . . . . . . . 243 E.4.1. Procedure Descriptions . . . . . . . . . . . 243 E.4.2. Initiator Algorithms . . . . . . . . . . . . 244 E.4.3. Target Algorithms. . . . . . . . . . . . . . 246 Appendix F. Clearing Effects of Various Events on Targets. . . . 249 F.1. Clearing Effects on iSCSI Objects. . . . . . . . . . . 249 F.2. Clearing Effects on SCSI Objects . . . . . . . . . . . 253 Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . 254 Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . . . 256 Full Copyright Statement. . . . . . . . . . . . . . . . . . . . . 257 1. Introduction The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. SCSI is a client-server architecture. Clients of a SCSI interface are called "initiators". Initiators issue SCSI "commands" to request services from components, logical units of a server known as a "target". A "SCSI transport" maps the client-server SCSI protocol to a specific interconnect. An Initiator is one endpoint of a SCSI transport and a target is the other endpoint. Satran, et al. Standards Track [Page 9] RFC 3720 iSCSI April 2004 The SCSI protocol has been mapped over various transports, including Parallel SCSI, IPI, IEEE-1394 (firewire) and Fibre Channel. These transports are I/O specific and have limited distance capabilities. The iSCSI protocol defined in this document describes a means of transporting SCSI packets over TCP/IP (see [RFC791], [RFC793], [RFC1035], [RFC1122]), providing for an interoperable solution which can take advantage of existing Internet infrastructure, Internet management facilities, and address distance limitations. 2. Definitions and Acronyms 2.1. Definitions - Alias: An alias string can also be associated with an iSCSI Node. The alias allows an organization to associate a user-friendly string with the iSCSI Name. However, the alias string is not a substitute for the iSCSI Name. - CID (Connection ID): Connections within a session are identified by a connection ID. It is a unique ID for this connection within the session for the initiator. It is generated by the initiator and presented to the target during login requests and during logouts that close connections. - Connection: A connection is a TCP connection. Communication between the initiator and target occurs over one or more TCP connections. The TCP connections carry control messages, SCSI commands, parameters, and data within iSCSI Protocol Data Units (iSCSI PDUs). - iSCSI Device: A SCSI Device using an iSCSI service delivery subsystem. Service Delivery Subsystem is defined by [SAM2] as a transport mechanism for SCSI commands and responses. - iSCSI Initiator Name: The iSCSI Initiator Name specifies the worldwide unique name of the initiator. - iSCSI Initiator Node: The "initiator". The word "initiator" has been appropriately qualified as either a port or a device in the rest of the document when the context is ambiguous. All unqualified usages of "initiator" refer to an initiator port (or device) depending on the context. - iSCSI Layer: This layer builds/receives iSCSI PDUs and relays/receives them to/from one or more TCP connections that form an initiator-target "session". Satran, et al. Standards Track [Page 10] RFC 3720 iSCSI April 2004 - iSCSI Name: The name of an iSCSI initiator or iSCSI target. - iSCSI Node: The iSCSI Node represents a single iSCSI initiator or iSCSI target. There are one or more iSCSI Nodes within a Network Entity. The iSCSI Node is accessible via one or more Network Portals. An iSCSI Node is identified by its iSCSI Name. The separation of the iSCSI Name from the addresses used by and for the iSCSI Node allows multiple iSCSI Nodes to use the same address, and the same iSCSI Node to use multiple addresses. - iSCSI Target Name: The iSCSI Target Name specifies the worldwide unique name of the target. - iSCSI Target Node: The "target". - iSCSI Task: An iSCSI task is an iSCSI request for which a response is expected. - iSCSI Transfer Direction: The iSCSI transfer direction is defined with regard to the initiator. Outbound or outgoing transfers are transfers from the initiator to the target, while inbound or incoming transfers are from the target to the initiator. - ISID: The initiator part of the Session Identifier. It is explicitly specified by the initiator during Login. - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + ',i,'+ ISID, iSCSI Target Name + ',t,'+ Portal Group Tag). - Network Entity: The Network Entity represents a device or gateway that is accessible from the IP network. A Network Entity must have one or more Network Portals, each of which can be used to gain access to the IP network by some iSCSI Nodes contained in that Network Entity. - Network Portal: The Network Portal is a component of a Network Entity that has a TCP/IP network address and that may be used by an iSCSI Node within that Network Entity for the connection(s) within one of its iSCSI sessions. A Network Portal in an initiator is identified by its IP address. A Network Portal in a target is identified by its IP address and its listening TCP port. Satran, et al. Standards Track [Page 11] RFC 3720 iSCSI April 2004 - Originator: In a negotiation or exchange, the party that initiates the negotiation or exchange. - PDU (Protocol Data Unit): The initiator and target divide their communications into messages. The term "iSCSI protocol data unit" (iSCSI PDU) is used for these messages. - Portal Groups: iSCSI supports multiple connections within the same session; some implementations will have the ability to combine connections in a session across multiple Network Portals. A Portal Group defines a set of Network Portals within an iSCSI Network Entity that collectively supports the capability of coordinating a session with connections spanning these portals. Not all Network Portals within a Portal Group need participate in every session connected through that Portal Group. One or more Portal Groups may provide access to an iSCSI Node. Each Network Portal, as utilized by a given iSCSI Node, belongs to exactly one portal group within that node. - Portal Group Tag: This 16-bit quantity identifies a Portal Group within an iSCSI Node. All Network Portals with the same portal group tag in the context of a given iSCSI Node are in the same Portal Group. - Recovery R2T: An R2T generated by a target upon detecting the loss of one or more Data-Out PDUs through one of the following means: a digest error, a sequence error, or a sequence reception timeout. A recovery R2T carries the next unused R2TSN, but requests all or part of the data burst that an earlier R2T (with a lower R2TSN) had already requested. - Responder: In a negotiation or exchange, the party that responds to the originator of the negotiation or exchange. - SCSI Device: This is the SAM2 term for an entity that contains one or more SCSI ports that are connected to a service delivery subsystem and supports a SCSI application protocol. For example, a SCSI Initiator Device contains one or more SCSI Initiator Ports and zero or more application clients. A Target Device contains one or more SCSI Target Ports and one or more device servers and associated logical units. For iSCSI, the SCSI Device is the component within an iSCSI Node that provides the SCSI functionality. As such, there can be at most, one SCSI Device within a given iSCSI Node. Access to the SCSI Device can only be achieved in an iSCSI normal operational session. The SCSI Device Name is defined to be the iSCSI Name of the node. Satran, et al. Standards Track [Page 12] RFC 3720 iSCSI April 2004 - SCSI Layer: This builds/receives SCSI CDBs (Command Descriptor Blocks) and relays/receives them with the remaining command execute [SAM2] parameters to/from the iSCSI Layer. - Session: The group of TCP connections that link an initiator with a target form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one and the same target. - SCSI Initiator Port: This maps to the endpoint of an iSCSI normal operational session. An iSCSI normal operational session is negotiated through the login process between an iSCSI initiator node and an iSCSI target node. At successful completion of this process, a SCSI Initiator Port is created within the SCSI Initiator Device. The SCSI Initiator Port Name and SCSI Initiator Port Identifier are both defined to be the iSCSI Initiator Name together with (a) a label that identifies it as an initiator port name/identifier and (b) the ISID portion of the session identifier. - SCSI Port: This is the SAM2 term for an entity in a SCSI Device that provides the SCSI functionality to interface with a service delivery subsystem. For iSCSI, the definition of the SCSI Initiator Port and the SCSI Target Port are different. - SCSI Port Name: A name made up as UTF-8 [RFC2279] characters and includes the iSCSI Name + 'i' or 't' + ISID or Portal Group Tag. - SCSI Target Port: This maps to an iSCSI Target Portal Group. - SCSI Target Port Name and SCSI Target Port Identifier: These are both defined to be the iSCSI Target Name together with (a) a label that identifies it as a target port name/identifier and (b) the portal group tag. - SSID (Session ID): A session between an iSCSI initiator and an iSCSI target is defined by a session ID that is a tuple composed of an initiator part (ISID) and a target part (Target Portal Group Tag). The ISID is explicitly specified by the initiator at session establishment. The Target Portal Group Tag is implied by the initiator through the selection of the TCP endpoint at connection establishment. The TargetPortalGroupTag key must also be returned by the target as a confirmation during connection establishment when TargetName is given. - Target Portal Group Tag: A numerical identifier (16-bit) for an iSCSI Target Portal Group. Satran, et al. Standards Track [Page 13] RFC 3720 iSCSI April 2004 - TSIH (Target Session Identifying Handle): A target assigned tag for a session with a specific named initiator. The target generates it during session establishment. Its internal format and content are not defined by this protocol, except for the value 0 that is reserved and used by the initiator to indicate a new session. It is given to the target during additional connection establishment for the same session. 2.2. Acronyms Acronym Definition ------------------------------------------------------------ 3DES Triple Data Encryption Standard ACA Auto Contingent Allegiance AEN Asynchronous Event Notification AES Advanced Encryption Standard AH Additional Header (not the IPsec AH!) AHS Additional Header Segment API Application Programming Interface ASC Additional Sense Code ASCII American Standard Code for Information Interchange ASCQ Additional Sense Code Qualifier BHS Basic Header Segment CBC Cipher Block Chaining CD Compact Disk CDB Command Descriptor Block CHAP Challenge Handshake Authentication Protocol CID Connection ID CO Connection Only CRC Cyclic Redundancy Check CRL Certificate Revocation List CSG Current Stage CSM Connection State Machine DES Data Encryption Standard DNS Domain Name Server DOI Domain of Interpretation DVD Digital Versatile Disk ESP Encapsulating Security Payload EUI Extended Unique Identifier FFP Full Feature Phase FFPO Full Feature Phase Only FIM Fixed Interval Marker Gbps Gigabits per Second HBA Host Bus Adapter HMAC Hashed Message Authentication Code I_T Initiator_Target I_T_L Initiator_Target_LUN IANA Internet Assigned Numbers Authority Satran, et al. Standards Track [Page 14] RFC 3720 iSCSI April 2004 ID Identifier IDN Internationalized Domain Name IEEE Institute of Electrical & Electronics Engineers IETF Internet Engineering Task Force IKE Internet Key Exchange I/O Input - Output IO Initialize Only IP Internet Protocol IPsec Internet Protocol Security IPv4 Internet Protocol Version 4 IPv6 Internet Protocol Version 6 IQN iSCSI Qualified Name ISID Initiator Session ID ITN iSCSI Target Name ITT Initiator Task Tag KRB5 Kerberos V5 LFL Lower Functional Layer LTDS Logical-Text-Data-Segment LO Leading Only LU Logical Unit LUN Logical Unit Number MAC Message Authentication Codes NA Not Applicable NIC Network Interface Card NOP No Operation NSG Next Stage OS Operating System PDU Protocol Data Unit PKI Public Key Infrastructure R2T Ready To Transfer R2TSN Ready To Transfer Sequence Number RDMA Remote Direct Memory Access RFC Request For Comments SAM SCSI Architecture Model SAM2 SCSI Architecture Model - 2 SAN Storage Area Network SCSI Small Computer Systems Interface SN Sequence Number SNACK Selective Negative Acknowledgment - also Sequence Number Acknowledgement for data SPKM Simple Public-Key Mechanism SRP Secure Remote Password SSID Session ID SW Session Wide TCB Task Control Block TCP Transmission Control Protocol TPGT Target Portal Group Tag TSIH Target Session Identifying Handle Satran, et al. Standards Track [Page 15] RFC 3720 iSCSI April 2004 TTT Target Transfer Tag UFL Upper Functional Layer ULP Upper Level Protocol URN Uniform Resource Names [RFC2396] UTF Universal Transformation Format WG Working Group 2.3. Conventions In examples, "I->" and "T->" show iSCSI PDUs sent by the initiator and target respectively. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119]. iSCSI messages - PDUs - are represented by diagrams as in the following example: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| Basic Header Segment (BHS) | +---------------+---------------+---------------+---------------+ ---------- +| | +---------------+---------------+---------------+---------------+ The diagrams include byte and bit numbering. The following representation and ordering rules are observed in this document: - Word Rule - Half-word Rule - Byte Rule 2.3.1. Word Rule A word holds four consecutive bytes. Whenever a word has numeric content, it is considered an unsigned number in base 2 positional representation with the lowest numbered byte (e.g., byte 0) bit 0 representing 2**31 and bit 1 representing 2**30 through lowest numbered byte + 3 (e.g., byte 3) bit 7 representing 2**0. Decimal and hexadecimal representation of word values map this representation to decimal or hexadecimal positional notation. Satran, et al. Standards Track [Page 16] RFC 3720 iSCSI April 2004 2.3.2. Half-Word Rule A half-word holds two consecutive bytes. Whenever a half-word has numeric content it is considered an unsigned number in base 2 positional representation with the lowest numbered byte (e.g., byte 0), bit 0 representing 2**15 and bit 1 representing 2**14 through lowest numbered byte + 1 (e.g., byte 1), bit 7 representing 2**0. Decimal and hexadecimal representation of half-word values map this representation to decimal or hexadecimal positional notation. 2.3.3. Byte Rule For every PDU, bytes are sent and received in increasing numbered order (network order). Whenever a byte has numerical content, it is considered an unsigned number in base 2 positional representation with bit 0 representing 2**7 and bit 1 representing 2**6 through bit 7 representing 2**0. 3. Overview 3.1. SCSI Concepts The SCSI Architecture Model-2 [SAM2] describes in detail the architecture of the SCSI family of I/O protocols. This section provides a brief background of the SCSI architecture and is intended to familiarize readers with its terminology. At the highest level, SCSI is a family of interfaces for requesting services from I/O devices, including hard drives, tape drives, CD and DVD drives, printers, and scanners. In SCSI terminology, an individual I/O device is called a "logical unit" (LU). SCSI is a client-server architecture. Clients of a SCSI interface are called "initiators". Initiators issue SCSI "commands" to request services from components, logical units, of a server known as a "target". The "device server" on the logical unit accepts SCSI commands and processes them. A "SCSI transport" maps the client-server SCSI protocol to a specific interconnect. Initiators are one endpoint of a SCSI transport. The "target" is the other endpoint. A target can contain multiple Logical Units (LUs). Each Logical Unit has an address within a target called a Logical Unit Number (LUN). A SCSI task is a SCSI command or possibly a linked set of SCSI commands. Some LUs support multiple pending (queued) tasks, but the Satran, et al. Standards Track [Page 17] RFC 3720 iSCSI April 2004 queue of tasks is managed by the logical unit. The target uses an initiator provided "task tag" to distinguish between tasks. Only one command in a task can be outstanding at any given time. Each SCSI command results in an optional data phase and a required response phase. In the data phase, information can travel from the initiator to target (e.g., WRITE), target to initiator (e.g., READ), or in both directions. In the response phase, the target returns the final status of the operation, including any errors. Command Descriptor Blocks (CDB) are the data structures used to contain the command parameters that an initiator sends to a target. The CDB content and structure is defined by [SAM2] and device-type specific SCSI standards. 3.2. iSCSI Concepts and Functional Overview The iSCSI protocol is a mapping of the SCSI remote procedure invocation model (see [SAM2]) over the TCP protocol. SCSI commands are carried by iSCSI requests and SCSI responses and status are carried by iSCSI responses. iSCSI also uses the request response mechanism for iSCSI protocol mechanisms. For the remainder of this document, the terms "initiator" and "target" refer to "iSCSI initiator node" and "iSCSI target node", respectively (see Section 3.4.1 iSCSI Architecture Model) unless otherwise qualified. In keeping with similar protocols, the initiator and target divide their communications into messages. This document uses the term "iSCSI protocol data unit" (iSCSI PDU) for these messages. For performance reasons, iSCSI allows a "phase-collapse". A command and its associated data may be shipped together from initiator to target, and data and responses may be shipped together from targets. The iSCSI transfer direction is defined with respect to the initiator. Outbound or outgoing transfers are transfers from an initiator to a target, while inbound or incoming transfers are from a target to an initiator. An iSCSI task is an iSCSI request for which a response is expected. In this document "iSCSI request", "iSCSI command", request, or (unqualified) command have the same meaning. Also, unless otherwise specified, status, response, or numbered response have the same meaning. Satran, et al. Standards Track [Page 18] RFC 3720 iSCSI April 2004 3.2.1. Layers and Sessions The following conceptual layering model is used to specify initiator and target actions and the way in which they relate to transmitted and received Protocol Data Units: a) the SCSI layer builds/receives SCSI CDBs (Command Descriptor Blocks) and passes/receives them with the remaining command execute parameters ([SAM2]) to/from b) the iSCSI layer that builds/receives iSCSI PDUs and relays/receives them to/from one or more TCP connections; the group of connections form an initiator-target "session". Communication between the initiator and target occurs over one or more TCP connections. The TCP connections carry control messages, SCSI commands, parameters, and data within iSCSI Protocol Data Units (iSCSI PDUs). The group of TCP connections that link an initiator with a target form a session (loosely equivalent to a SCSI I_T nexus, see Section 3.4.2 SCSI Architecture Model). A session is defined by a session ID that is composed of an initiator part and a target part. TCP connections can be added and removed from a session. Each connection within a session is identified by a connection ID (CID). Across all connections within a session, an initiator sees one "target image". All target identifying elements, such as LUN, are the same. A target also sees one "initiator image" across all connections within a session. Initiator identifying elements, such as the Initiator Task Tag, are global across the session regardless of the connection on which they are sent or received. iSCSI targets and initiators MUST support at least one TCP connection and MAY support several connections in a session. For error recovery purposes, targets and initiators that support a single active connection in a session SHOULD support two connections during recovery. 3.2.2. Ordering and iSCSI Numbering iSCSI uses Command and Status numbering schemes and a Data sequencing scheme. Command numbering is session-wide and is used for ordered command delivery over multiple connections. It can also be used as a mechanism for command flow control over a session. Satran, et al. Standards Track [Page 19] RFC 3720 iSCSI April 2004 Status numbering is per connection and is used to enable missing status detection and recovery in the presence of transient or permanent communication errors. Data sequencing is per command or part of a command (R2T triggered sequence) and is used to detect missing data and/or R2T PDUs due to header digest errors. Typically, fields in the iSCSI PDUs communicate the Sequence Numbers between the initiator and target. During periods when traffic on a connection is unidirectional, iSCSI NOP-Out/In PDUs may be utilized to synchronize the command and status ordering counters of the target and initiator. The iSCSI session abstraction is equivalent to the SCSI I_T nexus, and the iSCSI session provides an ordered command delivery from the SCSI initiator to the SCSI target. For detailed design considerations that led to the iSCSI session model as it is defined here and how it relates the SCSI command ordering features defined in SCSI specifications to the iSCSI concepts see [CORD]. 3.2.2.1. Command Numbering and Acknowledging iSCSI performs ordered command delivery within a session. All commands (initiator-to-target PDUs) in transit from the initiator to the target are numbered. iSCSI considers a task to be instantiated on the target in response to every request issued by the initiator. A set of task management operations including abort and reassign (see Section 10.5 Task Management Function Request) may be performed on any iSCSI task. Some iSCSI tasks are SCSI tasks, and many SCSI activities are related to a SCSI task ([SAM2]). In all cases, the task is identified by the Initiator Task Tag for the life of the task. The command number is carried by the iSCSI PDU as CmdSN (Command Sequence Number). The numbering is session-wide. Outgoing iSCSI PDUs carry this number. The iSCSI initiator allocates CmdSNs with a 32-bit unsigned counter (modulo 2**32). Comparisons and arithmetic on CmdSN use Serial Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32. Commands meant for immediate delivery are marked with an immediate delivery flag; they MUST also carry the current CmdSN. CmdSN does not advance after a command marked for immediate delivery is sent. Satran, et al. Standards Track [Page 20] RFC 3720 iSCSI April 2004 Command numbering starts with the first login request on the first connection of a session (the leading login on the leading connection) and command numbers are incremented by 1 for every non-immediate command issued afterwards. If immediate delivery is used with task management commands, these commands may reach the target before the tasks on which they are supposed to act. However their CmdSN serves as a marker of their position in the stream of commands. The initiator and target must ensure that the task management commands act as specified by [SAM2]. For example, both commands and responses appear as if delivered in order. Whenever CmdSN for an outgoing PDU is not specified by an explicit rule, CmdSN will carry the current value of the local CmdSN variable (see later in this section). The means by which an implementation decides to mark a PDU for immediate delivery or by which iSCSI decides by itself to mark a PDU for immediate delivery are beyond the scope of this document. The number of commands used for immediate delivery is not limited and their delivery for execution is not acknowledged through the numbering scheme. Immediate commands MAY be rejected by the iSCSI target layer due to a lack of resources. An iSCSI target MUST be able to handle at least one immediate task management command and one immediate non-task-management iSCSI command per connection at any time. In this document, delivery for execution means delivery to the SCSI execution engine or an iSCSI protocol specific execution engine (e.g., for text requests with public or private extension keys involving an execution component). With the exception of the commands marked for immediate delivery, the iSCSI target layer MUST deliver the commands for execution in the order specified by CmdSN. Commands marked for immediate delivery may be delivered by the iSCSI target layer for execution as soon as detected. iSCSI may avoid delivering some commands to the SCSI target layer if required by a prior SCSI or iSCSI action (e.g., CLEAR TASK SET Task Management request received before all the commands on which it was supposed to act). On any connection, the iSCSI initiator MUST send the commands in increasing order of CmdSN, except for commands that are retransmitted due to digest error recovery and connection recovery. For the numbering mechanism, the initiator and target maintain the following three variables for each session: Satran, et al. Standards Track [Page 21] RFC 3720 iSCSI April 2004 - CmdSN - the current command Sequence Number, advanced by 1 on each command shipped except for commands marked for immediate delivery. CmdSN always contains the number to be assigned to the next Command PDU. - ExpCmdSN - the next expected command by the target. The target acknowledges all commands up to, but not including, this number. The initiator treats all commands with CmdSN less than ExpCmdSN as acknowledged. The target iSCSI layer sets the ExpCmdSN to the largest non-immediate CmdSN that it can deliver for execution plus 1 (no holes in the CmdSN sequence). - MaxCmdSN - the maximum number to be shipped. The queuing capacity of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1. The initiator's ExpCmdSN and MaxCmdSN are derived from target-to-initiator PDU fields. Comparisons and arithmetic on ExpCmdSN and MaxCmdSN MUST use Serial Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32. The target MUST NOT transmit a MaxCmdSN that is less than ExpCmdSN-1. For non-immediate commands, the CmdSN field can take any value from ExpCmdSN to MaxCmdSN inclusive. The target MUST silently ignore any non-immediate command outside of this range or non- immediate duplicates within the range. The CmdSN carried by immediate commands may lie outside the ExpCmdSN to MaxCmdSN range. For example, if the initiator has previously sent a non-immediate command carrying the CmdSN equal to MaxCmdSN, the target window is closed. For group task management commands issued as immediate commands, CmdSN indicates the scope of the group action (e.g., on ABORT TASK SET indicates which commands are aborted). MaxCmdSN and ExpCmdSN fields are processed by the initiator as follows: - If the PDU MaxCmdSN is less than the PDU ExpCmdSN-1 (in Serial Arithmetic Sense), they are both ignored. - If the PDU MaxCmdSN is greater than the local MaxCmdSN (in Serial Arithmetic Sense), it updates the local MaxCmdSN; otherwise, it is ignored. - If the PDU ExpCmdSN is greater than the local ExpCmdSN (in Serial Arithmetic Sense), it updates the local ExpCmdSN; otherwise, it is ignored. This sequence is required because updates may arrive out of order (e.g., the updates are sent on different TCP connections). iSCSI initiators and targets MUST support the command numbering scheme. Satran, et al. Standards Track [Page 22] RFC 3720 iSCSI April 2004 A numbered iSCSI request will not change its allocated CmdSN, regardless of the number of times and circumstances in which it is reissued (see Section 6.2.1 Usage of Retry). At the target, CmdSN is only relevant when the command has not created any state related to its execution (execution state); afterwards, CmdSN becomes irrelevant. Testing for the execution state (represented by identifying the Initiator Task Tag) MUST precede any other action at the target. If no execution state is found, it is followed by ordering and delivery. If an execution state is found, it is followed by delivery. If an initiator issues a command retry for a command with CmdSN R on a connection when the session CmdSN value is Q, it MUST NOT advance the CmdSN past R + 2**31 -1 unless the connection is no longer operational (i.e., it has returned to the FREE state, see Section 7.1.3 Standard Connection State Diagram for an Initiator), the connection has been reinstated (see Section 5.3.4 Connection Reinstatement), or a non-immediate command with CmdSN equal or greater than Q was issued subsequent to the command retry on the same connection and the reception of that command is acknowledged by the target (see Section 9.4 Command Retry and Cleaning Old Command Instances). A target MUST NOT issue a command response or Data-In PDU with status before acknowledging the command. However, the acknowledgement can be included in the response or Data-In PDU. 3.2.2.2. Response/Status Numbering and Acknowledging Responses in transit from the target to the initiator are numbered. The StatSN (Status Sequence Number) is used for this purpose. StatSN is a counter maintained per connection. ExpStatSN is used by the initiator to acknowledge status. The status sequence number space is 32-bit unsigned-integers and the arithmetic operations are the regular mod(2**32) arithmetic. Status numbering starts with the Login response to the first Login request of the connection. The Login response includes an initial value for status numbering (any initial value is valid). To enable command recovery, the target MAY maintain enough state information for data and status recovery after a connection failure. A target doing so can safely discard all of the state information maintained for recovery of a command after the delivery of the status for the command (numbered StatSN) is acknowledged through ExpStatSN. A large absolute difference between StatSN and ExpStatSN may indicate a failed connection. Initiators MUST undertake recovery actions if Satran, et al. Standards Track [Page 23] RFC 3720 iSCSI April 2004 the difference is greater than an implementation defined constant that MUST NOT exceed 2**31-1. Initiators and Targets MUST support the response-numbering scheme. 3.2.2.3. Data Sequencing Data and R2T PDUs transferred as part of some command execution MUST be sequenced. The DataSN field is used for data sequencing. For input (read) data PDUs, DataSN starts with 0 for the first data PDU of an input command and advances by 1 for each subsequent data PDU. For output data PDUs, DataSN starts with 0 for the first data PDU of a sequence (the initial unsolicited sequence or any data PDU sequence issued to satisfy an R2T) and advances by 1 for each subsequent data PDU. R2Ts are also sequenced per command. For example, the first R2T has an R2TSN of 0 and advances by 1 for each subsequent R2T. For bidirectional commands, the target uses the DataSN/R2TSN to sequence Data-In and R2T PDUs in one continuous sequence (undifferentiated). Unlike command and status, data PDUs and R2Ts are not acknowledged by a field in regular outgoing PDUs. Data-In PDUs can be acknowledged on demand by a special form of the SNACK PDU. Data and R2T PDUs are implicitly acknowledged by status for the command. The DataSN/R2TSN field enables the initiator to detect missing data or R2T PDUs. For any read or bidirectional command, a target MUST issue less than 2**32 combined R2T and Data-In PDUs. Any output data sequence MUST contain less than 2**32 Data-Out PDUs. 3.2.3. iSCSI Login The purpose of the iSCSI login is to enable a TCP connection for iSCSI use, authentication of the parties, negotiation of the session's parameters and marking of the connection as belonging to an iSCSI session. A session is used to identify to a target all the connections with a given initiator that belong to the same I_T nexus. (For more details on how a session relates to an I_T nexus, see Section 3.4.2 SCSI Architecture Model). The targets listen on a well-known TCP port or other TCP port for incoming connections. The initiator begins the login process by connecting to one of these TCP ports. As part of the login process, the initiator and target SHOULD authenticate each other and MAY set a security association protocol for the session. This can occur in many different ways and is subject to negotiation. Satran, et al. Standards Track [Page 24] RFC 3720 iSCSI April 2004 To protect the TCP connection, an IPsec security association MAY be established before the Login request. For information on using IPsec security for iSCSI see Chapter 8 and [RFC3723]. The iSCSI Login Phase is carried through Login requests and responses. Once suitable authentication has occurred and operational parameters have been set, the session transitions to the Full Feature Phase and the initiator may start to send SCSI commands. The security policy for whether, and by what means, a target chooses to authorize an initiator is beyond the scope of this document. For a more detailed description of the Login Phase, see Chapter 5. The login PDU includes the ISID part of the session ID (SSID). The target portal group that services the login is implied by the selection of the connection endpoint. For a new session, the TSIH is zero. As part of the response, the target generates a TSIH. During session establishment, the target identifies the SCSI initiator port (the "I" in the "I_T nexus") through the value pair (InitiatorName, ISID). We describe InitiatorName later in this section. Any persistent state (e.g., persistent reservations) on the target that is associated with a SCSI initiator port is identified based on this value pair. Any state associated with the SCSI target port (the "T" in the "I_T nexus") is identified externally by the TargetName and portal group tag (see Section 3.4.1 iSCSI Architecture Model). ISID is subject to reuse restrictions because it is used to identify a persistent state (see Section 3.4.3 Consequences of the Model). Before the Full Feature Phase is established, only Login Request and Login Response PDUs are allowed. Login requests and responses MUST be used exclusively during Login. On any connection, the login phase MUST immediately follow TCP connection establishment and a subsequent Login Phase MUST NOT occur before tearing down a connection. A target receiving any PDU except a Login request before the Login phase is started MUST immediately terminate the connection on which the PDU was received. Once the Login phase has started, if the target receives any PDU except a Login request, it MUST send a Login reject (with Status "invalid during login") and then disconnect. If the initiator receives any PDU except a Login response, it MUST immediately terminate the connection. 3.2.4. iSCSI Full Feature Phase Once the initiator is authorized to do so, the iSCSI session is in the iSCSI Full Feature Phase. A session is in Full Feature Phase after successfully finishing the Login Phase on the first (leading) Satran, et al. Standards Track [Page 25] RFC 3720 iSCSI April 2004 connection of a session. A connection is in Full Feature Phase if the session is in Full Feature Phase and the connection login has completed successfully. An iSCSI connection is not in Full Feature Phase a) when it does not have an established transport connection, OR b) when it has a valid transport connection, but a successful login was not performed or the connection is currently logged out. In a normal Full Feature Phase, the initiator may send SCSI commands and data to the various LUs on the target by encapsulating them in iSCSI PDUs that go over the established iSCSI session. 3.2.4.1. Command Connection Allegiance For any iSCSI request issued over a TCP connection, the corresponding response and/or other related PDU(s) MUST be sent over the same connection. We call this "connection allegiance". If the original connection fails before the command is completed, the connection allegiance of the command may be explicitly reassigned to a different transport connection as described in detail in Section 6.2 Retry and Reassign in Recovery. Thus, if an initiator issues a READ command, the target MUST send the requested data, if any, followed by the status to the initiator over the same TCP connection that was used to deliver the SCSI command. If an initiator issues a WRITE command, the initiator MUST send the data, if any, for that command over the same TCP connection that was used to deliver the SCSI command. The target MUST return Ready To Transfer (R2T), if any, and the status over the same TCP connection that was used to deliver the SCSI command. Retransmission requests (SNACK PDUs) and the data and status that they generate MUST also use the same connection. However, consecutive commands that are part of a SCSI linked command-chain task (see [SAM2]) MAY use different connections. Connection allegiance is strictly per-command and not per-task. During the iSCSI Full Feature Phase, the initiator and target MAY interleave unrelated SCSI commands, their SCSI Data, and responses over the session. Satran, et al. Standards Track [Page 26] RFC 3720 iSCSI April 2004 3.2.4.2. Data Transfer Overview Outgoing SCSI data (initiator to target user data or command parameters) is sent as either solicited data or unsolicited data. Solicited data are sent in response to R2T PDUs. Unsolicited data can be sent as part of an iSCSI command PDU ("immediate data") or in separate iSCSI data PDUs. Immediate data are assumed to originate at offset 0 in the initiator SCSI write-buffer (outgoing data buffer). All other Data PDUs have the buffer offset set explicitly in the PDU header. An initiator may send unsolicited data up to FirstBurstLength as immediate (up to the negotiated maximum PDU length), in a separate PDU sequence or both. All subsequent data MUST be solicited. The maximum length of an individual data PDU or the immediate-part of the first unsolicited burst MAY be negotiated at login. The maximum amount of unsolicited data that can be sent with a command is negotiated at login through the FirstBurstLength key. A target MAY separately enable immediate data (through the ImmediateData key) without enabling the more general (separate data PDUs) form of unsolicited data (through the InitialR2T key). Unsolicited data on write are meant to reduce the effect of latency on throughput (no R2T is needed to start sending data). In addition, immediate data is meant to reduce the protocol overhead (both bandwidth and execution time). An iSCSI initiator MAY choose not to send unsolicited data, only immediate data or FirstBurstLength bytes of unsolicited data with a command. If any non-immediate unsolicited data is sent, the total unsolicited data MUST be either FirstBurstLength, or all of the data if the total amount is less than the FirstBurstLength. It is considered an error for an initiator to send unsolicited data PDUs to a target that operates in R2T mode (only solicited data are allowed). It is also an error for an initiator to send more unsolicited data, whether immediate or as separate PDUs, than FirstBurstLength. An initiator MUST honor an R2T data request for a valid outstanding command (i.e., carrying a valid Initiator Task Tag) and deliver all the requested data provided the command is supposed to deliver outgoing data and the R2T specifies data within the command bounds. The initiator action is unspecified for receiving an R2T request that specifies data, all or part, outside of the bounds of the command. Satran, et al. Standards Track [Page 27] RFC 3720 iSCSI April 2004 A target SHOULD NOT silently discard data and then request retransmission through R2T. Initiators SHOULD NOT keep track of the data transferred to or from the target (scoreboarding). SCSI targets perform residual count calculation to check how much data was actually transferred to or from the device by a command. This may differ from the amount the initiator sent and/or received for reasons such as retransmissions and errors. Read or bidirectional commands implicitly solicit the transmission of the entire amount of data covered by the command. SCSI data packets are matched to their corresponding SCSI commands by using tags specified in the protocol. In addition, iSCSI initiators and targets MUST enforce some ordering rules. When unsolicited data is used, the order of the unsolicited data on each connection MUST match the order in which the commands on that connection are sent. Command and unsolicited data PDUs may be interleaved on a single connection as long as the ordering requirements of each are maintained (e.g., command N+1 MAY be sent before the unsolicited Data-Out PDUs for command N, but the unsolicited Data-Out PDUs for command N MUST precede the unsolicited Data-Out PDUs of command N+1). A target that receives data out of order MAY terminate the session. 3.2.4.3. Tags and Integrity Checks Initiator tags for pending commands are unique initiator-wide for a session. Target tags are not strictly specified by the protocol. It is assumed that target tags are used by the target to tag (alone or in combination with the LUN) the solicited data. Target tags are generated by the target and "echoed" by the initiator. These mechanisms are designed to accomplish efficient data delivery along with a large degree of control over the data flow. As the Initiator Task Tag is used to identify a task during its execution, the iSCSI initiator and target MUST verify that all other fields used in task-related PDUs have values that are consistent with the values used at the task instantiation based on the Initiator Task Tag (e.g., the LUN used in an R2T PDU MUST be the same as the one used in the SCSI command PDU used to instantiate the task). Using inconsistent field values is considered a protocol error. 3.2.4.4. Task Management SCSI task management assumes that individual tasks and task groups can be aborted solely based on the task tags (for individual tasks) or the timing of the task management command (for task groups), and that the task management action is executed synchronously - i.e., no message involving an aborted task will be seen by the SCSI initiator after receiving the task management response. In iSCSI initiators Satran, et al. Standards Track [Page 28] RFC 3720 iSCSI April 2004 and targets interact asynchronously over several connections. iSCSI specifies the protocol mechanism and implementation requirements needed to present a synchronous view while using an asynchronous infrastructure. 3.2.5. iSCSI Connection Termination An iSCSI connection may be terminated by use of a transport connection shutdown or a transport reset. Transport reset is assumed to be an exceptional event. Graceful TCP connection shutdowns are done by sending TCP FINs. A graceful transport connection shutdown SHOULD only be initiated by either party when the connection is not in iSCSI Full Feature Phase. A target MAY terminate a Full Feature Phase connection on internal exception events, but it SHOULD announce the fact through an Asynchronous Message PDU. Connection termination with outstanding commands may require recovery actions. If a connection is terminated while in Full Feature Phase, connection cleanup (see section 7) is required prior to recovery. By doing connection cleanup before starting recovery, the initiator and target will avoid receiving stale PDUs after recovery. 3.2.6. iSCSI Names Both targets and initiators require names for the purpose of identification. In addition, names enable iSCSI storage resources to be managed regardless of location (address). An iSCSI node name is also the SCSI device name of an iSCSI device. The iSCSI name of a SCSI device is the principal object used in authentication of targets to initiators and initiators to targets. This name is also used to identify and manage iSCSI storage resources. iSCSI names must be unique within the operational domain of the end user. However, because the operational domain of an IP network is potentially worldwide, the iSCSI name formats are architected to be worldwide unique. To assist naming authorities in the construction of worldwide unique names, iSCSI provides two name formats for different types of naming authorities. iSCSI names are associated with iSCSI nodes, and not iSCSI network adapter cards, to ensure that the replacement of network adapter cards does not require reconfiguration of all SCSI and iSCSI resource allocation information. Satran, et al. Standards Track [Page 29] RFC 3720 iSCSI April 2004 Some SCSI commands require that protocol-specific identifiers be communicated within SCSI CDBs. See Section 3.4.2 SCSI Architecture Model for the definition of the SCSI port name/identifier for iSCSI ports. An initiator may discover the iSCSI Target Names to which it has access, along with their addresses, using the SendTargets text request, or other techniques discussed in [RFC3721]. 3.2.6.1. iSCSI Name Properties Each iSCSI node, whether an initiator or target, MUST have an iSCSI name. Initiators and targets MUST support the receipt of iSCSI names of up to the maximum length of 223 bytes. The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session or connection. The only exception is if a discovery session (see Section 2.3 iSCSI Session Types) is to be established. In this case, the iSCSI Initiator Name is still required, but the iSCSI Target Name MAY be omitted. iSCSI names have the following properties: a) iSCSI names are globally unique. No two initiators or targets can have the same name. b) iSCSI names are permanent. An iSCSI initiator node or target node has the same name for its lifetime. c) iSCSI names do not imply a location or address. An iSCSI initiator or target can move, or have multiple addresses. A change of address does not imply a change of name. d) iSCSI names do not rely on a central name broker; the naming authority is distributed. e) iSCSI names support integration with existing unique naming schemes. f) iSCSI names rely on existing naming authorities. iSCSI does not create any new naming authority. The encoding of an iSCSI name has the following properties: a) iSCSI names have the same encoding method regardless of the underlying protocols. b) iSCSI names are relatively simple to compare. The algorithm for comparing two iSCSI names for equivalence does not rely on an external server. Satran, et al. Standards Track [Page 30] RFC 3720 iSCSI April 2004 c) iSCSI names are composed only of displayable characters. iSCSI names allow the use of international character sets but are not case sensitive. No whitespace characters are used in iSCSI names. d) iSCSI names may be transported using both binary and ASCII-based protocols. An iSCSI name really names a logical software entity, and is not tied to a port or other hardware that can be changed. For instance, an initiator name should name the iSCSI initiator node, not a particular NIC or HBA. When multiple NICs are used, they should generally all present the same iSCSI initiator name to the targets, because they are simply paths to the same SCSI layer. In most operating systems, the named entity is the operating system image. Similarly, a target name should not be tied to hardware interfaces that can be changed. A target name should identify the logical target and must be the same for the target regardless of the physical portion being addressed. This assists iSCSI initiators in determining that the two targets it has discovered are really two paths to the same target. The iSCSI name is designed to fulfill the functional requirements for Uniform Resource Names (URN) [RFC1737]. For example, it is required that the name have a global scope, be independent of address or location, and be persistent and globally unique. Names must be extensible and scalable with the use of naming authorities. The name encoding should be both human and machine readable. See [RFC1737] for further requirements. 3.2.6.2. iSCSI Name Encoding An iSCSI name MUST be a UTF-8 encoding of a string of Unicode characters with the following properties: - It is in Normalization Form C (see "Unicode Normalization Forms" [UNICODE]). - It only contains characters allowed by the output of the iSCSI stringprep template (described in [RFC3722]). - The following characters are used for formatting iSCSI names: - dash ('-'=U+002d) - dot ('.'=U+002e) - colon (':'=U+003a) - The UTF-8 encoding of the name is not larger than 223 bytes. Satran, et al. Standards Track [Page 31] RFC 3720 iSCSI April 2004 The stringprep process is described in [RFC3454]; iSCSI's use of the stringprep process is described in [RFC3722]. Stringprep is a method designed by the Internationalized Domain Name (IDN) working group to translate human-typed strings into a format that can be compared as opaque strings. Strings MUST NOT include punctuation, spacing, diacritical marks, or other characters that could get in the way of readability. The stringprep process also converts strings into equivalent strings of lower-case characters. The stringprep process does not need to be implemented if the names are only generated using numeric and lower-case (any character set) alphabetic characters. Once iSCSI names encoded in UTF-8 are "normalized" they may be safely compared byte-for-byte. 3.2.6.3. iSCSI Name Structure An iSCSI name consists of two parts--a type designator followed by a unique name string. The iSCSI name does not define any new naming authorities. Instead, it supports two existing ways of designating naming authorities: an iSCSI-Qualified Name, using domain names to identify a naming authority, and the EUI format, where the IEEE Registration Authority assists in the formation of worldwide unique names (EUI-64 format). The type designator strings currently defined are: iqn. - iSCSI Qualified name eui. - Remainder of the string is an IEEE EUI-64 identifier, in ASCII-encoded hexadecimal. These two naming authority designators were considered sufficient at the time of writing this document. The creation of additional naming type designators for iSCSI may be considered by the IETF and detailed in separate RFCs. 3.2.6.3.1. Type "iqn." (iSCSI Qualified Name) This iSCSI name type can be used by any organization that owns a domain name. This naming format is useful when an end user or service provider wishes to assign iSCSI names for targets and/or initiators. To generate names of this type, the person or organization generating the name must own a registered domain name. This domain name does not have to be active, and does not have to resolve to an address; it Satran, et al. Standards Track [Page 32] RFC 3720 iSCSI April 2004 just needs to be reserved to prevent others from generating iSCSI names using the same domain name. Since a domain name can expire, be acquired by another entity, or may be used to generate iSCSI names by both owners, the domain name must be additionally qualified by a date during which the naming authority owned the domain name. For this reason, a date code is provided as part of the "iqn." format. The iSCSI qualified name string consists of: - The string "iqn.", used to distinguish these names from "eui." formatted names. - A date code, in yyyy-mm format. This date MUST be a date during which the naming authority owned the domain name used in this format, and SHOULD be the first month in which the domain name was owned by this naming authority at 00:01 GMT of the first day of the month. This date code uses the Gregorian calendar. All four digits in the year must be present. Both digits of the month must be present, with January == "01" and December == "12". The dash must be included. - A dot "." - The reversed domain name of the naming authority (person or organization) creating this iSCSI name. - An optional, colon (:) prefixed, string within the character set and length boundaries that the owner of the domain name deems appropriate. This may contain product types, serial numbers, host identifiers, or software keys (e.g., it may include colons to separate organization boundaries). With the exception of the colon prefix, the owner of the domain name can assign everything after the reversed domain name as desired. It is the responsibility of the entity that is the naming authority to ensure that the iSCSI names it assigns are worldwide unique. For example, "Example Storage Arrays, Inc.", might own the domain name "example.com". The following are examples of iSCSI qualified names that might be generated by "EXAMPLE Storage Arrays, Inc." Naming String defined by Type Date Auth "example.com" naming authority +--++-----+ +---------+ +--------------------------------+ | || | | | | | iqn.2001-04.com.example:storage:diskarrays-sn-a8675309 iqn.2001-04.com.example iqn.2001-04.com.example:storage.tape1.sys1.xyz iqn.2001-04.com.example:storage.disk2.sys1.xyz Satran, et al. Standards Track [Page 33] RFC 3720 iSCSI April 2004 3.2.6.3.2. Type "eui." (IEEE EUI-64 format) The IEEE Registration Authority provides a service for assigning globally unique identifiers [EUI]. The EUI-64 format is used to build a global identifier in other network protocols. For example, Fibre Channel defines a method of encoding it into a WorldWideName. For more information on registering for EUI identifiers, see [OUI]. The format is "eui." followed by an EUI-64 identifier (16 ASCII-encoded hexadecimal digits). Example iSCSI name: Type EUI-64 identifier (ASCII-encoded hexadecimal) +--++--------------+ | || | eui.02004567A425678D The IEEE EUI-64 iSCSI name format might be used when a manufacturer is already registered with the IEEE Registration Authority and uses EUI-64 formatted worldwide unique names for its products. More examples of name construction are discussed in [RFC3721]. 3.2.7. Persistent State iSCSI does not require any persistent state maintenance across sessions. However, in some cases, SCSI requires persistent identification of the SCSI initiator port name (See Section 3.4.2 SCSI Architecture Model and Section 3.4.3 Consequences of the Model). iSCSI sessions do not persist through power cycles and boot operations. All iSCSI session and connection parameters are re-initialized upon session and connection creation. Commands persist beyond connection termination if the session persists and command recovery within the session is supported. However, when a connection is dropped, command execution, as perceived by iSCSI (i.e., involving iSCSI protocol exchanges for the affected task), is suspended until a new allegiance is established by the 'task reassign' task management function. (See Section 10.5 Task Management Function Request.) Satran, et al. Standards Track [Page 34] RFC 3720 iSCSI April 2004 3.2.8. Message Synchronization and Steering iSCSI presents a mapping of the SCSI protocol onto TCP. This encapsulation is accomplished by sending iSCSI PDUs of varying lengths. Unfortunately, TCP does not have a built-in mechanism for signaling message boundaries at the TCP layer. iSCSI overcomes this obstacle by placing the message length in the iSCSI message header. This serves to delineate the end of the current message as well as the beginning of the next message. In situations where IP packets are delivered in order from the network, iSCSI message framing is not an issue and messages are processed one after the other. In the presence of IP packet reordering (i.e., frames being dropped), legacy TCP implementations store the "out of order" TCP segments in temporary buffers until the missing TCP segments arrive, upon which the data must be copied to the application buffers. In iSCSI, it is desirable to steer the SCSI data within these out of order TCP segments into the pre-allocated SCSI buffers rather than store them in temporary buffers. This decreases the need for dedicated reassembly buffers as well as the latency and bandwidth related to extra copies. Relying solely on the "message length" information from the iSCSI message header may make it impossible to find iSCSI message boundaries in subsequent TCP segments due to the loss of a TCP segment that contains the iSCSI message length. The missing TCP segment(s) must be received before any of the following segments can be steered to the correct SCSI buffers (due to the inability to determine the iSCSI message boundaries). Since these segments cannot be steered to the correct location, they must be saved in temporary buffers that must then be copied to the SCSI buffers. Different schemes can be used to recover synchronization. To make these schemes work, iSCSI implementations have to make sure that the appropriate protocol layers are provided with enough information to implement a synchronization and/or data steering mechanism. One of these schemes is detailed in Appendix A. - Sync and Steering with Fixed Interval Markers -. The Fixed Interval Markers (FIM) scheme works by inserting markers in the payload stream at fixed intervals that contain the offset for the start of the next iSCSI PDU. Under normal circumstances (no PDU loss or data reception out of order), iSCSI data steering can be accomplished by using the identifying tag and the data offset fields in the iSCSI header in addition to the TCP sequence number from the TCP header. The Satran, et al. Standards Track [Page 35] RFC 3720 iSCSI April 2004 identifying tag helps associate the PDU with a SCSI buffer address while the data offset and TCP sequence number are used to determine the offset within the buffer. When the part of the TCP data stream containing an iSCSI PDU header is delayed or lost, markers may be used to minimize the damage as follows: - Markers indicate where the next iSCSI PDU starts and enable continued processing when iSCSI headers have to be dropped due to data errors discovered at the iSCSI level (e.g., iSCSI header CRC errors). - Markers help minimize the amount of data that has to be kept by the TCP/iSCSI layer while waiting for a late TCP packet arrival or recovery, because later they might help find iSCSI PDU headers and use the information contained in those to steer data to SCSI buffers. 3.2.8.1. Sync/Steering and iSCSI PDU Length When a large iSCSI message is sent, the TCP segment(s) that contain the iSCSI header may be lost. The remaining TCP segment(s), up to the next iSCSI message, must be buffered (in temporary buffers) because the iSCSI header that indicates to which SCSI buffers the data are to be steered was lost. To minimize the amount of buffering, it is recommended that the iSCSI PDU length be restricted to a small value (perhaps a few TCP segments in length). During login, each end of the iSCSI session specifies the maximum iSCSI PDU length it will accept. 3.3. iSCSI Session Types iSCSI defines two types of sessions: a) Normal operational session - an unrestricted session. b) Discovery-session - a session only opened for target discovery. The target MUST ONLY accept text requests with the SendTargets key and a logout request with the reason "close the session". All other requests MUST be rejected. The session type is defined during login with the key=value parameter in the login command. Satran, et al. Standards Track [Page 36] RFC 3720 iSCSI April 2004 3.4. SCSI to iSCSI Concepts Mapping Model The following diagram shows an example of how multiple iSCSI Nodes (targets in this case) can coexist within the same Network Entity and can share Network Portals (IP addresses and TCP ports). Other more complex configurations are also possible. For detailed descriptions of the components of these diagrams, see Section 3.4.1 iSCSI Architecture Model. +-----------------------------------+ | Network Entity (iSCSI Client) | | | | +-------------+ | | | iSCSI Node | | | | (Initiator) | | | +-------------+ | | | | | | +--------------+ +--------------+ | | |Network Portal| |Network Portal| | | | 10.1.30.4 | | 10.1.40.6 | | +-+--------------+-+--------------+-+ | | | IP Networks | | | +-+--------------+-+--------------+-+ | |Network Portal| |Network Portal| | | | 10.1.30.21 | | 10.1.40.3 | | | | TCP Port 3260| | TCP Port 3260| | | +--------------+ +--------------+ | | | | | | ----------------- | | | | | | +-------------+ +--------------+ | | | iSCSI Node | | iSCSI Node | | | | (Target) | | (Target) | | | +-------------+ +--------------+ | | | | Network Entity (iSCSI Server) | +-----------------------------------+ 3.4.1. iSCSI Architecture Model This section describes the part of the iSCSI architecture model that has the most bearing on the relationship between iSCSI and the SCSI Architecture Model. Satran, et al. Standards Track [Page 37] RFC 3720 iSCSI April 2004 a) Network Entity - represents a device or gateway that is accessible from the IP network. A Network Entity must have one or more Network Portals (see item d), each of which can be used by some iSCSI Nodes (see item (b)) contained in that Network Entity to gain access to the IP network. b) iSCSI Node - represents a single iSCSI initiator or iSCSI target. There are one or more iSCSI Nodes within a Network Entity. The iSCSI Node is accessible via one or more Network Portals (see item d). An iSCSI Node is identified by its iSCSI Name (see Section 3.2.6 iSCSI Names and Chapter 12). The separation of the iSCSI Name from the addresses used by and for the iSCSI node allows multiple iSCSI nodes to use the same addresses, and the same iSCSI node to use multiple addresses. c) An alias string may also be associated with an iSCSI Node. The alias allows an organization to associate a user friendly string with the iSCSI Name. However, the alias string is not a substitute for the iSCSI Name. d) Network Portal - a component of a Network Entity that has a TCP/IP network address and that may be used by an iSCSI Node within that Network Entity for the connection(s) within one of its iSCSI sessions. In an initiator, it is identified by its IP address. In a target, it is identified by its IP address and its listening TCP port. e) Portal Groups - iSCSI supports multiple connections within the same session; some implementations will have the ability to combine connections in a session across multiple Network Portals. A Portal Group defines a set of Network Portals within an iSCSI Node that collectively supports the capability of coordinating a session with connections that span these portals. Not all Network Portals within a Portal Group need to participate in every session connected through that Portal Group. One or more Portal Groups may provide access to an iSCSI Node. Each Network Portal, as utilized by a given iSCSI Node, belongs to exactly one portal group within that node. Portal Groups are identified within an iSCSI Node by a portal group tag, a simple unsigned-integer between 0 and 65535 (see Section 12.3 SendTargets). All Network Portals with the same portal group tag in the context of a given iSCSI Node are in the same Portal Group. Satran, et al. Standards Track [Page 38] RFC 3720 iSCSI April 2004 Both iSCSI Initiators and iSCSI Targets have portal groups, though only the iSCSI Target Portal Groups are used directly in the iSCSI protocol (e.g., in SendTargets). For references to the initiator Portal Groups, see Section 9.1.1 Conservative Reuse of ISIDs. f) Portals within a Portal Group should support similar session parameters, because they may participate in a common session. The following diagram shows an example of one such configuration on a target and how a session that shares Network Portals within a Portal Group may be established. ----------------------------IP Network--------------------- | | | +----|---------------|-----+ +----|---------+ | +---------+ +---------+ | | +---------+ | | | Network | | Network | | | | Network | | | | Portal | | Portal | | | | Portal | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | Portal | | | | Portal | | | Group 1 | | | | Group 2 | +--------------------------+ +--------------+ | | | +--------|---------------|--------------------|--------------------+ | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (TSIH = 56) | | (TSIH = 48) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +------------------------------------------------------------------+ 3.4.2. SCSI Architecture Model This section describes the relationship between the SCSI Architecture Model [SAM2] and the constructs of the SCSI device, SCSI port and I_T nexus, and the iSCSI constructs described in Section 3.4.1 iSCSI Architecture Model. This relationship implies implementation requirements in order to conform to the SAM2 model and other SCSI operational functions. These requirements are detailed in Section 3.4.3 Consequences of the Model. Satran, et al. Standards Track [Page 39] RFC 3720 iSCSI April 2004 The following list outlines mappings of SCSI architectural elements to iSCSI. a) SCSI Device - the SAM2 term for an entity that contains one or more SCSI ports that are connected to a service delivery subsystem and supports a SCSI application protocol. For example, a SCSI Initiator Device contains one or more SCSI Initiator Ports and zero or more application clients. A SCSI Target Device contains one or more SCSI Target Ports and one or more logical units. For iSCSI, the SCSI Device is the component within an iSCSI Node that provides the SCSI functionality. As such, there can be one SCSI Device, at most, within an iSCSI Node. Access to the SCSI Device can only be achieved in an iSCSI normal operational session (see Section 3.3 iSCSI Session Types). The SCSI Device Name is defined to be the iSCSI Name of the node and MUST be used in the iSCSI protocol. b) SCSI Port - the SAM2 term for an entity in a SCSI Device that provides the SCSI functionality to interface with a service delivery subsystem or transport. For iSCSI, the definition of SCSI Initiator Port and SCSI Target Port are different. SCSI Initiator Port: This maps to one endpoint of an iSCSI normal operational session (see Section 3.3 iSCSI Session Types). An iSCSI normal operational session is negotiated through the login process between an iSCSI initiator node and an iSCSI target node. At successful completion of this process, a SCSI Initiator Port is created within the SCSI Initiator Device. The SCSI Initiator Port Name and SCSI Initiator Port Identifier are both defined to be the iSCSI Initiator Name together with (a) a label that identifies it as an initiator port name/identifier and (b) the ISID portion of the session identifier. SCSI Target Port: This maps to an iSCSI Target Portal Group. The SCSI Target Port Name and the SCSI Target Port Identifier are both defined to be the iSCSI Target Name together with (a) a label that identifies it as a target port name/identifier and (b) the portal group tag. The SCSI Port Name MUST be used in iSCSI. When used in SCSI parameter data, the SCSI port name MUST be encoded as: - The iSCSI Name in UTF-8 format, followed by - a comma separator (1 byte), followed by - the ASCII character 'i' (for SCSI Initiator Port) or the ASCII character 't' (for SCSI Target Port) (1 byte), followed by Satran, et al. Standards Track [Page 40] RFC 3720 iSCSI April 2004 - a comma separator (1 byte), followed by - a text encoding as a hex-constant (see Section 5.1 Text Format) of the ISID (for SCSI initiator port) or the portal group tag (for SCSI target port) including the initial 0X or 0x and the terminating null (15 bytes). The ASCII character 'i' or 't' is the label that identifies this port as either a SCSI Initiator Port or a SCSI Target Port. c) I_T nexus - a relationship between a SCSI Initiator Port and a SCSI Target Port, according to [SAM2]. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names or by the iSCSI session identifier SSID. iSCSI defines the I_T nexus identifier to be the tuple (iSCSI Initiator Name + 'i' + ISID, iSCSI Target Name + 't' + Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). 3.4.3. Consequences of the Model This section describes implementation and behavioral requirements that result from the mapping of SCSI constructs to the iSCSI constructs defined above. Between a given SCSI initiator port and a given SCSI target port, only one I_T nexus (session) can exist. No more than one nexus relationship (parallel nexus) is allowed by [SAM2]. Therefore, at any given time, only one session can exist between a given iSCSI initiator node and an iSCSI target node, with the same session identifier (SSID). These assumptions lead to the following conclusions and requirements: ISID RULE: Between a given iSCSI Initiator and iSCSI Target Portal Group (SCSI target port), there can only be one session with a given value for ISID that identifies the SCSI initiator port. See Section 10.12.5 ISID. The structure of the ISID that contains a naming authority component (see Section 10.12.5 ISID and [RFC3721]) provides a mechanism to facilitate compliance with the ISID rule. (See Section 9.1.1 Conservative Reuse of ISIDs.) Satran, et al. Standards Track [Page 41] RFC 3720 iSCSI April 2004 The iSCSI Initiator Node should manage the assignment of ISIDs prior to session initiation. The "ISID RULE" does not preclude the use of the same ISID from the same iSCSI Initiator with different Target Portal Groups on the same iSCSI target or on other iSCSI targets (see Section 9.1.1 Conservative Reuse of ISIDs). Allowing this would be analogous to a single SCSI Initiator Port having relationships (nexus) with multiple SCSI target ports on the same SCSI target device or SCSI target ports on other SCSI target devices. It is also possible to have multiple sessions with different ISIDs to the same Target Portal Group. Each such session would be considered to be with a different initiator even when the sessions originate from the same initiator device. The same ISID may be used by a different iSCSI initiator because it is the iSCSI Name together with the ISID that identifies the SCSI Initiator Port. NOTE: A consequence of the ISID RULE and the specification for the I_T nexus identifier is that two nexus with the same identifier should never exist at the same time. TSIH RULE: The iSCSI Target selects a non-zero value for the TSIH at session creation (when an initiator presents a 0 value at Login). After being selected, the same TSIH value MUST be used whenever the initiator or target refers to the session and a TSIH is required. 3.4.3.1. I_T Nexus State Certain nexus relationships contain an explicit state (e.g., initiator-specific mode pages) that may need to be preserved by the device server [SAM2] in a logical unit through changes or failures in the iSCSI layer (e.g., session failures). In order for that state to be restored, the iSCSI initiator should reestablish its session (re-login) to the same Target Portal Group using the previous ISID. That is, it should perform session recovery as described in Chapter 6. This is because the SCSI initiator port identifier and the SCSI target port identifier (or relative target port) form the datum that the SCSI logical unit device server uses to identify the I_T nexus. 3.5. Request/Response Summary This section lists and briefly describes all the iSCSI PDU types (request and responses). All iSCSI PDUs are built as a set of one or more header segments (basic and auxiliary) and zero or one data segments. The header group and the data segment may each be followed by a CRC (digest). The basic header segment has a fixed length of 48 bytes. Satran, et al. Standards Track [Page 42] RFC 3720 iSCSI April 2004 3.5.1. Request/Response Types Carrying SCSI Payload 3.5.1.1. SCSI-Command This request carries the SCSI CDB and all the other SCSI execute command procedure call (see [SAM2]) IN arguments such as task attributes, Expected Data Transfer Length for one or both transfer directions (the latter for bidirectional commands), and Task Tag (as part of the I_T_L_x nexus). The I_T_L nexus is derived by the initiator and target from the LUN field in the request and the I_T nexus is implicit in the session identification. In addition, the SCSI-command PDU carries information required for the proper operation of the iSCSI protocol - the command sequence number (CmdSN) for the session and the expected status number (ExpStatSN) for the connection. All or part of the SCSI output (write) data associated with the SCSI command may be sent as part of the SCSI-Command PDU as a data segment. 3.5.1.2. SCSI-Response The SCSI-Response carries all the SCSI execute-command procedure call (see [SAM2]) OUT arguments and the SCSI execute-command procedure call return value. The SCSI-Response contains the residual counts from the operation, if any, an indication of whether the counts represent an overflow or an underflow, and the SCSI status if the status is valid or a response code (a non-zero return value for the execute-command procedure call) if the status is not valid. For a valid status that indicates that the command has been processed, but resulted in an exception (e.g., a SCSI CHECK CONDITION), the PDU data segment contains the associated sense data. The use of Autosense ([SAM2]) is REQUIRED by iSCSI. Some data segment content may also be associated (in the data segment) with a non-zero response code. In addition, the SCSI-Response PDU carries information required for the proper operation of the iSCSI protocol: - The number of Data-In PDUs that a target has sent (to enable the initiator to check that all have arrived). - StatSN - the Status Sequence Number on this connection. Satran, et al. Standards Track [Page 43] RFC 3720 iSCSI April 2004 - ExpCmdSN - the next Expected Command Sequence Number at the target. - MaxCmdSN - the maximum CmdSN acceptable at the target from this initiator. 3.5.1.3 Task Management Function Request The Task Management function request provides an initiator with a way to explicitly control the execution of one or more SCSI Tasks or iSCSI functions. The PDU carries a function identifier (which task management function to perform) and enough information to unequivocally identify the task or task-set on which to perform the action, even if the task(s) to act upon has not yet arrived or has been discarded due to an error. The referenced tag identifies an individual task if the function refers to an individual task. The I_T_L nexus identifies task sets. In iSCSI the I_T_L nexus is identified by the LUN and the session identification (the session identifies an I_T nexus). For task sets, the CmdSN of the Task Management function request helps identify the tasks upon which to act, namely all tasks associated with a LUN and having a CmdSN preceding the Task Management function request CmdSN. For a Task Management function, the coordination between responses to the tasks affected and the Task Management function response is done by the target. 3.5.1.4. Task Management Function Response The Task Management function response carries an indication of function completion for a Task Management function request including how it was completed (response and qualifier) and additional information for failure responses. After the Task Management response indicates Task Management function completion, the initiator will not receive any additional responses from the affected tasks. 3.5.1.5. SCSI Data-Out and SCSI Data-In SCSI Data-Out and SCSI Data-In are the main vehicles by which SCSI data payload is carried between initiator and target. Data payload is associated with a specific SCSI command through the Initiator Task Tag. For target convenience, outgoing solicited data also carries a Satran, et al. Standards Track [Page 44] RFC 3720 iSCSI April 2004 Target Transfer Tag (copied from R2T) and the LUN. Each PDU contains the payload length and the data offset relative to the buffer address contained in the SCSI execute command procedure call. In each direction, the data transfer is split into "sequences". An end-of-sequence is indicated by the F bit. An outgoing sequence is either unsolicited (only the first sequence can be unsolicited) or consists of all the Data-Out PDUs sent in response to an R2T. Input sequences are built to enable the direction switching for bidirectional commands. For input, the target may request positive acknowledgement of input data. This is limited to sessions that support error recovery and is implemented through the A bit in the SCSI Data-In PDU header. Data-In and Data-Out PDUs also carry the DataSN to enable the initiator and target to detect missing PDUs (discarded due to an error). In addition, StatSN is carried by the Data-In PDUs. To enable a SCSI command to be processed while involving a minimum number of messages, the last SCSI Data-In PDU passed for a command may also contain the status if the status indicates termination with no exceptions (no sense or response involved). 3.5.1.6. Ready To Transfer (R2T) R2T is the mechanism by which the SCSI target "requests" the initiator for output data. R2T specifies to the initiator the offset of the requested data relative to the buffer address from the execute command procedure call and the length of the solicited data. To help the SCSI target associate the resulting Data-Out with an R2T, the R2T carries a Target Transfer Tag that will be copied by the initiator in the solicited SCSI Data-Out PDUs. There are no protocol specific requirements with regard to the value of these tags, but it is assumed that together with the LUN, they will enable the target to associate data with an R2T. Satran, et al. Standards Track [Page 45] RFC 3720 iSCSI April 2004 R2T also carries information required for proper operation of the iSCSI protocol, such as: - R2TSN (to enable an initiator to detect a missing R2T) - StatSN - ExpCmdSN - MaxCmdSN 3.5.2. Requests/Responses carrying SCSI and iSCSI Payload 3.5.2.1. Asynchronous Message Asynchronous Messages are used to carry SCSI asynchronous events (AEN) and iSCSI asynchronous messages. When carrying an AEN, the event details are reported as sense data in the data segment. 3.5.3. Requests/Responses Carrying iSCSI Only Payload 3.5.3.1. Text Request and Text Response Text requests and responses are designed as a parameter negotiation vehicle and as a vehicle for future extension. In the data segment, Text Requests/Responses carry text information using a simple "key=value" syntax. Text Request/Responses may form extended sequences using the same Initiator Task Tag. The initiator uses the F (Final) flag bit in the text request header to indicate its readiness to terminate a sequence. The target uses the F (Final) flag bit in the text response header to indicate its consent to sequence termination. Text Request and Responses also use the Target Transfer Tag to indicate continuation of an operation or a new beginning. A target that wishes to continue an operation will set the Target Transfer Tag in a Text Response to a value different from the default 0xffffffff. An initiator willing to continue will copy this value into the Target Transfer Tag of the next Text Request. If the initiator wants to restart the current target negotiation (start fresh) will set the Target Transfer Tag to 0xffffffff. Although a complete exchange is always started by the initiator, specific parameter negotiations may be initiated by the initiator or target. Satran, et al. Standards Track [Page 46] RFC 3720 iSCSI April 2004 3.5.3.2. Login Request and Login Response Login Requests and Responses are used exclusively during the Login Phase of each connection to set up the session and connection parameters. (The Login Phase consists of a sequence of login requests and responses carrying the same Initiator Task Tag.) A connection is identified by an arbitrarily selected connection-ID (CID) that is unique within a session. Similar to the Text Requests and Responses, Login Requests/Responses carry key=value text information with a simple syntax in the data segment. The Login Phase proceeds through several stages (security negotiation, operational parameter negotiation) that are selected with two binary coded fields in the header -- the "current stage" (CSG) and the "next stage" (NSG) with the appearance of the latter being signaled by the "transit" flag (T). The first Login Phase of a session plays a special role, called the leading login, which determines some header fields (e.g., the version number, the maximum number of connections, and the session identification). The CmdSN initial value is also set by the leading login. StatSN for each connection is initiated by the connection login. A login request may indicate an implied logout (cleanup) of the connection to be logged in (a connection restart) by using the same Connection ID (CID) as an existing connection, as well as the same session identifying elements of the session to which the old connection was associated. 3.5.3.3. Logout Request and Response Logout Requests and Responses are used for the orderly closing of connections for recovery or maintenance. The logout request may be issued following a target prompt (through an asynchronous message) or at an initiators initiative. When issued on the connection to be logged out, no other request may follow it. The Logout Response indicates that the connection or session cleanup is completed and no other responses will arrive on the connection (if received on the logging out connection). In addition, the Logout Response indicates how long the target will continue to hold resources for recovery (e.g., command execution that continues on a Satran, et al. Standards Track [Page 47] RFC 3720 iSCSI April 2004 new connection) in the text key Time2Retain and how long the initiator must wait before proceeding with recovery in the text key Time2Wait. 3.5.3.4. SNACK Request With the SNACK Request, the initiator requests retransmission of numbered-responses or data from the target. A single SNACK request covers a contiguous set of missing items, called a run, of a given type of items. The type is indicated in a type field in the PDU header. The run is composed of an initial item (StatSN, DataSN, R2TSN) and the number of missed Status, Data, or R2T PDUs. For long Data-In sequences, the target may request (at predefined minimum intervals) a positive acknowledgement for the data sent. A SNACK request with a type field that indicates ACK and the number of Data-In PDUs acknowledged conveys this positive acknowledgement. 3.5.3.5. Reject Reject enables the target to report an iSCSI error condition (e.g., protocol, unsupported option) that uses a Reason field in the PDU header and includes the complete header of the bad PDU in the Reject PDU data segment. 3.5.3.6. NOP-Out Request and NOP-In Response This request/response pair may be used by an initiator and target as a "ping" mechanism to verify that a connection/session is still active and all of its components are operational. Such a ping may be triggered by the initiator or target. The triggering party indicates that it wants a reply by setting a value different from the default 0xffffffff in the corresponding Initiator/Target Transfer Tag. NOP-In/NOP-Out may also be used "unidirectional" to convey to the initiator/target command, status or data counter values when there is no other "carrier" and there is a need to update the initiator/ target. 4. SCSI Mode Parameters for iSCSI There are no iSCSI specific mode pages. 5. Login and Full Feature Phase Negotiation iSCSI parameters are negotiated at session or connection establishment by using Login Requests and Responses (see Section 3.2.3 iSCSI Login) and during the Full Feature Phase (Section 3.2.4 iSCSI Full Feature Phase) by using Text Requests and Responses. In Satran, et al. Standards Track [Page 48] RFC 3720 iSCSI April 2004 both cases the mechanism used is an exchange of iSCSI-text-key=value pairs. For brevity iSCSI-text-keys are called just keys in the rest of this document. Keys are either declarative or require negotiation and the key description indicates if the key is declarative or requires negotiation. For the declarative keys, the declaring party sets a value for the key. The key specification indicates if the key can be declared by the initiator, target or both. For the keys that require negotiation one of the parties (the proposing party) proposes a value or set of values by including the key=value in the data part of a Login or Text Request or Response PDUs. The other party (the accepting party) makes a selection based on the value or list of values proposed and includes the selected value in a key=value in the data part of one of the following Login or Text Response or Request PDUs. For most of the keys both the initiator and target can be proposing parties. The login process proceeds in two stages - the security negotiation stage and the operational parameter negotiation stage. Both stages are optional but at least one of them has to be present to enable the setting of some mandatory parameters. If present, the security negotiation stage precedes the operational parameter negotiation stage. Progression from stage to stage is controlled by the T (Transition) bit in the Login Request/Response PDU header. Through the T bit set to 1, the initiator indicates that it would like to transition. The target agrees to the transition (and selects the next stage) when ready. A field in the Login PDU header indicates the current stage (CSG) and during transition, another field indicates the next stage (NSG) proposed (initiator) and selected (target). The text negotiation process is used to negotiate or declare operational parameters. The negotiation process is controlled by the F (final) bit in the PDU header. During text negotiations, the F bit is used by the initiator to indicate that it is ready to finish the negotiation and by the Target to acquiesce the end of negotiation. Since some key=value pairs may not fit entirely in a single PDU, the C (continuation) bit is used (both in Login and Text) to indicate that "more follows". Satran, et al. Standards Track [Page 49] RFC 3720 iSCSI April 2004 The text negotiation uses an additional mechanism by which a target may deliver larger amounts of data to an enquiring initiator. The target sets a Target Task Tag to be used as a bookmark that when returned by the initiator, means "go on". If reset to a "neutral value", it means "forget about the rest". This chapter details types of keys and values used, the syntax rules for parameter formation, and the negotiation schemes to be used with different types of parameters. 5.1. Text Format The initiator and target send a set of key=value pairs encoded in UTF-8 Unic