idnits 2.17.1 draft-ietf-ips-iscsi-reqmts-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 111: '... MUST allow implementations to equal...' RFC 2119 keyword, line 114: '... MUST enable cost competitive implem...' RFC 2119 keyword, line 116: '... SHOULD minimize control overhead to...' RFC 2119 keyword, line 118: '... MUST provide high bandwidth and ban...' RFC 2119 keyword, line 120: '... MUST have low host CPU utilizations...' (135 more instances...) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 2001) is 8321 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 19 -- Looks like a reference, but probably isn't: '2' on line 71 -- Looks like a reference, but probably isn't: '10' on line 689 -- Looks like a reference, but probably isn't: '11' on line 689 == Missing Reference: 'SAM2' is mentioned on line 783, but not defined -- Looks like a reference, but probably isn't: '12' on line 943 == Missing Reference: '99-245r9' is mentioned on line 965, but not defined == Unused Reference: 'SPC-2' is defined on line 1098, but no explicit reference was found in the text == Unused Reference: 'CAM-3' is defined on line 1100, but no explicit reference was found in the text == Unused Reference: '99-245r8' is defined on line 1103, but no explicit reference was found in the text == Unused Reference: 'RFC1783' is defined on line 1114, but no explicit reference was found in the text ** Obsolete normative reference: RFC 1783 (Obsoleted by RFC 2348) Summary: 6 errors (**), 0 flaws (~~), 7 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 IP Storage Working Group M. Krueger 3 R. Haagens 4 Internet Draft Hewlett-Packard 5 Corporation 6 Category: Informational 7 C. Sapuntzakis 8 M. Bakke 9 Cisco Systems 11 Document: draft-ietf-ips-iscsi-reqmts-05.txt July 2001 13 iSCSI Requirements and Design Considerations 15 Status of this Memo 17 This document is an Internet-Draft and is in full conformance with 18 all provisions of Section 10 of RFC2026 [1]. 20 Internet-Drafts are working documents of the Internet Engineering 21 Task Force (IETF), its areas, and its working groups. Note that 22 other groups may also distribute working documents as Internet- 23 Drafts. 25 Internet-Drafts are draft documents valid for a maximum of six 26 months and may be updated, replaced, or obsoleted by other documents 27 at any time. It is inappropriate to use Internet-Drafts as 28 reference material or to cite them other than as "work in progress." 30 The list of current Internet-Drafts can be accessed at 31 http://www.ietf.org/ietf/1id-abstracts.txt 33 The list of Internet-Draft Shadow Directories can be accessed at 34 http://www.ietf.org/shadow.html. 36 Abstract 38 The IP Storage Working group is chartered with developing 39 comprehensive technology to transport block storage data over IP 40 protocols. This effort includes a protocol to transport the Small 41 Computer Systems Interface (SCSI) protocol over the Internet 42 (iSCSI). The initial version of the iSCSI protocol will define a 43 mapping of SCSI transport protocol over TCP/IP so that SCSI storage 44 controllers (principally disk and tape arrays and libraries) can be 45 attached to IP networks, notably Gigabit Ethernet (GbE) and 10 46 Gigabit Ethernet (10 GbE). 48 This document specifies the requirements iSCSI and it's related 49 infrastructure should satisfy and the design considerations guiding 50 the iSCSI protocol development efforts. In the interest of timely 51 adoption of the iSCSI protocol, the IPS group has chosen to focus 52 iSCSI Reqmnts and Design Considerations Nov. 2000 54 the first version of the protocol to work with the existing SCSI 55 architecture and commands, and the existing TCP/IP transport layer. 56 Both these protocols are widely-deployed and well-understood. The 57 thought is that using these mature protocols will entail a minimum 58 of new invention, the most rapid possible adoption, and the greatest 59 compatibility with Internet architecture, protocols, and equipment. 61 The iSCSI protocol is a mapping of SCSI to TCP, and constitutes a 62 "SCSI transport" as defined by the ANSI T10 document SCSI SAM-2 63 document [SAM2, p. 3, "Transport Protocols"]. 65 Conventions used in this document 67 This document describes the requirements for a protocol design, 68 but does not define a protocol standard. Nevertheless, the 69 key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 70 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 71 this document are to be interpreted as described in RFC-2119 [2]. 73 Table of Contents 75 1. Summary of Requirements...........................................3 76 2. iSCSI Design Considerations.......................................7 77 2.1. General Discussion..............................................7 78 2.2. Performance/Cost................................................8 79 2.3. Framing.........................................................10 80 2.4. High bandwidth, bandwidth aggregation...........................11 81 3. Ease of implementation/complexity of protocol.....................13 82 4. Reliability and Availability......................................13 83 4.1. Detection of Data Corruption....................................14 84 4.2. Recovery........................................................14 85 5. Interoperability..................................................15 86 5.1. Internet infrastructure.........................................15 87 5.2. SCSI............................................................15 88 6. Security Considerations...........................................16 89 6.1. Extensible Security.............................................17 90 6.2. Authentication..................................................17 91 6.3. Data Integrity..................................................18 92 6.4. Data Confidentiality............................................18 93 7. Management........................................................18 94 7.1. Naming..........................................................18 95 7.2. Discovery.......................................................19 96 8. Internet Accessibility............................................20 97 8.1. Denial of Service...............................................20 98 8.2. Firewalls and Proxy servers.....................................20 99 8.3. Congestion Control and Transport Selection......................21 100 9. Definitions.......................................................21 101 10. References........................................................21 102 11. Acknowledgements..................................................22 103 12. Author's Addresses................................................22 104 iSCSI Reqmnts and Design Considerations Nov. 2000 106 1. Summary of Requirements 108 The iSCSI standard: 110 >From section 2.2 Performance/Cost: 111 MUST allow implementations to equal or improve on the current state 112 of the art for SCSI interconnects. 114 MUST enable cost competitive implementations. 116 SHOULD minimize control overhead to enable low delay communications. 118 MUST provide high bandwidth and bandwidth aggregation. 120 MUST have low host CPU utilizations, equal to or better than current 121 technology. 123 MUST be possible to build I/O adapters that handle the entire SCSI 124 task. 126 SHOULD permit direct data placement architectures. 128 MUST NOT impose complex operations on host software. 130 MUST provide for full utilization of available link bandwidth. 132 MUST allow an implementation to exploit parallelism (multiple 133 connections) at the device interfaces and within the interconnect 134 fabric. 136 >From section 2.4 High Bandwidth/Bandwidth Aggregation: 137 MUST operate over a single TCP connection. 139 SHOULD support 'connection binding', and it MUST be optional to 140 implement. 142 >From section 3 Ease of Implementation/Complexity of Protocol: 143 SHOULD keep the protocol simple. 145 SHOULD minimize optional features. 147 MUST specify feature negotiation at session establishment (login). 149 MUST operate correctly when no optional features are negotiated as 150 well as when individual option negotions are unsuccessful. 152 >From section 4.1 Detection of Data Corruption: 153 MUST support a data integrity check format for use in digest 154 generation. 156 iSCSI Reqmnts and Design Considerations Nov. 2000 158 MAY use separate digest for data and headers. 160 iSCSI header format SHOULD be extensible to include other data 161 integrity digest calculation methods. 163 >From section 4.2 Recovery: 164 MUST specify mechanisms to recover in a timely fashion from 165 failures on the initiator, target, or connecting infrastructure. 167 MUST specify recovery methods for non-idempotent requests. 169 SHOULD take into account fail-over schemes for mirrored targets or 170 highly available storage configurations. 172 SHOULD provide a method for sessions to be gracefully terminated and 173 restarted that can be initiated by either the initiator or target. 175 >From section 5 Interoperability: 176 iSCSI protocol document MUST be clear and unambiguous. 178 >From section 5.1 Internet Infrastructure: 179 MUST: 180 -- be compatible with both IPv4 and IPv6 181 -- use TCP connections conservatively, keeping in mind there may be 182 many other users of TCP on a given machine. 184 MUST NOT require changes to existing Internet protocols. 186 SHOULD minimize required changes to existing TCP/IP 187 implementations. 189 >From section 5.2 SCSI: 190 Any feature SAM2 requires in a valid transport mapping MUST be 191 specified by iSCSI. 193 MUST specify strictly ordered delivery of SCSI commands over an 194 iSCSI session between an initiator/target pair. 196 The command ordering mechanism SHOULD seek to minimize the amount of 197 communication necessary across multiple adapters doing transport 198 off-load. 200 MUST specify for each feature whether it is OPTIONAL, RECOMMENDED or 201 REQUIRED to implement and/or use. 203 MUST NOT require changes to the SCSI-3 command sets and SCSI client 204 code except except where SCSI specifications point to "transport 205 dependant" fields and behavior. 207 SHOULD track changes to SCSI and the SCSI Architecture Model. 209 iSCSI Reqmnts and Design Considerations Nov. 2000 211 MUST be capable of supporting all SCSI-3 command sets and device 212 types. 214 SHOULD support ACA implementation. 216 MUST allow for the construction of gateways to other SCSI transports 218 MUST reliably transport SCSI commands from the initiator to the 219 target. 221 MUST correctly deal with iSCSI packet drop, duplication, corruption, 222 stale packets, and re-ordering. 224 >From section 6.1 Extensible Security: 225 SHOULD require minimal configuration and overhead in the insecure 226 operation. 228 SHOULD provide for strong authentication when increased security is 229 required. 231 SHOULD allow integration of new security mechanisms without breaking 232 backwards compatible operation. 234 >From section 6.2 Authentication: 235 MAY support various levels of authentication security. 237 MUST support private authenticated login. 239 iSCSI authenticated login MUST be resilient against passive attacks. 241 MUST support data origin authentication of its communications; data 242 origin authentication MAY be optional to use. 244 >From section 6.3 Data Integrity: 245 SHOULD NOT preclude use of additional data integrity protection 246 protocols (IPSec, TLS). 248 >From section 6.4 Data Confidentiality: 249 MAY use a data encryption protocol such as TLS or IPsec ESP to 250 provide data confidentiality between iSCSI endpoints. 252 >From section 7 Management: 253 SHOULD be manageable using standard IP-based management protocols 254 (eg. SNMP, RMI, etc). 256 iSCSI protocol document MUST NOT define the management architecture 257 for iSCSI, or make explicit references to management objects such as 258 MIB variables. 260 >From section 7.1 Naming: 261 MUST support the naming architecture of SAM-2. 263 iSCSI Reqmnts and Design Considerations Nov. 2000 265 The means by which an iSCSI resource is located MUST use or extend 266 existing Internet standard resource location methods. 268 MUST provide a means of identifying iSCSI targets by a unique 269 identifier that is independent of the path on which it is found. 271 The format for the iSCSI names MUST use existing naming authorities. 273 An iSCSI name SHOULD be a human readable string in an international 274 character set encoding. 276 Standard Internet lookup services SHOULD be used to resolve iSCSI 277 names. 279 SHOULD deal with the complications of the new SCSI security 280 architecture. 282 iSCSI naming architecture MUST address support of SCSI 3rd party 283 operations such as EXTENDED COPY. 285 >From section 7.2 Discovery: 287 MUST have no impact on the use of current IP network discovery 288 techniques. 290 MUST provide some means of determining whether an iSCSI service is 291 available through an IP address. 293 SCSI protocol-dependent techniques SHOULD be used for further 294 discovery beyond the iSCSI layer. 296 MUST provide a method of discovering, given an IP end point on its 297 well-known port, the list of SCSI targets available to the 298 requestor. The use of this discovery service MUST be optional. 300 >From section 8 Internet Accessability. 302 SHOULD be scrutinized for denial of service issues and they should 303 be addressed. 305 >From section 8.2 Firewalls and Proxy Servers 307 SHOULD allow deployment where functional and optimizing middle-boxes 308 such as firewalls, proxy servers and NATs are present. 310 use of IP addresses and TCP ports SHOULD be firewall friendly. 312 >From section 8.3 Congestion Control and Transport Selection 314 MUST be a good network citizen with TCP-compatible congestion 315 control (as defined in RFC 2309). 317 iSCSI Reqmnts and Design Considerations Nov. 2000 319 iSCSI implementations MUST NOT use multiple connections as a means 320 to avoid transport-layer congestion control. 322 2. iSCSI Design Considerations 323 2.1. General Discussion 325 Traditionally, storage controllers (e.g., disk array controllers, 326 tape library controllers) have supported the SCSI-3 protocol and 327 have been attached to computers by SCSI parallel bus or Fibre 328 Channel. 330 The IP infrastructure offers compelling advantages for volume/block- 331 oriented storage attachment. It offers the opportunity to take 332 advantage of the performance/cost benefits provided by competition 333 in the Internet marketplace. This could reduce the cost of storage 334 network infrastructure by providing economies arising from the need 335 to install and operate only a single type of network. 337 In addition, the IP protocol suite offers the opportunity for a rich 338 array of management, security and QoS solutions. Organizations may 339 initially choose to operate storage networks based on iSCSI that are 340 independent of (isolated from) their current data networks except 341 for secure routing of storage management traffic. These 342 organizations anticipate benefits from the high performance/cost of 343 IP equipment and a the opportunity for a unified management 344 architecture. As security and QoS evolve, it becomes reasonable to 345 build combined networks with shared infrastructure; nevertheless, it 346 is likely that sophisticated users will choose to keep their storage 347 sub-networks isolated to afford the best control of security and QoS 348 to ensure a high-performance environment tuned to storage traffic. 350 Mapping SCSI over IP also provides: 352 -- Extended distance ranges 353 -- Connectivity to "carrier class" services that support IP 355 The following applications for iSCSI are contemplated: 357 -- Local storage access, consolidation, clustering and pooling (as 358 in the data center) 359 -- Network client access to remote storage (eg. a "storage service 360 provider") 361 -- Local and remote synchronous and asynchronous mirroring between 362 storage controllers 363 -- Local and remote backup and recovery 365 iSCSI will support the following topologies: 367 -- Point-to-point direct connections 368 iSCSI Reqmnts and Design Considerations Nov. 2000 370 -- Dedicated storage LAN, consisting of one or more LAN segments 371 -- Shared LAN, carrying a mix of traditional LAN traffic plus 372 storage traffic 373 -- LAN-to-WAN extension using IP routers or carrier-provided "IP 374 Datatone" 375 -- Private networks and the public Internet 377 IP LAN-WAN routers may be used to extend the IP storage network to 378 the wide area, permitting remote disk access (as for a storage 379 utility), synchronous and asynchronous remote mirroring, and remote 380 backup and restore (as for tape vaulting). In the WAN, using TCP 381 end-to-end avoids the need for specialized equipment for protocol 382 conversion, ensures data reliability, copes with network congestion, 383 and provides retransmission strategies adapted to WAN delays. 385 The iSCSI technology deployment will involve the following elements: 386 (1) Conclusion of a complete protocol standard and supporting 387 implementations; 388 (2) Development of Ethernet storage NICs and related driver and 389 protocol software; [NOTE: high-speed applications of iSCSI are 390 expected to require significant portions of the iSCSI/TCP/IP 391 implementation in hardware to achieve the necessary 392 throughput.] 393 (3) Development of compatible storage controllers; and 394 (4) The likely development of translating gateways to provide 395 connectivity between the Ethernet storage network and the 396 Fibre Channel and/or parallel-bus SCSI domains. 397 (5) Development of specifications for iSCSI device management such 398 as MIBs, LDAP or XML schemas, etc. 399 (6) Development of management and directory service applications 400 to support a robust SAN infrastructure. 402 Products could initially be offered for Gigabit Ethernet attachment, 403 with rapid migration to 10 GbE. For performance competitive with 404 alternative SCSI transports, it will be necessary to implement the 405 performance path of the full protocol stack in hardware. These new 406 storage NICs might perform full-stack processing of a complete SCSI 407 task, analogous to today's SCSI and Fibre Channel HBAs, and might 408 also support all host protocols that use TCP (NFS, CIFS, HTTP, etc). 410 The charter of the IETF IP Storage Working Group (IPSWG) describes 411 the broad goal of mapping SCSI to IP using a transport that has 412 proven congestion avoidance behavior and broad implementation on a 413 variety of platforms. Within that broad charter, several transport 414 alternatives may be considered. Initial IPS work focuses on TCP, 415 and this requirements document is restricted to that domain of 416 interest. 418 2.2. Performance/Cost 419 iSCSI Reqmnts and Design Considerations Nov. 2000 421 In general, iSCSI MUST allow implementations to equal or improve on 422 the current state of the art for SCSI interconnects. This goal 423 breaks down into several types of requirement: 425 Cost competitive with alternative storage network technologies: 427 In order to be adopted by vendors and the user community, the iSCSI 428 protocol MUST enable cost competitive implementations when compared 429 to other SCSI transports (Fibre Channel). 431 Low delay communication: 433 Conventional storage access is of a stop-and-wait or remote 434 procedure call type. Applications typically employ very little 435 pipelining of their storage accesses, and so storage access delay 436 directly impacts performance. The delay imposed by current storage 437 interconnects, including protocol processing, is generally in the 438 range of 100 microseconds. The use of caching in storage 439 controllers means that many storage accesses complete almost 440 instantly, and so the delay of the interconnect can have a high 441 relative impact on overall performance. When stop-and-wait IO is 442 used, the delay of the interconnect will affect performance. The 443 iSCSI protocol SHOULD minimize control overhead, which adds to 444 delay. 446 Low host CPU utilization, equal to or better than current 447 technology: 449 For competitive performance, the iSCSI protocol MUST allow three key 450 implementation goals to be realized: 452 (1) iSCSI MUST make it possible to build I/O adapters that handle 453 an entire SCSI task, as alternative SCSI transport 454 implementations do. 455 (2) The protocol SHOULD permit direct data placement ("zero-copy" 456 memory architectures, where the I/O adapter reads or writes 457 host memory exactly once per disk transaction. 458 (3) The protocol SHOULD NOT impose complex operations on the host 459 software, which would increase host instruction path length 460 relative to alternatives. 462 Direct data placement (zero-copy iSCSI): 464 Direct data placement refers to iSCSI data being placed directly 465 "off the wire" into the allocated location in memory with no 466 intermediate copies. Direct data placement significantly reduces 467 the memory bus and I/O bus loading in the endpoint systems, allowing 468 improved performance. It reduces the memory required for NICs, 469 possibly reducing the cost of these solutions. 471 This is an important implementation goal. In an iSCSI system, each 472 of the end nodes (for example host computer and storage controller) 473 iSCSI Reqmnts and Design Considerations Nov. 2000 475 should have ample memory, but the intervening nodes (NIC, switches) 476 typically will not. 478 High bandwidth, bandwidth aggregation: 480 The bandwidth (transfer rate, MB/sec) supported by storage 481 controllers is rapidly increasing, due to several factors: 483 1. Increase in disk spindle and controller performance; 484 2. Use of ever-larger caches, and improved caching algorithms; 485 3. Increased scale of storage controllers (number of supported 486 spindles, speed of interconnects). 488 The iSCSI protocol MUST provide for full utilization of available 489 link bandwidth. The protocol MUST also allow an implementation to 490 exploit parallelism (multiple connections) at the device interfaces 491 and within the interconnect fabric. 493 The next two sections further discuss the need for direct data 494 placement and high bandwidth. 496 2.3. Framing 498 Framing refers to the addition of information in a header, or the 499 data stream to allow implementations to locate the boundaries of an 500 iSCSI protocol data unit (PDU) within the TCP byte stream. There 501 are two technical requirements driving framing: interfacing needs, 502 and accelerated processing needs. 504 A framing solution that addresses the "interfacing needs" of the 505 iSCSI protocol will facilitate the implementation of a message-based 506 upper layer protocol (iSCSI) on top of an underlying byte streaming 507 protocol (TCP). Since TCP is a reliable transport, this can be 508 accomplished by including a length field in the iSCSI header. 509 Finding the protocol frame assumes that the receiver will parse from 510 the beginning of the TCP data stream, and never make a mistake (lose 511 alignment on packet headers). 513 The other technical requirement for framing, "accelerated 514 processing", stems from the need to handle increasingly higher data 515 rates in the physical media interface. Two needs arise from higher 516 data rates: 518 (1) LAN environment - NIC vendors seek ways to provide "zero-copy" 519 methods of moving data directly from the wire into application 520 buffers. 522 (2) WAN environment- the emergence of high bandwidth, high latency, 523 low bit error rate physical media places huge buffer 524 requirements on the physical interface solutions. 526 iSCSI Reqmnts and Design Considerations Nov. 2000 528 First, vendors are producing network processing hardware that 529 offloads network protocols to hardware solutions to achieve higher 530 data rates. The concept of "zero-copy" seeks to store blocks of 531 data in appropriate memory locations (aligned) directly off the 532 wire, even when data is reordered due to packet loss. This is 533 necessary to drive actual data rates of 10 Gigabit/sec and beyond. 535 Secondly, in order for iSCSI to be successful in the WAN arena it 536 must be possible to operate efficiently in high bandwidth, high 537 delay networks. The emergence of multi-gigabit IP networks with 538 latencies in the tens to hundreds of milliseconds presents a 539 challenge. To fill such large pipes, it is necessary to have tens of 540 megabytes of outstanding requests from the application. In addition, 541 some protocols potentially require tens of megabytes at the 542 transport layer to deal with buffering for reassembly of data when 543 packets are received out-of-order. 545 In both cases, the issue is the desire to minimize the amount of 546 memory and memory bandwidth required for iSCSI hardware solutions. 548 Consider that a network pipe at 10 Gbps x 200 msec holds 250 MB. 549 [Assume land-based communication with a spot half way around the 550 world at the equator. Ignore additional distance due to cable 551 routing. Ignore repeater and switching delays; consider only a 552 speed-of-light delay of 5 microsec/km. The circumference of the 553 globe at the equator is approx. 40000 km (round-trip delay must be 554 considered to keep the pipe full). 10 Gb/sec x 40000 km x 5 555 microsec/km x B / 8b = 250 MB]. In a conventional TCP 556 implementation, loss of a TCP segment means that stream processing 557 MUST stop until that segment is recovered, which takes at least a 558 time of to accomplish. Following the example 559 above, an implementation would be obliged to catch 250 MB of data 560 into an anonymous buffer before resuming stream processing; later, 561 this data would need to be moved to its proper location. Some 562 proponents of iSCSI seek some means of putting data directly where 563 it belongs, and avoiding extra data movement in the case of segment 564 drop. This is a key concept in understanding the debate behind 565 framing methodologies. 567 The framing of the iSCSI protocol impacts both the "interfacing 568 needs" and the "accelerated processing needs", however, while 569 including a length in a header may suffice for the "interfacing 570 needs", it will not serve the direct data placement needs. The 571 framing mechanism developed should allow resynchronization of packet 572 boundaries even in the case where a packet is temporarily missing in 573 the incoming data stream. 575 2.4. High bandwidth, bandwidth aggregation 577 At today's block storage transport throughput, any single link can 578 be saturated by the volume of storage traffic. Scientific data 579 iSCSI Reqmnts and Design Considerations Nov. 2000 581 applications and data replication are examples of storage 582 applications that push the limits of throughput. 584 Some applications, such as log updates, streaming tape, and 585 replication, require ordering of updates and thus ordering of SCSI 586 commands. An initiator may maintain ordering by waiting for each 587 update to complete before issuing the next (a.k.a. synchronous 588 updates). However, the throughput of synchronous updates decreases 589 inversely with increases in network distances. 591 For greater throughput, the SCSI task queuing mechanism allows an 592 initiator to have multiple commands outstanding at the target 593 simultaneously and to express ordering constraints on the execution 594 of those commands. The task queuing mechanism is only effective if 595 the commands arrive at the target in the order they were presented 596 to the initiator (FIFO order). The iSCSI standard must provide an 597 ordered transport of SCSI commands, even when commands are sent 598 along different network paths (see Section 5.2 SCSI). This is 599 referred to as "command ordering". 601 The iSCSI protocol MUST operate over a single TCP connection to 602 accomodate lower cost implementations. To enable higher performance 603 storage devices, the protocol should specify a means to allow 604 operation over multiple connections while maintaining the behavior 605 of a single SCSI port. This would allow the initiator and target 606 to use multiple network interfaces and multiple paths through the 607 network for increased throughput. There are a few potential ways to 608 satisfy the multiple path and ordering requirements. 610 A popular way to satisfy the multiple-path requirement is to have a 611 driver above the SCSI layer instantiate multiple copies of the SCSI 612 transport, each communicating to the target along a different path. 613 "Wedge" drivers use this technique today to attain high performance. 614 Unfortunately, wedge drivers must wait for acknowledgement of 615 completion of each request (stop-and-wait) to ensure ordered 616 updates. 618 Another approach might be for iSCSI protocol to use multiple 619 instances of its underlying transport (e.g. TCP). The iSCSI layer 620 would make these independent transport instances appear as one SCSI 621 transport instance and maintain the ability to do ordered SCSI 622 command queuing. The document will refer to this technique as 623 "connection binding" for convenience. 625 The iSCSI protocol SHOULD support connection binding, and it MUST be 626 optional to implement. 628 In the presence of connection binding, there are two ways to assign 629 features to connections. In the symmetric approach, all the 630 connections are identical from a feature standpoint. In the 631 asymmetric model, connections have different features. For example, 632 some connections may be used primarily for data transfers whereas 633 others are used primarily for SCSI commands. 635 iSCSI Reqmnts and Design Considerations Nov. 2000 637 Since the iSCSI protocol must support the case where there was only 638 one transport connection, the protocol must have command, data, and 639 status travel over the same connection. 641 In the case of multiple connections, the iSCSI protocol must keep 642 the command and its associated data and status on the same 643 connection (connection allegiance). Sending data and status on the 644 same connection is desirable because this guarantees that status is 645 received after the data (TCP provides ordered delivery). In the case 646 where each connection is managed by a separate processor, allegiance 647 decreases the need for inter-processor communication. This 648 symmetric approach is a natural extension of the single connection 649 approach. 651 An alternate approach that was extensively discussed involved 652 sending all commands on a single connection and the associated data 653 and status on a different connection (asymetric approach). In this 654 scheme, the transport ensures the commands arrive in order. The 655 protocol on the data and status connections is simpler, perhaps 656 lending itself to a simpler realization in hardware. One 657 disadvantage of this approach is that the recovery procedure is 658 different if a command connection fails vs. a data connection. Some 659 argued that this approach would require greater inter-processor 660 communication when connections are spread across processors. 662 The reader may reference the mail archives of the IPS mailing list 663 between June and September of 2000 for extensive discussions on 664 symmetric vs asymmetric connection models. 666 3. Ease of implementation/complexity of protocol 668 Experience has shown that adoption of a protocol by the Internet 669 community is inversely proportional to its complexity. In addition, 670 the simpler the protocol, the easier it is to diagnose problems. 671 The designers of iSCSI SHOULD strive to fulfill the requirements of 672 the creating a SCSI transport over IP, while keeping the protocol as 673 simple as possible. 675 In the interest of simplicity, iSCSI SHOULD minimize optional 676 features. When features are deemed necessary, the protocol MUST 677 specify feature negotiation at session establishment (login). The 678 iSCSI transport MUST operate correctly when no optional features are 679 negotiated as well as when individual option negotions are 680 unsuccessful. 682 4. Reliability and Availability 683 iSCSI Reqmnts and Design Considerations Nov. 2000 685 4.1. Detection of Data Corruption 687 There have been several research papers that suggest that the TCP 688 checksum calculation allows a certain number of bit errors to pass 689 undetected [10] [11]. 691 In order to protect against data corruption, the iSCSI protocol MUST 692 support a data integrity check format for use in digest generation. 694 The iSCSI protocol MAY use separate digests for data and headers. In 695 an iSCSI proxy or gateway situation, the iSCSI headers are removed 696 and re-built, and the TCP stream is terminated on either side. This 697 means that even the TCP checksum is removed and recomputed within 698 the gateway. To ensure the protection of commands, data, and status 699 the iSCSI protocol MUST include a CRC or other digest mechanism that 700 is computed on the SCSI data block itself, as well as on each 701 command and status message. Since gateways may strip iSCSI headers 702 and rebuild them, a separate header CRC is required. Two header 703 digests, one for invariant portions of the header (addresses) and 704 one for the variant portion would provide protection against changes 705 to portions of the header that should never be changed by middle 706 boxes (eg, addresses). 708 The iSCSI header format SHOULD be extensible to include other digest 709 calculation methods. 711 4.2. Recovery 713 The SCSI protocol was originally designed for a parallel bus 714 transport that was highly reliable. SCSI applications tend to 715 assume that transport errors never happen, and when they do, SCSI 716 application recovery tends to be expensive in terms of time and 717 computational resources. 719 iSCSI protocol design, while placing an emphasis on simplicity, MUST 720 lead to timely recovery from failure of initiator, target, or 721 connecting network infrastructure (cabling, data path equipment such 722 as routers, etc). 724 iSCSI MUST specify recovery methods for non-idempotent requests, 725 such as operations on tape drives. 727 The iSCSI protocol error recover mechanism SHOULD take into account 728 fail-over schemes for mirrored targets or highly available storage 729 configurations that provide paths to target data through multiple 730 "storage servers". This would provide a basis for layered 731 technologies like high availability and clustering. 733 The iSCSI protocol SHOULD also provide a method for sessions to be 734 gracefully terminated and restarted that can be initiated by either 735 the initiator or target. This provides the ability to gracefully 736 iSCSI Reqmnts and Design Considerations Nov. 2000 738 fail over an initiator or target, or reset a target after performing 739 maintenance tasks such as upgrading software. 741 5. Interoperability 743 It must be possible for initiators and targets that implement the 744 required portions of the iSCSI specification to interoperate. While 745 this requirement is so obvious that it doesn't seem worth 746 mentioning, if the protocol specification contains ambiguous 747 wording, different implementations may not interoperate. The iSCSI 748 protocol document MUST be clear and unambiguous. 750 5.1. Internet infrastructure 752 The iSCSI protocol MUST: 753 -- be compatible with both IPv4 and IPv6. 754 -- use TCP connections conservatively, keeping in mind there may be 755 many other users of TCP on a given machine. 757 The iSCSI protocol MUST NOT require changes to existing Internet 758 protocols and SHOULD minimize required changes to existing TCP/IP 759 implementations. 761 5.2. SCSI 763 In order to be considered a SCSI transport, the iSCSI standard must 764 comply with the requirements of the SCSI Architecture Model [SAM2] 765 for a SCSI transport. Any feature SAM2 requires in a valid 766 transport mapping MUST be specified by iSCSI. The iSCSI protocol 767 document MUST specify for each feature whether it is OPTIONAL, 768 RECOMMENDED or REQUIRED to implement and/or use. 770 The SCSI Architectural Model [SAM-2] indicates an expectation that 771 the SCSI transport provides ordering of commands on an initiator- 772 target-LUN granularity. There has been much discussion on the IPS 773 reflector and in working group meetings regarding the means to 774 ensure this ordering. The rough consensus is that iSCSI MUST 775 specify strictly ordered delivery of SCSI commands over an iSCSI 776 session between an initiator/target pair, even in the presence of 777 transport errors. This command ordering mechanism SHOULD seek to 778 minimize the amount of communication necessary across multiple 779 adapters doing transport off-load. If an iSCSI implementation does 780 not require ordering it can instantiate multiple sessions per 781 initiator-target pair. 783 iSCSI is intended to be a new SCSI "transport" [SAM2]. As a mapping 784 of SCSI over TCP, iSCSI requires interaction with both T10 and IETF. 785 However, the iSCSI protocol MUST NOT require changes to the SCSI-3 786 iSCSI Reqmnts and Design Considerations Nov. 2000 788 command sets and SCSI client code except where SCSI specifications 789 point to "transport dependant" fields and behavior. For example, 790 changes to SCSI documents will be necessary to reflect lengthier 791 iSCSI target names and potentially lengthier timeouts. 792 Collaboration with T10 will be necessary to achieve this 793 requirement. 795 The iSCSI protocol SHOULD track changes to SCSI and the SCSI 796 Architecture Model. 798 The iSCSI protocol MUST be capable of supporting all SCSI-3 command 799 sets and device types. The primary focus is on supporting 'larger' 800 devices: host computers and storage controllers (disk arrays, tape 801 libraries). However, other command sets (printers, scanners) must 802 be supported. These requirements MUST NOT be construed to mean that 803 iSCSI must be natively implementable on all of today's SCSI devices, 804 which might have limited processing power or memory. 806 ACA (Auto Contingent Allegiance) is an optional SCSI mechanism that 807 stops execution of a sequence of dependent SCSI commands when one of 808 them fails. The situation surrounding it is complex - T10 specifies 809 ACA in SAM2, and hence iSCSI must support it and endeavor to make 810 sure that ACA gets implemented sufficiently (two independent 811 interoperable implementations) to avoid dropping ACA in the 812 transition from Proposed Standard to Draft Standard. This implies 813 iSCSI SHOULD support ACA implementation. 815 The iSCSI protocol MUST allow for the construction of gateways to 816 other SCSI transports, including parallel SCSI [SPI-X] and to SCSI- 817 FCP[FCP, FCP-2]. It MUST be possible to construct "translating" 818 gateways so that iSCSI hosts can interoperate with SCSI-X devices; 819 so that SCSI-X devices can communicate over an iSCSI network; and so 820 that SCSI-X hosts can use iSCSI targets (where SCSI-X refers to 821 parallel SCSI, SCSI-FCP, or SCSI over any other transport). This 822 requirement is implied by support for SAM-2, but is worthy of 823 emphasis. These are true application protocol gateways, and not just 824 bridge/routers. The different standards have only the SCSI-3 825 command set layer in common. These gateways are not mere packet 826 forwarders. 828 The iSCSI protocol MUST reliably transport SCSI commands from the 829 initiator to the target. According to [SAM-2, p. 17.] "The function 830 of the service delivery subsystem is to transport an error-free copy 831 of the request or response between the sender and the receiver" 832 [SAM-2, p. 22]. The iSCSI protocol MUST correctly deal with iSCSI 833 packet drop, duplication, corruption, stale packets, and re- 834 ordering. 836 6. Security Considerations 837 iSCSI Reqmnts and Design Considerations Nov. 2000 839 In the past, directly attached storage systems have implemented 840 minimal security checks because the physical connection offered 841 little chance for attack. Transporting block storage (SCSI) over 842 IP opens a whole new opportunity for a variety of malicious attacks. 843 Attacks can take the active form (identity spoofing, man-in-the- 844 middle) or the passive form (eavesdropping). 846 6.1. Extensible Security 848 The security services required for communications depends on the 849 individual network configurations and environments. Organizations 850 are setting up Virtual Private Networks(VPN), also known as 851 Intranets, that will require one set of security functions for 852 communications within the VPN and possibly many different security 853 functions for communications outside the VPN to support 854 geographically separate components. The iSCSI protocol is 855 applicable to a wide range of internetworking environments that may 856 employ different security policies. The protocol SHOULD require 857 minimal configuration and overhead in the insecure operation, 858 provide for strong authentication when increased security is 859 required, and allow integration of new security mechanisms without 860 breaking backwards compatible operation. 862 6.2. Authentication 864 The iSCSI protocol MAY support various levels of authentication 865 security, ranging from no authentication to secure authentication 866 using public or private keys. 868 The iSCSI protocol MUST support private authenticated login. 869 Authenticated login aids the target in blocking the unauthorized use 870 of SCSI resources. "Private" authenticated login mandates protected 871 identity exchange (no clear text passwords at a minimum). Since 872 block storage confidentiality is considered critical in enterprises 873 and many IP networks may have access holes, organizations will want 874 to protect their iSCSI resources. 876 The iSCSI authenticated login MUST be resilient against passive 877 attacks since many IP networks are vulnerable to packet inspection. 879 In addition, the iSCSI protocol MUST support data origin 880 authentication of its communications; data origin authentication MAY 881 be optional to use. Data origin authentication is critical since IP 882 networks are vulnerable to source spoofing, where a malicious third 883 party pretends to send packets from the initiator's IP address. 885 These requirements should be met using standard Internet protocols 886 such as IPsec or TLS. The endpoints may negotiate the authentication 887 method, optionally none. 889 iSCSI Reqmnts and Design Considerations Nov. 2000 891 6.3. Data Integrity 893 The iSCSI protocol SHOULD NOT preclude use of additional data 894 integrity protection protocols (IPSec, TLS). 896 6.4. Data Confidentiality 898 Block storage is used for storing sensitive information, where data 899 confidentiality is critical. An application may encrypt the data 900 blocks before writing them to storage - this provides the best 901 protection for the application. Even if the storage or 902 communications are compromised, the attacker will have difficulty 903 reading the data. 905 In certain environments, encryption may be desired to provide an 906 extra assurance of confidentiality. An iSCSI implementation MAY use 907 a data encryption protocol such as TLS or IPsec ESP to provide data 908 confidentiality between iSCSI endpoints. 910 7. Management 912 iSCSI implementations SHOULD be manageable using standard IP-based 913 management protocols (eg. SNMP, RMI, etc). However, the iSCSI 914 protocol document MUST NOT define the management architecture for 915 iSCSI within the network infrastructure. iSCSI will be yet another 916 resource service within a complex environment of network resources 917 (printers, file servers, NAS, application servers, etc). There will 918 certainly be efforts to design how the "block storage service" that 919 iSCSI devices provide is integrated into a comprehensive, shared 920 model, network management environment. A "network administrator" 921 (or "storage administrator") will desire to have integrated 922 applications for assigning user names, resource names, etc. and 923 indicating access rights. iSCSI devices presumably will want to 924 interact with these integrated network management applications. The 925 iSCSI protocol document will not attempt to solve that set of 926 problems, or specify means for devices to provide management agents. 927 In fact, there should be no mention of MIBs or any other means of 928 managing iSCSI devices as explicit references in the iSCSI protocol 929 document, because management data and protocols change with the 930 needs of the environment and the business models of the management 931 applications. 933 7.1. Naming 935 Whenever possible, iSCSI MUST support the naming architecture of 936 SAM-2. Deviations and uncertainties MUST be made explicit, and 937 comments and resolutions worked out between ANSI T10 and the IPS 938 working group. 940 iSCSI Reqmnts and Design Considerations Nov. 2000 942 The means by which an iSCSI resource is located MUST use or extend 943 existing Internet standard resource location methods. RFC 1783 [12] 944 specifies URL syntax and semantics which should be sufficiently 945 extensible for the iSCSI resource. 947 The iSCSI protocol MUST provide a means of identifying an iSCSI 948 storage device by a unique identifier that is independent of the 949 path on which it is found. This name will be used to correlate 950 alternate paths to the same device. The format for the iSCSI names 951 MUST use existing naming authorities, to avoid creating new central 952 administrative tasks. An iSCSI name SHOULD be a human readable 953 string in an international character set encoding. 955 Standard Internet lookup services SHOULD be used to resolve names. 956 For example, Domain Name Services (DNS) MAY be used to resolve the 957 portion of a URL to one or multiple IP addresses. When a 958 hostname resolves to multiple addresses, these addresses should be 959 equivalent for functional (possibly not performance) purposes. This 960 means that the addresses can be used interchangeably as long as 961 performance isn't a concern. For example, the same set of SCSI 962 targets MUST be accessible from each of these addresses. 964 An iSCSI device naming scheme MUST interact correctly with the 965 proposed SCSI security architecture [99-245r9]. Particular 966 attention must be directed to the proxy naming architecture defined 967 by the new security model. In this new model, a host is identified 968 by an Access ID, and SCSI Logical Unit Numbers (LUNs) can be mapped 969 in a manner that gives each AccessID a unique LU map. Thus, a given 970 LU within a target may be addressed by different LUNs. 972 The iSCSI naming architecture MUST address support of SCSI 3rd party 973 operations such as EXTENDED COPY. The key issue here relates to the 974 naming architecture for SCSI LUs - iSCSI must provide a means of 975 passing a name or handle between parties. iSCSI must specify a means 976 of providing a name or handle that could be used in the XCOPY 977 command and fit within the available space allocated by that 978 command. And it must be possible, of course, for the XCOPY target 979 (the third party) to dereference the name to the correct target and 980 LU. 982 7.2. Discovery 984 iSCSI MUST have no impact on the use of current IP network discovery 985 techniques. Network management platforms discover IP addresses and 986 have various methods of probing the services available through these 987 IP addresses. An iSCSI service should be evident using similar 988 techniques. 990 The iSCSI specifications MUST provide some means of determining 991 whether an iSCSI service is available through an IP address. It is 992 iSCSI Reqmnts and Design Considerations Nov. 2000 994 expected that iSCSI will be a point of service in a host, just as 995 SNMP, etc are points of services, associated with a well known port 996 number. 998 SCSI protocol-dependent techniques SHOULD be used for further 999 discovery beyond the iSCSI layer. Discovery is a complex, multi- 1000 layered process. The SCSI protocol specifications provide specific 1001 commands for discovering LUs and the commands associated with this 1002 process will also work over iSCSI. 1004 The iSCSI protocol MUST provide a method of discovering, given an IP 1005 end point on its well-known port, the list of SCSI targets available 1006 to the requestor. The use of this discovery service MUST be 1007 optional. 1009 Further discovery guidelines are outside the scope of this document 1010 and may be addressed in separate Informational drafts. 1012 8. Internet Accessibility 1013 8.1. Denial of Service 1015 As with all services, the denial of service by either incorrect 1016 implementations or malicious agents is always a concern. All 1017 aspects of the iSCSI protocol SHOULD be scrutinized for potential 1018 denial of service issues, and guarded against as much as possible. 1020 8.2. NATs, Firewalls and Proxy servers 1022 NATs (Network Address Translator), firewalls, and proxy servers are 1023 a reality in today's Internet. These devices present a number of 1024 challenges to device access methods being developed for iSCSI. For 1025 example, specifying a URL syntax for iSCSI resource connection 1026 allows an initiator to address an iSCSI target device both directly 1027 and through an iSCSI proxy server or NAT. iSCSI SHOULD allow 1028 deployment where functional and optimizing middle-boxes such as 1029 firewalls, proxy servers and NATs are present. 1031 The iSCSI protocols use of IP addressing and TCP port numbers MUST 1032 be firewall friendly. This means that all connection requests should 1033 normally be addressed to a specific, well-known TCP port. That way, 1034 firewalls can filter based on source and destination IP addresses, 1035 and destination (target) port number. Additional TCP connections 1036 would require different source port numbers (for uniqueness), but 1037 could be opened after a security dialogue on the control channel. 1039 It's important that iSCSI operate through a firewall to provide a 1040 possible means of defending against Denial of Service (DoS) assaults 1041 iSCSI Reqmnts and Design Considerations Nov. 2000 1043 from less-trusted areas of the network. It is assumed that a 1044 firewall will have much greater processing power for dismissing 1045 bogus connection requests than end nodes. 1047 8.3. Congestion Control and Transport Selection 1049 The iSCSI protocol MUST be a good network citizen with proven 1050 congestion control (as defined in RFC 2309). In addition, iSCSI 1051 implementations MUST NOT use multiple connections as a means to 1052 avoid transport-layer congestion control. 1054 9. Definitions 1056 Certain definitions are offered here, with references to the 1057 original document where applicable, in order to clarify the 1058 discussion of requirements. Definitions without references are the 1059 work of the authors and reviewers of this document. 1061 Logical Unit (LU): A target-resident entity that implements a device 1062 model and executes SCSI commands sent by an application client [SAM- 1063 2, sec. 3.1.50, p. 7]. 1065 Logical Unit Number (LUN): A 64-bit identifier for a logical unit 1066 [SAM-2, sec. 3.1.52, p. 7]. 1068 SCSI Device: A device that is connected to a service delivery 1069 subsystem and supports a SCSI application protocol [SAM-2, sec. 1070 3.1.78, p. 9]. 1072 Service Delivery Port (SDP): A device-resident interface used by the 1073 application client, device server, or task manager to enter and 1074 retrieve requests and responses from the service delivery subsystem. 1075 Synonymous with port (SAM-2 sec. 3.1.61) [SAM-2, sec. 3.1.89, p. 9]. 1077 Target: A SCSI device that receives a SCSI command and directs it to 1078 one or more logical units for execution [SAM-2 sec. 3.1.97, p. 10]. 1080 Task: An object within the logical unit representing the work 1081 associated with a command or a group of linked commands [SAM-2, sec. 1082 3.1.98, p. 10]. 1084 Transaction: A cooperative interaction between two objects, 1085 involving the exchange of information or the execution of some 1086 service by one object on behalf of the other [SAM-2, sec. 3.1.109, 1087 p. 10]. 1089 10. References 1090 iSCSI Reqmnts and Design Considerations Nov. 2000 1092 1 Bradner, S., "The Internet Standards Process -- Revision 3", BCP 1093 9, RFC 2026, October 1996. 1094 2 Bradner, S., "Key words for use in RFCs to Indicate Requirement 1095 Levels", BCP 14, RFC 2119, March 1997 1096 3 [SAM-2] ANSI NCITS. Weber, Ralph O., editor. SCSI Architecture 1097 Model -2 (SAM-2). T10 Project 1157-D. rev 13, 22 Mar 2000. 1098 4 [SPC-2] ANSI NCITS. Weber, Ralph O., editor. SCSI Primary 1099 Commands 2 (SPC-2). T10 Project 1236-D. rev 18, 21 May 2000. 1100 5 [CAM-3] ANSI NCITS. Dallas, William D., editor. Information 1101 Technology - Common Access Method - 3 (CAM-3)). X3T10 Project 1102 990D. rev 3, 16 Mar 1998. 1103 6 [99-245r8] Hafner, Jim. A Detailed Proposal for Access Controls. 1104 T10/99-245 revision 8, 26 Apr 2000. 1105 7 [SPI-X] ANSI NCITS. SCSI Parallel Interface - X. 1106 8 [FCP] ANSI NCITS. SCSI-3 Fibre Channel Protocol [ANSI 1107 X3.269:1996]. 1108 9 [FCP-2] ANSI NCITS. SCSI-3 Fibre Channel Protocol - 2 [T10/1144- 1109 D]. 1110 10 Paxon, V. End-to-end internet packet dynamics, IEEE Transactions 1111 on Networking 7,3 (June 1999) pg 277-292. 1112 11 Stone J., Partridge, C. When the CRC and TCP checksum disagree, 1113 ACM Sigcomm (Sept. 2000). 1114 12 [RFC1783] Berners-Lee, t., et.al.,"Uniform Resource Locators", RFC 1115 1783, December 1994. 1117 11. Acknowledgements 1119 Special thanks to Julian Satran, IBM and David Black, EMC for their 1120 extensive review comments. 1122 12. Author's Addresses 1124 Address comments to: 1126 Marjorie Krueger 1127 Hewlett-Packard Corporation 1128 8000 Foothills Blvd 1129 Roseville, CA 95747-5668, USA 1130 Phone: +1 916 785-2656 1131 Email: marjorie_krueger@hp.com 1133 Randy Haagens 1134 Hewlett-Packard Corporation 1135 8000 Foothills Blvd 1136 iSCSI Reqmnts and Design Considerations Nov. 2000 1138 Roseville, CA 95747-5668, USA 1139 Phone: +1 916 785-4578 1140 Email: Randy_Haagens@hp.com 1142 Costa Sapuntzakis 1143 Cisco Systems, Inc. 1144 170 W. Tasman Dr. 1145 San Jose, CA 95134, USA 1146 Phone: +1 408 525-5497 1147 Email: csapuntz@cisco.com 1149 Mark Bakke 1150 Cisco Systems, Inc. 1151 6450 Wedgwood Road 1152 Maple Grove, MN 55311 1153 Phone: +1 763 398-1054 1154 Email: mbakke@cisco.com 1155 iSCSI Reqmnts and Design Considerations Nov. 2000 1157 Full Copyright Statement 1159 "Copyright (C) The Internet Society (2001). All Rights Reserved. 1160 This document and translations of it may be copied and furnished to 1161 others, and derivative works that comment on or otherwise explain it 1162 or assist in its implementation may be prepared, copied, published 1163 and distributed, in whole or in part, without restriction of any 1164 kind, provided that the above copyright notice and this paragraph 1165 are included on all such copies and derivative works. However, this 1166 document itself may not be modified in any way, such as by removing 1167 the copyright notice or references to the Internet Society or other 1168 Internet organizations, except as needed for the purpose of 1169 developing Internet standards in which case the procedures for 1170 copyrights defined in the Internet Standards process must be 1171 followed, or as required to translate it into languages other than 1172 English. 1174 The limited permissions granted above are perpetual and will not be 1175 revoked by the Internet Society or its successors or assigns.