idnits 2.17.1 draft-ietf-nfsv4-pnfs-block-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 17. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 1109. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1086. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1093. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1099. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 59 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year == Line 212 has weird spacing: '... opaque conte...' -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (November 18, 2007) is 6002 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 Working Group David L. Black 3 Internet Draft Stephen Fridella 4 Expires: May 21, 2008 Jason Glasgow 5 Intended Status: Proposed Standard EMC Corporation 6 November 18, 2007 8 pNFS Block/Volume Layout 9 draft-ietf-nfsv4-pnfs-block-05.txt 11 Status of this Memo 13 By submitting this Internet-Draft, each author represents that 14 any applicable patent or other IPR claims of which he or she is 15 aware have been or will be disclosed, and any of which he or she 16 becomes aware will be disclosed, in accordance with Section 6 of 17 BCP 79. 19 Internet-Drafts are working documents of the Internet Engineering 20 Task Force (IETF), its areas, and its working groups. Note that 21 other groups may also distribute working documents as Internet- 22 Drafts. 24 Internet-Drafts are draft documents valid for a maximum of six months 25 and may be updated, replaced, or obsoleted by other documents at any 26 time. It is inappropriate to use Internet-Drafts as reference 27 material or to cite them other than as "work in progress." 29 The list of current Internet-Drafts can be accessed at 30 http://www.ietf.org/ietf/1id-abstracts.txt 32 The list of Internet-Draft Shadow Directories can be accessed at 33 http://www.ietf.org/shadow.html 35 This Internet-Draft will expire in September 2007. 37 Abstract 39 Parallel NFS (pNFS) extends NFSv4 to allow clients to directly access 40 file data on the storage used by the NFSv4 server. This ability to 41 bypass the server for data access can increase both performance and 42 parallelism, but requires additional client functionality for data 43 access, some of which is dependent on the class of storage used. The 44 main pNFS operations draft specifies storage-class-independent 45 extensions to NFS; this draft specifies the additional extensions 46 (primarily data structures) for use of pNFS with block and volume 47 based storage. 49 Conventions used in this document 51 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 52 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 53 document are to be interpreted as described in RFC-2119 [RFC2119]. 55 Table of Contents 57 1. Introduction...................................................3 58 1.1. General Definitions.......................................3 59 2. Block Layout Description.......................................4 60 2.1. Background and Architecture...............................4 61 2.2. GETDEVICELIST and GETDEVICEINFO...........................5 62 2.2.1. Volume Identification................................5 63 2.2.2. Volume Topology......................................6 64 2.2.3. GETDEVICELIST and GETDEVICEINFO deviceid4............9 65 2.3. Data Structures: Extents and Extent Lists.................9 66 2.3.1. Layout Requests and Extent Lists....................12 67 2.3.2. Layout Commits......................................13 68 2.3.3. Layout Returns......................................13 69 2.3.4. Client Copy-on-Write Processing.....................14 70 2.3.5. Extents are Permissions.............................15 71 2.3.6. End-of-file Processing..............................17 72 2.3.7. Layout Hints........................................17 73 2.3.8. Client Fencing......................................18 74 2.4. Crash Recovery Issues....................................20 75 2.5. Recalling resources: CB_RECALL_ANY.......................20 76 2.6. Transient and Permanent Errors...........................21 77 3. Security Considerations.......................................21 78 4. Conclusions...................................................23 79 5. IANA Considerations...........................................23 80 6. Acknowledgments...............................................23 81 7. References....................................................23 82 7.1. Normative References.....................................23 83 7.2. Informative References...................................24 84 Author's Addresses...............................................24 85 Intellectual Property Statement..................................25 86 Disclaimer of Validity...........................................25 87 Copyright Statement..............................................25 88 Acknowledgment...................................................25 90 1. Introduction 92 Figure 1 shows the overall architecture of a pNFS system: 94 +-----------+ 95 |+-----------+ +-----------+ 96 ||+-----------+ | | 97 ||| | NFSv4.1 + pNFS | | 98 +|| Clients |<------------------------------>| Server | 99 +| | | | 100 +-----------+ | | 101 ||| +-----------+ 102 ||| | 103 ||| | 104 ||| +-----------+ | 105 ||| |+-----------+ | 106 ||+----------------||+-----------+ | 107 |+-----------------||| | | 108 +------------------+|| Storage |------------+ 109 +| Systems | 110 +-----------+ 112 Figure 1 pNFS Architecture 114 The overall approach is that pNFS-enhanced clients obtain sufficient 115 information from the server to enable them to access the underlying 116 storage (on the Storage Systems) directly. See the pNFS portion of 117 [NFSV4.1] for more details. This draft is concerned with access from 118 pNFS clients to Storage Systems over storage protocols based on 119 blocks and volumes, such as the SCSI protocol family (e.g., parallel 120 SCSI, FCP for Fibre Channel, iSCSI, SAS, and FCoE). This class of 121 storage is referred to as block/volume storage. While the Server to 122 Storage System protocol is not of concern for interoperability here, 123 it will typically also be a block/volume protocol when clients use 124 block/ volume protocols. 126 1.1. General Definitions 128 The following definitions are provided for the purpose of providing 129 an appropriate context for the reader. 131 Byte 133 This document defines a byte as an octet, i.e. a datum exactly 8 134 bits in length. 136 Client 138 The "client" is the entity that accesses the NFS server's 139 resources. The client may be an application which contains the 140 logic to access the NFS server directly. The client may also be 141 the traditional operating system client that provides remote file 142 system services for a set of applications. 144 Server 146 The "Server" is the entity responsible for coordinating client 147 access to a set of file systems and is identified by a Server 148 owner. 150 2. Block Layout Description 152 2.1. Background and Architecture 154 The fundamental storage abstraction supported by block/volume storage 155 is a storage volume consisting of a sequential series of fixed size 156 blocks. This can be thought of as a logical disk; it may be realized 157 by the Storage System as a physical disk, a portion of a physical 158 disk or something more complex (e.g., concatenation, striping, RAID, 159 and combinations thereof) involving multiple physical disks or 160 portions thereof. 162 A pNFS layout for this block/volume class of storage is responsible 163 for mapping from an NFS file (or portion of a file) to the blocks of 164 storage volumes that contain the file. The blocks are expressed as 165 extents with 64 bit offsets and lengths using the existing NFSv4 166 offset4 and length4 types. Clients must be able to perform I/O to 167 the block extents without affecting additional areas of storage 168 (especially important for writes), therefore extents MUST be aligned 169 to 512-byte boundaries, and SHOULD be aligned to the block size used 170 by the NFSv4 server in managing the actual file system (4 kilobytes 171 and 8 kilobytes are common block sizes). This block size is 172 available as the NFSv4.1 layout_blksize attribute. [NFSV4.1] 174 The pNFS operation for requesting a layout (LAYOUTGET) includes the 175 "layoutiomode4 loga_iomode" argument which indicates whether the 176 requested layout is for read-only use or read-write use. A read-only 177 layout may contain holes that are read as zero, whereas a read-write 178 layout will contain allocated, but un-initialized storage in those 179 holes (read as zero, can be written by client). This draft also 180 supports client participation in copy on write (e.g. for file systems 181 with snapshots) by providing both read-only and un-initialized 182 storage for the same range in a layout. Reads are initially 183 performed on the read-only storage, with writes going to the un- 184 initialized storage. After the first write that initializes the un- 185 initialized storage, all reads are performed to that now-initialized 186 writeable storage, and the corresponding read-only storage is no 187 longer used. 189 2.2. GETDEVICELIST and GETDEVICEINFO 191 2.2.1. Volume Identification 193 Storage Systems such as storage arrays can have multiple physical 194 network ports that need not be connected to a common network, 195 resulting in a pNFS client having simultaneous multipath access to 196 the same storage volumes via different ports on different networks. 197 The networks may not even be the same technology - for example, 198 access to the same volume via both iSCSI and Fibre Channel is 199 possible, hence network address are difficult to use for volume 200 identification. For this reason, this pNFS block layout identifies 201 storage volumes by content, for example providing the means to match 202 (unique portions of) labels used by volume managers. Any block pNFS 203 system using this layout MUST support a means of content-based unique 204 volume identification that can be employed via the data structure 205 given here. 207 struct pnfs_block_sig_component4 { /* disk signature component */ 209 int64_t sig_offset; /* byte offset of component on 210 volume*/ 212 opaque contents<>; /* contents of this component of 213 the signature (this is 214 opaque) */ 216 }; 218 Note that the opaque "contents" field in the 219 "pnfs_block_sig_component4" structure MUST NOT be interpreted as a 220 zero-terminated string, as it may contain embedded zero-valued bytes. 221 There are no restrictions on alignment (e.g., neither sig_offset nor 222 the length are required to be multiples of 4). The sig_offset is a 223 signed quantity which when positive represents an byte offset from 224 the start of the volume, and when negative represents an byte offset 225 from the end of the volume. 227 Negative offsets are permitted in order to simplify the client 228 implementation on systems where the device label is found at a fixed 229 offset from the end of the volume. If the server uses negative 230 offsets to describe the signature, then the client and server MUST 231 NOT see different volume sizes. Negative offsets SHOULD NOT be used 232 in systems that dynamically resize volumes unless care is taken to 233 ensure that the device label is always present at the offset from the 234 end of the volume as seen by the clients. 236 A signature is an array up to "PNFS_BLOCK_MAX_SIG_COMP" (defined 237 below) signature components. The client MUST NOT assume that all 238 signature components are colocated within a single sector on a block 239 device. 241 The pNFS client block layout driver uses this volume identification 242 to map pnfs_block_volume_type4 PNFS_BLOCK_VOLUME_SIMPLE deviceid4s to 243 its local view of a LUN. 245 2.2.2. Volume Topology 247 The pNFS block server volume topology is expressed as an arbitrary 248 combination of base volume types enumerated in the following data 249 structures. 251 enum pnfs_block_volume_type4 { 253 PNFS_BLOCK_VOLUME_SIMPLE = 0, /* volume maps to a single LU */ 255 PNFS_BLOCK_VOLUME_SLICE = 1, /* volume is a slice of another 256 volume */ 258 PNFS_BLOCK_VOLUME_CONCAT = 2, /* volume is a concatenation of 259 multiple volumes */ 261 PNFS_BLOCK_VOLUME_STRIPE = 3 /* volume is striped across 262 multiple volumes */ 264 }; 266 const PNFS_BLOCK_MAX_SIG_COMP = 16; /* maximum components per 267 signature */ 269 struct pnfs_block_simple_volume_info4 { 271 deviceid4 vol_id; /* this volume id */ 272 pnfs_block_sig_component4 ds; 273 /* disk signature */ 275 }; 277 struct pnfs_block_slice_volume_info4 { 279 deviceid4 vol_id; /* this volume id */ 281 offset4 start; /* offset of the start of the 282 slice in bytes */ 284 length4 length; /* length of slice in bytes */ 286 deviceid4 volume; /* volume which is sliced */ 288 }; 290 struct pnfs_block_concat_volume_info4 { 292 deviceid4 vol_id; /* this volume id */ 294 deviceid4 volumes<>; /* volumes which are 295 concatenated */ 297 }; 299 struct pnfs_block_stripe_volume_info4 { 301 deviceid4 vol_id; /* this volume id */ 303 length4 stripe_unit; /* size of stripe in octects */ 305 deviceid4 volumes<>; /* volumes which are striped 306 across -- MUST be same size 307 */ 309 }; 310 union pnfs_block_volume4 switch (pnfs_block_volume_type4 type) { 312 case PNFS_BLOCK_VOLUME_SIMPLE: 314 pnfs_block_simple_volume_info4 simple_info; 316 case PNFS_BLOCK_VOLUME_SLICE: 318 pnfs_block_slice_volume_info4 slice_info; 320 case PNFS_BLOCK_VOLUME_CONCAT: 322 pnfs_block_concat_volume_info4 concat_info; 324 case PNFS_BLOCK_VOLUME_STRIPE: 326 pnfs_block_stripe_volume_info4 stripe_info; 328 }; 330 struct pnfs_block_deviceaddr4 { 332 pnfs_block_volume4 volumes<>; /* array of volumes */ 334 }; 336 The "pnfs_block_deviceaddr4" data structure is a structure that 337 allows arbitrarily complex nested volume structures to be encoded. 338 The types of aggregations that are allowed are stripes, 339 concatenations, and slices. Note that the volume topology expressed 340 in the pnfs_block_deviceaddr4 data structure will always resolve to a 341 set of pnfs_block_volume_type4 PNFS_BLOCK_VOLUME_SIMPLE. The array 342 of volumes is ordered such that the root volume is the last element 343 of the array. Concat, slice and stripe volumes MUST refer to volumes 344 defined by lower indexed elements of the array. 346 The "pnfs_block_device_addr4" data structure is returned by the 347 server as the storage-protocol-specific opaque field da_addr_body in 348 the "device_addr4" structure by successful GETDEVICELIST and 349 GETDEVICEINFO operations. [NFSV4.1]. Typically the server in 350 response to a GETDEVICELIST request will return a single 351 "devlist_item4" in the gdlr_devinfo_list array. This is because the 352 "opaque da_addr_body" field inside the "device_addr4" encodes the 353 entire volume hierarchy. In the case of copy-on-write file systems, 354 the "gdlr_devinfo_list" array will contain two devices_item4's, one 355 describing the read-only volume hierarchy, and one describing the 356 writable volume hierarchy. There is no required ordering of the 357 readable and writable volumes in the array as the volumes are 358 uniquely identified by their deviceid4, and are referred to by 359 layouts using the deviceid4. Another example of the server returning 360 multiple device items occurs when the file handle represents the root 361 of a name space spanning multiple physical file systems on the 362 server, each with a different volume hierarchy. 364 As noted above, all device_addr4 structures eventually resolve to a 365 set of volumes of type PNFS_BLOCK_VOLUME_SIMPLE. These volumes are 366 each uniquely identified by a set of signature components. 367 Complicated volume hierarchies may be composed of dozens of volumes 368 each with several signature components, thus the device address may 369 require several kilobytes. The client SHOULD be prepared to allocate 370 a large buffer to contain the result, and in the case of the server 371 returning NFS4ERR_TOOSMALL the client SHOULD be prepared to allocate 372 a large enough buffer to contain the expected result. 374 2.2.3. GETDEVICELIST and GETDEVICEINFO deviceid4 376 The "deviceid4 dli_id" returned in the devlist_item4 of a successful 377 GETDEVICELIST operation is a shorthand id used to reference the whole 378 volume topology. Decoding the "pnfs_block_deviceaddr4" results in a 379 flat ordering of data blocks mapped to PNFS_BLOCK_VOLUME_SIMPLE 380 deviceid4s. Combined with the deviceid4 mapping to a client LUN 381 described in 2.2.1 Volume Identification, a logical volume offset can 382 be mapped to a block on a pNFS client LUN. [NFSV4.1] With the 383 exception of the root volume id, the device ids returned in the 384 volumes array of a pnfs_block_deviceaddr4 data structure should not 385 be passed as arguments in a GETDEVICEINFO request. These non-root 386 volume device ids are never returned by LAYOUTGET in the 387 "pnfs_block_layout4 vol_id" field. If a non-root device id is passed 388 as an argument in a GETDEVICEINFO request, the server SHOULD return 389 NFS4ERR_INVAL. 391 2.3. Data Structures: Extents and Extent Lists 393 A pNFS block layout is a list of extents within a flat array of data 394 blocks in a logical volume. The details of the volume topology can 395 be determined by using the GETDEVICEINFO or GETDEVICELIST operation 396 (see discussion of volume identification, section 2.2 above). The 397 block layout describes the individual block extents on the volume 398 that make up the file. The offsets and length contained in an extent 399 are specified in units of bytes. 401 enum pnfs_block_extent_state4 { 403 PNFS_BLOCK_READWRITE_DATA = 0, /* the data located by this 404 extent is valid 405 for reading and writing. */ 407 PNFS_BLOCK_READ_DATA = 1, /* the data located by this 408 extent is valid for reading 409 only; it may not be written. 410 */ 412 PNFS_BLOCK_INVALID_DATA = 2, /* the location is valid; the 413 data is invalid. It is a 414 newly (pre-) allocated 415 extent. There is physical 416 space on the volume. */ 418 PNFS_BLOCK_NONE_DATA = 3 /* the location is invalid. It 419 is a hole in the file. There 420 is no physical space on the 421 volume. */ 423 }; 425 struct pnfs_block_extent4 { 427 deviceid4 vol_id; /* id of logical volume on which 428 extent of file is stored. */ 430 offset4 file_offset; /* the starting byte offset in 431 the file */ 433 length4 extent_length; /* the size in bytes of the 434 extent */ 436 offset4 storage_offset; /* the starting byte offset in 437 the volume */ 439 pnfs_block_extent_state4 es; /* the state of this extent */ 441 }; 442 struct pnfs_block_layout4 { 444 pnfs_block_extent4 extents<>; /* extents which make up this 445 layout. */ 447 }; 449 The block layout consists of a list of extents which map the logical 450 regions of the file to physical locations on a volume. The "storage 451 offset" field within each extent identifies a location on the logical 452 volume specified by the "vol_id" field in the extent. The vol_id 453 itself is shorthand for the whole topology of the logical volume on 454 which the file is stored. The client is responsible for translating 455 this logical offset into an offset on the appropriate underlying SAN 456 logical unit. In most cases all extents in a layout will reside on 457 the same volume and thus have the same vol_id. In the case of copy 458 on write file systems, the PNFS_BLOCK_READ_DATA extents may have a 459 different vol_id from the writable extents. 461 Each extent maps a logical region of the file onto a portion of the 462 specified logical volume. The file_offset, extent_length, and es 463 fields for an extent returned from the server are always valid. The 464 interpretation of the storage_offset field depends on the value of es 465 as follows (in increasing order): 467 o PNFS_BLOCK_READ_WRITE_DATA means that storage_offset is valid, and 468 points to valid/initialized data that can be read and written. 470 o PNFS_BLOCK_READ_DATA means that storage_offset is valid and points 471 to valid/ initialized data which can only be read. Write 472 operations are prohibited; the client may need to request a read- 473 write layout. 475 o PNFS_BLOCK_INVALID_DATA means that storage_offset is valid, but 476 points to invalid un-initialized data. This data must not be 477 physically read from the disk until it has been initialized. A 478 read request for a PNFS_BLOCK_INVALID_DATA extent must fill the 479 user buffer with zeros. Write requests must write whole server- 480 sized blocks to the disk; bytes not initialized by the user must 481 be set to zero. Any write to storage in a PNFS_BLOCK_INVALID_DATA 482 extent changes the written portion of the extent to 483 PNFS_BLOCK_READ_WRITE_DATA; the pNFS client is responsible for 484 reporting this change via LAYOUTCOMMIT. 486 o PNFS_BLOCK_NONE_DATA means that storage_offset is not valid, and 487 this extent may not be used to satisfy write requests. Read 488 requests may be satisfied by zero-filling as for 489 PNFS_BLOCK_INVALID_DATA. PNFS_BLOCK_NONE_DATA extents may be 490 returned by requests for readable extents; they are never returned 491 if the request was for a writeable extent. 493 An extent list lists all relevant extents in increasing order of the 494 file_offset of each extent; any ties are broken by increasing order 495 of the extent state (es). 497 2.3.1. Layout Requests and Extent Lists 499 Each request for a layout specifies at least three parameters: file 500 offset, desired size, and minimum size. If the status of a request 501 indicates success, the extent list returned must meet the following 502 criteria: 504 o A request for a readable (but not writeable) layout returns only 505 PNFS_BLOCK_READ_DATA or PNFS_BLOCK_NONE_DATA extents (but not 506 PNFS_BLOCK_INVALID_DATA or PNFS_BLOCK_READ_WRITE_DATA extents). 508 o A request for a writeable layout returns 509 PNFS_BLOCK_READ_WRITE_DATA or PNFS_BLOCK_INVALID_DATA extents (but 510 not PNFS_BLOCK_NONE_DATA extents). It may also return 511 PNFS_BLOCK_READ_DATA extents only when the offset ranges in those 512 extents are also covered by PNFS_BLOCK_INVALID_DATA extents to 513 permit writes. 515 o The first extent in the list MUST contain the starting offset. 517 o The total size of extents in the extent list MUST cover at least 518 the minimum size. One exception is allowed: the total size MAY be 519 smaller if only readable extents were requested and EOF is 520 encountered. 522 o Extents in the extent list MUST be logically contiguous for a 523 read-only layout. For a read-write layout, the set of writable 524 extents (i.e., excluding PNFS_BLOCK_READ_DATA extents) MUST be 525 logically contiguous. Every PNFS_BLOCK_READ_DATA extent in a 526 read-write layout MUST be covered by a PNFS_BLOCK_INVALID_DATA 527 extent. This overlap of PNFS_BLOCK_READ_DATA and 528 PNFS_BLOCK_INVALID_DATA extents is the only permitted extent 529 overlap. 531 o Extents MUST be ordered in the list by starting offset, with 532 PNFS_BLOCK_READ_DATA extents preceding PNFS_BLOCK_INVALID_DATA 533 extents in the case of equal file_offsets. 535 2.3.2. Layout Commits 537 struct pnfs_block_layoutupdate4 { 539 pnfs_block_extent4 commit_list<>; /* list of extents which now 540 contain valid data. */ 542 }; 544 The "pnfs_block_layoutupdate4" structure is used by the client as the 545 block-protocol specific argument in a LAYOUTCOMMIT operation. The 546 "commit_list" field is an extent list covering regions of the file 547 layout that were previously in the PNFS_BLOCK_INVALID_DATA state, but 548 have been written by the client and should now be considered in the 549 PNFS_BLOCK_READ_WRITE_DATA state. The es field of each extent in the 550 commit_list MUST be set to PNFS_BLOCK_READ_WRITE_DATA. Implementers 551 should be aware that a server may be unable to commit regions at a 552 granularity smaller than a file-system block (typically 4KB or 8KB). 553 As noted above, the block-size that the server uses is available as 554 an NFSv4 attribute, and any extents included in the "commit_list" 555 MUST be aligned to this granularity and have a size that is a 556 multiple of this granularity. If the client believes that its 557 actions have moved the end-of-file into the middle of a block being 558 committed, the client MUST write zeroes from the end-of-file to the 559 end of that block before committing the block. Failure to do so may 560 result in junk (uninitialized data) appearing in that area if the 561 file is subsequently extended by moving the end-of-file. 563 2.3.3. Layout Returns 565 The LAYOUTRETURN operation is done without any block layout specific 566 data. When the LAYOUTRETURN operation specifies a 567 LAYOUTRETURN4_FILE_return type, then the layoutreturn_file4 data 568 structure specifies the region of the file layout that is no longer 569 needed by the client. The opaque "lrf_body" field of the 570 "layoutreturn_file4" data structure MUST have length zero. A 571 LAYOUTRETURN operation represents an explicit release of resources by 572 the client, usually done for the purpose of avoiding unnecessary 573 CB_LAYOUTRECALL operations in the future. The client may return 574 disjoint regions of the file by using multiple LAYOUTRETURN 575 operations within a single COMPOUND operation. 577 Note that the block/volume layout supports unilateral layout 578 revocation. When a layout is unilaterally revoked by the server, 579 usually due to the client's lease time expiring, or a delegation 580 being recalled, or the client failing to return a layout in a timely 581 manner, it is important for the sake of correctness that any in- 582 flight I/Os that the client issued before the layout was revoked are 583 rejected at the storage. For the block/volume protocol, this is 584 possible by fencing a client with an expired layout timer from the 585 physical storage. Note, however, that the granularity of this 586 operation can only be at the host/logical-unit level. Thus, if one 587 of a client's layouts is unilaterally revoked by the server, it will 588 effectively render useless *all* of the client's layouts for files 589 located on the storage units comprising the logical volume. This may 590 render useless the client's layouts for files in other file systems. 592 2.3.4. Client Copy-on-Write Processing 594 Copy-on-write is a mechanism used to support file and/or file system 595 snapshots. When writing to unaligned regions, or to regions smaller 596 than a file system block, the writer must copy the portions of the 597 original file data to a new location on disk. This behavior can 598 either be implemented on the client or the server. The paragraphs 599 below describe how a pNFS block layout client implements access to a 600 file which requires copy-on-write semantics. 602 Distinguishing the PNFS_BLOCK_READ_WRITE_DATA and 603 PNFS_BLOCK_READ_DATA extent types in combination with the allowed 604 overlap of PNFS_BLOCK_READ_DATA extents with PNFS_BLOCK_INVALID_DATA 605 extents allows copy-on-write processing to be done by pNFS clients. 606 In classic NFS, this operation would be done by the server. Since 607 pNFS enables clients to do direct block access, it is useful for 608 clients to participate in copy-on-write operations. All block/volume 609 pNFS clients MUST support this copy-on-write processing. 611 When a client wishes to write data covered by a PNFS_BLOCK_READ_DATA 612 extent, it MUST have requested a writable layout from the server; 613 that layout will contain PNFS_BLOCK_INVALID_DATA extents to cover all 614 the data ranges of that layout's PNFS_BLOCK_READ_DATA extents. More 615 precisely, for any file_offset range covered by one or more 616 PNFS_BLOCK_READ_DATA extents in a writable layout, the server MUST 617 include one or more PNFS_BLOCK_INVALID_DATA extents in the layout 618 that cover the same file_offset range. When performing a write to 619 such an area of a layout, the client MUST effectively copy the data 620 from the PNFS_BLOCK_READ_DATA extent for any partial blocks of 621 file_offset and range, merge in the changes to be written, and write 622 the result to the PNFS_BLOCK_INVALID_DATA extent for the blocks for 623 that file_offset and range. That is, if entire blocks of data are to 624 be overwritten by an operation, the corresponding 625 PNFS_BLOCK_READ_DATA blocks need not be fetched, but any partial- 626 block writes must be merged with data fetched via 627 PNFS_BLOCK_READ_DATA extents before storing the result via 628 PNFS_BLOCK_INVALID_DATA extents. For the purposes of this 629 discussion, "entire blocks" and "partial blocks" refer to the 630 server's file-system block size. Storing of data in a 631 PNFS_BLOCK_INVALID_DATA extent converts the written portion of the 632 PNFS_BLOCK_INVALID_DATA extent to a PNFS_BLOCK_READ_WRITE_DATA 633 extent; all subsequent reads MUST be performed from this extent; the 634 corresponding portion of the PNFS_BLOCK_READ_DATA extent MUST NOT be 635 used after storing data in a PNFS_BLOCK_INVALID_DATA extent. If a 636 client writes only a portion of an extent, the extent may be split at 637 block aligned boundaries. 639 When a client wishes to write data to a PNFS_BLOCK_INVALID_DATA 640 extent that is not covered by a PNFS_BLOCK_READ_DATA extent, it MUST 641 treat this write identically to a write to a file not involved with 642 copy-on-write semantics. Thus, data must be written in at least 643 block size increments, aligned to multiples of block sized offsets, 644 and unwritten portions of blocks must be zero filled. 646 In the LAYOUTCOMMIT operation that normally sends updated layout 647 information back to the server, for writable data, some 648 PNFS_BLOCK_INVALID_DATA extents may be committed as 649 PNFS_BLOCK_READ_WRITE_DATA extents, signifying that the storage at 650 the corresponding storage_offset values has been stored into and is 651 now to be considered as valid data to be read. PNFS_BLOCK_READ_DATA 652 extents are not committed to the server. For extents that the client 653 receives via LAYOUTGET as PNFS_BLOCK_INVALID_DATA and returns via 654 LAYOUTCOMMIT as PNFS_BLOCK_READ_WRITE_DATA, the server will 655 understand that the PNFS_BLOCK_READ_DATA mapping for that extent is 656 no longer valid or necessary for that file. 658 2.3.5. Extents are Permissions 660 Layout extents returned to pNFS clients grant permission to read or 661 write; PNFS_BLOCK_READ_DATA and PNFS_BLOCK_NONE_DATA are read-only 662 (PNFS_BLOCK_NONE_DATA reads as zeroes), PNFS_BLOCK_READ_WRITE_DATA 663 and PNFS_BLOCK_INVALID_DATA are read/write, (PNFS_BLOCK_INVALID_DATA 664 reads as zeros, any write converts it to PNFS_BLOCK_READ_WRITE_DATA). 665 This is the only client means of obtaining permission to perform 666 direct I/O to storage devices; a pNFS client MUST NOT perform direct 667 I/O operations that are not permitted by an extent held by the 668 client. Client adherence to this rule places the pNFS server in 669 control of potentially conflicting storage device operations, 670 enabling the server to determine what does conflict and how to avoid 671 conflicts by granting and recalling extents to/from clients. 673 Block/volume class storage devices are not required to perform read 674 and write operations atomically. Overlapping concurrent read and 675 write operations to the same data may cause the read to return a 676 mixture of before-write and after-write data. Overlapping write 677 operations can be worse, as the result could be a mixture of data 678 from the two write operations; data corruption can occur if the 679 underlying storage is striped and the operations complete in 680 different orders on different stripes. A pNFS server can avoid these 681 conflicts by implementing a single writer XOR multiple readers 682 concurrency control policy when there are multiple clients who wish 683 to access the same data. This policy SHOULD be implemented when 684 storage devices do not provide atomicity for concurrent read/write 685 and write/write operations to the same data. 687 If a client makes a layout request that conflicts with an existing 688 layout delegation, the request will be rejected with the error 689 NFS4ERR_LAYOUTTRYLATER. This client is then expected to retry the 690 request after a short interval. During this interval the server 691 SHOULD recall the conflicting portion of the layout delegation from 692 the client that currently holds it. This reject-and-retry approach 693 does not prevent client starvation when there is contention for the 694 layout of a particular file. For this reason a pNFS server SHOULD 695 implement a mechanism to prevent starvation. One possibility is that 696 the server can maintain a queue of rejected layout requests. Each 697 new layout request can be checked to see if it conflicts with a 698 previous rejected request, and if so, the newer request can be 699 rejected. Once the original requesting client retries its request, 700 its entry in the rejected request queue can be cleared, or the entry 701 in the rejected request queue can be removed when it reaches a 702 certain age. 704 NFSv4 supports mandatory locks and share reservations. These are 705 mechanisms that clients can use to restrict the set of I/O operations 706 that are permissible to other clients. Since all I/O operations 707 ultimately arrive at the NFSv4 server for processing, the server is 708 in a position to enforce these restrictions. However, with pNFS 709 layouts, I/Os will be issued from the clients that hold the layouts 710 directly to the storage devices that host the data. These devices 711 have no knowledge of files, mandatory locks, or share reservations, 712 and are not in a position to enforce such restrictions. For this 713 reason the NFSv4 server MUST NOT grant layouts that conflict with 714 mandatory locks or share reservations. Further, if a conflicting 715 mandatory lock request or a conflicting open request arrives at the 716 server, the server MUST recall the part of the layout in conflict 717 with the request before granting the request. 719 2.3.6. End-of-file Processing 721 The end-of-file location can be changed in two ways: implicitly as 722 the result of a WRITE or LAYOUTCOMMIT beyond the current end-of-file, 723 or explicitly as the result of a SETATTR request. Typically, when a 724 file is truncated by an NFSv4 client via the SETATTR call, the server 725 frees any disk blocks belonging to the file which are beyond the new 726 end-of-file byte, and MUST write zeros to the portion of the new end- 727 of-file block beyond the new end-of-file byte. These actions render 728 any pNFS layouts which refer to the blocks that are freed or written 729 semantically invalid. Therefore, the server MUST recall from clients 730 the portions of any pNFS layouts which refer to blocks that will be 731 freed or written by the server before processing the truncate 732 request. These recalls may take time to complete; as explained in 733 [NFSv4.1], if the server cannot respond to the client SETATTR request 734 in a reasonable amount of time, it SHOULD reply to the client with 735 the error NFS4ERR_DELAY. 737 Blocks in the PNFS_BLOCK_INVALID_DATA state which lie beyond the new 738 end-of-file block present a special case. The server has reserved 739 these blocks for use by a pNFS client with a writable layout for the 740 file, but the client has yet to commit the blocks, and they are not 741 yet a part of the file mapping on disk. The server MAY free these 742 blocks while processing the SETATTR request. If so, the server MUST 743 recall any layouts from pNFS clients which refer to the blocks before 744 processing the truncate. If the server does not free the 745 PNFS_BLOCK_INVALID_DATA blocks while processing the SETATTR request, 746 it need not recall layouts which refer only to the PNFS_BLOCK_INVALID 747 DATA blocks. 749 When a file is extended implicitly by a WRITE or LAYOUTCOMMIT beyond 750 the current end-of-file, or extended explicitly by a SETATTR request, 751 the server need not recall any portions of any pNFS layouts. 753 2.3.7. Layout Hints 755 The SETATTR operation supports a layout hint attribute [NFSv4.1]. 756 When the client sets a layout hint (data type layouthint4) with a 757 layout type of LAYOUT4_BLOCK_VOLUME (the loh_type field), the 758 loh_body field contains a value of data type pnfs_block_layouthint4. 760 struct pnfs_block_layouthint4 { 761 uint64_t maximum_io_time; /* maximum i/o time in seconds 762 */ 764 }; 766 The block layout client uses the layout hint data structure to 767 communicate to the server the maximum time that it may take an I/O to 768 execute on the client. Clients using block layouts it MUST set the 769 layout hint attribute before using LAYOUTGET operations. 771 2.3.8. Client Fencing 773 The pNFS block protocol must handle situations in which a system 774 failure, typically a network connectivity issue, requires the server 775 to unilaterally revoke extents from one client in order to transfer 776 the extents to another client. The pNFS server implementation MUST 777 ensure that when resources are transferred to another client, they 778 are not used by the client originally owning them, and this must be 779 ensured against any possible combination of partitions and delays 780 among all of the participants to the protocol (server, storage and 781 client). Two approaches to guaranteeing this isolation are possible 782 and are discussed below. 784 One implementation choice for fencing the block client from the block 785 storage is the use of LUN (Logical Unit Number) masking or mapping at 786 the storage systems or storage area network to disable access by the 787 client to be isolated. This requires server access to a management 788 interface for the storage system and authorization to perform LUN 789 masking and management operations. For example, SMI-S [SMIS] 790 provides a means to discover and mask LUNs, including a means of 791 associating clients with the necessary World Wide Names or Initiator 792 names to be masked. 794 In the absence of support for LUN masking, the server has to rely on 795 the clients to implement a timed lease I/O fencing mechanism. 796 Because clients do not know if the server is using LUN masking, in 797 all cases the client MUST implement timed lease fencing. In timed 798 lease fencing we define two time periods, the first, "lease_time" is 799 the length of a lease as defined by the server's lease_time attribute 800 (see [NFSV4.1]), and the second, "maximum_io_time" is the maximum 801 time it can take for a client I/O to the storage system to either 802 complete or fail; this value is often 30 seconds or 60 seconds, but 803 may be longer in some environments. If the maximum client I/O time 804 cannot be bounded, the client MUST use a value of all 1s as the 805 maximum_io_time. 807 The client MUST use SETATTR with a layout hint of type 808 LAYOUT4_BLOCK_VOLUME to inform the server of its maximum_I/O time 809 prior to issuing the first LAYOUTGET operation. The maximum io time 810 hint is a per client attribute, and as such the server SHOULD 811 maintain the value set by each client. A server which implements 812 fencing via LUN masking SHOULD accept any maximum io time value from 813 a client. A server which does not implement fencing may return an 814 error NFS4ERR_INVAL to the SETATTR operation. Such a server SHOULD 815 return NFS4ERR_INVAL when a client sends an unbounded maximum I/O 816 time (all 1s), or when the maximum I/O time is significantly greater 817 than that of other clients using block layouts with pNFS. 819 When a client receives the error NFS4ERR_INVAL in response to the 820 SETATTR operation for a layout hint, the client MUST NOT use the 821 LAYOUTGET operation. After responding with NFS4ERR_INVAL to the 822 SETATTR for layout hint, the server MUST return the error 823 NFS4ERR_LAYOUTUNAVAILABLE to all subsequent LAYOUTGET operations from 824 that client. Thus the server, by returning either NFS4ERR_INVAL or 825 NFS4_OK determines whether or not a client with a large, or an 826 unbounded maximum I/O time may use pNFS. 828 Using the lease time and the maximum i/o time values, we specify the 829 behavior of the client and server as follows. 831 When a client receives layout information via a LAYOUTGET operation, 832 those layouts are valid for at most "lease_time" seconds from when 833 the server granted them. A layout is renewed by any successful 834 SEQUEUNCE operation, or whenever a new stateid is created or updated 835 (see the section "Lease Renewal" of [NFSV4.1]). If the layout lease 836 is not renewed prior to expiration, the client MUST cease to use the 837 layout after "lease_time" seconds from when it either sent the 838 original LAYOUTGET command, or sent the last operation renewing the 839 lease. In other words, the client may not issue any I/O to blocks 840 specified by an expired layout. In the presence of large 841 communication delays between the client and server it is even 842 possible for the lease to expire prior to the server response 843 arriving at the client. In such a situation the client MUST NOT use 844 the expired layouts, and SHOULD revert to using standard NFSv41 READ 845 and WRITE operations. Furthermore, the client must be configured 846 such that I/O operations complete within the "maximum_io_time" even 847 in the presence of multipath drivers that will retry I/Os via 848 multiple paths. 850 As stated in the section "Dealing with Lease Expiration on the 851 Client" of [NFSV4.1], if any SEQUENCE operation is successful, but 852 sr_status_flag has SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, 853 SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, or 854 SEQ4_STATUS_ADMIN_STATE_REVOKED set, the client MUST immediately 855 cease to use all layouts and device id to device address mappings 856 associated with the corresponding server. 858 In the absence of known two way communication between the client and 859 the server on the fore channel, the server must wait for at least the 860 time period "lease_time" plus "maximum_io_time" before transferring 861 layouts from the original client to any other client. The server, 862 like the client, must take a conservative approach, and start the 863 lease expiration timer from the time that it received the operation 864 which last renewed the lease. 866 2.4. Crash Recovery Issues 868 When the server crashes while the client holds a writable layout, and 869 the client has written data to blocks covered by the layout, and the 870 blocks are still in the PNFS_BLOCK_INVALID_DATA state, the client has 871 two options for recovery. If the data that has been written to these 872 blocks is still cached by the client, the client can simply re-write 873 the data via NFSv4, once the server has come back online. However, 874 if the data is no longer in the client's cache, the client MUST NOT 875 attempt to source the data from the data servers. Instead, it should 876 attempt to commit the blocks in question to the server during the 877 server's recovery grace period, by sending a LAYOUTCOMMIT with the 878 "loca_reclaim" flag set to true. This process is described in detail 879 in [NFSv4.1] section 18.42.4. 881 2.5. Recalling resources: CB_RECALL_ANY 883 The server may decide that it cannot hold all of the state for 884 layouts without running out of resources. In such a case, it is free 885 to recall individual layouts using CB_LAYOUTRECALL to reduce the 886 load, or it may choose to request that the client return any layout. 888 For the block layout we define the following bit 890 const RCA4_BLK_LAYOUT_RECALL_ANY_LAYOUTS = 4; 892 When the server sends a CB_RECALL_ANY request to a client specifying 893 the RCA4_BLK_LAYOUT_RECALL_ANY_LAYOUTS bit in craa_type_mask, the 894 client should immediately respond with NFS4_OK, and then 895 asynchronously return complete file layouts until the number of files 896 with layouts cached on the client is less the craa_object_to_keep. 898 The block layout does not currently use bits 5, 6 or 7. If any of 899 these bits are set, the client should return NFS4ERR_INVAL. 901 2.6. Transient and Permanent Errors 903 The server may respond to LAYOUTGET with a variety of error statuses. 904 These errors can convey transient conditions or more permanent 905 conditions that are unlikely to be resolved soon. 907 The transient errors, NFS4ERR_RECALLCONFLICT and NFS4ERR_TRYLATER are 908 used to indicate that the server cannot immediately grant the layout 909 to the client. In the former case this is because the server has 910 recently issued a CB_LAYOUTRECALL to the requesting client, whereas 911 in the case of NFS4ERR_TRYLATER, the server cannot grant the request 912 possibly due to sharing conflicts with other clients. In either 913 case, a reasonable approach for the client is to wait several 914 milliseconds and retry the request. The client SHOULD track the 915 number of retries, and if forward progress is not made, the client 916 SHOULD send the READ or WRITE operation directly to the server. 918 The error NFS4ERR_LAYOUTUNAVAILABLE may be returned by the server if 919 layouts are not supported for the requested file or its containing 920 file system. The server may also return this error code if the 921 server is the progress of migrating the file from secondary storage, 922 or for any other reason which causes the server to be unable to 923 supply the layout. As a result of receiving 924 NFS4ERR_LAYOUTUNAVAILABLE, the client SHOULD send future READ and 925 WRITE requests directly to the server. It is expected that a client 926 will not cache the file's layoutunavailable state forever, particular 927 if the file is closed, and thus eventually, the client MAY reissue a 928 LAYOUTGET operation. 930 3. Security Considerations 932 Typically, SAN disk arrays and SAN protocols provide access control 933 mechanisms (access-logics, lun masking, etc.) which operate at the 934 granularity of individual hosts. The functionality provided by such 935 mechanisms makes it possible for the server to "fence" individual 936 client machines from certain physical disks---that is to say, to 937 prevent individual client machines from reading or writing to certain 938 physical disks. Finer-grained access control methods are not 939 generally available. For this reason, certain security 940 responsibilities are delegated to pNFS clients for block/volume 941 layouts. Block/volume storage systems generally control access at a 942 volume granularity, and hence pNFS clients have to be trusted to only 943 perform accesses allowed by the layout extents they currently hold 944 (e.g., and not access storage for files on which a layout extent is 945 not held). In general, the server will not be able to prevent a 946 client which holds a layout for a file from accessing parts of the 947 physical disk not covered by the layout. Similarly, the server will 948 not be able to prevent a client from accessing blocks covered by a 949 layout that it has already returned. This block-based level of 950 protection must be provided by the client software. 952 An alternative method of block/volume protocol use is for the storage 953 devices to export virtualized block addresses, which do reflect the 954 files to which blocks belong. These virtual block addresses are 955 exported to pNFS clients via layouts. This allows the storage device 956 to make appropriate access checks, while mapping virtual block 957 addresses to physical block addresses. In environments where the 958 security requirements are such that client-side protection from 959 access to storage outside of the layout is not sufficient pNFS 960 block/volume storage layouts for pNFS SHOULD NOT be used, unless the 961 storage device is able to implement the appropriate access checks, 962 via use of virtualized block addresses, or other means. 964 This also has implications for some NFSv4 functionality outside pNFS. 965 For instance, if a file is covered by a mandatory read-only lock, the 966 server can ensure that only readable layouts for the file are granted 967 to pNFS clients. However, it is up to each pNFS client to ensure 968 that the readable layout is used only to service read requests, and 969 not to allow writes to the existing parts of the file. Since 970 block/volume storage systems are generally not capable of enforcing 971 such file-based security, in environments where pNFS clients cannot 972 be trusted to enforce such policies, pNFS block/volume storage 973 layouts SHOULD NOT be used. 975 Access to block/volume storage is logically at a lower layer of the 976 I/O stack than NFSv4, and hence NFSv4 security is not directly 977 applicable to protocols that access such storage directly. Depending 978 on the protocol, some of the security mechanisms provided by NFSv4 979 (e.g., encryption, cryptographic integrity) may not be available, or 980 may be provided via different means. At one extreme, pNFS with 981 block/volume storage can be used with storage access protocols (e.g., 982 parallel SCSI) that provide essentially no security functionality. 983 At the other extreme, pNFS may be used with storage protocols such as 984 iSCSI that provide significant functionality. It is the 985 responsibility of those administering and deploying pNFS with a 986 block/volume storage access protocol to ensure that appropriate 987 protection is provided to that protocol (physical security is a 988 common means for protocols not based on IP). In environments where 989 the security requirements for the storage protocol cannot be met, 990 pNFS block/volume storage layouts SHOULD NOT be used. 992 When security is available for a storage protocol, it is generally at 993 a different granularity and with a different notion of identity than 994 NFSv4 (e.g., NFSv4 controls user access to files, iSCSI controls 995 initiator access to volumes). The responsibility for enforcing 996 appropriate correspondences between these security layers is placed 997 upon the pNFS client. As with the issues in the first paragraph of 998 this section, in environments where the security requirements are 999 such that client-side protection from access to storage outside of 1000 the layout is not sufficient, pNFS block/volume storage layouts 1001 SHOULD NOT be used. 1003 4. Conclusions 1005 This draft specifies the block/volume layout type for pNFS and 1006 associated functionality. 1008 5. IANA Considerations 1010 There are no IANA considerations in this document. All pNFS IANA 1011 Considerations are covered in [NFSV4.1]. 1013 6. Acknowledgments 1015 This draft draws extensively on the authors' familiarity with the 1016 mapping functionality and protocol in EMC's HighRoad system 1017 [HighRoad]. The protocol used by HighRoad is called FMP (File 1018 Mapping Protocol); it is an add-on protocol that runs in parallel 1019 with file system protocols such as NFSv3 to provide pNFS-like 1020 functionality for block/volume storage. While drawing on HighRoad 1021 FMP, the data structures and functional considerations in this draft 1022 differ in significant ways, based on lessons learned and the 1023 opportunity to take advantage of NFSv4 features such as COMPOUND 1024 operations. The design to support pNFS client participation in copy- 1025 on-write is based on text and ideas contributed by Craig Everhart 1026 (formerly with IBM). 1028 7. References 1030 7.1. Normative References 1032 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1033 Requirement Levels", BCP 14, RFC 2119, March 1997. 1035 [NFSV4.1] Shepler, S., Eisler, M., and Noveck, D. ed., "NFSv4 Minor 1036 Version 1", draft-ietf-nfsv4-minorversion1-14.txt, Internet 1037 Draft, July 2007. 1039 7.2. Informative References 1041 [HighRoad] EMC Corporation, "EMC Celerra HighRoad", EMC C819.1 white 1042 paper, available at: 1043 http://www.emc.com/pdf/products/celerra_file_server/HighRoad_wp.pdf 1044 link checked 29 August 2006. 1046 [SMIS] SNIA, "SNIA Storage Management Initiative Specification", 1047 version 1.0.2, available at: 1048 http://www.snia.org/smi/tech_activities/smi_spec_pr/spec/SMIS_1_0_2_f 1049 inal.pdf 1051 Author's Addresses 1053 David L. Black 1054 EMC Corporation 1055 176 South Street 1056 Hopkinton, MA 01748 1058 Phone: +1 (508) 293-7953 1059 Email: black_david@emc.com 1061 Stephen Fridella 1062 EMC Corporation 1063 228 South Street 1064 Hopkinton, MA 01748 1066 Phone: +1 (508) 249-3528 1067 Email: fridella_stephen@emc.com 1069 Jason Glasgow 1070 EMC Corporation 1071 32 Coslin Drive 1072 Southboro, MA 01772 1074 Phone: +1 (508) 305 8831 1075 Email: glasgow_jason@emc.com 1077 Intellectual Property Statement 1079 The IETF takes no position regarding the validity or scope of any 1080 Intellectual Property Rights or other rights that might be claimed to 1081 pertain to the implementation or use of the technology described in 1082 this document or the extent to which any license under such rights 1083 might or might not be available; nor does it represent that it has 1084 made any independent effort to identify any such rights. Information 1085 on the procedures with respect to rights in RFC documents can be 1086 found in BCP 78 and BCP 79. 1088 Copies of IPR disclosures made to the IETF Secretariat and any 1089 assurances of licenses to be made available, or the result of an 1090 attempt made to obtain a general license or permission for the use of 1091 such proprietary rights by implementers or users of this 1092 specification can be obtained from the IETF on-line IPR repository at 1093 http://www.ietf.org/ipr. 1095 The IETF invites any interested party to bring to its attention any 1096 copyrights, patents or patent applications, or other proprietary 1097 rights that may cover technology that may be required to implement 1098 this standard. Please address the information to the IETF at ietf- 1099 ipr@ietf.org. 1101 Disclaimer of Validity 1103 This document and the information contained herein are provided on an 1104 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1105 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND 1106 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS 1107 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 1108 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1109 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1111 Copyright Statement 1113 Copyright (C) The IETF Trust (2007). 1115 This document is subject to the rights, licenses and restrictions 1116 contained in BCP 78, and except as set forth therein, the authors 1117 retain all their rights. 1119 Acknowledgment 1121 Funding for the RFC Editor function is currently provided by the 1122 Internet Society.