idnits 2.17.1 draft-ietf-nfsv4-flex-files-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 773 has weird spacing: '...stateid lor...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: However, what SHOULD not be involved in that calculation is a perceived network distance between the client and the storage device. The client is better situated for making that determination based on past interaction with the storage device over the different available network intefaces bewteen the two. I.e., the metadata server might not know about a transient outage between the client and storage device because it has no presence on the given subnet. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: The metadata server is then responsible for determining if it wants to remove the errant mirror from the layout, if the mirror has recovered from some transient error, etc. When the client tries to get a new layout, the metadata server informs it of the decision by the contents of the layout. The client MUST not make any assumptions that the contents of the previous layout will match those of the new one. If it has updates that were not committed, it MUST resend those updates to all mirrors. -- The document date (October 07, 2014) is 3488 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-41) exists of draft-ietf-nfsv4-minorversion2-22 ** Obsolete normative reference: RFC 3530 (Obsoleted by RFC 7530) ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 2 errors (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 B. Halevy 3 Internet-Draft T. Haynes 4 Intended status: Informational Primary Data 5 Expires: April 10, 2015 October 07, 2014 7 Parallel NFS (pNFS) Flexible File Layout 8 draft-ietf-nfsv4-flex-files-02.txt 10 Abstract 12 The Parallel Network File System (pNFS) allows a separation between 13 the metadata and data for a file. The metadata file access is 14 handled via Network File System version 4 (NFSv4) minor version 1 15 (NFSv4.1) and the data file access is specific to the protocol being 16 used between the client and storage device. The client is informed 17 by the metadata server as to which protocol to use via a Layout Type. 18 The Flexible File Layout Type is defined in this document as an 19 extension to NFSv4.1 to allow the use of storage devices which need 20 not be tightly coupled to the metadata server. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at http://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on April 10, 2015. 39 Copyright Notice 41 Copyright (c) 2014 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 57 1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3 58 1.2. Difference Between a Data Server and a Storage Device . . 5 59 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 5 60 2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 5 61 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6 62 2.2. Security Models . . . . . . . . . . . . . . . . . . . . . 6 63 2.3. State and Locking Models . . . . . . . . . . . . . . . . 6 64 3. XDR Description of the Flexible File Layout Type . . . . . . 7 65 3.1. Code Components Licensing Notice . . . . . . . . . . . . 8 66 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 9 67 4.1. ff_device_addr . . . . . . . . . . . . . . . . . . . . . 9 68 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 10 69 5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 11 70 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 12 71 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 14 72 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 14 73 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 15 74 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 15 75 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 16 76 8.3. Metadata Server Resilvering of the File . . . . . . . . . 16 77 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 16 78 9.1. ff_ioerr . . . . . . . . . . . . . . . . . . . . . . . . 17 79 9.2. ff_iostats . . . . . . . . . . . . . . . . . . . . . . . 18 80 9.3. ff_layoutreturn . . . . . . . . . . . . . . . . . . . . . 19 81 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 19 82 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 19 83 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 20 84 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 20 85 13. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 20 86 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 21 87 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 21 88 15. Security Considerations . . . . . . . . . . . . . . . . . . . 22 89 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 90 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 91 17.1. Normative References . . . . . . . . . . . . . . . . . . 23 92 17.2. Informative References . . . . . . . . . . . . . . . . . 23 93 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 24 94 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 24 95 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 97 1. Introduction 99 In the parallel Network File System (pNFS), the metadata server 100 returns Layout Type structures that describe where file data is 101 located. There are different Layout Types for different storage 102 systems and methods of arranging data on storage devices. This 103 document defines the Flexible File Layout Type used with file-based 104 data servers that are accessed using the Network File System (NFS) 105 protocols: NFSv3 [RFC1813], NFSv4 [RFC3530], NFSv4.1 [RFC5661], and 106 NFSv4.2 [NFSv42]. 108 To provide a global state model equivalent to that of the Files 109 Layout Type, a back-end control protocol MAY be implemented between 110 the metadata server and NFSv4.1 storage devices. It is out of scope 111 for this document to specify the wire protocol of such a protocol, 112 yet the requirements for the protocol are specified in [RFC5661] and 113 clarified in [pNFSLayouts]. 115 1.1. Definitions 117 control protocol: is a set of requirements for the communication of 118 information on layouts, stateids, file metadata, and file data 119 between the metadata server and the storage devices (see 120 [pNFSLayouts]). 122 data file: is that part of the file system object which describes 123 the payload and not the object. E.g., it is the file contents. 125 Data Server (DS): is one of the pNFS servers which provide the 126 contents of a file system object which is a regular file. 127 Depending on the layout, there might be one or more data servers 128 over which the data is striped. Note that while the metadata 129 server is strictly accessed over the NFSv4.1 protocol, depending 130 on the Layout Type, the data server could be accessed via any 131 protocol that meets the pNFS requirements. 133 fencing: is when the metadata server prevents the storage devices 134 from processing I/O from a specific client to a specific file. 136 File Layout Type: is a Layout Type in which the storage devices are 137 accessed via the NFSv4.1 protocol. It is defined in Section 13 of 138 [RFC5661]. 140 layout: informs a client of which storage devices it needs to 141 communicate with (and over which protocol) to perform I/O on a 142 file. The layout might also provide some hints about how the 143 storage is physically organized. 145 layout iomode: describes whether the layout granted to the client is 146 for read or read/write I/O. 148 layout stateid: is a 128-bit quantity returned by a server that 149 uniquely defines the layout state provided by the server for a 150 specific layout that describes a Layout Type and file (see 151 Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 describes 152 the difference between a layout stateid and a normal stateid. 154 Layout Type: describes both the storage protocol used to access the 155 data and the aggregation scheme used to lays out the file data on 156 the underlying storage devices. 158 loose coupling: is when the metadata server and the storage devices 159 do not have a control protocol present. 161 metadata file: is that part of the file system object which 162 describes the object and not the payload. E.g., it could be the 163 time since last modification, access, etc. 165 Metadata Server (MDS): is the pNFS server which provides metadata 166 information for a file system object. It also is responsible for 167 generating layouts for file system objects. Note that the MDS is 168 responsible for directory-based operations. 170 Mirror: is a copy of a file. While mirroring can be used for 171 backing up a file, the copies can be distrbuted such that each 172 remote site has a locally cached copy. Note that if one copy of 173 the mirror is updated, then all copies must be updated. 175 Object Layout Type: is a Layout Type in which the storage devices 176 are accessed via the OSD protocol [ANSI400-2004]. It is defined 177 in [RFC5664]. 179 recalling a layout: is when the metadata server uses a back channel 180 to inform the client that the layout is to be returned in a 181 graceful manner. Note that the client could be able to flush any 182 writes, etc., before replying to the metadata server. 184 revoking a layout: is when the metadata server invalidates the 185 layout such that neither the metadata server nor any storage 186 device will accept any access from the client with that layout. 188 resilvering: is the act of rebuilding a mirrored copy of a file from 189 a known good copy of the file. Note that this can also be done to 190 create a new mirrored copy of the file. 192 rsize: is the data transfer buffer size used for reads. 194 stateid: is a 128-bit quantity returned by a server that uniquely 195 defines the open and locking states provided by the server for a 196 specific open-owner or lock-owner/open-owner pair for a specific 197 file and type of lock. 199 storage device: is another term used almost interchangeably with 200 data server. See Section 1.2 for the nuances between the two. 202 tight coupling: is when the metadata server and the storage devices 203 do have a control protocol present. 205 wsize: is the data transfer buffer size used for writes. 207 1.2. Difference Between a Data Server and a Storage Device 209 We defined a data server as a pNFS server, which implies that it can 210 utilize the NFSv4.1 protocol to communicate with the client. As 211 such, only the File Layout Type would currently meet this 212 requirement. The more generic concept is a storage device, which can 213 use any protocol to communicate with the client. The requirements 214 for a storage device to act together with the metadata server to 215 provide data to a client are that there is a Layout Type 216 specification for the given protocol and that the metadata server has 217 granted a layout to the client. Note that nothing precludes there 218 being multiple supported Layout Types (i.e., protocols) between a 219 metadata server, storage devices, and client. 221 As storage device is the more encompassing terminology, this document 222 utilizes it over data server. 224 1.3. Requirements Language 226 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 227 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 228 document are to be interpreted as described in [RFC2119]. 230 2. Coupling of Storage Devices 232 The coupling of the metadata server with the storage devices can be 233 either tight or loose. In a tight coupling, there is a control 234 protocol present to manage security, LAYOUTCOMMITs, etc. With a 235 loose coupling, the only control protocol might be a version of NFS. 236 As such, semantics for managing security, state, and locking models 237 MUST be defined. 239 A file is split into metadata and data. The "metadata file" is that 240 part of the file stored on the metadata server. The "data file" is 241 that part of the file stored on the storage device. And the "file" 242 is the combination of the two. 244 2.1. LAYOUTCOMMIT 246 With a tightly coupled system, when the metadata server receives a 247 LAYOUTCOMMIT (see Section 18.42 of [RFC5661]), the semantics of the 248 File Layout Type MUST be met (see Section 12.5.4 of [RFC5661]). With 249 a loosely coupled system, a LAYOUTCOMMIT to the metadata server MUST 250 be proceeded with a COMMIT to the storage device. I.e., it is the 251 responsibility of the client to make sure the data file is stable 252 before the metadata server begins to query the storage devices about 253 the changes to the file. Note that if the client has not done a 254 COMMIT to the storage device, then the LAYOUTCOMMIT might not be 255 synchronized to the last WRITE operation to the storage device. 257 2.2. Security Models 259 With loosely coupled storage devices, the metadata server uses 260 synthetic uids and gids for the data file, where the uid owner of the 261 data file is allowed read/write access and the gid owner is allowed 262 read only access. As part of the layout, the client is provided with 263 the rpc credentials to be used (see ffm_auth in Section 5.1) to 264 access the data file. Fencing off clients is achieved by using 265 SETATTR by the server to change the uid and/or gid owners of the data 266 file to implicitly revoke the outstanding rpc credentials. Note: it 267 is recommended to implement common access control methods at the 268 storage device filesystem exports level to allow only the metadata 269 server root (super user) access to the storage device, and to set the 270 owner of all directories holding data files to the root user. This 271 security method, when using weak auth flavors such as AUTH_SYS, 272 provides a practical model to enforce access control and fence off 273 cooperative clients, but it can not protect against malicious 274 clients; hence it provides a level of security equivalent to NFSv3. 276 With tightly coupled storage devices, the metadata server sets the 277 user and group owners, mode bits, and ACL of the data file to be the 278 same as the metadata file. And the client must authenticate with the 279 storage device and go through the same authorization process it would 280 go through via the metadata server. 282 2.3. State and Locking Models 284 Metadata file OPEN, LOCK, and DELEGATION operations are always 285 executed only against the metadata server. 287 With NFSv4 storage devices, the metadata server, in response to the 288 state changing operation, executes them against the respective data 289 files on the storage devices. It then sends the storage device open 290 stateid as part of the layout (see the ffm_stateid in Section 5.1) 291 and it is then used by the client for executing READ/WRITE operations 292 against the storage device. 294 Standalone NFSv4.1 storage devices that do not return the 295 EXCHGID4_FLAG_USE_PNFS_DS flag to EXCHANGE_ID are used the same way 296 as NFSv4 storage devices. 298 NFSv4.1 clustered storage devices that do identify themselves with 299 the EXCHGID4_FLAG_USE_PNFS_DS flag to EXCHANGE_ID use a back-end 300 control protocol as described in [RFC5661] to implement a global 301 stateid model as defined there. 303 3. XDR Description of the Flexible File Layout Type 305 This document contains the external data representation (XDR) 306 [RFC4506] description of the Flexible File Layout Type. The XDR 307 description is embedded in this document in a way that makes it 308 simple for the reader to extract into a ready-to-compile form. The 309 reader can feed this document into the following shell script to 310 produce the machine readable XDR description of the Flexible File 311 Layout Type: 313 #!/bin/sh 314 grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' 316 That is, if the above script is stored in a file called "extract.sh", 317 and this document is in a file called "spec.txt", then the reader can 318 do: 320 sh extract.sh < spec.txt > flex_files_prot.x 322 The effect of the script is to remove leading white space from each 323 line, plus a sentinel sequence of "///". 325 The embedded XDR file header follows. Subsequent XDR descriptions, 326 with the sentinel sequence are embedded throughout the document. 328 Note that the XDR code contained in this document depends on types 329 from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs 330 types that end with a 4, such as offset4, length4, etc., as well as 331 more generic types such as uint32_t and uint64_t. 333 3.1. Code Components Licensing Notice 335 Both the XDR description and the scripts used for extracting the XDR 336 description are Code Components as described in Section 4 of "Legal 337 Provisions Relating to IETF Documents" [LEGAL]. These Code 338 Components are licensed according to the terms of that document. 340 /// /* 341 /// * Copyright (c) 2012 IETF Trust and the persons identified 342 /// * as authors of the code. All rights reserved. 343 /// * 344 /// * Redistribution and use in source and binary forms, with 345 /// * or without modification, are permitted provided that the 346 /// * following conditions are met: 347 /// * 348 /// * o Redistributions of source code must retain the above 349 /// * copyright notice, this list of conditions and the 350 /// * following disclaimer. 351 /// * 352 /// * o Redistributions in binary form must reproduce the above 353 /// * copyright notice, this list of conditions and the 354 /// * following disclaimer in the documentation and/or other 355 /// * materials provided with the distribution. 356 /// * 357 /// * o Neither the name of Internet Society, IETF or IETF 358 /// * Trust, nor the names of specific contributors, may be 359 /// * used to endorse or promote products derived from this 360 /// * software without specific prior written permission. 361 /// * 362 /// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS 363 /// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED 364 /// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 365 /// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 366 /// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 367 /// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 368 /// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 369 /// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 370 /// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 371 /// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 372 /// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 373 /// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 374 /// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 375 /// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 376 /// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 377 /// * 378 /// * This code was derived from RFCTBD10. 379 /// * Please reproduce this note if possible. 380 /// */ 381 /// 382 /// /* 383 /// * flex_files_prot.x 384 /// */ 385 /// 386 /// /* 387 /// * The following include statements are for example only. 388 /// * The actual XDR definition files are generated separately 389 /// * and independently and are likely to have a different name. 390 /// * %#include 391 /// * %#include 392 /// */ 393 /// 395 4. Device Addressing and Discovery 397 Data operations to a storage device require the client to know the 398 network address of the storage device. The NFSv4.1 GETDEVICEINFO 399 operation (Section 18.40 of [RFC5661]) is used by the client to 400 retrieve that information. 402 4.1. ff_device_addr 404 The ff_device_addr data structure is returned by the server as the 405 storage protocol specific opaque field da_addr_body in the 406 device_addr4 structure by a successful GETDEVICEINFO operation. 408 /// struct ff_device_addr { 409 /// multipath_list4 ffda_netaddrs; 410 /// uint32_t ffda_version; 411 /// uint32_t ffda_minorversion; 412 /// uint32_t ffda_rsize; 413 /// uint32_t ffda_wsize; 414 /// bool ffda_tightly_coupled; 415 /// }; 416 /// 418 The ffda_netaddrs field is used to locate the storage device. It 419 MUST be set by the server to a list holding one or more of the device 420 network addresses. 422 The ffda_version and ffda_minorversion represent the NFS protocol to 423 be used to access the storage device. This layout specification 424 defines the semantics for ffda_versions 3 and 4. If ffda_version 425 equals 3 then server MUST set ffda_minorversion to 0 and the client 426 MUST access the storage device using the NFSv3 protocol [RFC1813]. 427 If ffda_version equals 4 then the server MUST set ffda_minorversion 428 to one of the NFSv4 minor version numbers and the client MUST access 429 the storage device using NFSv4. 431 The ffda_rsize and ffda_wsize are used to communicate the maximum 432 rsize and wsize supported by the storage device. As the storage 433 device can have a different rsize or wsize than the metadata server, 434 the ffda_rsize and ffda_wsize allow the metadata server to 435 communicate that information on behalf of the storage device. 437 ffda_tightly_coupled informs the client as to whether the metadata 438 server is tightly coupled with the storage devices or not. Note that 439 even if the data protocol is at least NFSv4.1, it may still be the 440 case that there is no control protocol present. If 441 ffda_tightly_coupled is not set, then the client MUST commit writes 442 to the storage devices for the file before sending a LAYOUTCOMMIT to 443 the metadata server. I.e., the writes MUST be committed by the 444 client to stable storage via issuing WRITEs with stable_how == 445 FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how != 446 FILE_SYNC (see Section 3.3.7 of [RFC1813]). 448 4.2. Storage Device Multipathing 450 The Flexible File Layout Type supports multipathing to multiple 451 storage device addresses. Storage device level multipathing is used 452 for bandwidth scaling via trunking and for higher availability of use 453 in the case of a storage device failure. Multipathing allows the 454 client to switch to another storage device address which may be that 455 of another storage device that is exporting the same data stripe 456 unit, without having to contact the metadata server for a new layout. 458 To support storage device multipathing, ffda_netaddrs contains an 459 array of one or more storage device network addresses. This array 460 (data type multipath_list4) represents a list of storage device (each 461 identified by a network address), with the possibility that some 462 storage device will appear in the list multiple times. 464 The client is free to use any of the network addresses as a 465 destination to send storage device requests. If some network 466 addresses are less optimal paths to the data than others, then the 467 MDS SHOULD NOT include those network addresses in ffda_netaddrs. If 468 less optimal network addresses exist to provide failover, the 469 RECOMMENDED method to offer the addresses is to provide them in a 470 replacement device-ID-to-device-address mapping, or a replacement 471 device ID. When a client finds no response from the storage device 472 using all addresses available in ffda_netaddrs, it SHOULD send a 473 GETDEVICEINFO to attempt to replace the existing device-ID-to-device- 474 address mappings. If the MDS detects that all network paths 475 represented by ffda_netaddrs are unavailable, the MDS SHOULD send a 476 CB_NOTIFY_DEVICEID (if the client has indicated it wants device ID 477 notifications for changed device IDs) to change the device-ID-to- 478 device-address mappings to the available addresses. If the device ID 479 itself will be replaced, the MDS SHOULD recall all layouts with the 480 device ID, and thus force the client to get new layouts and device ID 481 mappings via LAYOUTGET and GETDEVICEINFO. 483 Generally, if two network addresses appear in ffda_netaddrs, they 484 will designate the same storage device. When the storage device is 485 accessed over NFSv4.1 or higher minor version the two storage device 486 addresses will support the implementation of client ID or session 487 trunking (the latter is RECOMMENDED) as defined in [RFC5661]. The 488 two storage device addresses will share the same server owner or 489 major ID of the server owner. It is not always necessary for the two 490 storage device addresses to designate the same storage device with 491 trunking being used. For example, the data could be read-only, and 492 the data consist of exact replicas. 494 5. Flexible File Layout Type 496 The layout4 type is defined in [RFC5662] as follows: 498 enum layouttype4 { 499 LAYOUT4_NFSV4_1_FILES = 1, 500 LAYOUT4_OSD2_OBJECTS = 2, 501 LAYOUT4_BLOCK_VOLUME = 3, 502 LAYOUT4_FLEX_FILES = 0x80000004 503 [[RFC Editor: please modify the LAYOUT4_FLEX_FILES 504 to be the layouttype assigned by IANA]] 505 }; 507 struct layout_content4 { 508 layouttype4 loc_type; 509 opaque loc_body<>; 510 }; 512 struct layout4 { 513 offset4 lo_offset; 514 length4 lo_length; 515 layoutiomode4 lo_iomode; 516 layout_content4 lo_content; 517 }; 519 [[AI10: Remember, using experimental version number to track changes 520 to the XDR via LAYOUT4_FLEX_FILES! --TH]] 522 This document defines structure associated with the layouttype4 value 523 LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an 524 XDR type "opaque". The opaque layout is uninterpreted by the generic 525 pNFS client layers, but obviously must be interpreted by the Flexible 526 File Layout Type implementation. This section defines the structure 527 of this opaque value, ff_layout4. 529 5.1. ff_layout4 531 /// struct ff_data_server4 { 532 /// deviceid4 ffds_deviceid; 533 /// uint32_t ffds_efficiency; 534 /// stateid4 ffds_stateid; 535 /// nfs_fh4 ffds_fhandle; 536 /// opaque_auth ffds_auth; 537 /// }; 538 /// 540 /// struct ff_mirror4 { 541 /// ff_data_server4 ffm_data_servers<>; 542 /// }; 543 /// 545 /// struct ff_layout4 { 546 /// length4 ffl_stripe_unit; 547 /// ff_mirror4 ffl_mirrors<>; 548 /// }; 549 /// 551 The ff_layout4 structure specifies a layout over a set of mirrored 552 copies of the data file. This mirroring protects against loss of 553 data files. 555 It is possible that the file is concatenated from more than one 556 layout segment. Each layout segment MAY represent different striping 557 parameters, applying respectively only to the layout segment byte 558 range. 560 The ffl_stripe_unit field is the stripe unit size in use for the 561 current layout segment. The number of stripes is given inside each 562 mirror by the number of elements in ffm_data_servers. If the number 563 of stripes is one, then the value for ffl_stripe_unit MUST default to 564 zero. The only supported mapping scheme is sparse and is detailed in 565 Section 6. Note that there is an assumption here that both the 566 stripe unit size and the number of of stripes is the same across all 567 mirrors. 569 The ffl_mirrors field is the array of mirrored storage devices which 570 provide the storage for the current stripe, see Figure 1. 572 +-----------+ 573 | | 574 | | 575 | File | 576 | | 577 | | 578 +-----+-----+ 579 | 580 +------------+------------+ 581 | | 582 +----+-----+ +-----+----+ 583 | Mirror 1 | | Mirror 2 | 584 +----+-----+ +-----+----+ 585 | | 586 +-----------+ +-----------+ 587 |+-----------+ |+-----------+ 588 ||+-----------+ ||+-----------+ 589 +|| Storage | +|| Storage | 590 +| Devices | +| Devices | 591 +-----------+ +-----------+ 593 Figure 1 595 The ffs_mirrors field represents an array of state information for 596 each mirrored copy of the file. Each element is described by a 597 ff_mirror4 type. 599 ffds_deviceid provides the deviceid of the storage device holding the 600 data file. 602 ffds_fhandle provides the filehandle of the data file on the given 603 storage device. For tight coupling, ffds_stateid provides the 604 stateid to be used by the client to access the file. For loose 605 coupling and a NFSv4 storage device, the client may use an anonymous 606 stateid to perform I/O on the storage device as there is no use for 607 the metadata server stateid (no control protocol). In such a 608 scenario, the server MUST set the ffds_stateid to be zero. 610 For NFSv3 storage devices, ffds_auth provides the RPC credentials to 611 be used by the client to access the data files. For NFSv4.x storage 612 devices, the server SHOULD use the AUTH_NONE flavor and a zero length 613 opaque body to minimize the returned structure length. The client 614 MUST ignore ffds_auth in this case. [[AI6: Even for tightly coupled 615 systems, that cannot be correct! --TH]] 617 ffds_efficiency describes the metadata server's evaluation as to the 618 effectiveness of each mirror. Note that this is per layout and not 619 per device as the metric may change due to perceived load, 620 availability to the metadata server, etc. Higher values denote 621 higher perceived utility. The way the client can select the best 622 mirror to access is discussed in Section 8.1. 624 6. Striping via Sparse Mapping 626 While other Layout Types support both dense and sparse mapping of 627 logical offsets to phyisical offsets within a file (see for example 628 Section 13.4 of [RFC5661]), the Flexible File Layout Type only 629 supports a sparse mapping. 631 With sparse mappings, the logical offset within a file (L) is also 632 the physical offset on the storage device. As detailed in 633 Section 13.4.4 of [RFC5661], this results in holes across each 634 storage device which does not contain the current stripe index. 636 L: logical offset into the file 638 W: stripe width 639 W = number of elements in ffm_data_servers 641 S: number of bytes in a stripe 642 S = W * ffl_stripe_unit 644 N: stripe number 645 N = L / S 647 7. Recovering from Client I/O Errors 649 The pNFS client may encounter errors when directly accessing the 650 storage devices. However, it is the responsibility of the metadata 651 server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES 652 layout type is used, the client MUST report the I/O errors to the 653 server at LAYOUTRETURN time using the ff_ioerr structure (see 654 Section 9.1). 656 The metadata server analyzes the error and determines the required 657 recovery operations such as recovering media failures or 658 reconstructing missing data files. 660 The metadata server SHOULD recall any outstanding layouts to allow it 661 exclusive write access to the stripes being recovered and to prevent 662 other clients from hitting the same error condition. In these cases, 663 the server MUST complete recovery before handing out any new layouts 664 to the affected byte ranges. 666 Although it MAY be acceptable for the client to propagate a 667 corresponding error to the application that initiated the I/O 668 operation and drop any unwritten data, the client SHOULD attempt to 669 retry the original I/O operation by requesting a new layout using 670 LAYOUTGET and retry the I/O operation(s) using the new layout, or the 671 client MAY just retry the I/O operation(s) using regular NFS READ or 672 WRITE operations via the metadata server. The client SHOULD attempt 673 to retrieve a new layout and retry the I/O operation using the 674 storage device first and only if the error persists, retry the I/O 675 operation via the metadata server. 677 8. Mirroring 679 The Flexible File Layout Type has a simple model in place for the 680 mirroring of files. There is no assumption that each copy of the 681 mirror is stored identically on the storage devices, i.e., one device 682 might employ compression or deduplication on the file. However, the 683 over the wire transfer of the file contents MUST appear identical. 684 Note, this is a construct of the selected XDR representation that 685 each mirrored copy of the file has the same striping pattern (see 686 Figure 1). 688 The metadata server is responsible for determining the number of 689 mirrored copies and the location of each mirror. While the client 690 may provide a hint to how many copies it wants (see Section 12), the 691 metadata server can ignore that hint and in any event, the client has 692 no means to dictate neither the storage device (which also means the 693 coupling and/or protocol levels to access the file) nor the location 694 of said storage device. 696 8.1. Selecting a Mirror 698 When the metadata server grants a layout to a client, it can let the 699 client know how fast it expects each mirror to be once the request 700 arrives at the storage devices via the ffds_efficiency member. While 701 the algorithms to calculate that value are left to the metadata 702 server implementations, factors that could contribute to that 703 calculation include speed of the storage device, physical memory 704 available to the device, operating system version, current load, etc. 706 However, what SHOULD not be involved in that calculation is a 707 perceived network distance between the client and the storage device. 708 The client is better situated for making that determination based on 709 past interaction with the storage device over the different available 710 network intefaces bewteen the two. I.e., the metadata server might 711 not know about a transient outage between the client and storage 712 device because it has no presence on the given subnet. 714 As such, it is the client which decides which mirror to access for 715 reading the file. The requirements for writing to a mirrored file 716 are presented below. 718 8.2. Writing to Mirrors 720 The client is responsible for updating all mirrored copies of the 721 file that it is given in the layout. If all but one copy is updated 722 successfully and the last one provides an error, then the client 723 needs to return the layout to the metadata server with an error 724 indicating that the update failed to that storage device. 726 The metadata server is then responsible for determining if it wants 727 to remove the errant mirror from the layout, if the mirror has 728 recovered from some transient error, etc. When the client tries to 729 get a new layout, the metadata server informs it of the decision by 730 the contents of the layout. The client MUST not make any assumptions 731 that the contents of the previous layout will match those of the new 732 one. If it has updates that were not committed, it MUST resend those 733 updates to all mirrors. 735 8.3. Metadata Server Resilvering of the File 737 The metadata server may elect to create a new mirror of the file at 738 any time. This might be to resilver a copy on a storage device which 739 was down for servicing, to provide a copy of the file on storage with 740 different storage performance characteristics, etc. As the client 741 will not be aware of the new mirror and the metadata server will not 742 be aware of updates that the client is making to the file, the 743 metadata server MUST recall the writable layout segment(s) that it is 744 resilvering. If the client issues a LAYOUTGET for a writable layout 745 segment which is in the process of being resilvered, then the 746 metadata server MUST deny that request with a NFS4ERR_LAYOUTTRYLATER. 747 The client can then perform the IO through the metadata server. 749 9. Flexible Files Layout Type Return 751 layoutreturn_file4 is used in the LAYOUTRETURN operation to convey 752 layout-type specific information to the server. It is defined in 753 [RFC5661] as follows: 755 struct layoutreturn_file4 { 756 offset4 lrf_offset; 757 length4 lrf_length; 758 stateid4 lrf_stateid; 759 /* layouttype4 specific data */ 760 opaque lrf_body<>; 761 }; 763 union layoutreturn4 switch(layoutreturn_type4 lr_returntype) { 764 case LAYOUTRETURN4_FILE: 765 layoutreturn_file4 lr_layout; 766 default: 767 void; 768 }; 770 struct LAYOUTRETURN4args { 771 /* CURRENT_FH: file */ 772 bool lora_reclaim; 773 layoutreturn_stateid lora_recallstateid; 774 layouttype4 lora_layout_type; 775 layoutiomode4 lora_iomode; 776 layoutreturn4 lora_layoutreturn; 777 }; 779 If the lora_layout_type layout type is LAYOUT4_FLEX_FILES, then the 780 lrf_body opaque value is defined by the ff_layoutreturn4 type. The 781 new type allows the client to report I/O error information or layout 782 usage statistics back to the metadata server as defined below. 784 9.1. ff_ioerr 786 /// struct ff_ioerr4 { 787 /// offset4 ffie_offset; 788 /// length4 ffie_length; 789 /// stateid4 ffie_stateid; 790 /// device_error4 ffie_errors; 791 /// }; 792 /// 794 Recall that [NFSv42] defines device_error4 as: 796 struct device_error4 { 797 deviceid4 de_deviceid; 798 nfsstat4 de_status; 799 nfs_opnum4 de_opnum; 800 }; 802 The ff_ioerr4 structure is used to return error indications for data 803 files that generated errors during data transfers. These are hints 804 to the metadata server that there are problems with that file. For 805 each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length 806 represent the storage device and byte range within the file in which 807 the error occurred; ffie_errors represents the operation and type of 808 error. The use of device_error4 is described in Section 16.6 of 809 [NFSv42]. 811 9.2. ff_iostats 813 /// struct ff_iostats4 { 814 /// offset4 ffis_offset; 815 /// length4 ffis_length; 816 /// stateid4 ffis_stateid; 817 /// uint32_t ffis_duration; 818 /// io_info4 ffis_read; 819 /// io_info4 ffis_write; 820 /// layoutupdate4 ffis_layoutupdate; 821 /// }; 822 /// 824 Recall that [NFSv42] defines io_info4 as: 826 struct io_info4 { 827 uint32_t ii_count; 828 uint64_t ii_bytes; 829 }; 831 With pNFS, the data transfers are performed directly between the pNFS 832 client and the storage devices. Therefore, the metadata server has 833 no visibility to the I/O stream and cannot use any statistical 834 information about client I/O to optimize data storage location. 835 ff_iostats4 MAY be used by the client to report I/O statistics back 836 to the metadata server upon returning the layout. Since it is 837 infeasible for the client to report every I/O that used the layout, 838 the client MAY identify "hot" byte ranges for which to report I/O 839 statistics. The definition and/or configuration mechanism of what is 840 considered "hot" and the size of the reported byte range is out of 841 the scope of this document. It is suggested for client 842 implementation to provide reasonable default values and an optional 843 run-time management interface to control these parameters. For 844 example, a client can define the default byte range resolution to be 845 1 MB in size and the thresholds for reporting to be 1 MB/second or 10 846 I/O operations per second. For each byte range, ffis_offset and 847 ffis_length represent the starting offset of the range and the range 848 length in bytes. ffis_duration represents the number of seconds the 849 reported burst of I/O lasted. ffis_read.ii_count, 850 ffis_read.ii_bytes, ffis_write.ii_count, and ffis_write.ii_bytes 851 represent, respectively, the number of contiguous read and write I/Os 852 and the respective aggregate number of bytes transferred within the 853 reported byte range. [[AI7: Need to define whether we are using 854 ffis_layoutupdate or not. --TH]] [[AI8: Actually, ffis_duration 855 might be what we plop down in there. In any event, ffis_duration 856 needs some work. --TH]] 858 9.3. ff_layoutreturn 860 /// struct ff_layoutreturn { 861 /// ff_ioerr4 fflr_ioerr_report<>; 862 /// ff_iostats4 fflr_iostats_report<>; 863 /// }; 864 /// 866 When data file I/O operations fail, fflr_ioerr_report<> is used to 867 report these errors to the metadata server as an array of elements of 868 type ff_ioerr4. Each element in the array represents an error that 869 occurred on the data file identified by ffie_errors.de_deviceid. If 870 no errors are to be reported, the size of the fflr_ioerr_report<> 871 array is set to zero. The client MAY also use fflr_iostats_report<> 872 to report a list of I/O statistics as an array of elements of type 873 ff_iostats4. Each element in the array represents statistics for a 874 particular byte range. Byte ranges are not guaranteed to be disjoint 875 and MAY repeat or intersect. 877 10. Flexible Files Layout Type LAYOUTERROR 879 If the client is using NFSv4.2 to communicate with the metadata 880 server, then instead of waiting for a LAYOUTRETURN to send error 881 information to the metadata server (see Section 9.1), it can use 882 LAYOUTERROR (see Section 16.6 of [NFSv42]) to communicate that 883 information. 885 11. Flexible Files Layout Type LAYOUTSTATS 887 If the client is using NFSv4.2 to communicate with the metadata 888 server, then instead of waiting for a LAYOUTRETURN to send I/O 889 statistics to the metadata server (see Section 9.2), it can use 890 LAYOUTSTATS (see Section 16.7 of [NFSv42]) to communicate that 891 information. 893 12. Flexible File Layout Type Creation Hint 895 The layouthint4 type is defined in the [RFC5661] as follows: 897 struct layouthint4 { 898 layouttype4 loh_type; 899 opaque loh_body<>; 900 }; 902 The layouthint4 structure is used by the client to pass a hint about 903 the type of layout it would like created for a particular file. If 904 the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body 905 opaque value is defined by the ff_layouthint4 type. 907 12.1. ff_layouthint4 909 /// union ff_mirrors_hint switch (bool ffmc_valid) { 910 /// case TRUE: 911 /// uint32_t ffmc_mirrors; 912 /// case FALSE: 913 /// void; 914 /// }; 915 /// 917 /// struct ff_layouthint4 { 918 /// ff_mirrors_hint fflh_mirrors_hint; 919 /// }; 920 /// 922 This type conveys hints for the desired data map. All parameters are 923 optional so the client can give values for only the parameter it 924 cares about. 926 13. Recalling Layouts 928 The Flexible File Layout Type metadata server should recall 929 outstanding layouts in the following cases: 931 o When the file's security policy changes, i.e., Access Control 932 Lists (ACLs) or permission mode bits are set. 934 o When the file's layout changes, rendering outstanding layouts 935 invalid. 937 o When there are sharing conflicts. 939 13.1. CB_RECALL_ANY 941 The metadata server can use the CB_RECALL_ANY callback operation to 942 notify the client to return some or all of its layouts. The 943 [RFC5661] defines the following types: 945 const RCA4_TYPE_MASK_FF_LAYOUT_MIN = -2; 946 const RCA4_TYPE_MASK_FF_LAYOUT_MAX = -1; 947 [[RFC Editor: please insert assigned constants]] 949 struct CB_RECALL_ANY4args { 950 uint32_t craa_layouts_to_keep; 951 bitmap4 craa_type_mask; 952 }; 954 Typically, CB_RECALL_ANY will be used to recall client state when the 955 server needs to reclaim resources. The craa_type_mask bitmap 956 specifies the type of resources that are recalled and the 957 craa_layouts_to_keep value specifies how many of the recalled 958 Flexible File Layouts the client is allowed to keep. The Flexible 959 File Layout Type mask flags are defined as follows: 961 /// enum ff_cb_recall_any_mask { 962 /// FF_RCA4_TYPE_MASK_READ = -2, 963 /// FF_RCA4_TYPE_MASK_RW = -1 964 [[RFC Editor: please insert assigned constants]] 965 /// }; 966 /// 968 They represent the iomode of the recalled layouts. In response, the 969 client SHOULD return layouts of the recalled iomode that it needs the 970 least, keeping at most craa_layouts_to_keep Flexible File Layouts. 972 The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return 973 layouts of iomode LAYOUTIOMODE4_READ. Similarly, the 974 PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts 975 of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client 976 is notified to return layouts of either iomode. 978 14. Client Fencing 980 In cases where clients are uncommunicative and their lease has 981 expired or when clients fail to return recalled layouts within a 982 lease period, at the least the server MAY revoke client layouts and/ 983 or device address mappings and reassign these resources to other 984 clients (see "Recalling a Layout" in [RFC5661]). To avoid data 985 corruption, the metadata server MUST fence off the revoked clients 986 from the respective data files as described in Section 2.2. 988 15. Security Considerations 990 The pNFS extension partitions the NFSv4 file system protocol into two 991 parts, the control path and the data path (storage protocol). The 992 control path contains all the new operations described by this 993 extension; all existing NFSv4 security mechanisms and features apply 994 to the control path. The combination of components in a pNFS system 995 is required to preserve the security properties of NFSv4 with respect 996 to an entity accessing data via a client, including security 997 countermeasures to defend against threats that NFSv4 provides 998 defenses for in environments where these threats are considered 999 significant. 1001 The metadata server enforces the file access-control policy at 1002 LAYOUTGET time. The client should use suitable authorization 1003 credentials for getting the layout for the requested iomode (READ or 1004 RW) and the server verifies the permissions and ACL for these 1005 credentials, possibly returning NFS4ERR_ACCESS if the client is not 1006 allowed the requested iomode. If the LAYOUTGET operation succeeds 1007 the client receives, as part of the layout, a set of credentials 1008 allowing it I/O access to the specified data files corresponding to 1009 the requested iomode. When the client acts on I/O operations on 1010 behalf of its local users, it MUST authenticate and authorize the 1011 user by issuing respective OPEN and ACCESS calls to the metadata 1012 server, similar to having NFSv4 data delegations. If access is 1013 allowed, the client uses the corresponding (READ or RW) credentials 1014 to perform the I/O operations at the data files storage devices. 1015 When the metadata server receives a request to change a file's 1016 permissions or ACL, it SHOULD recall all layouts for that file and it 1017 MUST fence off the clients holding outstanding layouts for the 1018 respective file by implicitly invalidating the outstanding 1019 credentials on all data files comprising before committing to the new 1020 permissions and ACL. Doing this will ensure that clients re- 1021 authorize their layouts according to the modified permissions and ACL 1022 by requesting new layouts. Recalling the layouts in this case is 1023 courtesy of the server intended to prevent clients from getting an 1024 error on I/Os done after the client was fenced off. 1026 16. IANA Considerations 1028 As described in [RFC5661], new layout type numbers have been assigned 1029 by IANA. This document defines the protocol associated with the 1030 existing layout type number, LAYOUT4_FLEX_FILES. 1032 17. References 1034 17.1. Normative References 1036 [LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents", 1037 November 2008, . 1040 [NFSv42] Haynes, T., "NFS Version 4 Minor Version 2", draft-ietf- 1041 nfsv4-minorversion2-22 (Work In Progress), April 2014. 1043 [RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813, 1044 June 1995. 1046 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1047 Requirement Levels", BCP 14, RFC 2119, March 1997. 1049 [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., 1050 Beame, C., Eisler, M., and D. Noveck, "Network File System 1051 (NFS) version 4 Protocol", RFC 3530, April 2003. 1053 [RFC4506] Eisler, M., "XDR: External Data Representation Standard", 1054 STD 67, RFC 4506, May 2006. 1056 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1057 "Network File System (NFS) Version 4 Minor Version 1 1058 Protocol", RFC 5661, January 2010. 1060 [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1061 "Network File System (NFS) Version 4 Minor Version 1 1062 External Data Representation Standard (XDR) Description", 1063 RFC 5662, January 2010. 1065 [RFC5664] Halevy, B., Ed., Welch, B., Ed., and J. Zelenka, Ed., 1066 "Object-Based Parallel NFS (pNFS) Operations", RFC 5664, 1067 January 2010. 1069 [pNFSLayouts] 1070 Haynes, T., "Considerations for a New pNFS Layout Type", 1071 draft-haynes-nfsv4-layout-types-02 (Work In Progress), 1072 April 2014. 1074 17.2. Informative References 1076 [ANSI400-2004] 1077 Weber, R., Ed., "ANSI INCITS 400-2004, Information 1078 Technology - SCSI Object-Based Storage Device Commands 1079 (OSD)", December 2004. 1081 Appendix A. Acknowledgments 1083 Those who provided miscellaneous comments to early drafts of this 1084 document include: Matt W. Benjamin, Adam Emerson, Tom Haynes, J. 1085 Bruce Fields, and Lev Solomonov. 1087 Appendix B. RFC Editor Notes 1089 [RFC Editor: please remove this section prior to publishing this 1090 document as an RFC] 1092 [RFC Editor: prior to publishing this document as an RFC, please 1093 replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the 1094 RFC number of this document] 1096 Authors' Addresses 1098 Benny Halevy 1099 Primary Data, Inc. 1101 Email: bhalevy@primarydata.com 1102 URI: http://www.primarydata.com 1104 Thomas Haynes 1105 Primary Data, Inc. 1106 4300 El Camino Real Ste 100 1107 Los Altos, CA 94022 1108 USA 1110 Phone: +1 408 215 1519 1111 Email: thomas.haynes@primarydata.com