idnits 2.17.1 draft-ietf-nfsv4-flex-files-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 338 has weird spacing: '... loghyr staff...' == Line 1047 has weird spacing: '...stateid lor...' == Line 1296 has weird spacing: '...rs_hint ffl...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: FF_FLAGS_NO_IO_THRU_MDS : can be set to indicate that the client SHOULD not send IO operations to the metadata server. I.e., even if a storage device is partitioned from the client, the client SHOULD not try to proxy the IO through the metadata server. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: FF_FLAGS_NO_READ_IO: can be set to indicate that the client SHOULD not send READ requests with the layouts of iomode LAYOUTIOMODE4_RW. Instead, it should request a layout of iomode LAYOUTIOMODE4_READ from the metadata server. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: NFS4ERR_DELAY: there is some issue preventing the layout from being granted. If the client already has an appropriate layout, it SHOULD not continue with IO to the storage devices. -- The document date (May 09, 2016) is 2909 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. 'LEGAL' ** Downref: Normative reference to an Informational RFC: RFC 1813 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 2 errors (**), 0 flaws (~~), 7 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 B. Halevy 3 Internet-Draft 4 Intended status: Standards Track T. Haynes 5 Expires: November 10, 2016 Primary Data 6 May 09, 2016 8 Parallel NFS (pNFS) Flexible File Layout 9 draft-ietf-nfsv4-flex-files-08.txt 11 Abstract 13 The Parallel Network File System (pNFS) allows a separation between 14 the metadata (onto a metadata server) and data (onto a storage 15 device) for a file. The flexible file layout type is defined in this 16 document as an extension to pNFS to allow the use of storage devices 17 in a fashion such that they require only a quite limited degree of 18 interaction with the metadata server, using already existing 19 protocols. Client side mirroring is also added to provide 20 replication of files. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at http://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on November 10, 2016. 39 Copyright Notice 41 Copyright (c) 2016 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 57 1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3 58 1.2. Difference Between a Data Server and a Storage Device . . 5 59 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6 60 2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6 61 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6 62 2.2. Fencing Clients from the Data Server . . . . . . . . . . 6 63 2.2.1. Implementation Notes for Synthetic uids/gids . . . . 7 64 2.2.2. Example of using Synthetic uids/gids . . . . . . . . 7 65 2.3. State and Locking Models . . . . . . . . . . . . . . . . 8 66 3. XDR Description of the Flexible File Layout Type . . . . . . 9 67 3.1. Code Components Licensing Notice . . . . . . . . . . . . 10 68 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 11 69 4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 11 70 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 13 71 5. Flexible File Layout type . . . . . . . . . . . . . . . . . . 14 72 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 14 73 5.1.1. Error codes from LAYOUTGET . . . . . . . . . . . . . 18 74 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS . . 18 75 5.2. Interactions Between Devices and Layouts . . . . . . . . 18 76 5.3. Handling Version Errors . . . . . . . . . . . . . . . . . 19 77 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 19 78 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 20 79 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 20 80 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 21 81 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 21 82 8.3. Metadata Server Resilvering of the File . . . . . . . . . 22 83 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 22 84 9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 23 85 9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 23 86 9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 24 87 9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 24 88 9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 25 89 9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 25 90 9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 26 91 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 27 92 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 27 93 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 27 94 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 28 95 13. Recalling a Layout . . . . . . . . . . . . . . . . . . . . . 28 96 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 28 98 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 29 99 15. Security Considerations . . . . . . . . . . . . . . . . . . . 30 100 15.1. Kerberized File Access . . . . . . . . . . . . . . . . . 30 101 15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 31 102 15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 31 103 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 104 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 31 105 17.1. Normative References . . . . . . . . . . . . . . . . . . 31 106 17.2. Informative References . . . . . . . . . . . . . . . . . 32 107 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 32 108 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 33 109 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 33 111 1. Introduction 113 In the parallel Network File System (pNFS), the metadata server 114 returns layout type structures that describe where file data is 115 located. There are different layout types for different storage 116 systems and methods of arranging data on storage devices. This 117 document defines the flexible file layout type used with file-based 118 data servers that are accessed using the Network File System (NFS) 119 protocols: NFSv3 [RFC1813], NFSv4.0 [RFC7530], NFSv4.1 [RFC5661], and 120 NFSv4.2 [RFC7862]. 122 To provide a global state model equivalent to that of the files 123 layout type, a back-end control protocol MAY be implemented between 124 the metadata server and NFSv4.1+ storage devices. It is out of scope 125 for this document to specify the wire protocol of such a protocol, 126 yet the requirements for the protocol are specified in [RFC5661] and 127 clarified in [pNFSLayouts]. 129 1.1. Definitions 131 control protocol: is a set of requirements for the communication of 132 information on layouts, stateids, file metadata, and file data 133 between the metadata server and the storage devices (see 134 [pNFSLayouts]). 136 client-side mirroring: is when the client and not the server is 137 responsible for updating all of the mirrored copies of a layout 138 segment. 140 data file: is that part of the file system object which describes 141 the payload and not the object. E.g., it is the file contents. 143 data server (DS): is one of the pNFS servers which provides the 144 contents of a file system object which is a regular file. 145 Depending on the layout, there might be one or more data servers 146 over which the data is striped. Note that while the metadata 147 server is strictly accessed over the NFSv4.1+ protocol, depending 148 on the layout type, the data server could be accessed via any 149 protocol that meets the pNFS requirements. 151 fencing: is when the metadata server prevents the storage devices 152 from processing I/O from a specific client to a specific file. 154 file layout type: is a layout type in which the storage devices are 155 accessed via the NFS protocol. 157 layout: informs a client of which storage devices it needs to 158 communicate with (and over which protocol) to perform I/O on a 159 file. The layout might also provide some hints about how the 160 storage is physically organized. 162 layout iomode: describes whether the layout granted to the client is 163 for read or read/write I/O. 165 layout segment: describes a sub-division of a layout. That sub- 166 division might be by the iomode (see Sections 3.3.20 and 12.2.9 of 167 [RFC5661]), a striping pattern (see Section 13.3 of [RFC5661]), or 168 requested byte range. 170 layout stateid: is a 128-bit quantity returned by a server that 171 uniquely defines the layout state provided by the server for a 172 specific layout that describes a layout type and file (see 173 Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 describes 174 the difference between a layout stateid and a normal stateid. 176 layout type: describes both the storage protocol used to access the 177 data and the aggregation scheme used to lay out the file data on 178 the underlying storage devices. 180 loose coupling: is when the metadata server and the storage devices 181 do not have a control protocol present. 183 metadata file: is that part of the file system object which 184 describes the object and not the payload. E.g., it could be the 185 time since last modification, access, etc. 187 metadata server (MDS): is the pNFS server which provides metadata 188 information for a file system object. It also is responsible for 189 generating layouts for file system objects. Note that the MDS is 190 responsible for directory-based operations. 192 mirror: is a copy of a layout segment. While mirroring can be used 193 for backing up a layout segment, the copies can be distributed 194 such that each remote site has a locally available copy. Note 195 that if one copy of the mirror is updated, then all copies must be 196 updated. 198 recalling a layout: is when the metadata server uses a back channel 199 to inform the client that the layout is to be returned in a 200 graceful manner. Note that the client could be able to flush any 201 writes, etc., before replying to the metadata server. 203 revoking a layout: is when the metadata server invalidates the 204 layout such that neither the metadata server nor any storage 205 device will accept any access from the client with that layout. 207 resilvering: is the act of rebuilding a mirrored copy of a layout 208 segment from a known good copy of the layout segment. Note that 209 this can also be done to create a new mirrored copy of the layout 210 segment. 212 rsize: is the data transfer buffer size used for reads. 214 stateid: is a 128-bit quantity returned by a server that uniquely 215 defines the open and locking states provided by the server for a 216 specific open-owner or lock-owner/open-owner pair for a specific 217 file and type of lock. 219 storage device: is another term used almost interchangeably with 220 data server. See Section 1.2 for the nuances between the two. 222 tight coupling: is when the metadata server and the storage devices 223 do have a control protocol present. 225 wsize: is the data transfer buffer size used for writes. 227 1.2. Difference Between a Data Server and a Storage Device 229 We defined a data server as a pNFS server, which implies that it can 230 utilize the NFSv4.1+ protocol to communicate with the client. As 231 such, only the file layout type would currently meet this 232 requirement. The more generic concept is a storage device, which can 233 use any protocol to communicate with the client. The requirements 234 for a storage device to act together with the metadata server to 235 provide data to a client are that there is a layout type 236 specification for the given protocol and that the metadata server has 237 granted a layout to the client. Note that nothing precludes there 238 being multiple supported layout types (i.e., protocols) between a 239 metadata server, storage devices, and client. 241 As storage device is the more encompassing terminology, this document 242 utilizes it over data server. 244 1.3. Requirements Language 246 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 247 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 248 document are to be interpreted as described in [RFC2119]. 250 2. Coupling of Storage Devices 252 The coupling of the metadata server with the storage devices can be 253 either tight or loose. In a tight coupling, there is a control 254 protocol present to manage security, LAYOUTCOMMITs, etc. With a 255 loose coupling, the only control protocol might be a version of NFS. 256 As such, semantics for managing security, state, and locking models 257 MUST be defined. 259 2.1. LAYOUTCOMMIT 261 With a tightly coupled system, when the metadata server receives a 262 LAYOUTCOMMIT (see Section 18.42 of [RFC5661]), the semantics of the 263 file layout type MUST be met (see Section 12.5.4 of [RFC5661]). It 264 is the responsibility of the client to make sure the data file is 265 stable before the metadata server begins to query the storage devices 266 about the changes to the file. With a loosely coupled system, if any 267 WRITE to a storage device did not result with stable_how equal to 268 FILE_SYNC, a LAYOUTCOMMIT to the metadata server MUST be preceded 269 with a COMMIT to the storage device. Note that if the client has not 270 done a COMMIT to the storage device, then the LAYOUTCOMMIT might not 271 be synchronized to the last WRITE operation to the storage device. 273 2.2. Fencing Clients from the Data Server 275 With loosely coupled storage devices, the metadata server uses 276 synthetic uids and gids for the data file, where the uid owner of the 277 data file is allowed read/write access and the gid owner is allowed 278 read only access. As part of the layout (see ffds_user and 279 ffds_group in Section 5.1), the client is provided with the user and 280 group to be used in the Remote Procedure Call (RPC) [RFC5531] 281 credentials needed to access the data file. Fencing off of clients 282 is achieved by the metadata server changing the synthetic uid and/or 283 gid owners of the data file on the storage device to implicitly 284 revoke the outstanding RPC credentials. 286 With this loosely coupled model, the metadata server is not able to 287 fence off a single client, it is forced to fence off all clients. 288 However, as the other clients react to the fencing, returning their 289 layouts and trying to get new ones, the metadata server can hand out 290 a new uid and gid to allow access. 292 Note: it is recommended to implement common access control methods at 293 the storage device filesystem to allow only the metadata server root 294 (super user) access to the storage device, and to set the owner of 295 all directories holding data files to the root user. This approach 296 provides a practical model to enforce access control and fence off 297 cooperative clients, but it can not protect against malicious 298 clients; hence it provides a level of security equivalent to 299 AUTH_SYS. 301 With tightly coupled storage devices, the metadata server sets the 302 user and group owners, mode bits, and ACL of the data file to be the 303 same as the metadata file. And the client must authenticate with the 304 storage device and go through the same authorization process it would 305 go through via the metadata server. 307 2.2.1. Implementation Notes for Synthetic uids/gids 309 The selection method for the synthetic uids and gids to be used for 310 fencing in loosely coupled storage devices is strictly an 311 implementation issue. I.e., an administrator might restrict a range 312 of such ids available to the Lightweight Directory Access Protocol 313 (LDAP) 'uid' field [RFC4519]. She might also be able to choose an id 314 that would never be used to grant acccess. Then when the metadata 315 server had a request to access a file, a SETATTR would be sent to the 316 storage device to set the owner and group of the data file. The user 317 and group might be selected in a round robin fashion from the range 318 of available ids. 320 Those ids would be sent back as ffds_user and ffds_group to the 321 client. And it would present them as the RPC credentials to the 322 storage device. When the client was done accessing the file and the 323 metadata server knew that no other client was accessing the file, it 324 could reset the owner and group to restrict access to the data file. 326 When the metadata server wanted to fence off a client, it would 327 change the synthetic uid and/or gid to the restricted ids. Note that 328 using a restricted id ensures that there is a change of owner and at 329 least one id available that never gets allowed access. 331 2.2.2. Example of using Synthetic uids/gids 333 The user loghyr creates a file "ompha.c" on the metadata server and 334 it creates a corresponding data file on the storage device. 336 The metadata server entry may look like: 338 -rw-r--r-- 1 loghyr staff 1697 Dec 4 11:31 ompha.c 340 On the storage device, it may be assigned some random synthetic uid/ 341 gid to deny access: 343 -rw-r----- 1 19452 28418 1697 Dec 4 11:31 data_ompha.c 345 When the file is opened on a client, since the layout knows nothing 346 about the user (and does not care), whether loghyr or garbo opens the 347 file does not matter. The owner and group are modified and those 348 values are returned. 350 -rw-r----- 1 1066 1067 1697 Dec 4 11:31 data_ompha.c 352 The set of synthetic gids on the storage device should be selected 353 such that there is no mapping in any of the name services used by the 354 storage device. I.e., each group should have no members. 356 If the layout segment has an iomode of LAYOUTIOMODE4_READ, then the 357 metadata server should return a synthetic uid that is not set on the 358 storage device. Only the synthetic gid would be valid. 360 The client is thus solely responsible for enforcing file permissions 361 in a loosely coupled model. To allow loghyr write access, it will 362 send an RPC to the storage device with a credential of 1066:1067. To 363 allow garbo read access, it will send an RPC to the storage device 364 with a credential of 1067:1067. The value of the uid does not matter 365 as long as it is not the synthetic uid granted it when getting the 366 layout. 368 While pushing the enforcement of permission checking onto the client 369 may seem to weaken security, the client may already be responsible 370 for enforcing permissions before modifications are sent to a server. 371 With cached writes, the client is always responsible for tracking who 372 is modifying a file and making sure to not coalesce requests from 373 multiple users into one request. 375 2.3. State and Locking Models 377 Metadata file OPEN, LOCK, and DELEGATION operations are always 378 executed only against the metadata server. 380 The metadata server responds to state changing operations by 381 executing them against the respective data files on the storage 382 devices. It then sends the storage device open stateid as part of 383 the layout (see the ffm_stateid in Section 5.1) and it is then used 384 by the client for executing READ/WRITE operations against the storage 385 device. 387 NFSv4.1+ storage devices that do not return the 388 EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are indicating that 389 they are loosely coupled. As such, they are treated the same way as 390 NFSv4 storage devices. 392 NFSv4.1+ storage devices that do identify themselves with the 393 EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are stongly 394 coupled. They will be using a back-end control protocol as described 395 in [RFC5661] to implement a global stateid model as defined there. 397 3. XDR Description of the Flexible File Layout Type 399 This document contains the external data representation (XDR) 400 [RFC4506] description of the flexible file layout type. The XDR 401 description is embedded in this document in a way that makes it 402 simple for the reader to extract into a ready-to-compile form. The 403 reader can feed this document into the following shell script to 404 produce the machine readable XDR description of the flexible file 405 layout type: 407 409 #!/bin/sh 410 grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' 412 414 That is, if the above script is stored in a file called "extract.sh", 415 and this document is in a file called "spec.txt", then the reader can 416 do: 418 sh extract.sh < spec.txt > flex_files_prot.x 420 The effect of the script is to remove leading white space from each 421 line, plus a sentinel sequence of "///". 423 The embedded XDR file header follows. Subsequent XDR descriptions, 424 with the sentinel sequence are embedded throughout the document. 426 Note that the XDR code contained in this document depends on types 427 from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs 428 types that end with a 4, such as offset4, length4, etc., as well as 429 more generic types such as uint32_t and uint64_t. 431 3.1. Code Components Licensing Notice 433 Both the XDR description and the scripts used for extracting the XDR 434 description are Code Components as described in Section 4 of "Legal 435 Provisions Relating to IETF Documents" [LEGAL]. These Code 436 Components are licensed according to the terms of that document. 438 440 /// /* 441 /// * Copyright (c) 2012 IETF Trust and the persons identified 442 /// * as authors of the code. All rights reserved. 443 /// * 444 /// * Redistribution and use in source and binary forms, with 445 /// * or without modification, are permitted provided that the 446 /// * following conditions are met: 447 /// * 448 /// * o Redistributions of source code must retain the above 449 /// * copyright notice, this list of conditions and the 450 /// * following disclaimer. 451 /// * 452 /// * o Redistributions in binary form must reproduce the above 453 /// * copyright notice, this list of conditions and the 454 /// * following disclaimer in the documentation and/or other 455 /// * materials provided with the distribution. 456 /// * 457 /// * o Neither the name of Internet Society, IETF or IETF 458 /// * Trust, nor the names of specific contributors, may be 459 /// * used to endorse or promote products derived from this 460 /// * software without specific prior written permission. 461 /// * 462 /// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS 463 /// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED 464 /// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 465 /// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 466 /// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 467 /// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 468 /// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 469 /// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 470 /// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 471 /// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 472 /// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 473 /// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 474 /// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 475 /// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 476 /// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 477 /// * 478 /// * This code was derived from RFCTBD10. 480 /// * Please reproduce this note if possible. 481 /// */ 482 /// 483 /// /* 484 /// * flex_files_prot.x 485 /// */ 486 /// 487 /// /* 488 /// * The following include statements are for example only. 489 /// * The actual XDR definition files are generated separately 490 /// * and independently and are likely to have a different name. 491 /// * %#include 492 /// * %#include 493 /// */ 494 /// 496 498 4. Device Addressing and Discovery 500 Data operations to a storage device require the client to know the 501 network address of the storage device. The NFSv4.1+ GETDEVICEINFO 502 operation (Section 18.40 of [RFC5661]) is used by the client to 503 retrieve that information. 505 4.1. ff_device_addr4 507 The ff_device_addr4 data structure is returned by the server as the 508 storage protocol specific opaque field da_addr_body in the 509 device_addr4 structure by a successful GETDEVICEINFO operation. 511 513 /// struct ff_device_versions4 { 514 /// uint32_t ffdv_version; 515 /// uint32_t ffdv_minorversion; 516 /// uint32_t ffdv_rsize; 517 /// uint32_t ffdv_wsize; 518 /// bool ffdv_tightly_coupled; 519 /// }; 520 /// 522 /// struct ff_device_addr4 { 523 /// multipath_list4 ffda_netaddrs; 524 /// ff_device_versions4 ffda_versions<>; 525 /// }; 526 /// 527 529 The ffda_netaddrs field is used to locate the storage device. It 530 MUST be set by the server to a list holding one or more of the device 531 network addresses. 533 The ffda_versions array allows the metadata server to present choices 534 as to NFS version, minor version, and coupling strength to the 535 client. The ffdv_version and ffdv_minorversion represent the NFS 536 protocol to be used to access the storage device. This layout 537 specification defines the semantics for ffdv_versions 3 and 4. If 538 ffdv_version equals 3 then the server MUST set ffdv_minorversion to 0 539 and ffdv_tightly_coupled to false. The client MUST then access the 540 storage device using the NFSv3 protocol [RFC1813]. If ffdv_version 541 equals 4 then the server MUST set ffdv_minorversion to one of the 542 NFSv4 minor version numbers and the client MUST access the storage 543 device using NFSv4. 545 Note that while the client might determine that it cannot use any of 546 the configured combinations of ffdv_version, ffdv_minorversion, and 547 ffdv_tightly_coupled, when it gets the device list from the metadata 548 server, there is no way to indicate to the metadata server as to 549 which device it is version incompatible. If however, the client 550 waits until it retrieves the layout from the metadata server, it can 551 at that time clearly identify the storage device in question (see 552 Section 5.3). 554 The ffdv_rsize and ffdv_wsize are used to communicate the maximum 555 rsize and wsize supported by the storage device. As the storage 556 device can have a different rsize or wsize than the metadata server, 557 the ffdv_rsize and ffdv_wsize allow the metadata server to 558 communicate that information on behalf of the storage device. 560 ffdv_tightly_coupled informs the client as to whether the metadata 561 server is tightly coupled with the storage devices or not. Note that 562 even if the data protocol is at least NFSv4.1, it may still be the 563 case that there is loose coupling is in effect. If 564 ffdv_tightly_coupled is not set, then the client MUST commit writes 565 to the storage devices for the file before sending a LAYOUTCOMMIT to 566 the metadata server. I.e., the writes MUST be committed by the 567 client to stable storage via issuing WRITEs with stable_how == 568 FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how != 569 FILE_SYNC (see Section 3.3.7 of [RFC1813]). 571 4.2. Storage Device Multipathing 573 The flexible file layout type supports multipathing to multiple 574 storage device addresses. Storage device level multipathing is used 575 for bandwidth scaling via trunking and for higher availability of use 576 in the event of a storage device failure. Multipathing allows the 577 client to switch to another storage device address which may be that 578 of another storage device that is exporting the same data stripe 579 unit, without having to contact the metadata server for a new layout. 581 To support storage device multipathing, ffda_netaddrs contains an 582 array of one or more storage device network addresses. This array 583 (data type multipath_list4) represents a list of storage devices 584 (each identified by a network address), with the possibility that 585 some storage device will appear in the list multiple times. 587 The client is free to use any of the network addresses as a 588 destination to send storage device requests. If some network 589 addresses are less desirable paths to the data than others, then the 590 MDS SHOULD NOT include those network addresses in ffda_netaddrs. If 591 less desirable network addresses exist to provide failover, the 592 RECOMMENDED method to offer the addresses is to provide them in a 593 replacement device-ID-to-device-address mapping, or a replacement 594 device ID. When a client finds no response from the storage device 595 using all addresses available in ffda_netaddrs, it SHOULD send a 596 GETDEVICEINFO to attempt to replace the existing device-ID-to-device- 597 address mappings. If the MDS detects that all network paths 598 represented by ffda_netaddrs are unavailable, the MDS SHOULD send a 599 CB_NOTIFY_DEVICEID (if the client has indicated it wants device ID 600 notifications for changed device IDs) to change the device-ID-to- 601 device-address mappings to the available addresses. If the device ID 602 itself will be replaced, the MDS SHOULD recall all layouts with the 603 device ID, and thus force the client to get new layouts and device ID 604 mappings via LAYOUTGET and GETDEVICEINFO. 606 Generally, if two network addresses appear in ffda_netaddrs, they 607 will designate the same storage device. When the storage device is 608 accessed over NFSv4.1 or a higher minor version, the two storage 609 device addresses will support the implementation of client ID or 610 session trunking (the latter is RECOMMENDED) as defined in [RFC5661]. 611 The two storage device addresses will share the same server owner or 612 major ID of the server owner. It is not always necessary for the two 613 storage device addresses to designate the same storage device with 614 trunking being used. For example, the data could be read-only, and 615 the data consist of exact replicas. 617 5. Flexible File Layout type 619 The layout4 type is defined in [RFC5662] as follows: 621 623 enum layouttype4 { 624 LAYOUT4_NFSV4_1_FILES = 1, 625 LAYOUT4_OSD2_OBJECTS = 2, 626 LAYOUT4_BLOCK_VOLUME = 3, 627 LAYOUT4_FLEX_FILES = 4 628 [[RFC Editor: please modify the LAYOUT4_FLEX_FILES 629 to be the layouttype assigned by IANA]] 630 }; 632 struct layout_content4 { 633 layouttype4 loc_type; 634 opaque loc_body<>; 635 }; 637 struct layout4 { 638 offset4 lo_offset; 639 length4 lo_length; 640 layoutiomode4 lo_iomode; 641 layout_content4 lo_content; 642 }; 644 646 This document defines structure associated with the layouttype4 value 647 LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an 648 XDR type "opaque". The opaque layout is uninterpreted by the generic 649 pNFS client layers, but is interpreted by the flexible file layout 650 type implementation. This section defines the structure of this 651 otherwise opaque value, ff_layout4. 653 5.1. ff_layout4 655 657 /// const FF_FLAGS_NO_LAYOUTCOMMIT = 0x00000001; 658 /// const FF_FLAGS_NO_IO_THRU_MDS = 0x00000002; 659 /// const FF_FLAGS_NO_READ_IO = 0x00000004; 661 /// typedef uint32_t ff_flags4; 662 /// 663 /// struct ff_data_server4 { 664 /// deviceid4 ffds_deviceid; 665 /// uint32_t ffds_efficiency; 666 /// stateid4 ffds_stateid; 667 /// nfs_fh4 ffds_fh_vers<>; 668 /// fattr4_owner ffds_user; 669 /// fattr4_owner_group ffds_group; 670 /// }; 671 /// 673 /// struct ff_mirror4 { 674 /// ff_data_server4 ffm_data_servers<>; 675 /// }; 676 /// 678 /// struct ff_layout4 { 679 /// length4 ffl_stripe_unit; 680 /// ff_mirror4 ffl_mirrors<>; 681 /// ff_flags4 ffl_flags; 682 /// uint32_t ffl_stats_collect_hint; 683 /// }; 684 /// 686 688 The ff_layout4 structure specifies a layout over a set of mirrored 689 copies of that portion of the data file described in the current 690 layout segment. This mirroring protects against loss of data in 691 layout segments. Note that while not explicitly shown in the above 692 XDR, each layout4 element returned in the logr_layout array of 693 LAYOUTGET4res (see Section 18.43.1 of [RFC5661]) descibes a layout 694 segment. Hence each ff_layout4 also descibes a layout segment. 696 It is possible that the file is concatenated from more than one 697 layout segment. Each layout segment MAY represent different striping 698 parameters, applying respectively only to the layout segment byte 699 range. 701 The ffl_stripe_unit field is the stripe unit size in use for the 702 current layout segment. The number of stripes is given inside each 703 mirror by the number of elements in ffm_data_servers. If the number 704 of stripes is one, then the value for ffl_stripe_unit MUST default to 705 zero. The only supported mapping scheme is sparse and is detailed in 706 Section 6. Note that there is an assumption here that both the 707 stripe unit size and the number of stripes is the same across all 708 mirrors. 710 The ffl_mirrors field is the array of mirrored storage devices which 711 provide the storage for the current stripe, see Figure 1. 713 The ffl_stats_collect_hint field provides a hint to the client on how 714 often the server wants it to report LAYOUTSTATS for a file. The time 715 is in seconds. 717 +-----------+ 718 | | 719 | | 720 | File | 721 | | 722 | | 723 +-----+-----+ 724 | 725 +------------+------------+ 726 | | 727 +----+-----+ +-----+----+ 728 | Mirror 1 | | Mirror 2 | 729 +----+-----+ +-----+----+ 730 | | 731 +-----------+ +-----------+ 732 |+-----------+ |+-----------+ 733 ||+-----------+ ||+-----------+ 734 +|| Storage | +|| Storage | 735 +| Devices | +| Devices | 736 +-----------+ +-----------+ 738 Figure 1 740 The ffs_mirrors field represents an array of state information for 741 each mirrored copy of the current layout segment. Each element is 742 described by a ff_mirror4 type. 744 ffds_deviceid provides the deviceid of the storage device holding the 745 data file. 747 ffds_fh_vers is an array of filehandles of the data file matching to 748 the available NFS versions on the given storage device. There MUST 749 be exactly as many elements in ffds_fh_vers as there are in 750 ffda_versions. Each element of the array corresponds to a particular 751 combination of ffdv_version, ffdv_minorversion, and 752 ffdv_tightly_coupled provided for the device. The array allows for 753 server implementations which have different filehandles for different 754 combinations of version, minor version, and coupling strength. See 755 Section 5.3 for how to handle versioning issues between the client 756 and storage devices. 758 For tight coupling, ffds_stateid provides the stateid to be used by 759 the client to access the file. For loose coupling and a NFSv4 760 storage device, the client may use an anonymous stateid to perform I/ 761 O on the storage device as there is no use for the metadata server 762 stateid (no control protocol). In such a scenario, the server MUST 763 set the ffds_stateid to be the anonymous stateid. 765 For loosely coupled storage devices, ffds_user and ffds_group provide 766 the synthetic user and group to be used in the RPC credentials that 767 the client presents to the storage device to access the data files. 768 For tightly coupled storage devices, the user and group on the 769 storage device will be the same as on the metadata server. I.e., if 770 ffdv_tightly_coupled (see Section 4.1) is set, then the client MUST 771 ignore both ffds_user and ffds_group. 773 The allowed values for both ffds_user and ffds_group are specified in 774 Section 5.9 of [RFC5661]. For NFSv3 compatibility, user and group 775 strings that consist of decimal numeric values with no leading zeros 776 can be given a special interpretation by clients and servers that 777 choose to provide such support. The receiver may treat such a user 778 or group string as representing the same user as would be represented 779 by an NFSv3 uid or gid having the corresponding numeric value. Note 780 that if using Kerberos for security, the expectation is that these 781 values will be a name@domain string. 783 ffds_efficiency describes the metadata server's evaluation as to the 784 effectiveness of each mirror. Note that this is per layout and not 785 per device as the metric may change due to perceived load, 786 availability to the metadata server, etc. Higher values denote 787 higher perceived utility. The way the client can select the best 788 mirror to access is discussed in Section 8.1. 790 ffl_flags is a bitmap that allows the metadata server to inform the 791 client of particular conditions that may result from the more or less 792 tight coupling of the storage devices. 794 FF_FLAGS_NO_LAYOUTCOMMIT: can be set to indicate that the client is 795 not required to send LAYOUTCOMMIT to the metadata server. 797 FF_FLAGS_NO_IO_THRU_MDS : can be set to indicate that the client 798 SHOULD not send IO operations to the metadata server. I.e., even 799 if a storage device is partitioned from the client, the client 800 SHOULD not try to proxy the IO through the metadata server. 802 FF_FLAGS_NO_READ_IO: can be set to indicate that the client SHOULD 803 not send READ requests with the layouts of iomode 804 LAYOUTIOMODE4_RW. Instead, it should request a layout of iomode 805 LAYOUTIOMODE4_READ from the metadata server. 807 5.1.1. Error codes from LAYOUTGET 809 [RFC5661] provides little guidance as to how the client is to proceed 810 with a LAYOUTEGT which returns an error of either 811 NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, and NFS4ERR_DELAY. 813 NFS4ERR_LAYOUTUNAVAILABLE: there is no layout available and the IO 814 is to go to the metadata server. Note that it is possible to have 815 had a layout before a recall and not after. 817 NFS4ERR_LAYOUTTRYLATER: there is some issue preventing the layout 818 from being granted. If the client already has an appropriate 819 layout, it SHOULD continue with IO to the storage devices. 821 NFS4ERR_DELAY: there is some issue preventing the layout from being 822 granted. If the client already has an appropriate layout, it 823 SHOULD not continue with IO to the storage devices. 825 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS 827 If the client does not ask for a layout for a file, then the IO will 828 go through the metadata server. Thus, even if the metadata server 829 sets the FF_FLAGS_NO_IO_THRU_MDS flag, it can recall the layout and 830 either not set the flag on the new layout or not provide a layout. 831 When a client encounters an error with a storage device, it typically 832 returns the layout to the metadata server and requests a new layout. 833 The client's IO would then proceed according to the status codes as 834 outlined in Section 5.1.1. 836 5.2. Interactions Between Devices and Layouts 838 In [RFC5661], the file layout type is defined such that the 839 relationship between multipathing and filehandles can result in 840 either 0, 1, or N filehandles (see Section 13.3). Some rationals for 841 this are clustered servers which share the same filehandle or 842 allowing for multiple read-only copies of the file on the same 843 storage device. In the flexible file layout type, while there is an 844 array of filehandles, they are independent of the multipathing being 845 used. If the metadata server wants to provide multiple read-only 846 copies of the same file on the same storage device, then it should 847 provide multiple ff_device_addr4, each as a mirror. The client can 848 then determine that since the ffds_fh_vers are different, then there 849 are multiple copies of the file for the current layout segment 850 available. 852 5.3. Handling Version Errors 854 When the metadata server provides the ffda_versions array in the 855 ff_device_addr4 (see Section 4.1), the client is able to determine if 856 it can not access a storage device with any of the supplied 857 combinations of ffdv_version, ffdv_minorversion, and 858 ffdv_tightly_coupled. However, due to the limitations of reporting 859 errors in GETDEVICEINFO (see Section 18.40 in [RFC5661], the client 860 is not able to specify which specific device it can not communicate 861 with over one of the provided ffdv_version and ffdv_minorversion 862 combinations. Using ff_ioerr4 (see Section 9.1.1 inside either the 863 LAYOUTRETURN (see Section 18.44 of [RFC5661]) or the LAYOUTERROR (see 864 Section 15.6 of [RFC7862] and Section 10 of this document), the 865 client can isolate the problematic storage device. 867 The error code to return for LAYOUTRETURN and/or LAYOUTERROR is 868 NFS4ERR_MINOR_VERS_MISMATCH. It does not matter whether the mismatch 869 is a major version (e.g., client can use NFSv3 but not NFSv4) or 870 minor version (e.g., client can use NFSv4.1 but not NFSv4.2), the 871 error indicates that for all the supplied combinations for 872 ffdv_version and ffdv_minorversion, the client can not communicate 873 with the storage device. The client can retry the GETDEVICEINFO to 874 see if the metadata server can provide a different combination or it 875 can fall back to doing the I/O through the metadata server. 877 6. Striping via Sparse Mapping 879 While other layout types support both dense and sparse mapping of 880 logical offsets to physical offsets within a file (see for example 881 Section 13.4 of [RFC5661]), the flexible file layout type only 882 supports a sparse mapping. 884 With sparse mappings, the logical offset within a file (L) is also 885 the physical offset on the storage device. As detailed in 886 Section 13.4.4 of [RFC5661], this results in holes across each 887 storage device which does not contain the current stripe index. 889 L: logical offset into the file 891 W: stripe width 892 W = number of elements in ffm_data_servers 894 S: number of bytes in a stripe 895 S = W * ffl_stripe_unit 897 N: stripe number 898 N = L / S 900 7. Recovering from Client I/O Errors 902 The pNFS client may encounter errors when directly accessing the 903 storage devices. However, it is the responsibility of the metadata 904 server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES 905 layout type is used, the client MUST report the I/O errors to the 906 server at LAYOUTRETURN time using the ff_ioerr4 structure (see 907 Section 9.1.1). 909 The metadata server analyzes the error and determines the required 910 recovery operations such as recovering media failures or 911 reconstructing missing data files. 913 The metadata server SHOULD recall any outstanding layouts to allow it 914 exclusive write access to the stripes being recovered and to prevent 915 other clients from hitting the same error condition. In these cases, 916 the server MUST complete recovery before handing out any new layouts 917 to the affected byte ranges. 919 Although it MAY be acceptable for the client to propagate a 920 corresponding error to the application that initiated the I/O 921 operation and drop any unwritten data, the client SHOULD attempt to 922 retry the original I/O operation by requesting a new layout using 923 LAYOUTGET and retry the I/O operation(s) using the new layout, or the 924 client MAY just retry the I/O operation(s) using regular NFS READ or 925 WRITE operations via the metadata server. The client SHOULD attempt 926 to retrieve a new layout and retry the I/O operation using the 927 storage device first and only if the error persists, retry the I/O 928 operation via the metadata server. 930 8. Mirroring 932 The flexible file layout type has a simple model in place for the 933 mirroring of the file data constrained by a layout segment. There is 934 no assumption that each copy of the mirror is stored identically on 935 the storage devices, i.e., one device might employ compression or 936 deduplication on the data. However, the over the wire transfer of 937 the file contents MUST appear identical. Note, this is a construct 938 of the selected XDR representation that each mirrored copy of the 939 layout segment has the same striping pattern (see Figure 1). 941 The metadata server is responsible for determining the number of 942 mirrored copies and the location of each mirror. While the client 943 may provide a hint to how many copies it wants (see Section 12), the 944 metadata server can ignore that hint and in any event, the client has 945 no means to dictate neither the storage device (which also means the 946 coupling and/or protocol levels to access the layout segments) nor 947 the location of said storage device. 949 The updating of mirrored layout segments is done via client-side 950 mirroring. With this approach, the client is responsible for making 951 sure modifications get to all copies of the layout segments it is 952 informed of via the layout. If a layout segment is being resilvered 953 to a storage device, that mirrored copy will not be in the layout. 954 Thus the metadata server MUST update that copy until the client is 955 presented it in a layout. Also, if the client is writing to the 956 layout segments via the metadata server, e.g., using an earlier 957 version of the protocol, then the metadata server MUST update all 958 copies of the mirror. As seen in Section 8.3, during the 959 resilvering, the layout is recalled, and the client has to make 960 modifications via the metadata server. 962 8.1. Selecting a Mirror 964 When the metadata server grants a layout to a client, it MAY let the 965 client know how fast it expects each mirror to be once the request 966 arrives at the storage devices via the ffds_efficiency member. While 967 the algorithms to calculate that value are left to the metadata 968 server implementations, factors that could contribute to that 969 calculation include speed of the storage device, physical memory 970 available to the device, operating system version, current load, etc. 972 However, what should not be involved in that calculation is a 973 perceived network distance between the client and the storage device. 974 The client is better situated for making that determination based on 975 past interaction with the storage device over the different available 976 network interfaces between the two. I.e., the metadata server might 977 not know about a transient outage between the client and storage 978 device because it has no presence on the given subnet. 980 As such, it is the client which decides which mirror to access for 981 reading the file. The requirements for writing to a mirrored layout 982 segments are presented below. 984 8.2. Writing to Mirrors 986 The client is responsible for updating all mirrored copies of the 987 layout segments that it is given in the layout. A single failed 988 update is suffcient to fail the entire operation. I.e., if all but 989 one copy is updated successfully and the last one provides an error, 990 then the client needs to return the layout to the metadata server 991 with an error indicating that the update failed to that storage 992 device. If the client is updating the mirrors serially, then it 993 SHOULD stop at the first error encountered and report that to the 994 metadata server. If the client is updating the mirrors in parallel, 995 then it SHOULD wait until all storage devices respond such that it 996 can report all errors encountered during the update. 998 The metadata server is then responsible for determining if it wants 999 to remove the errant mirror from the layout, if the mirror has 1000 recovered from some transient error, etc. When the client tries to 1001 get a new layout, the metadata server informs it of the decision by 1002 the contents of the layout. The client MUST NOT make any assumptions 1003 that the contents of the previous layout will match those of the new 1004 one. If it has updates that were not committed, it MUST resend those 1005 updates to all mirrors. 1007 8.3. Metadata Server Resilvering of the File 1009 The metadata server may elect to create a new mirror of the layout 1010 segments at any time. This might be to resilver a copy on a storage 1011 device which was down for servicing, to provide a copy of the layout 1012 segments on storage with different storage performance 1013 characteristics, etc. As the client will not be aware of the new 1014 mirror and the metadata server will not be aware of updates that the 1015 client is making to the layout segments, the metadata server MUST 1016 recall the writable layout segment(s) that it is resilvering. If the 1017 client issues a LAYOUTGET for a writable layout segment which is in 1018 the process of being resilvered, then the metadata server MUST deny 1019 that request with a NFS4ERR_LAYOUTTRYLATER. The client can then 1020 perform the I/O through the metadata server. 1022 9. Flexible Files Layout Type Return 1024 layoutreturn_file4 is used in the LAYOUTRETURN operation to convey 1025 layout-type specific information to the server. It is defined in 1026 [RFC5661] as follows: 1028 1030 struct layoutreturn_file4 { 1031 offset4 lrf_offset; 1032 length4 lrf_length; 1033 stateid4 lrf_stateid; 1034 /* layouttype4 specific data */ 1035 opaque lrf_body<>; 1036 }; 1038 union layoutreturn4 switch(layoutreturn_type4 lr_returntype) { 1039 case LAYOUTRETURN4_FILE: 1040 layoutreturn_file4 lr_layout; 1041 default: 1042 void; 1043 }; 1044 struct LAYOUTRETURN4args { 1045 /* CURRENT_FH: file */ 1046 bool lora_reclaim; 1047 layoutreturn_stateid lora_recallstateid; 1048 layouttype4 lora_layout_type; 1049 layoutiomode4 lora_iomode; 1050 layoutreturn4 lora_layoutreturn; 1051 }; 1053 1055 If the lora_layout_type layout type is LAYOUT4_FLEX_FILES, then the 1056 lrf_body opaque value is defined by ff_layoutreturn4 (See 1057 Section 9.3). It allows the client to report I/O error information 1058 or layout usage statistics back to the metadata server as defined 1059 below. 1061 9.1. I/O Error Reporting 1063 9.1.1. ff_ioerr4 1065 1067 /// struct ff_ioerr4 { 1068 /// offset4 ffie_offset; 1069 /// length4 ffie_length; 1070 /// stateid4 ffie_stateid; 1071 /// device_error4 ffie_errors<>; 1072 /// }; 1073 /// 1075 1077 Recall that [RFC7862] defines device_error4 as: 1079 1081 struct device_error4 { 1082 deviceid4 de_deviceid; 1083 nfsstat4 de_status; 1084 nfs_opnum4 de_opnum; 1085 }; 1087 1089 The ff_ioerr4 structure is used to return error indications for data 1090 files that generated errors during data transfers. These are hints 1091 to the metadata server that there are problems with that file. For 1092 each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length 1093 represent the storage device and byte range within the file in which 1094 the error occurred; ffie_errors represents the operation and type of 1095 error. The use of device_error4 is described in Section 15.6 of 1096 [RFC7862]. 1098 Even though the storage device might be accessed via NFSv3 and 1099 reports back NFSv3 errors to the client, the client is responsible 1100 for mapping these to appropriate NFSv4 status codes as de_status. 1101 Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 1102 operations. 1104 9.2. Layout Usage Statistics 1106 9.2.1. ff_io_latency4 1108 1110 /// struct ff_io_latency4 { 1111 /// uint64_t ffil_ops_requested; 1112 /// uint64_t ffil_bytes_requested; 1113 /// uint64_t ffil_ops_completed; 1114 /// uint64_t ffil_bytes_completed; 1115 /// uint64_t ffil_bytes_not_delivered; 1116 /// nfstime4 ffil_total_busy_time; 1117 /// nfstime4 ffil_aggregate_completion_time; 1118 /// }; 1119 /// 1121 1123 Both operation counts and bytes transferred are kept in the 1124 ff_io_latency4. READ operations are used for read latencies. Both 1125 WRITE and COMMIT operations are used for write latencies. 1126 "Requested" counters track what the client is attempting to do and 1127 "completed" counters track what was done. Note that there is no 1128 requirement that the client only report completed results that have 1129 matching requested results from the reported period. 1131 ffil_bytes_not_delivered is used to track the aggregate number of 1132 bytes requested by not fulfilled due to error conditions. 1133 ffil_total_busy_time is the aggregate time spent with outstanding RPC 1134 calls, ffil_aggregate_completion_time is the sum of all latencies for 1135 completed RPC calls. 1137 Note that LAYOUTSTATS are cumulative, i.e., not reset each time the 1138 operation is sent. If two LAYOUTSTATS ops for the same file, layout 1139 stateid, and originating from the same NFS client are processed at 1140 the same time by the metadata server, then the one containing the 1141 larger values contains the most recent time series data. 1143 9.2.2. ff_layoutupdate4 1145 1147 /// struct ff_layoutupdate4 { 1148 /// netaddr4 ffl_addr; 1149 /// nfs_fh4 ffl_fhandle; 1150 /// ff_io_latency4 ffl_read; 1151 /// ff_io_latency4 ffl_write; 1152 /// nfstime4 ffl_duration; 1153 /// bool ffl_local; 1154 /// }; 1155 /// 1157 1159 ffl_addr differentiates which network address the client connected to 1160 on the storage device. In the case of multipathing, ffl_fhandle 1161 indicates which read-only copy was selected. ffl_read and ffl_write 1162 convey the latencies respectively for both read and write operations. 1163 ffl_duration is used to indicate the time period over which the 1164 statistics were collected. ffl_local if true indicates that the I/O 1165 was serviced by the client's cache. This flag allows the client to 1166 inform the metadata server about "hot" access to a file it would not 1167 normally be allowed to report on. 1169 9.2.3. ff_iostats4 1171 1173 /// struct ff_iostats4 { 1174 /// offset4 ffis_offset; 1175 /// length4 ffis_length; 1176 /// stateid4 ffis_stateid; 1177 /// io_info4 ffis_read; 1178 /// io_info4 ffis_write; 1179 /// deviceid4 ffis_deviceid; 1180 /// ff_layoutupdate4 ffis_layoutupdate; 1181 /// }; 1182 /// 1184 1186 Recall that [RFC7862] defines io_info4 as: 1188 1190 struct io_info4 { 1191 uint64_t ii_count; 1192 uint64_t ii_bytes; 1193 }; 1195 1197 With pNFS, the data transfers are performed directly between the pNFS 1198 client and the storage devices. Therefore, the metadata server has 1199 no visibility to the I/O stream and cannot use any statistical 1200 information about client I/O to optimize data storage location. 1201 ff_iostats4 MAY be used by the client to report I/O statistics back 1202 to the metadata server upon returning the layout. Since it is 1203 infeasible for the client to report every I/O that used the layout, 1204 the client MAY identify "hot" byte ranges for which to report I/O 1205 statistics. The definition and/or configuration mechanism of what is 1206 considered "hot" and the size of the reported byte range is out of 1207 the scope of this document. It is suggested for client 1208 implementation to provide reasonable default values and an optional 1209 run-time management interface to control these parameters. For 1210 example, a client can define the default byte range resolution to be 1211 1 MB in size and the thresholds for reporting to be 1 MB/second or 10 1212 I/O operations per second. For each byte range, ffis_offset and 1213 ffis_length represent the starting offset of the range and the range 1214 length in bytes. ffis_read.ii_count, ffis_read.ii_bytes, 1215 ffis_write.ii_count, and ffis_write.ii_bytes represent, respectively, 1216 the number of contiguous read and write I/Os and the respective 1217 aggregate number of bytes transferred within the reported byte range. 1219 The combination of ffis_deviceid and ffl_addr uniquely identify both 1220 the storage path and the network route to it. Finally, the 1221 ffl_fhandle allows the metadata server to differentiate between 1222 multiple read-only copies of the file on the same storage device. 1224 9.3. ff_layoutreturn4 1226 1228 /// struct ff_layoutreturn4 { 1229 /// ff_ioerr4 fflr_ioerr_report<>; 1230 /// ff_iostats4 fflr_iostats_report<>; 1231 /// }; 1232 /// 1234 1235 When data file I/O operations fail, fflr_ioerr_report<> is used to 1236 report these errors to the metadata server as an array of elements of 1237 type ff_ioerr4. Each element in the array represents an error that 1238 occurred on the data file identified by ffie_errors.de_deviceid. If 1239 no errors are to be reported, the size of the fflr_ioerr_report<> 1240 array is set to zero. The client MAY also use fflr_iostats_report<> 1241 to report a list of I/O statistics as an array of elements of type 1242 ff_iostats4. Each element in the array represents statistics for a 1243 particular byte range. Byte ranges are not guaranteed to be disjoint 1244 and MAY repeat or intersect. 1246 10. Flexible Files Layout Type LAYOUTERROR 1248 If the client is using NFSv4.2 to communicate with the metadata 1249 server, then instead of waiting for a LAYOUTRETURN to send error 1250 information to the metadata server (see Section 9.1), it MAY use 1251 LAYOUTERROR (see Section 15.6 of [RFC7862]) to communicate that 1252 information. For the flexible files layout type, this means that 1253 LAYOUTERROR4args is treated the same as ff_ioerr4. 1255 11. Flexible Files Layout Type LAYOUTSTATS 1257 If the client is using NFSv4.2 to communicate with the metadata 1258 server, then instead of waiting for a LAYOUTRETURN to send I/O 1259 statistics to the metadata server (see Section 9.2), it MAY use 1260 LAYOUTSTATS (see Section 15.7 of [RFC7862]) to communicate that 1261 information. For the flexible files layout type, this means that 1262 LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same 1263 contents as in ffis_layoutupdate. 1265 12. Flexible File Layout Type Creation Hint 1267 The layouthint4 type is defined in the [RFC5661] as follows: 1269 1271 struct layouthint4 { 1272 layouttype4 loh_type; 1273 opaque loh_body<>; 1274 }; 1276 1278 The layouthint4 structure is used by the client to pass a hint about 1279 the type of layout it would like created for a particular file. If 1280 the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body 1281 opaque value is defined by the ff_layouthint4 type. 1283 12.1. ff_layouthint4 1285 1287 /// union ff_mirrors_hint switch (bool ffmc_valid) { 1288 /// case TRUE: 1289 /// uint32_t ffmc_mirrors; 1290 /// case FALSE: 1291 /// void; 1292 /// }; 1293 /// 1295 /// struct ff_layouthint4 { 1296 /// ff_mirrors_hint fflh_mirrors_hint; 1297 /// }; 1298 /// 1300 1302 This type conveys hints for the desired data map. All parameters are 1303 optional so the client can give values for only the parameter it 1304 cares about. 1306 13. Recalling a Layout 1308 While Section 12.5.5 of [RFC5661] discusses layout type independent 1309 reasons for recalling a layout, the flexible file layout type 1310 metadata server should recall outstanding layouts in the following 1311 cases: 1313 o When the file's security policy changes, i.e., Access Control 1314 Lists (ACLs) or permission mode bits are set. 1316 o When the file's layout changes, rendering outstanding layouts 1317 invalid. 1319 o When there are sharing conflicts. 1321 o When a file is being resilvered, either due to being repaired 1322 after a write error or to load balance. 1324 13.1. CB_RECALL_ANY 1326 The metadata server can use the CB_RECALL_ANY callback operation to 1327 notify the client to return some or all of its layouts. The 1328 [RFC5661] defines the following types: 1330 1331 const RCA4_TYPE_MASK_FF_LAYOUT_MIN = -2; 1332 const RCA4_TYPE_MASK_FF_LAYOUT_MAX = -1; 1333 [[RFC Editor: please insert assigned constants]] 1335 struct CB_RECALL_ANY4args { 1336 uint32_t craa_layouts_to_keep; 1337 bitmap4 craa_type_mask; 1338 }; 1340 1342 [[AI13: No, 5661 does not define these above values. The ask here is 1343 to create these and _add_ them to 5661. --TH]] 1345 Typically, CB_RECALL_ANY will be used to recall client state when the 1346 server needs to reclaim resources. The craa_type_mask bitmap 1347 specifies the type of resources that are recalled and the 1348 craa_layouts_to_keep value specifies how many of the recalled 1349 flexible file layouts the client is allowed to keep. The flexible 1350 file layout type mask flags are defined as follows: 1352 1354 /// enum ff_cb_recall_any_mask { 1355 /// FF_RCA4_TYPE_MASK_READ = -2, 1356 /// FF_RCA4_TYPE_MASK_RW = -1 1357 [[RFC Editor: please insert assigned constants]] 1358 /// }; 1359 /// 1361 1363 They represent the iomode of the recalled layouts. In response, the 1364 client SHOULD return layouts of the recalled iomode that it needs the 1365 least, keeping at most craa_layouts_to_keep Flexible File Layouts. 1367 The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return 1368 layouts of iomode LAYOUTIOMODE4_READ. Similarly, the 1369 PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts 1370 of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client 1371 is notified to return layouts of either iomode. 1373 14. Client Fencing 1375 In cases where clients are uncommunicative and their lease has 1376 expired or when clients fail to return recalled layouts within a 1377 lease period, at the least the server MAY revoke client layouts and 1378 reassign these resources to other clients (see Section 12.5.5 in 1380 [RFC5661]). To avoid data corruption, the metadata server MUST fence 1381 off the revoked clients from the respective data files as described 1382 in Section 2.2. 1384 15. Security Considerations 1386 The pNFS extension partitions the NFSv4.1+ file system protocol into 1387 two parts, the control path and the data path (storage protocol). 1388 The control path contains all the new operations described by this 1389 extension; all existing NFSv4 security mechanisms and features apply 1390 to the control path. The combination of components in a pNFS system 1391 is required to preserve the security properties of NFSv4.1+ with 1392 respect to an entity accessing data via a client, including security 1393 countermeasures to defend against threats that NFSv4.1+ provides 1394 defenses for in environments where these threats are considered 1395 significant. 1397 The metadata server enforces the file access-control policy at 1398 LAYOUTGET time. The client should use suitable authorization 1399 credentials for getting the layout for the requested iomode (READ or 1400 RW) and the server verifies the permissions and ACL for these 1401 credentials, possibly returning NFS4ERR_ACCESS if the client is not 1402 allowed the requested iomode. If the LAYOUTGET operation succeeds 1403 the client receives, as part of the layout, a set of credentials 1404 allowing it I/O access to the specified data files corresponding to 1405 the requested iomode. When the client acts on I/O operations on 1406 behalf of its local users, it MUST authenticate and authorize the 1407 user by issuing respective OPEN and ACCESS calls to the metadata 1408 server, similar to having NFSv4 data delegations. If access is 1409 allowed, the client uses the corresponding (READ or RW) credentials 1410 to perform the I/O operations at the data file's storage devices. 1411 When the metadata server receives a request to change a file's 1412 permissions or ACL, it SHOULD recall all layouts for that file and it 1413 MUST fence off the clients holding outstanding layouts for the 1414 respective file by implicitly invalidating the outstanding 1415 credentials on all data files comprising before committing to the new 1416 permissions and ACL. Doing this will ensure that clients re- 1417 authorize their layouts according to the modified permissions and ACL 1418 by requesting new layouts. Recalling the layouts in this case is 1419 courtesy of the server intended to prevent clients from getting an 1420 error on I/Os done after the client was fenced off. 1422 15.1. Kerberized File Access 1423 15.1.1. Loosely Coupled 1425 Under this coupling model, the principal used to authenticate the 1426 metadata file is different than that used to authenticate the data 1427 file. I.e., the synthetic principals generated to control access to 1428 the data file could prove to be difficult to manage. 1430 While RPCSEC_GSS version 3 (RPCSEC_GSSv3) [rpcsec_gssv3] could be 1431 used to authorize the client to the storage device on behalf of the 1432 metadata server, such a requirement exceeds the loose coupling model. 1433 I.e., each of the metadata server, storage device, and client would 1434 have to implement RPCSEC_GSSv3. 1436 In all, while either an elaborate schema could be used to 1437 automatically authenticate principals or RPCSEC_GSSv3 aware clients, 1438 metadata server, and storage devices could be deployed, if more 1439 secure authentication is desired, tight coupling should be considered 1440 as described in the next section. 1442 15.1.2. Tightly Coupled 1444 With tight coupling, the principal used to access the metadata file 1445 is exactly the same as used to access the data file. Thus there are 1446 no security issues related to using Kerberos with a tightly coupled 1447 system. 1449 16. IANA Considerations 1451 As described in [RFC5661], new layout type numbers have been assigned 1452 by IANA. This document defines the protocol associated with the 1453 existing layout type number, LAYOUT4_FLEX_FILES. 1455 17. References 1457 17.1. Normative References 1459 [LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents", 1460 November 2008, . 1463 [RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813, 1464 June 1995. 1466 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1467 Requirement Levels", BCP 14, RFC 2119, March 1997. 1469 [RFC4506] Eisler, M., "XDR: External Data Representation Standard", 1470 STD 67, RFC 4506, May 2006. 1472 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 1473 Specification Version 2", RFC 5531, May 2009. 1475 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1476 "Network File System (NFS) Version 4 Minor Version 1 1477 Protocol", RFC 5661, January 2010. 1479 [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1480 "Network File System (NFS) Version 4 Minor Version 1 1481 External Data Representation Standard (XDR) Description", 1482 RFC 5662, January 2010. 1484 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 1485 version 4 Protocol", RFC 7530, March 2015. 1487 [RFC7862] Haynes, T., "NFS Version 4 Minor Version 2", RFC 7862, May 1488 2016. 1490 [pNFSLayouts] 1491 Haynes, T., "Requirements for pNFS Layout Types", draft- 1492 ietf-nfsv4-layout-types-04 (Work In Progress), January 1493 2016. 1495 17.2. Informative References 1497 [RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol 1498 (LDAP): Schema for User Applications", RFC 4519, DOI 1499 10.17487/RFC4519, June 2006, 1500 . 1502 [rpcsec_gssv3] 1503 Adamson, W. and N. Williams, "Remote Procedure Call (RPC) 1504 Security Version 3", November 2014. 1506 Appendix A. Acknowledgments 1508 Those who provided miscellaneous comments to early drafts of this 1509 document include: Matt W. Benjamin, Adam Emerson, J. Bruce Fields, 1510 and Lev Solomonov. 1512 Those who provided miscellaneous comments to the final drafts of this 1513 document include: Anand Ganesh, Robert Wipfel, Gobikrishnan 1514 Sundharraj, and Trond Myklebust. 1516 Idan Kedar caught a nasty bug in the interaction of client side 1517 mirroring and the minor versioning of devices. 1519 Dave Noveck provided a comprehensive review of the document during 1520 the working group last call. 1522 Olga Kornievskaia lead the charge against the use of a credential 1523 versus a principal in the fencing approach. Andy Adamson and 1524 Benjamin Kaduk helped to sharpen the focus. 1526 Tigran Mkrtchyan provided the use case for not allowing the client to 1527 proxy the IO through the data server. 1529 Appendix B. RFC Editor Notes 1531 [RFC Editor: please remove this section prior to publishing this 1532 document as an RFC] 1534 [RFC Editor: prior to publishing this document as an RFC, please 1535 replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the 1536 RFC number of this document] 1538 Authors' Addresses 1540 Benny Halevy 1542 Email: bhalevy@gmail.com 1544 Thomas Haynes 1545 Primary Data, Inc. 1546 4300 El Camino Real Ste 100 1547 Los Altos, CA 94022 1548 USA 1550 Phone: +1 408 215 1519 1551 Email: thomas.haynes@primarydata.com