idnits 2.17.1 draft-ietf-nfsv4-flex-files-13.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 353 has weird spacing: '... loghyr staff...' == Line 1295 has weird spacing: '...stateid lor...' == Line 1561 has weird spacing: '...rs_hint ffl...' -- The document date (August 07, 2017) is 2451 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. 'LEGAL' ** Downref: Normative reference to an Informational RFC: RFC 1813 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 B. Halevy 3 Internet-Draft 4 Intended status: Standards Track T. Haynes 5 Expires: February 8, 2018 Primary Data 6 August 07, 2017 8 Parallel NFS (pNFS) Flexible File Layout 9 draft-ietf-nfsv4-flex-files-13.txt 11 Abstract 13 The Parallel Network File System (pNFS) allows a separation between 14 the metadata (onto a metadata server) and data (onto a storage 15 device) for a file. The flexible file layout type is defined in this 16 document as an extension to pNFS which allows the use of storage 17 devices in a fashion such that they require only a quite limited 18 degree of interaction with the metadata server, using already 19 existing protocols. Client side mirroring is also added to provide 20 replication of files. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at http://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on February 8, 2018. 39 Copyright Notice 41 Copyright (c) 2017 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 57 1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3 58 1.2. Requirements Language . . . . . . . . . . . . . . . . . . 5 59 2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6 60 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6 61 2.2. Fencing Clients from the Storage Device . . . . . . . . . 6 62 2.2.1. Implementation Notes for Synthetic uids/gids . . . . 7 63 2.2.2. Example of using Synthetic uids/gids . . . . . . . . 8 64 2.3. State and Locking Models . . . . . . . . . . . . . . . . 9 65 2.3.1. Loosely Coupled Locking Model . . . . . . . . . . . . 9 66 2.3.2. Tightly Coupled Locking Model . . . . . . . . . . . . 10 67 3. XDR Description of the Flexible File Layout Type . . . . . . 12 68 3.1. Code Components Licensing Notice . . . . . . . . . . . . 13 69 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 14 70 4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 14 71 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 16 72 5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 17 73 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 17 74 5.1.1. Error Codes from LAYOUTGET . . . . . . . . . . . . . 21 75 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS . . 21 76 5.2. Interactions Between Devices and Layouts . . . . . . . . 22 77 5.3. Handling Version Errors . . . . . . . . . . . . . . . . . 22 78 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 23 79 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 23 80 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 24 81 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 24 82 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 25 83 8.2.1. Single Storage Device Updates Mirrors . . . . . . . . 25 84 8.2.2. Single Storage Device Updates Mirrors . . . . . . . . 25 85 8.2.3. Handling Write Errors . . . . . . . . . . . . . . . . 25 86 8.2.4. Handling Write COMMITs . . . . . . . . . . . . . . . 26 87 8.3. Metadata Server Resilvering of the File . . . . . . . . . 27 88 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 27 89 9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 28 90 9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 28 91 9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 29 92 9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 29 93 9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 30 94 9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 31 95 9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 32 96 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 32 97 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 33 98 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 33 99 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 33 100 13. Recalling a Layout . . . . . . . . . . . . . . . . . . . . . 34 101 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 34 102 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 35 103 15. Security Considerations . . . . . . . . . . . . . . . . . . . 35 104 15.1. RPCSEC_GSS and Security Services . . . . . . . . . . . . 36 105 15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 36 106 15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 36 107 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 37 108 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 38 109 17.1. Normative References . . . . . . . . . . . . . . . . . . 38 110 17.2. Informative References . . . . . . . . . . . . . . . . . 39 111 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 39 112 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 39 113 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40 115 1. Introduction 117 In the parallel Network File System (pNFS), the metadata server 118 returns layout type structures that describe where file data is 119 located. There are different layout types for different storage 120 systems and methods of arranging data on storage devices. This 121 document defines the flexible file layout type used with file-based 122 data servers that are accessed using the Network File System (NFS) 123 protocols: NFSv3 [RFC1813], NFSv4.0 [RFC7530], NFSv4.1 [RFC5661], and 124 NFSv4.2 [RFC7862]. 126 To provide a global state model equivalent to that of the files 127 layout type, a back-end control protocol MAY be implemented between 128 the metadata server and NFSv4.1+ storage devices. It is out of scope 129 for this document to specify such a protocol, yet the requirements 130 for the protocol are specified in [RFC5661] and clarified in 131 [pNFSLayouts]. 133 1.1. Definitions 135 control communication requirements: defines for a layout type the 136 details regarding information on layouts, stateids, file metadata, 137 and file data which must be communicated between the metadata 138 server and the storage devices. 140 control protocol: defines a particular mechanism that an 141 implementation of a layout type would use to meet the control 142 communication requirement for that layout type. This need not be 143 a protocol as normally understood. In some cases the same 144 protocol may be used as a control protocol and data access 145 protocol. 147 client-side mirroring: is when the client and not the server is 148 responsible for updating all of the mirrored copies of a layout 149 segment. 151 data file: is that part of the file system object which contains the 152 content. 154 data server (DS): is another term for storage device. 156 fencing: is when the metadata server prevents the storage devices 157 from processing I/O from a specific client to a specific file. 159 file layout type: is a layout type in which the storage devices are 160 accessed via the NFS protocol (see Section 13 of [RFC5661]). 162 layout: informs a client of which storage devices it needs to 163 communicate with (and over which protocol) to perform I/O on a 164 file. The layout might also provide some hints about how the 165 storage is physically organized. 167 layout iomode: describes whether the layout granted to the client is 168 for read or read/write I/O. 170 layout segment: describes a sub-division of a layout. That sub- 171 division might be by the iomode (see Sections 3.3.20 and 12.2.9 of 172 [RFC5661]), a striping pattern (see Section 13.3 of [RFC5661]), or 173 requested byte range. 175 layout stateid: is a 128-bit quantity returned by a server that 176 uniquely defines the layout state provided by the server for a 177 specific layout that describes a layout type and file (see 178 Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 of 179 [RFC5661] describes the difference between a layout stateid and a 180 normal stateid. 182 layout type: describes both the storage protocol used to access the 183 data and the aggregation scheme used to lay out the file data on 184 the underlying storage devices. 186 loose coupling: is when the metadata server and the storage devices 187 do not have a control protocol present. 189 metadata file: is that part of the file system object which 190 describes the object and not the content. E.g., it could be the 191 time since last modification, access, etc. 193 metadata server (MDS): is the pNFS server which provides metadata 194 information for a file system object. It also is responsible for 195 generating layouts for file system objects. Note that the MDS is 196 responsible for directory-based operations. 198 mirror: is a copy of a layout segment. Note that if one copy of the 199 mirror is updated, then all copies must be updated. 201 recalling a layout: is when the metadata server uses a back channel 202 to inform the client that the layout is to be returned in a 203 graceful manner. Note that the client has the opportunity to 204 flush any writes, etc., before replying to the metadata server. 206 revoking a layout: is when the metadata server invalidates the 207 layout such that neither the metadata server nor any storage 208 device will accept any access from the client with that layout. 210 resilvering: is the act of rebuilding a mirrored copy of a layout 211 segment from a known good copy of the layout segment. Note that 212 this can also be done to create a new mirrored copy of the layout 213 segment. 215 rsize: is the data transfer buffer size used for reads. 217 stateid: is a 128-bit quantity returned by a server that uniquely 218 defines the open and locking states provided by the server for a 219 specific open-owner or lock-owner/open-owner pair for a specific 220 file and type of lock. 222 storage device: designates the target to which clients may direct I/ 223 O requests when they hold an appropriate layout. See Section 2.1 224 of [pNFSLayouts] for further discussion of the difference between 225 a data store and a storage device. 227 tight coupling: is when the metadata server and the storage devices 228 do have a control protocol present. 230 wsize: is the data transfer buffer size used for writes. 232 1.2. Requirements Language 234 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 235 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 236 document are to be interpreted as described in [RFC2119]. 238 2. Coupling of Storage Devices 240 The coupling of the metadata server with the storage devices can be 241 either tight or loose. In a tight coupling, there is a control 242 protocol present to manage security, LAYOUTCOMMITs, etc. With a 243 loose coupling, the only control protocol might be a version of NFS. 244 As such, semantics for managing security, state, and locking models 245 MUST be defined. 247 2.1. LAYOUTCOMMIT 249 Regardless of the coupling model, the metadata server has the 250 responsibility, upon receiving a LAYOUTCOMMIT (see Section 18.42 of 251 [RFC5661]), of ensuring that the semantics of pNFS are respected (see 252 Section 12.5.4 of [RFC5661]). These do include a requirement that 253 data written to data storage device be stable before the occurrence 254 of the LAYOUTCOMMIT. 256 It is the responsibility of the client to make sure the data file is 257 stable before the metadata server begins to query the storage devices 258 about the changes to the file. If any WRITE to a storage device did 259 not result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the 260 metadata server MUST be preceded by a COMMIT to the storage devices 261 written to. Note that if the client has not done a COMMIT to the 262 storage device, then the LAYOUTCOMMIT might not be synchronized to 263 the last WRITE operation to the storage device. 265 2.2. Fencing Clients from the Storage Device 267 With loosely coupled storage devices, the metadata server uses 268 synthetic uids and gids for the data file, where the uid owner of the 269 data file is allowed read/write access and the gid owner is allowed 270 read only access. As part of the layout (see ffds_user and 271 ffds_group in Section 5.1), the client is provided with the user and 272 group to be used in the Remote Procedure Call (RPC) [RFC5531] 273 credentials needed to access the data file. Fencing off of clients 274 is achieved by the metadata server changing the synthetic uid and/or 275 gid owners of the data file on the storage device to implicitly 276 revoke the outstanding RPC credentials. A client presenting the 277 wrong credential for the desired access will get a NFS4ERR_ACCESS 278 error. 280 With this loosely coupled model, the metadata server is not able to 281 fence off a single client, it is forced to fence off all clients. 282 However, as the other clients react to the fencing, returning their 283 layouts and trying to get new ones, the metadata server can hand out 284 a new uid and gid to allow access. 286 Note: it is recommended to implement common access control methods at 287 the storage device filesystem to allow only the metadata server root 288 (super user) access to the storage device, and to set the owner of 289 all directories holding data files to the root user. This approach 290 provides a practical model to enforce access control and fence off 291 cooperative clients, but it can not protect against malicious 292 clients; hence it provides a level of security equivalent to 293 AUTH_SYS. 295 With tightly coupled storage devices, the metadata server sets the 296 user and group owners, mode bits, and ACL of the data file to be the 297 same as the metadata file. And the client must authenticate with the 298 storage device and go through the same authorization process it would 299 go through via the metadata server. In the case of tight coupling, 300 fencing is the responsibility of the control protocol and is not 301 described in detail here. However, implementations of the tight 302 coupling locking model (see Section 2.3), will need a way to prevent 303 access by certain clients to specific files by invalidating the 304 corresponding stateids on the storage device. In such a scenario, 305 the client will be given an error of NFS4ERR_BAD_STATEID. 307 The client need not know the model used between the metadata server 308 and the storage device. It need only react consistently to any 309 errors in interacting with the storage device. It should both return 310 the layout and error to the metadata server and ask for a new layout. 311 At that point, the metadata server can either hand out a new layout, 312 hand out no layout (forcing the I/O through it), or deny the client 313 further access to the file. 315 2.2.1. Implementation Notes for Synthetic uids/gids 317 The selection method for the synthetic uids and gids to be used for 318 fencing in loosely coupled storage devices is strictly an 319 implementation issue. I.e., an administrator might restrict a range 320 of such ids available to the Lightweight Directory Access Protocol 321 (LDAP) 'uid' field [RFC4519]. She might also be able to choose an id 322 that would never be used to grant access. Then when the metadata 323 server had a request to access a file, a SETATTR would be sent to the 324 storage device to set the owner and group of the data file. The user 325 and group might be selected in a round robin fashion from the range 326 of available ids. 328 Those ids would be sent back as ffds_user and ffds_group to the 329 client. And it would present them as the RPC credentials to the 330 storage device. When the client was done accessing the file and the 331 metadata server knew that no other client was accessing the file, it 332 could reset the owner and group to restrict access to the data file. 334 When the metadata server wanted to fence off a client, it would 335 change the synthetic uid and/or gid to the restricted ids. Note that 336 using a restricted id ensures that there is a change of owner and at 337 least one id available that never gets allowed access. 339 Under an AUTH_SYS security model, synthetic uids and gids of 0 SHOULD 340 be avoided. These typically either grant super access to files on a 341 storage device or are mapped to an anonymous id. In the first case, 342 even if the data file is fenced, the client might still be able to 343 access the file. In the second case, multiple ids might be mapped to 344 the anonymous ids. 346 2.2.2. Example of using Synthetic uids/gids 348 The user loghyr creates a file "ompha.c" on the metadata server and 349 it creates a corresponding data file on the storage device. 351 The metadata server entry may look like: 353 -rw-r--r-- 1 loghyr staff 1697 Dec 4 11:31 ompha.c 355 On the storage device, it may be assigned some random synthetic uid/ 356 gid to deny access: 358 -rw-r----- 1 19452 28418 1697 Dec 4 11:31 data_ompha.c 360 When the file is opened on a client, since the layout knows nothing 361 about the user (and does not care), whether loghyr or garbo opens the 362 file does not matter. The owner and group are modified and those 363 values are returned. 365 -rw-r----- 1 1066 1067 1697 Dec 4 11:31 data_ompha.c 367 The set of synthetic gids on the storage device should be selected 368 such that there is no mapping in any of the name services used by the 369 storage device. I.e., each group should have no members. 371 If the layout segment has an iomode of LAYOUTIOMODE4_READ, then the 372 metadata server should return a synthetic uid that is not set on the 373 storage device. Only the synthetic gid would be valid. 375 The client is thus solely responsible for enforcing file permissions 376 in a loosely coupled model. To allow loghyr write access, it will 377 send an RPC to the storage device with a credential of 1066:1067. To 378 allow garbo read access, it will send an RPC to the storage device 379 with a credential of 1067:1067. The value of the uid does not matter 380 as long as it is not the synthetic uid granted it when getting the 381 layout. 383 While pushing the enforcement of permission checking onto the client 384 may seem to weaken security, the client may already be responsible 385 for enforcing permissions before modifications are sent to a server. 386 With cached writes, the client is always responsible for tracking who 387 is modifying a file and making sure to not coalesce requests from 388 multiple users into one request. 390 2.3. State and Locking Models 392 The choice of locking models is governed by the following rules: 394 o Storage devices implementing the NFSv3 and NFSv4.0 protocols are 395 always treated as loosely coupled. 397 o NFSv4.1+ storage devices that do not return the 398 EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are indicating 399 that they are to be treated as loosely coupled. From the locking 400 viewpoint they are treated in the same way as NFSv4.0 storage 401 devices. 403 o NFSv4.1+ storage devices that do identify themselves with the 404 EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are considered 405 tightly coupled. They would use a back-end control protocol to 406 implement the global stateid model as described in [RFC5661]. 408 2.3.1. Loosely Coupled Locking Model 410 When locking-related operations are requested, they are primarily 411 dealt with by the metadata server, which generates the appropriate 412 stateids. When an NFSv4 version is used as the data access protocol, 413 the metadata server may make stateid-related requests of the storage 414 devices. However, it is not required to do so and the resulting 415 stateids are known only to the metadata server and the storage 416 device. 418 Given this basic structure, locking-related operations are handled as 419 follows: 421 o OPENs are dealt with by the metadata server. Stateids are 422 selected by the metadata server and associated with the client id 423 describing the client's connection to the metadata server. The 424 metadata server may need to interact with the storage device to 425 locate the file to be opened, but no locking-related functionality 426 need be used on the storage device. 428 OPEN_DOWNGRADE and CLOSE only require local execution on the 429 metadata server. 431 o Advisory byte-range locks can be implemented locally on the 432 metadata server. As in the case of OPENs, the stateids associated 433 with byte-range locks are assigned by the metadata server and only 434 used on the metadata server. 436 o Delegations are assigned by the metadata server which initiates 437 recalls when conflicting OPENs are processed. No storage device 438 involvement is required. 440 o TEST_STATEID and FREE_STATEID are processed locally on the 441 metadata server, without storage device involvement. 443 All I/O operations to the storage device are done using the anonymous 444 stateid. Thus the storage device has no information about the 445 openowner and lockowner responsible for issuing a particular I/O 446 operation. As a result: 448 o Mandatory byte-range locking cannot be supported because the 449 storage device has no way of distinguishing I/O done on behalf of 450 the lock owner from those done by others. 452 o Enforcement of share reservations is the responsibility of the 453 client. Even though I/O is done using the anonymous stateid, the 454 client must ensure that it has a valid stateid associated with the 455 openowner, that allows the I/O being done before issuing the I/O. 457 In the event that a stateid is revoked, the metadata server is 458 responsible for preventing client access, since it has no way of 459 being sure that the client is aware that the stateid in question has 460 been revoked. 462 As the client never receives a stateid generated by a storage device, 463 there is no client lease on the storage device and no prospect of 464 lease expiration, even when access is via NFSv4 protocols. Clients 465 will have leases on the metadata server. In dealing with lease 466 expiration, the metadata server may need to use fencing to prevent 467 revoked stateids from being relied upon by a client unaware of the 468 fact that they have been revoked. 470 2.3.2. Tightly Coupled Locking Model 472 When locking-related operations are requested, they are primarily 473 dealt with by the metadata server, which generates the appropriate 474 stateids. These stateids must be made known to the storage device 475 using control protocol facilities, the details of which are not 476 discussed in this document. 478 Given this basic structure, locking-related operations are handled as 479 follows: 481 o OPENs are dealt with primarily on the metadata server. Stateids 482 are selected by the metadata server and associated with the client 483 id describing the client's connection to the metadata server. The 484 metadata server needs to interact with the storage device to 485 locate the file to be opened, and to make the storage device aware 486 of the association between the metadata-server-chosen stateid and 487 the client and openowner that it represents. 489 OPEN_DOWNGRADE and CLOSE are executed initially on the metadata 490 server but the state change made must be propagated to the storage 491 device. 493 o Advisory byte-range locks can be implemented locally on the 494 metadata server. As in the case of OPENs, the stateids associated 495 with byte-range locks, are assigned by the metadata server and are 496 available for use on the metadata server. Because I/O operations 497 are allowed to present lock stateids, the metadata server needs 498 the ability to make the storage device aware of the association 499 between the metadata-server-chosen stateid and the corresponding 500 open stateid it is associated with. 502 o Mandatory byte-range locks can be supported when both the metadata 503 server and the storage devices have the appropriate support. As 504 in the case of advisory byte-range locks, these are assigned by 505 the metadata server and are available for use on the metadata 506 server. To enable mandatory lock enforcement on the storage 507 device, the metadata server needs the ability to make the storage 508 device aware of the association between the metadata-server-chosen 509 stateid and the client, openowner, and lock (i.e., lockowner, 510 byte-range, lock-type) that it represents. Because I/O operations 511 are allowed to present lock stateids, this information needs to be 512 propagated to all storage devices to which I/O might be directed 513 rather than only to storage device that contain the locked region. 515 o Delegations are assigned by the metadata server which initiates 516 recalls when conflicting OPENs are processed. Because I/O 517 operations are allowed to present delegation stateids, the 518 metadata server requires the ability to make the storage device 519 aware of the association between the metadata-server-chosen 520 stateid and the filehandle and delegation type it represents, and 521 to break such an association. 523 o TEST_STATEID is processed locally on the metadata server, without 524 storage device involvement. 526 o FREE_STATEID is processed on the metadata server but the metadata 527 server requires the ability to propagate the request to the 528 corresponding storage devices. 530 Because the client will possess and use stateids valid on the storage 531 device, there will be a client lease on the storage device and the 532 possibility of lease expiration does exist. The best approach for 533 the storage device is to retain these locks as a courtesy. However, 534 if it does not do so, control protocol facilities need to provide the 535 means to synchronize lock state between the metadata server and 536 storage device. 538 Clients will also have leases on the metadata server, which are 539 subject to expiration. In dealing with lease expiration, the 540 metadata server would be expected to use control protocol facilities 541 enabling it to invalidate revoked stateids on the storage device. In 542 the event the client is not responsive, the metadata server may need 543 to use fencing to prevent revoked stateids from being acted upon by 544 the storage device. 546 3. XDR Description of the Flexible File Layout Type 548 This document contains the external data representation (XDR) 549 [RFC4506] description of the flexible file layout type. The XDR 550 description is embedded in this document in a way that makes it 551 simple for the reader to extract into a ready-to-compile form. The 552 reader can feed this document into the following shell script to 553 produce the machine readable XDR description of the flexible file 554 layout type: 556 558 #!/bin/sh 559 grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' 561 563 That is, if the above script is stored in a file called "extract.sh", 564 and this document is in a file called "spec.txt", then the reader can 565 do: 567 sh extract.sh < spec.txt > flex_files_prot.x 569 The effect of the script is to remove leading white space from each 570 line, plus a sentinel sequence of "///". 572 The embedded XDR file header follows. Subsequent XDR descriptions, 573 with the sentinel sequence are embedded throughout the document. 575 Note that the XDR code contained in this document depends on types 576 from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs 577 types that end with a 4, such as offset4, length4, etc., as well as 578 more generic types such as uint32_t and uint64_t. 580 3.1. Code Components Licensing Notice 582 Both the XDR description and the scripts used for extracting the XDR 583 description are Code Components as described in Section 4 of "Legal 584 Provisions Relating to IETF Documents" [LEGAL]. These Code 585 Components are licensed according to the terms of that document. 587 589 /// /* 590 /// * Copyright (c) 2012 IETF Trust and the persons identified 591 /// * as authors of the code. All rights reserved. 592 /// * 593 /// * Redistribution and use in source and binary forms, with 594 /// * or without modification, are permitted provided that the 595 /// * following conditions are met: 596 /// * 597 /// * o Redistributions of source code must retain the above 598 /// * copyright notice, this list of conditions and the 599 /// * following disclaimer. 600 /// * 601 /// * o Redistributions in binary form must reproduce the above 602 /// * copyright notice, this list of conditions and the 603 /// * following disclaimer in the documentation and/or other 604 /// * materials provided with the distribution. 605 /// * 606 /// * o Neither the name of Internet Society, IETF or IETF 607 /// * Trust, nor the names of specific contributors, may be 608 /// * used to endorse or promote products derived from this 609 /// * software without specific prior written permission. 610 /// * 611 /// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS 612 /// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED 613 /// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 614 /// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 615 /// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 616 /// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 617 /// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 618 /// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 619 /// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 620 /// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 621 /// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 622 /// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 623 /// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 624 /// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 625 /// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 626 /// * 627 /// * This code was derived from RFCTBD10. 628 /// * Please reproduce this note if possible. 629 /// */ 630 /// 631 /// /* 632 /// * flex_files_prot.x 633 /// */ 634 /// 635 /// /* 636 /// * The following include statements are for example only. 637 /// * The actual XDR definition files are generated separately 638 /// * and independently and are likely to have a different name. 639 /// * %#include 640 /// * %#include 641 /// */ 642 /// 644 646 4. Device Addressing and Discovery 648 Data operations to a storage device require the client to know the 649 network address of the storage device. The NFSv4.1+ GETDEVICEINFO 650 operation (Section 18.40 of [RFC5661]) is used by the client to 651 retrieve that information. 653 4.1. ff_device_addr4 655 The ff_device_addr4 data structure is returned by the server as the 656 storage protocol specific opaque field da_addr_body in the 657 device_addr4 structure by a successful GETDEVICEINFO operation. 659 661 /// struct ff_device_versions4 { 662 /// uint32_t ffdv_version; 663 /// uint32_t ffdv_minorversion; 664 /// uint32_t ffdv_rsize; 665 /// uint32_t ffdv_wsize; 666 /// bool ffdv_tightly_coupled; 667 /// }; 668 /// 669 /// struct ff_device_addr4 { 670 /// multipath_list4 ffda_netaddrs; 671 /// ff_device_versions4 ffda_versions<>; 672 /// }; 673 /// 675 677 The ffda_netaddrs field is used to locate the storage device. It 678 MUST be set by the server to a list holding one or more of the device 679 network addresses. 681 The ffda_versions array allows the metadata server to present choices 682 as to NFS version, minor version, and coupling strength to the 683 client. The ffdv_version and ffdv_minorversion represent the NFS 684 protocol to be used to access the storage device. This layout 685 specification defines the semantics for ffdv_versions 3 and 4. If 686 ffdv_version equals 3 then the server MUST set ffdv_minorversion to 0 687 and ffdv_tightly_coupled to false. The client MUST then access the 688 storage device using the NFSv3 protocol [RFC1813]. If ffdv_version 689 equals 4 then the server MUST set ffdv_minorversion to one of the 690 NFSv4 minor version numbers and the client MUST access the storage 691 device using NFSv4 with the specified minor version. 693 Note that while the client might determine that it cannot use any of 694 the configured combinations of ffdv_version, ffdv_minorversion, and 695 ffdv_tightly_coupled, when it gets the device list from the metadata 696 server, there is no way to indicate to the metadata server as to 697 which device it is version incompatible. If however, the client 698 waits until it retrieves the layout from the metadata server, it can 699 at that time clearly identify the storage device in question (see 700 Section 5.3). 702 The ffdv_rsize and ffdv_wsize are used to communicate the maximum 703 rsize and wsize supported by the storage device. As the storage 704 device can have a different rsize or wsize than the metadata server, 705 the ffdv_rsize and ffdv_wsize allow the metadata server to 706 communicate that information on behalf of the storage device. 708 ffdv_tightly_coupled informs the client as to whether the metadata 709 server is tightly coupled with the storage devices or not. Note that 710 even if the data protocol is at least NFSv4.1, it may still be the 711 case that there is loose coupling in effect. If ffdv_tightly_coupled 712 is not set, then the client MUST commit writes to the storage devices 713 for the file before sending a LAYOUTCOMMIT to the metadata server. 714 I.e., the writes MUST be committed by the client to stable storage 715 via issuing WRITEs with stable_how == FILE_SYNC or by issuing a 716 COMMIT after WRITEs with stable_how != FILE_SYNC (see Section 3.3.7 717 of [RFC1813]). 719 4.2. Storage Device Multipathing 721 The flexible file layout type supports multipathing to multiple 722 storage device addresses. Storage device level multipathing is used 723 for bandwidth scaling via trunking and for higher availability of use 724 in the event of a storage device failure. Multipathing allows the 725 client to switch to another storage device address which may be that 726 of another storage device that is exporting the same data stripe 727 unit, without having to contact the metadata server for a new layout. 729 To support storage device multipathing, ffda_netaddrs contains an 730 array of one or more storage device network addresses. This array 731 (data type multipath_list4) represents a list of storage devices 732 (each identified by a network address), with the possibility that 733 some storage device will appear in the list multiple times. 735 The client is free to use any of the network addresses as a 736 destination to send storage device requests. If some network 737 addresses are less desirable paths to the data than others, then the 738 MDS SHOULD NOT include those network addresses in ffda_netaddrs. If 739 less desirable network addresses exist to provide failover, the 740 RECOMMENDED method to offer the addresses is to provide them in a 741 replacement device-ID-to-device-address mapping, or a replacement 742 device ID. When a client finds no response from the storage device 743 using all addresses available in ffda_netaddrs, it SHOULD send a 744 GETDEVICEINFO to attempt to replace the existing device-ID-to-device- 745 address mappings. If the MDS detects that all network paths 746 represented by ffda_netaddrs are unavailable, the MDS SHOULD send a 747 CB_NOTIFY_DEVICEID (if the client has indicated it wants device ID 748 notifications for changed device IDs) to change the device-ID-to- 749 device-address mappings to the available addresses. If the device ID 750 itself will be replaced, the MDS SHOULD recall all layouts with the 751 device ID, and thus force the client to get new layouts and device ID 752 mappings via LAYOUTGET and GETDEVICEINFO. 754 Generally, if two network addresses appear in ffda_netaddrs, they 755 will designate the same storage device. When the storage device is 756 accessed over NFSv4.1 or a higher minor version, the two storage 757 device addresses will support the implementation of client ID or 758 session trunking (the latter is RECOMMENDED) as defined in [RFC5661]. 759 The two storage device addresses will share the same server owner or 760 major ID of the server owner. It is not always necessary for the two 761 storage device addresses to designate the same storage device with 762 trunking being used. For example, the data could be read-only, and 763 the data consist of exact replicas. 765 5. Flexible File Layout Type 767 The layout4 type is defined in [RFC5662] as follows: 769 771 enum layouttype4 { 772 LAYOUT4_NFSV4_1_FILES = 1, 773 LAYOUT4_OSD2_OBJECTS = 2, 774 LAYOUT4_BLOCK_VOLUME = 3, 775 LAYOUT4_FLEX_FILES = 4 776 [[RFC Editor: please modify the LAYOUT4_FLEX_FILES 777 to be the layouttype assigned by IANA]] 778 }; 780 struct layout_content4 { 781 layouttype4 loc_type; 782 opaque loc_body<>; 783 }; 785 struct layout4 { 786 offset4 lo_offset; 787 length4 lo_length; 788 layoutiomode4 lo_iomode; 789 layout_content4 lo_content; 790 }; 792 794 This document defines structures associated with the layouttype4 795 value LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure 796 as an XDR type "opaque". The opaque layout is uninterpreted by the 797 generic pNFS client layers, but is interpreted by the flexible file 798 layout type implementation. This section defines the structure of 799 this otherwise opaque value, ff_layout4. 801 5.1. ff_layout4 803 805 /// const FF_FLAGS_NO_LAYOUTCOMMIT = 0x00000001; 806 /// const FF_FLAGS_NO_IO_THRU_MDS = 0x00000002; 807 /// const FF_FLAGS_NO_READ_IO = 0x00000004; 808 /// const FF_FLAGS_WRITE_ONE_MIRROR = 0x00000008; 810 /// typedef uint32_t ff_flags4; 811 /// 812 /// struct ff_data_server4 { 813 /// deviceid4 ffds_deviceid; 814 /// uint32_t ffds_efficiency; 815 /// stateid4 ffds_stateid; 816 /// nfs_fh4 ffds_fh_vers<>; 817 /// fattr4_owner ffds_user; 818 /// fattr4_owner_group ffds_group; 819 /// }; 820 /// 822 /// struct ff_mirror4 { 823 /// ff_data_server4 ffm_data_servers<>; 824 /// }; 825 /// 827 /// struct ff_layout4 { 828 /// length4 ffl_stripe_unit; 829 /// ff_mirror4 ffl_mirrors<>; 830 /// ff_flags4 ffl_flags; 831 /// uint32_t ffl_stats_collect_hint; 832 /// }; 833 /// 835 837 The ff_layout4 structure specifies a layout over a set of mirrored 838 copies of that portion of the data file described in the current 839 layout segment. This mirroring protects against loss of data in 840 layout segments. Note that while not explicitly shown in the above 841 XDR, each layout4 element returned in the logr_layout array of 842 LAYOUTGET4res (see Section 18.43.1 of [RFC5661]) describes a layout 843 segment. Hence each ff_layout4 also describes a layout segment. 845 It is possible that the file is concatenated from more than one 846 layout segment. Each layout segment MAY represent different striping 847 parameters, applying respectively only to the layout segment byte 848 range. 850 The ffl_stripe_unit field is the stripe unit size in use for the 851 current layout segment. The number of stripes is given inside each 852 mirror by the number of elements in ffm_data_servers. If the number 853 of stripes is one, then the value for ffl_stripe_unit MUST default to 854 zero. The only supported mapping scheme is sparse and is detailed in 855 Section 6. Note that there is an assumption here that both the 856 stripe unit size and the number of stripes is the same across all 857 mirrors. 859 The ffl_mirrors field is the array of mirrored storage devices which 860 provide the storage for the current stripe, see Figure 1. 862 The ffl_stats_collect_hint field provides a hint to the client on how 863 often the server wants it to report LAYOUTSTATS for a file. The time 864 is in seconds. 866 +-----------+ 867 | | 868 | | 869 | File | 870 | | 871 | | 872 +-----+-----+ 873 | 874 +------------+------------+ 875 | | 876 +----+-----+ +-----+----+ 877 | Mirror 1 | | Mirror 2 | 878 +----+-----+ +-----+----+ 879 | | 880 +-----------+ +-----------+ 881 |+-----------+ |+-----------+ 882 ||+-----------+ ||+-----------+ 883 +|| Storage | +|| Storage | 884 +| Devices | +| Devices | 885 +-----------+ +-----------+ 887 Figure 1 889 The ffs_mirrors field represents an array of state information for 890 each mirrored copy of the current layout segment. Each element is 891 described by a ff_mirror4 type. 893 ffds_deviceid provides the deviceid of the storage device holding the 894 data file. 896 ffds_fh_vers is an array of filehandles of the data file matching to 897 the available NFS versions on the given storage device. There MUST 898 be exactly as many elements in ffds_fh_vers as there are in 899 ffda_versions. Each element of the array corresponds to a particular 900 combination of ffdv_version, ffdv_minorversion, and 901 ffdv_tightly_coupled provided for the device. The array allows for 902 server implementations which have different filehandles for different 903 combinations of version, minor version, and coupling strength. See 904 Section 5.3 for how to handle versioning issues between the client 905 and storage devices. 907 For tight coupling, ffds_stateid provides the stateid to be used by 908 the client to access the file. For loose coupling and a NFSv4 909 storage device, the client may use an anonymous stateid to perform I/ 910 O on the storage device as there is no use for the metadata server 911 stateid (no control protocol). In such a scenario, the server MUST 912 set the ffds_stateid to be the anonymous stateid. 914 This specification of the ffds_stateid restricts both models for 915 NFSv4.x storage protocols: 917 loosely couple: the stateid has to be an anonymous stateid, 919 tightly couple: the stateid has to be a global stateid. 921 These stem from a mismatch of ffds_stateid being a singleton and 922 ffds_fh_vers being an array - each open file on the storage device 923 might need an open stateid. As there are established loosely coupled 924 implementations of this version of the protocol, it can not be fixed. 925 If an implementation needs a different stateid per file handle, then 926 this issue will require a new version of the protocol. 928 For loosely coupled storage devices, ffds_user and ffds_group provide 929 the synthetic user and group to be used in the RPC credentials that 930 the client presents to the storage device to access the data files. 931 For tightly coupled storage devices, the user and group on the 932 storage device will be the same as on the metadata server. I.e., if 933 ffdv_tightly_coupled (see Section 4.1) is set, then the client MUST 934 ignore both ffds_user and ffds_group. 936 The allowed values for both ffds_user and ffds_group are specified in 937 Section 5.9 of [RFC5661]. For NFSv3 compatibility, user and group 938 strings that consist of decimal numeric values with no leading zeros 939 can be given a special interpretation by clients and servers that 940 choose to provide such support. The receiver may treat such a user 941 or group string as representing the same user as would be represented 942 by an NFSv3 uid or gid having the corresponding numeric value. Note 943 that if using Kerberos for security, the expectation is that these 944 values will be a name@domain string. 946 ffds_efficiency describes the metadata server's evaluation as to the 947 effectiveness of each mirror. Note that this is per layout and not 948 per device as the metric may change due to perceived load, 949 availability to the metadata server, etc. Higher values denote 950 higher perceived utility. The way the client can select the best 951 mirror to access is discussed in Section 8.1. 953 ffl_flags is a bitmap that allows the metadata server to inform the 954 client of particular conditions that may result from the more or less 955 tight coupling of the storage devices. 957 FF_FLAGS_NO_LAYOUTCOMMIT: can be set to indicate that the client is 958 not required to send LAYOUTCOMMIT to the metadata server. 960 F_FLAGS_NO_IO_THRU_MDS: can be set to indicate that the client 961 should not send I/O operations to the metadata server. I.e., even 962 if the client could determine that there was a network disconnect 963 to a storage device, the client should not try to proxy the I/O 964 through the metadata server. 966 FF_FLAGS_NO_READ_IO: can be set to indicate that the client should 967 not send READ requests with the layouts of iomode 968 LAYOUTIOMODE4_RW. Instead, it should request a layout of iomode 969 LAYOUTIOMODE4_READ from the metadata server. 971 FF_FLAGS_WRITE_ONE_MIRROR: can be set to indicate that the client 972 only needs to update one of the mirrors (see Section 8.2). 974 5.1.1. Error Codes from LAYOUTGET 976 [RFC5661] provides little guidance as to how the client is to proceed 977 with a LAYOUTGET which returns an error of either 978 NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, and NFS4ERR_DELAY. 979 Within the context of this document: 981 NFS4ERR_LAYOUTUNAVAILABLE: there is no layout available and the I/O 982 is to go to the metadata server. Note that it is possible to have 983 had a layout before a recall and not after. 985 NFS4ERR_LAYOUTTRYLATER: there is some issue preventing the layout 986 from being granted. If the client already has an appropriate 987 layout, it should continue with I/O to the storage devices. 989 NFS4ERR_DELAY: there is some issue preventing the layout from being 990 granted. If the client already has an appropriate layout, it 991 should not continue with I/O to the storage devices. 993 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS 995 Even if the metadata server provides the FF_FLAGS_NO_IO_THRU_MDS, 996 flag, the client can still perform I/O to the metadata server. The 997 flag is at best a hint. The flag is indicating to the client that 998 the metadata server most likely wants to separate the metadata I/O 999 from the data I/O to increase the performance of the metadata 1000 operations. If the metadata server detects that the client is 1001 performing I/O against it despite the use of the 1002 FF_FLAGS_NO_IO_THRU_MDS flag, it can recall the layout and either not 1003 set the flag on the new layout or not provide a layout (perhaps the 1004 intent was for the server to temporarily prevent data I/O to meet 1005 some goal). The client's I/O would then proceed according to the 1006 status codes as outlined in Section 5.1.1. 1008 5.2. Interactions Between Devices and Layouts 1010 In [RFC5661], the file layout type is defined such that the 1011 relationship between multipathing and filehandles can result in 1012 either 0, 1, or N filehandles (see Section 13.3). Some rationals for 1013 this are clustered servers which share the same filehandle or 1014 allowing for multiple read-only copies of the file on the same 1015 storage device. In the flexible file layout type, while there is an 1016 array of filehandles, they are independent of the multipathing being 1017 used. If the metadata server wants to provide multiple read-only 1018 copies of the same file on the same storage device, then it should 1019 provide multiple ff_device_addr4, each as a mirror. The client can 1020 then determine that since the ffds_fh_vers are different, then there 1021 are multiple copies of the file for the current layout segment 1022 available. 1024 5.3. Handling Version Errors 1026 When the metadata server provides the ffda_versions array in the 1027 ff_device_addr4 (see Section 4.1), the client is able to determine if 1028 it can not access a storage device with any of the supplied 1029 combinations of ffdv_version, ffdv_minorversion, and 1030 ffdv_tightly_coupled. However, due to the limitations of reporting 1031 errors in GETDEVICEINFO (see Section 18.40 in [RFC5661], the client 1032 is not able to specify which specific device it can not communicate 1033 with over one of the provided ffdv_version and ffdv_minorversion 1034 combinations. Using ff_ioerr4 (see Section 9.1.1 inside either the 1035 LAYOUTRETURN (see Section 18.44 of [RFC5661]) or the LAYOUTERROR (see 1036 Section 15.6 of [RFC7862] and Section 10 of this document), the 1037 client can isolate the problematic storage device. 1039 The error code to return for LAYOUTRETURN and/or LAYOUTERROR is 1040 NFS4ERR_MINOR_VERS_MISMATCH. It does not matter whether the mismatch 1041 is a major version (e.g., client can use NFSv3 but not NFSv4) or 1042 minor version (e.g., client can use NFSv4.1 but not NFSv4.2), the 1043 error indicates that for all the supplied combinations for 1044 ffdv_version and ffdv_minorversion, the client can not communicate 1045 with the storage device. The client can retry the GETDEVICEINFO to 1046 see if the metadata server can provide a different combination or it 1047 can fall back to doing the I/O through the metadata server. 1049 6. Striping via Sparse Mapping 1051 While other layout types support both dense and sparse mapping of 1052 logical offsets to physical offsets within a file (see for example 1053 Section 13.4 of [RFC5661]), the flexible file layout type only 1054 supports a sparse mapping. 1056 With sparse mappings, the logical offset within a file (L) is also 1057 the physical offset on the storage device. As detailed in 1058 Section 13.4.4 of [RFC5661], this results in holes across each 1059 storage device which does not contain the current stripe index. 1061 L: logical offset into the file 1063 W: stripe width 1064 W = number of elements in ffm_data_servers 1066 S: number of bytes in a stripe 1067 S = W * ffl_stripe_unit 1069 N: stripe number 1070 N = L / S 1072 7. Recovering from Client I/O Errors 1074 The pNFS client may encounter errors when directly accessing the 1075 storage devices. However, it is the responsibility of the metadata 1076 server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES 1077 layout type is used, the client MUST report the I/O errors to the 1078 server at LAYOUTRETURN time using the ff_ioerr4 structure (see 1079 Section 9.1.1). 1081 The metadata server analyzes the error and determines the required 1082 recovery operations such as recovering media failures or 1083 reconstructing missing data files. 1085 The metadata server SHOULD recall any outstanding layouts to allow it 1086 exclusive write access to the stripes being recovered and to prevent 1087 other clients from hitting the same error condition. In these cases, 1088 the server MUST complete recovery before handing out any new layouts 1089 to the affected byte ranges. 1091 Although the client implementation has the option to propagate a 1092 corresponding error to the application that initiated the I/O 1093 operation and drop any unwritten data, the client should attempt to 1094 retry the original I/O operation by either requesting a new layout or 1095 sending the I/O via regular NFSv4.1+ READ or WRITE operations to the 1096 metadata server. The client SHOULD attempt to retrieve a new layout 1097 and retry the I/O operation using the storage device first and only 1098 if the error persists, retry the I/O operation via the metadata 1099 server. 1101 8. Mirroring 1103 The flexible file layout type has a simple model in place for the 1104 mirroring of the file data constrained by a layout segment. There is 1105 no assumption that each copy of the mirror is stored identically on 1106 the storage devices. For example, one device might employ 1107 compression or deduplication on the data. However, the over the wire 1108 transfer of the file contents MUST appear identical. Note, this is a 1109 constraint of the selected XDR representation in which each mirrored 1110 copy of the layout segment has the same striping pattern (see 1111 Figure 1). 1113 The metadata server is responsible for determining the number of 1114 mirrored copies and the location of each mirror. While the client 1115 may provide a hint to how many copies it wants (see Section 12), the 1116 metadata server can ignore that hint and in any event, the client has 1117 no means to dictate either the storage device (which also means the 1118 coupling and/or protocol levels to access the layout segments) or the 1119 location of said storage device. 1121 The updating of mirrored layout segments is done via client-side 1122 mirroring. With this approach, the client is responsible for making 1123 sure modifications are made on all copies of the layout segments it 1124 is informed of via the layout. If a layout segment is being 1125 resilvered to a storage device, that mirrored copy will not be in the 1126 layout. Thus the metadata server MUST update that copy until the 1127 client is presented it in a layout. If the FF_FLAGS_WRITE_ONE_MIRROR 1128 is set in ffl_flags, the client need only update one of the mirrors 1129 (see Section 8.2). If the client is writing to the layout segments 1130 via the metadata server, then the metadata server MUST update all 1131 copies of the mirror. As seen in Section 8.3, during the 1132 resilvering, the layout is recalled, and the client has to make 1133 modifications via the metadata server. 1135 8.1. Selecting a Mirror 1137 When the metadata server grants a layout to a client, it MAY let the 1138 client know how fast it expects each mirror to be once the request 1139 arrives at the storage devices via the ffds_efficiency member. While 1140 the algorithms to calculate that value are left to the metadata 1141 server implementations, factors that could contribute to that 1142 calculation include speed of the storage device, physical memory 1143 available to the device, operating system version, current load, etc. 1145 However, what should not be involved in that calculation is a 1146 perceived network distance between the client and the storage device. 1147 The client is better situated for making that determination based on 1148 past interaction with the storage device over the different available 1149 network interfaces between the two. I.e., the metadata server might 1150 not know about a transient outage between the client and storage 1151 device because it has no presence on the given subnet. 1153 As such, it is the client which decides which mirror to access for 1154 reading the file. The requirements for writing to mirrored layout 1155 segments are presented below. 1157 8.2. Writing to Mirrors 1159 8.2.1. Single Storage Device Updates Mirrors 1161 If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is set, the client 1162 only needs to update one of the copies of the layout segment. For 1163 this case, the storage device MUST ensure that all copies of the 1164 mirror are updated when any one of the mirrors is updated. If the 1165 storage device gets an error when updating one of the mirrors, then 1166 it MUST inform the client that the original WRITE had an error. The 1167 client then MUST inform the metadata server (see Section 8.2.3). The 1168 client's responsibility with respect to COMMIT is explained in 1169 Section 8.2.4. The client may choose any one of the mirrors and may 1170 use ffds_efficiency in the same manner as for reading when making 1171 this choice. 1173 8.2.2. Single Storage Device Updates Mirrors 1175 If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is not set, the 1176 client is responsible for updating all mirrored copies of the layout 1177 segments that it is given in the layout. A single failed update is 1178 sufficient to fail the entire operation. If all but one copy is 1179 updated successfully and the last one provides an error, then the 1180 client needs to inform the metadata server about the error via either 1181 LAYOUTRETURN or LAYOUTERROR that the update failed to that storage 1182 device. If the client is updating the mirrors serially, then it 1183 SHOULD stop at the first error encountered and report that to the 1184 metadata server. If the client is updating the mirrors in parallel, 1185 then it SHOULD wait until all storage devices respond such that it 1186 can report all errors encountered during the update. 1188 8.2.3. Handling Write Errors 1190 When the client reports a write error to the metadata server, the 1191 metadata server is responsible for determining if it wants to remove 1192 the errant mirror from the layout, if the mirror has recovered from 1193 some transient error, etc. When the client tries to get a new 1194 layout, the metadata server informs it of the decision by the 1195 contents of the layout. The client MUST NOT make any assumptions 1196 that the contents of the previous layout will match those of the new 1197 one. If it has updates that were not committed to all mirrors, then 1198 it MUST resend those updates to all mirrors. 1200 There is no provision in the protocol for the metadata server to 1201 directly determine that the client has or has not recovered from an 1202 error. I.e., assume that the storage device was network partitioned 1203 from the client and all of the copies are successfully updated after 1204 the error was reported. There is no mechanism for the client to 1205 report that fact and the metadata server is forced to repair the file 1206 across the mirror. 1208 If the client supports NFSv4.2, it can use LAYOUTERROR and 1209 LAYOUTRETURN to provide hints to the metadata server about the 1210 recovery efforts. A LAYOUTERROR on a file is for a non-fatal error. 1211 A subsequent LAYOUTRETURN without a ff_ioerr4 indicates that the 1212 client successfully replayed the I/O to all mirrors. Any 1213 LAYOUTRETURN with a ff_ioerr4 is an error that the metadata server 1214 needs to repair. The client MUST be prepared for the LAYOUTERROR to 1215 trigger a CB_LAYOUTRECALL if the metadata server determines it needs 1216 to start repairing the file. 1218 8.2.4. Handling Write COMMITs 1220 When stable writes are done to the metadata server or to a single 1221 replica (if allowed by the use of FF_FLAGS_WRITE_ONE_MIRROR ), it is 1222 the responsibility of the receiving node to propagate the written 1223 data stably, before replying to the client. 1225 In the corresponding cases in which unstable writes are done, the 1226 receiving node does not have any such obligation, although it may 1227 choose to asynchronously propagate the updates. However, once a 1228 COMMIT is replied to, all replicas must reflect the writes that have 1229 been done, and this data must have been committed to stable storage 1230 on all replicas. 1232 In order to avoid situations in which stale data is read from 1233 replicas to which writes have not been propagated: 1235 o A client which has outstanding unstable writes made to single node 1236 (metadata server or storage device) MUST do all reads from that 1237 same node. 1239 o When writes are flushed to the server, for example to implement, 1240 close-to-open semantics, a COMMIT must be done by the client to 1241 ensure that up-to-date written data will be available irrespective 1242 of the particular replica read. 1244 8.3. Metadata Server Resilvering of the File 1246 The metadata server may elect to create a new mirror of the layout 1247 segments at any time. This might be to resilver a copy on a storage 1248 device which was down for servicing, to provide a copy of the layout 1249 segments on storage with different storage performance 1250 characteristics, etc. As the client will not be aware of the new 1251 mirror and the metadata server will not be aware of updates that the 1252 client is making to the layout segments, the metadata server MUST 1253 recall the writable layout segment(s) that it is resilvering. If the 1254 client issues a LAYOUTGET for a writable layout segment which is in 1255 the process of being resilvered, then the metadata server can deny 1256 that request with a NFS4ERR_LAYOUTUNAVAILABLE. The client would then 1257 have to perform the I/O through the metadata server. 1259 9. Flexible Files Layout Type Return 1261 layoutreturn_file4 is used in the LAYOUTRETURN operation to convey 1262 layout-type specific information to the server. It is defined in 1263 Section 18.44.1 of [RFC5661] as follows: 1265 1267 /* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */ 1268 const LAYOUT4_RET_REC_FILE = 1; 1269 const LAYOUT4_RET_REC_FSID = 2; 1270 const LAYOUT4_RET_REC_ALL = 3; 1272 enum layoutreturn_type4 { 1273 LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE, 1274 LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID, 1275 LAYOUTRETURN4_ALL = LAYOUT4_RET_REC_ALL 1276 }; 1278 struct layoutreturn_file4 { 1279 offset4 lrf_offset; 1280 length4 lrf_length; 1281 stateid4 lrf_stateid; 1282 /* layouttype4 specific data */ 1283 opaque lrf_body<>; 1284 }; 1285 union layoutreturn4 switch(layoutreturn_type4 lr_returntype) { 1286 case LAYOUTRETURN4_FILE: 1287 layoutreturn_file4 lr_layout; 1288 default: 1289 void; 1290 }; 1292 struct LAYOUTRETURN4args { 1293 /* CURRENT_FH: file */ 1294 bool lora_reclaim; 1295 layoutreturn_stateid lora_recallstateid; 1296 layouttype4 lora_layout_type; 1297 layoutiomode4 lora_iomode; 1298 layoutreturn4 lora_layoutreturn; 1299 }; 1301 1303 If the lora_layout_type layout type is LAYOUT4_FLEX_FILES and the 1304 lr_returntype is LAYOUTRETURN4_FILE, then the lrf_body opaque value 1305 is defined by ff_layoutreturn4 (See Section 9.3). It allows the 1306 client to report I/O error information or layout usage statistics 1307 back to the metadata server as defined below. Note that while the 1308 data structures are built on concepts introduced in NFSv4.2, the 1309 effective discriminated union (lora_layout_type combined with 1310 ff_layoutreturn4) allows for a NFSv4.1 metadata server to utilize the 1311 data. 1313 9.1. I/O Error Reporting 1315 9.1.1. ff_ioerr4 1317 1319 /// struct ff_ioerr4 { 1320 /// offset4 ffie_offset; 1321 /// length4 ffie_length; 1322 /// stateid4 ffie_stateid; 1323 /// device_error4 ffie_errors<>; 1324 /// }; 1325 /// 1327 1329 Recall that [RFC7862] defines device_error4 as: 1331 1332 struct device_error4 { 1333 deviceid4 de_deviceid; 1334 nfsstat4 de_status; 1335 nfs_opnum4 de_opnum; 1336 }; 1338 1340 The ff_ioerr4 structure is used to return error indications for data 1341 files that generated errors during data transfers. These are hints 1342 to the metadata server that there are problems with that file. For 1343 each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length 1344 represent the storage device and byte range within the file in which 1345 the error occurred; ffie_errors represents the operation and type of 1346 error. The use of device_error4 is described in Section 15.6 of 1347 [RFC7862]. 1349 Even though the storage device might be accessed via NFSv3 and 1350 reports back NFSv3 errors to the client, the client is responsible 1351 for mapping these to appropriate NFSv4 status codes as de_status. 1352 Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 1353 operations. 1355 9.2. Layout Usage Statistics 1357 9.2.1. ff_io_latency4 1359 1361 /// struct ff_io_latency4 { 1362 /// uint64_t ffil_ops_requested; 1363 /// uint64_t ffil_bytes_requested; 1364 /// uint64_t ffil_ops_completed; 1365 /// uint64_t ffil_bytes_completed; 1366 /// uint64_t ffil_bytes_not_delivered; 1367 /// nfstime4 ffil_total_busy_time; 1368 /// nfstime4 ffil_aggregate_completion_time; 1369 /// }; 1370 /// 1372 1374 Both operation counts and bytes transferred are kept in the 1375 ff_io_latency4. As seen in ff_layoutupdate4 (See Section 9.2.2) read 1376 and write operations are aggregated separately. READ operations are 1377 used for the ff_io_latency4 ffl_read. Both WRITE and COMMIT 1378 operations are used for the ff_io_latency4 ffl_write. "Requested" 1379 counters track what the client is attempting to do and "completed" 1380 counters track what was done. There is no requirement that the 1381 client only report completed results that have matching requested 1382 results from the reported period. 1384 ffil_bytes_not_delivered is used to track the aggregate number of 1385 bytes requested by not fulfilled due to error conditions. 1386 ffil_total_busy_time is the aggregate time spent with outstanding RPC 1387 calls. ffil_aggregate_completion_time is the sum of all round trip 1388 times for completed RPC calls. 1390 In Section 3.3.1 of [RFC5661], the nfstime4 is defined as the number 1391 of seconds and nanoseconds since midnight or zero hour January 1, 1392 1970 Coordinated Universal Time (UTC). The use of nfstime4 in 1393 ff_io_latency4 is to store time since the start of the first I/O from 1394 the client after receiving the layout. In other words, these are to 1395 be decoded as duration and not as a date and time. 1397 Note that LAYOUTSTATS are cumulative, i.e., not reset each time the 1398 operation is sent. If two LAYOUTSTATS ops for the same file, layout 1399 stateid, and originating from the same NFS client are processed at 1400 the same time by the metadata server, then the one containing the 1401 larger values contains the most recent time series data. 1403 9.2.2. ff_layoutupdate4 1405 1407 /// struct ff_layoutupdate4 { 1408 /// netaddr4 ffl_addr; 1409 /// nfs_fh4 ffl_fhandle; 1410 /// ff_io_latency4 ffl_read; 1411 /// ff_io_latency4 ffl_write; 1412 /// nfstime4 ffl_duration; 1413 /// bool ffl_local; 1414 /// }; 1415 /// 1417 1419 ffl_addr differentiates which network address the client connected to 1420 on the storage device. In the case of multipathing, ffl_fhandle 1421 indicates which read-only copy was selected. ffl_read and ffl_write 1422 convey the latencies respectively for both read and write operations. 1423 ffl_duration is used to indicate the time period over which the 1424 statistics were collected. ffl_local if true indicates that the I/O 1425 was serviced by the client's cache. This flag allows the client to 1426 inform the metadata server about "hot" access to a file it would not 1427 normally be allowed to report on. 1429 9.2.3. ff_iostats4 1431 1433 /// struct ff_iostats4 { 1434 /// offset4 ffis_offset; 1435 /// length4 ffis_length; 1436 /// stateid4 ffis_stateid; 1437 /// io_info4 ffis_read; 1438 /// io_info4 ffis_write; 1439 /// deviceid4 ffis_deviceid; 1440 /// ff_layoutupdate4 ffis_layoutupdate; 1441 /// }; 1442 /// 1444 1446 Recall that [RFC7862] defines io_info4 as: 1448 1450 struct io_info4 { 1451 uint64_t ii_count; 1452 uint64_t ii_bytes; 1453 }; 1455 1457 With pNFS, data transfers are performed directly between the pNFS 1458 client and the storage devices. Therefore, the metadata server has 1459 no direct knowledge to the I/O operations being done and thus can not 1460 create on its own statistical information about client I/O to 1461 optimize data storage location. ff_iostats4 MAY be used by the 1462 client to report I/O statistics back to the metadata server upon 1463 returning the layout. 1465 Since it is not feasible for the client to report every I/O that used 1466 the layout, the client MAY identify "hot" byte ranges for which to 1467 report I/O statistics. The definition and/or configuration mechanism 1468 of what is considered "hot" and the size of the reported byte range 1469 is out of the scope of this document. It is suggested for client 1470 implementation to provide reasonable default values and an optional 1471 run-time management interface to control these parameters. For 1472 example, a client can define the default byte range resolution to be 1473 1 MB in size and the thresholds for reporting to be 1 MB/second or 10 1474 I/O operations per second. 1476 For each byte range, ffis_offset and ffis_length represent the 1477 starting offset of the range and the range length in bytes. 1478 ffis_read.ii_count, ffis_read.ii_bytes, ffis_write.ii_count, and 1479 ffis_write.ii_bytes represent, respectively, the number of contiguous 1480 read and write I/Os and the respective aggregate number of bytes 1481 transferred within the reported byte range. 1483 The combination of ffis_deviceid and ffl_addr uniquely identifies 1484 both the storage path and the network route to it. Finally, the 1485 ffl_fhandle allows the metadata server to differentiate between 1486 multiple read-only copies of the file on the same storage device. 1488 9.3. ff_layoutreturn4 1490 1492 /// struct ff_layoutreturn4 { 1493 /// ff_ioerr4 fflr_ioerr_report<>; 1494 /// ff_iostats4 fflr_iostats_report<>; 1495 /// }; 1496 /// 1498 1500 When data file I/O operations fail, fflr_ioerr_report<> is used to 1501 report these errors to the metadata server as an array of elements of 1502 type ff_ioerr4. Each element in the array represents an error that 1503 occurred on the data file identified by ffie_errors.de_deviceid. If 1504 no errors are to be reported, the size of the fflr_ioerr_report<> 1505 array is set to zero. The client MAY also use fflr_iostats_report<> 1506 to report a list of I/O statistics as an array of elements of type 1507 ff_iostats4. Each element in the array represents statistics for a 1508 particular byte range. Byte ranges are not guaranteed to be disjoint 1509 and MAY repeat or intersect. 1511 10. Flexible Files Layout Type LAYOUTERROR 1513 If the client is using NFSv4.2 to communicate with the metadata 1514 server, then instead of waiting for a LAYOUTRETURN to send error 1515 information to the metadata server (see Section 9.1), it MAY use 1516 LAYOUTERROR (see Section 15.6 of [RFC7862]) to communicate that 1517 information. For the flexible files layout type, this means that 1518 LAYOUTERROR4args is treated the same as ff_ioerr4. 1520 11. Flexible Files Layout Type LAYOUTSTATS 1522 If the client is using NFSv4.2 to communicate with the metadata 1523 server, then instead of waiting for a LAYOUTRETURN to send I/O 1524 statistics to the metadata server (see Section 9.2), it MAY use 1525 LAYOUTSTATS (see Section 15.7 of [RFC7862]) to communicate that 1526 information. For the flexible files layout type, this means that 1527 LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same 1528 contents as in ffis_layoutupdate. 1530 12. Flexible File Layout Type Creation Hint 1532 The layouthint4 type is defined in the [RFC5661] as follows: 1534 1536 struct layouthint4 { 1537 layouttype4 loh_type; 1538 opaque loh_body<>; 1539 }; 1541 1543 The layouthint4 structure is used by the client to pass a hint about 1544 the type of layout it would like created for a particular file. If 1545 the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body 1546 opaque value is defined by the ff_layouthint4 type. 1548 12.1. ff_layouthint4 1550 1552 /// union ff_mirrors_hint switch (bool ffmc_valid) { 1553 /// case TRUE: 1554 /// uint32_t ffmc_mirrors; 1555 /// case FALSE: 1556 /// void; 1557 /// }; 1558 /// 1560 /// struct ff_layouthint4 { 1561 /// ff_mirrors_hint fflh_mirrors_hint; 1562 /// }; 1563 /// 1565 1566 This type conveys hints for the desired data map. All parameters are 1567 optional so the client can give values for only the parameter it 1568 cares about. 1570 13. Recalling a Layout 1572 While Section 12.5.5 of [RFC5661] discusses layout type independent 1573 reasons for recalling a layout, the flexible file layout type 1574 metadata server should recall outstanding layouts in the following 1575 cases: 1577 o When the file's security policy changes, i.e., Access Control 1578 Lists (ACLs) or permission mode bits are set. 1580 o When the file's layout changes, rendering outstanding layouts 1581 invalid. 1583 o When existing layouts are inconsistent with the need to enforce 1584 locking constraints. 1586 o When existing layouts are inconsistent with the requirements 1587 regarding resilvering as described in Section 8.3. 1589 13.1. CB_RECALL_ANY 1591 The metadata server can use the CB_RECALL_ANY callback operation to 1592 notify the client to return some or all of its layouts. Section 22.3 1593 of [RFC5661] defines the allowed types of the "NFSv4 Recallable 1594 Object Types Registry". 1596 1598 /// const RCA4_TYPE_MASK_FF_LAYOUT_MIN = 16; 1599 /// const RCA4_TYPE_MASK_FF_LAYOUT_MAX = 17; 1600 [[RFC Editor: please insert assigned constants]] 1601 /// 1603 struct CB_RECALL_ANY4args { 1604 uint32_t craa_layouts_to_keep; 1605 bitmap4 craa_type_mask; 1606 }; 1608 1610 Typically, CB_RECALL_ANY will be used to recall client state when the 1611 server needs to reclaim resources. The craa_type_mask bitmap 1612 specifies the type of resources that are recalled and the 1613 craa_layouts_to_keep value specifies how many of the recalled 1614 flexible file layouts the client is allowed to keep. The flexible 1615 file layout type mask flags are defined as follows: 1617 1619 /// enum ff_cb_recall_any_mask { 1620 /// FF_RCA4_TYPE_MASK_READ = -2, 1621 /// FF_RCA4_TYPE_MASK_RW = -1 1622 [[RFC Editor: please insert assigned constants]] 1623 /// }; 1624 /// 1626 1628 They represent the iomode of the recalled layouts. In response, the 1629 client SHOULD return layouts of the recalled iomode that it needs the 1630 least, keeping at most craa_layouts_to_keep Flexible File Layouts. 1632 The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return 1633 layouts of iomode LAYOUTIOMODE4_READ. Similarly, the 1634 PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts 1635 of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client 1636 is notified to return layouts of either iomode. 1638 14. Client Fencing 1640 In cases where clients are uncommunicative and their lease has 1641 expired or when clients fail to return recalled layouts within a 1642 lease period, at the least the server MAY revoke client layouts and 1643 reassign these resources to other clients (see Section 12.5.5 in 1644 [RFC5661]). To avoid data corruption, the metadata server MUST fence 1645 off the revoked clients from the respective data files as described 1646 in Section 2.2. 1648 15. Security Considerations 1650 The pNFS extension partitions the NFSv4.1+ file system protocol into 1651 two parts, the control path and the data path (storage protocol). 1652 The control path contains all the new operations described by this 1653 extension; all existing NFSv4 security mechanisms and features apply 1654 to the control path (see Sections 1.7.1 and 2.2.1 of [RFC5661]). The 1655 combination of components in a pNFS system is required to preserve 1656 the security properties of NFSv4.1+ with respect to an entity 1657 accessing data via a client, including security countermeasures to 1658 defend against threats that NFSv4.1+ provides defenses for in 1659 environments where these threats are considered significant. 1661 The metadata server enforces the file access-control policy at 1662 LAYOUTGET time. The client should use RPC authorization credentials 1663 for getting the layout for the requested iomode (READ or RW) and the 1664 server verifies the permissions and ACL for these credentials, 1665 possibly returning NFS4ERR_ACCESS if the client is not allowed the 1666 requested iomode. If the LAYOUTGET operation succeeds the client 1667 receives, as part of the layout, a set of credentials allowing it I/O 1668 access to the specified data files corresponding to the requested 1669 iomode. When the client acts on I/O operations on behalf of its 1670 local users, it MUST authenticate and authorize the user by issuing 1671 respective OPEN and ACCESS calls to the metadata server, similar to 1672 having NFSv4 data delegations. 1674 If access is allowed, the client uses the corresponding (READ or RW) 1675 credentials to perform the I/O operations at the data file's storage 1676 devices. When the metadata server receives a request to change a 1677 file's permissions or ACL, it SHOULD recall all layouts for that file 1678 and then MUST fence off any clients still holding outstanding layouts 1679 for the respective files by implicitly invalidating the previously 1680 distributed credential on all data file comprising the file in 1681 question. It is REQUIRED that this be done before committing to the 1682 new permissions and/or ACL. By requesting new layouts, the clients 1683 will reauthorize access against the modified access control metadata. 1684 Recalling the layouts in this case is intended to prevent clients 1685 from getting an error on I/Os done after the client was fenced off. 1687 15.1. RPCSEC_GSS and Security Services 1689 15.1.1. Loosely Coupled 1691 RPCSEC_GSS version 3 (RPCSEC_GSSv3) [RFC7861] could be used to 1692 authorize the client to the storage device on behalf of the metadata 1693 server. This would require that each of the metadata server, storage 1694 device, and client would have to implement RPCSEC_GSSv3 via an RPC- 1695 application-defined structured privilege assertion in a manner 1696 described in Section 4.9.1 of [RFC7862]. These requirements do not 1697 match the intent of the loosely coupled model that the storage device 1698 need not be modified. (Note that this does not preclude the use of 1699 RPCSEC_GSSv3 in a loosely coupled model.) 1701 15.1.2. Tightly Coupled 1703 With tight coupling, the principal used to access the metadata file 1704 is exactly the same as used to access the data file. The storage 1705 device can use the control protocol to validate any RPC credentials. 1706 As a result there are no security issues related to using RPCSEC_GSS 1707 with a tightly coupled system. For example, if Kerberos V5 GSS-API 1708 [RFC4121] is used as the security mechanism, then the storage device 1709 could use a control protocol to validate the RPC credentials to the 1710 metadata server. 1712 16. IANA Considerations 1714 [RFC5661] introduced a registry for "pNFS Layout Types Registry" and 1715 as such, new layout type numbers need to be assigned by IANA. This 1716 document defines the protocol associated with the existing layout 1717 type number, LAYOUT4_FLEX_FILES (see Table 1). 1719 +--------------------+-------+----------+-----+----------------+ 1720 | Layout Type Name | Value | RFC | How | Minor Versions | 1721 +--------------------+-------+----------+-----+----------------+ 1722 | LAYOUT4_FLEX_FILES | 0x4 | RFCTBD10 | L | 1 | 1723 +--------------------+-------+----------+-----+----------------+ 1725 Table 1: Layout Type Assignments 1727 [RFC5661] also introduced a registry called "NFSv4 Recallable Object 1728 Types Registry". This document defines new recallable objects for 1729 RCA4_TYPE_MASK_FF_LAYOUT_MIN and RCA4_TYPE_MASK_FF_LAYOUT_MAX (see 1730 Table 2). 1732 +------------------------------+-------+----------+-----+-----------+ 1733 | Recallable Object Type Name | Value | RFC | How | Minor | 1734 | | | | | Versions | 1735 +------------------------------+-------+----------+-----+-----------+ 1736 | RCA4_TYPE_MASK_FF_LAYOUT_MIN | 16 | RFCTBD10 | L | 1 | 1737 | RCA4_TYPE_MASK_FF_LAYOUT_MAX | 17 | RFCTBD10 | L | 1 | 1738 +------------------------------+-------+----------+-----+-----------+ 1740 Table 2: Recallable Object Type Assignments 1742 Note, [RFC5661] should have also defined (see Table 3): 1744 +-------------------------------+------+-----------+-----+----------+ 1745 | Recallable Object Type Name | Valu | RFC | How | Minor | 1746 | | e | | | Versions | 1747 +-------------------------------+------+-----------+-----+----------+ 1748 | RCA4_TYPE_MASK_OTHER_LAYOUT_M | 12 | [RFC5661] | L | 1 | 1749 | IN | | | | | 1750 | RCA4_TYPE_MASK_OTHER_LAYOUT_M | 15 | [RFC5661] | L | 1 | 1751 | AX | | | | | 1752 +-------------------------------+------+-----------+-----+----------+ 1754 Table 3: Recallable Object Type Assignments 1756 17. References 1758 17.1. Normative References 1760 [LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents", 1761 November 2008, . 1764 [RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813, 1765 June 1995. 1767 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1768 Requirement Levels", BCP 14, RFC 2119, March 1997. 1770 [RFC4121] Zhu, L., Jaganathan, K., and S. Hartman, "The Kerberos 1771 Version 5 Generic Security Service Application Program 1772 Interface (GSS-API) Mechanism Version 2", RFC 4121, July 1773 2005. 1775 [RFC4506] Eisler, M., "XDR: External Data Representation Standard", 1776 STD 67, RFC 4506, May 2006. 1778 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 1779 Specification Version 2", RFC 5531, May 2009. 1781 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1782 "Network File System (NFS) Version 4 Minor Version 1 1783 Protocol", RFC 5661, January 2010. 1785 [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1786 "Network File System (NFS) Version 4 Minor Version 1 1787 External Data Representation Standard (XDR) Description", 1788 RFC 5662, January 2010. 1790 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 1791 version 4 Protocol", RFC 7530, March 2015. 1793 [RFC7862] Haynes, T., "NFS Version 4 Minor Version 2", RFC 7862, 1794 November 2016. 1796 [pNFSLayouts] 1797 Haynes, T., "Requirements for pNFS Layout Types", draft- 1798 ietf-nfsv4-layout-types-05 (Work In Progress), July 2017. 1800 17.2. Informative References 1802 [RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol 1803 (LDAP): Schema for User Applications", RFC 4519, DOI 1804 10.17487/RFC4519, June 2006, 1805 . 1807 [RFC7861] Adamson, W. and N. Williams, "Remote Procedure Call (RPC) 1808 Security Version 3", November 2016. 1810 Appendix A. Acknowledgments 1812 Those who provided miscellaneous comments to early drafts of this 1813 document include: Matt W. Benjamin, Adam Emerson, J. Bruce Fields, 1814 and Lev Solomonov. 1816 Those who provided miscellaneous comments to the final drafts of this 1817 document include: Anand Ganesh, Robert Wipfel, Gobikrishnan 1818 Sundharraj, Trond Myklebust, Rick Macklem, and Jim Sermersheim. 1820 Idan Kedar caught a nasty bug in the interaction of client side 1821 mirroring and the minor versioning of devices. 1823 Dave Noveck provided comprehensive reviews of the document during the 1824 working group last calls. He also rewrote Section 2.3. 1826 Olga Kornievskaia made a convincing case against the use of a 1827 credential versus a principal in the fencing approach. Andy Adamson 1828 and Benjamin Kaduk helped to sharpen the focus. 1830 Benjamin Kaduk and Olga Kornievskaia also helped provide concrete 1831 scenarios for loosely coupled security mechanisms. And in the end, 1832 Olga proved that as defined, the loosely coupled model would not work 1833 with RPCSEC_GSS. 1835 Tigran Mkrtchyan provided the use case for not allowing the client to 1836 proxy the I/O through the data server. 1838 Rick Macklem provided the use case for only writing to a single 1839 mirror. 1841 Appendix B. RFC Editor Notes 1843 [RFC Editor: please remove this section prior to publishing this 1844 document as an RFC] 1846 [RFC Editor: prior to publishing this document as an RFC, please 1847 replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the 1848 RFC number of this document] 1850 Authors' Addresses 1852 Benny Halevy 1854 Email: bhalevy@gmail.com 1856 Thomas Haynes 1857 Primary Data, Inc. 1858 4300 El Camino Real Ste 100 1859 Los Altos, CA 94022 1860 USA 1862 Phone: +1 408 215 1519 1863 Email: thomas.haynes@primarydata.com