idnits 2.17.1 draft-ietf-nfsv4-flex-files-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 355 has weird spacing: '... loghyr staff...' == Line 1249 has weird spacing: '...stateid lor...' == Line 1505 has weird spacing: '...rs_hint ffl...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: F_FLAGS_NO_IO_THRU_MDS: can be set to indicate that the client SHOULD not send I/O operations to the metadata server. I.e., even if the client could determine that there was a network diconnect to a storage device, the client SHOULD not try to proxy the I/O through the metadata server. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: FF_FLAGS_NO_READ_IO: can be set to indicate that the client SHOULD not send READ requests with the layouts of iomode LAYOUTIOMODE4_RW. Instead, it should request a layout of iomode LAYOUTIOMODE4_READ from the metadata server. -- The document date (July 17, 2017) is 2446 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. 'LEGAL' ** Downref: Normative reference to an Informational RFC: RFC 1813 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 2 errors (**), 0 flaws (~~), 6 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 B. Halevy 3 Internet-Draft 4 Intended status: Standards Track T. Haynes 5 Expires: January 18, 2018 Primary Data 6 July 17, 2017 8 Parallel NFS (pNFS) Flexible File Layout 9 draft-ietf-nfsv4-flex-files-10.txt 11 Abstract 13 The Parallel Network File System (pNFS) allows a separation between 14 the metadata (onto a metadata server) and data (onto a storage 15 device) for a file. The flexible file layout type is defined in this 16 document as an extension to pNFS which allows the use of storage 17 devices in a fashion such that they require only a quite limited 18 degree of interaction with the metadata server, using already 19 existing protocols. Client side mirroring is also added to provide 20 replication of files. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at http://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on January 18, 2018. 39 Copyright Notice 41 Copyright (c) 2017 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (http://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 57 1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3 58 1.2. Difference Between a Data Server and a Storage Device . . 5 59 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6 60 2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6 61 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6 62 2.2. Fencing Clients from the Storage Device . . . . . . . . . 6 63 2.2.1. Implementation Notes for Synthetic uids/gids . . . . 7 64 2.2.2. Example of using Synthetic uids/gids . . . . . . . . 8 65 2.3. State and Locking Models . . . . . . . . . . . . . . . . 9 66 2.3.1. Loosely Coupled Locking Model . . . . . . . . . . . . 9 67 2.3.2. Tighly Coupled Locking Model . . . . . . . . . . . . 10 68 3. XDR Description of the Flexible File Layout Type . . . . . . 12 69 3.1. Code Components Licensing Notice . . . . . . . . . . . . 13 70 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 14 71 4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 14 72 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 16 73 5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 17 74 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 17 75 5.1.1. Error Codes from LAYOUTGET . . . . . . . . . . . . . 21 76 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS . . 21 77 5.2. Interactions Between Devices and Layouts . . . . . . . . 22 78 5.3. Handling Version Errors . . . . . . . . . . . . . . . . . 22 79 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 23 80 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 23 81 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 24 82 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 24 83 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 25 84 8.3. Metadata Server Resilvering of the File . . . . . . . . . 26 85 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 26 86 9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 27 87 9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 27 88 9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 28 89 9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 28 90 9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 29 91 9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 29 92 9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 31 93 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 31 94 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 31 95 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 32 96 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 32 98 13. Recalling a Layout . . . . . . . . . . . . . . . . . . . . . 33 99 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 33 100 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 34 101 15. Security Considerations . . . . . . . . . . . . . . . . . . . 34 102 15.1. Kerberized File Access . . . . . . . . . . . . . . . . . 35 103 15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 35 104 15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 35 105 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 36 106 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 36 107 17.1. Normative References . . . . . . . . . . . . . . . . . . 36 108 17.2. Informative References . . . . . . . . . . . . . . . . . 37 109 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 37 110 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 37 111 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37 113 1. Introduction 115 In the parallel Network File System (pNFS), the metadata server 116 returns layout type structures that describe where file data is 117 located. There are different layout types for different storage 118 systems and methods of arranging data on storage devices. This 119 document defines the flexible file layout type used with file-based 120 data servers that are accessed using the Network File System (NFS) 121 protocols: NFSv3 [RFC1813], NFSv4.0 [RFC7530], NFSv4.1 [RFC5661], and 122 NFSv4.2 [RFC7862]. 124 To provide a global state model equivalent to that of the files 125 layout type, a back-end control protocol MAY be implemented between 126 the metadata server and NFSv4.1+ storage devices. It is out of scope 127 for this document to specify such a protocol, yet the requirements 128 for the protocol are specified in [RFC5661] and clarified in 129 [pNFSLayouts]. 131 1.1. Definitions 133 control protocol: is a set of requirements for the communication of 134 information on layouts, stateids, file metadata, and file data 135 between the metadata server and the storage devices (see 136 [pNFSLayouts]). 138 client-side mirroring: is when the client and not the server is 139 responsible for updating all of the mirrored copies of a layout 140 segment. 142 data file: is that part of the file system object which contains the 143 content. 145 data server (DS): is one of the pNFS servers which provides the 146 contents of a file system object which is a regular file. 147 Depending on the layout, there might be one or more data servers 148 over which the data is striped. Note that while the metadata 149 server is strictly accessed over the NFSv4.1+ protocol, depending 150 on the layout type, the data server could be accessed via any 151 protocol that meets the pNFS requirements. 153 fencing: is when the metadata server prevents the storage devices 154 from processing I/O from a specific client to a specific file. 156 file layout type: is a layout type in which the storage devices are 157 accessed via the NFS protocol (see Section 13 of [RFC5661]). 159 layout: informs a client of which storage devices it needs to 160 communicate with (and over which protocol) to perform I/O on a 161 file. The layout might also provide some hints about how the 162 storage is physically organized. 164 layout iomode: describes whether the layout granted to the client is 165 for read or read/write I/O. 167 layout segment: describes a sub-division of a layout. That sub- 168 division might be by the iomode (see Sections 3.3.20 and 12.2.9 of 169 [RFC5661]), a striping pattern (see Section 13.3 of [RFC5661]), or 170 requested byte range. 172 layout stateid: is a 128-bit quantity returned by a server that 173 uniquely defines the layout state provided by the server for a 174 specific layout that describes a layout type and file (see 175 Section 12.5.2 of [RFC5661]). Further, Section 12.5.3 of 176 [RFC5661] describes the difference between a layout stateid and a 177 normal stateid. 179 layout type: describes both the storage protocol used to access the 180 data and the aggregation scheme used to lay out the file data on 181 the underlying storage devices. 183 loose coupling: is when the metadata server and the storage devices 184 do not have a control protocol present. 186 metadata file: is that part of the file system object which 187 describes the object and not the content. E.g., it could be the 188 time since last modification, access, etc. 190 metadata server (MDS): is the pNFS server which provides metadata 191 information for a file system object. It also is responsible for 192 generating layouts for file system objects. Note that the MDS is 193 responsible for directory-based operations. 195 mirror: is a copy of a layout segment. Note that if one copy of the 196 mirror is updated, then all copies must be updated. 198 recalling a layout: is when the metadata server uses a back channel 199 to inform the client that the layout is to be returned in a 200 graceful manner. Note that the client has the opportunity to 201 flush any writes, etc., before replying to the metadata server. 203 revoking a layout: is when the metadata server invalidates the 204 layout such that neither the metadata server nor any storage 205 device will accept any access from the client with that layout. 207 resilvering: is the act of rebuilding a mirrored copy of a layout 208 segment from a known good copy of the layout segment. Note that 209 this can also be done to create a new mirrored copy of the layout 210 segment. 212 rsize: is the data transfer buffer size used for reads. 214 stateid: is a 128-bit quantity returned by a server that uniquely 215 defines the open and locking states provided by the server for a 216 specific open-owner or lock-owner/open-owner pair for a specific 217 file and type of lock. 219 storage device: is another term used almost interchangeably with 220 data server. See Section 1.2 for the nuances between the two. 222 tight coupling: is when the metadata server and the storage devices 223 do have a control protocol present. 225 wsize: is the data transfer buffer size used for writes. 227 1.2. Difference Between a Data Server and a Storage Device 229 We defined a data server as a pNFS server, which implies that it can 230 utilize the NFSv4.1+ protocol to communicate with the client. As 231 such, only the file layout type would currently meet this 232 requirement. The more generic concept is a storage device, which can 233 use any protocol to communicate with the client. The requirements 234 for a storage device to act together with the metadata server to 235 provide data to a client are that there is a layout type 236 specification for the given protocol and that the metadata server has 237 granted a layout to the client. Note that nothing precludes there 238 being multiple supported layout types (i.e., protocols) between a 239 metadata server, storage devices, and client. 241 As storage device is the more encompassing terminology, this document 242 utilizes it over data server. 244 1.3. Requirements Language 246 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 247 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 248 document are to be interpreted as described in [RFC2119]. 250 2. Coupling of Storage Devices 252 The coupling of the metadata server with the storage devices can be 253 either tight or loose. In a tight coupling, there is a control 254 protocol present to manage security, LAYOUTCOMMITs, etc. With a 255 loose coupling, the only control protocol might be a version of NFS. 256 As such, semantics for managing security, state, and locking models 257 MUST be defined. 259 2.1. LAYOUTCOMMIT 261 When tightly coupled storage devices are used, the metadata server 262 has the responsibility, upon receiving a LAYOUTCOMMIT (see 263 Section 18.42 of [RFC5661]), of ensuring that the semantics of pNFS 264 are respected (see Section 12.5.4 of [RFC5661]). These do not 265 include a requirement that data written to data storage device be 266 stable upon completion of the LAYOUTCOMMIT. 268 In the case of loosely coupled storage devices, it is the 269 responsibility of the client to make sure the data file is stable 270 before the metadata server begins to query the storage devices about 271 the changes to the file. If any WRITE to a storage device did not 272 result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the 273 metadata server MUST be preceded by a COMMIT to the storage devices 274 written to. Note that if the client has not done a COMMIT to the 275 storage device, then the LAYOUTCOMMIT might not be synchronized to 276 the last WRITE operation to the storage device. 278 2.2. Fencing Clients from the Storage Device 280 With loosely coupled storage devices, the metadata server uses 281 synthetic uids and gids for the data file, where the uid owner of the 282 data file is allowed read/write access and the gid owner is allowed 283 read only access. As part of the layout (see ffds_user and 284 ffds_group in Section 5.1), the client is provided with the user and 285 group to be used in the Remote Procedure Call (RPC) [RFC5531] 286 credentials needed to access the data file. Fencing off of clients 287 is achieved by the metadata server changing the synthetic uid and/or 288 gid owners of the data file on the storage device to implicitly 289 revoke the outstanding RPC credentials. 291 With this loosely coupled model, the metadata server is not able to 292 fence off a single client, it is forced to fence off all clients. 293 However, as the other clients react to the fencing, returning their 294 layouts and trying to get new ones, the metadata server can hand out 295 a new uid and gid to allow access. 297 Note: it is recommended to implement common access control methods at 298 the storage device filesystem to allow only the metadata server root 299 (super user) access to the storage device, and to set the owner of 300 all directories holding data files to the root user. This approach 301 provides a practical model to enforce access control and fence off 302 cooperative clients, but it can not protect against malicious 303 clients; hence it provides a level of security equivalent to 304 AUTH_SYS. 306 With tightly coupled storage devices, the metadata server sets the 307 user and group owners, mode bits, and ACL of the data file to be the 308 same as the metadata file. And the client must authenticate with the 309 storage device and go through the same authorization process it would 310 go through via the metadata server. In the case of tight coupling, 311 fencing is the responsibility of the control protocol and is not 312 described in detail here. However, implementations of the tight 313 coupling locking model (see Section 2.3), will need a way to prevent 314 access by certain clients to specific files by invalidating the 315 corresponding stateids on the storage device. 317 2.2.1. Implementation Notes for Synthetic uids/gids 319 The selection method for the synthetic uids and gids to be used for 320 fencing in loosely coupled storage devices is strictly an 321 implementation issue. I.e., an administrator might restrict a range 322 of such ids available to the Lightweight Directory Access Protocol 323 (LDAP) 'uid' field [RFC4519]. She might also be able to choose an id 324 that would never be used to grant acccess. Then when the metadata 325 server had a request to access a file, a SETATTR would be sent to the 326 storage device to set the owner and group of the data file. The user 327 and group might be selected in a round robin fashion from the range 328 of available ids. 330 Those ids would be sent back as ffds_user and ffds_group to the 331 client. And it would present them as the RPC credentials to the 332 storage device. When the client was done accessing the file and the 333 metadata server knew that no other client was accessing the file, it 334 could reset the owner and group to restrict access to the data file. 336 When the metadata server wanted to fence off a client, it would 337 change the synthetic uid and/or gid to the restricted ids. Note that 338 using a restricted id ensures that there is a change of owner and at 339 least one id available that never gets allowed access. 341 Under an AUTH_SYS security model, synthetic uids and gids of 0 SHOULD 342 be avoided. These typically either grant super access to files on a 343 storage device or are mapped to an anonymous id. In the first case, 344 even if the data file is fenced, the client might still be able to 345 access the file. In the second case, multiple ids might be mapped to 346 the anonymous ids. 348 2.2.2. Example of using Synthetic uids/gids 350 The user loghyr creates a file "ompha.c" on the metadata server and 351 it creates a corresponding data file on the storage device. 353 The metadata server entry may look like: 355 -rw-r--r-- 1 loghyr staff 1697 Dec 4 11:31 ompha.c 357 On the storage device, it may be assigned some random synthetic uid/ 358 gid to deny access: 360 -rw-r----- 1 19452 28418 1697 Dec 4 11:31 data_ompha.c 362 When the file is opened on a client, since the layout knows nothing 363 about the user (and does not care), whether loghyr or garbo opens the 364 file does not matter. The owner and group are modified and those 365 values are returned. 367 -rw-r----- 1 1066 1067 1697 Dec 4 11:31 data_ompha.c 369 The set of synthetic gids on the storage device should be selected 370 such that there is no mapping in any of the name services used by the 371 storage device. I.e., each group should have no members. 373 If the layout segment has an iomode of LAYOUTIOMODE4_READ, then the 374 metadata server should return a synthetic uid that is not set on the 375 storage device. Only the synthetic gid would be valid. 377 The client is thus solely responsible for enforcing file permissions 378 in a loosely coupled model. To allow loghyr write access, it will 379 send an RPC to the storage device with a credential of 1066:1067. To 380 allow garbo read access, it will send an RPC to the storage device 381 with a credential of 1067:1067. The value of the uid does not matter 382 as long as it is not the synthetic uid granted it when getting the 383 layout. 385 While pushing the enforcement of permission checking onto the client 386 may seem to weaken security, the client may already be responsible 387 for enforcing permissions before modifications are sent to a server. 388 With cached writes, the client is always responsible for tracking who 389 is modifying a file and making sure to not coalesce requests from 390 multiple users into one request. 392 2.3. State and Locking Models 394 The choice of locking models is governed by the following rules: 396 o Storage devices implementing the NFSv3 and NFSv4.0 protocols are 397 always treated as loosely coupled. 399 o NFSv4.1+ storage devices that do not return the 400 EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are indicating 401 that they are to be treated as loosely coupled. From the locking 402 viewpoint they are treated in the same way as NFSv4.0 storage 403 devices. 405 o NFSv4.1+ storage devices that do identify themselves with the 406 EXCHGID4_FLAG_USE_PNFS_DS flag set to EXCHANGE_ID are considered 407 strongly coupled. They would use a back-end control protocol to 408 implement the global stateid model as described in [RFC5661]. 410 2.3.1. Loosely Coupled Locking Model 412 When locking-related operations are requested, they are primarily 413 dealt with by the metadata server, which generates the appropriate 414 stateids. When an NFSv4 version is used as the data access protocol, 415 the metadata server may make stateid-related requests of the storage 416 devices. However, it is not required to do so and the resulting 417 stateids are known only to the metadata server and the storage 418 device. 420 Given this basic structure, locking-related operations are handled as 421 follows: 423 o OPENs are dealt with by the metadata server. Stateids are 424 selected by the metadata server and associated with the client id 425 describing the client's connection to the metadata server. The 426 metadata server may need to interact with the storage device to 427 locate the file to be opened, but no locking-related functionality 428 need be used on the storage device. 430 OPEN_DOWNGRADE and CLOSE only require local execution on the 431 metadata sever. 433 o Advisory byte-range locks can be implemented locally on the 434 metadata server. As in the case of OPENs, the stateids associated 435 with byte-range locks are assigned by the metadata server and only 436 used on the metadata server. 438 o Delegations are assigned by the metadata server which initiates 439 recalls when conflicting OPENs are processed. No storage device 440 involvement is required. 442 o TEST_STATEID and FREE_STATEID are processed locally on the 443 metadata server, without storage device involvement. 445 All I/O operations to the storage device are done using the anonymous 446 stateid. Thus the storage device has no information about the 447 openowner and lockowner responsible for issuing a particular I/O 448 operation. As a result: 450 o Mandatory byte-range locking cannot be supported because the 451 storage device has no way of distinguishing I/O done on behalf of 452 the lock owner from those done by others. 454 o Enforcement of share reservations is the responsibility of the 455 client. Even though I/O is done using the anonymous stateid, the 456 client must ensure that it has a valid stateid associated with the 457 openowner, that allows the I/O being done before issuing the I/O. 459 In the event that a stateid is revoked, the metadata server is 460 responsible for preventing client access, since it has no way of 461 being sure that the client is aware that the stateid in question has 462 been revoked. 464 As the client never receives a stateid generated by a storage device, 465 there is no client lease on the storage device and no prospect of 466 lease expiration, even when access is via NFSv4 protocols. Clients 467 will have leases on the metadata server. In dealing with lease 468 expiration, the metadata server may need to use fencing to prevent 469 revoked stateids from being relied upon by a client unaware of the 470 fact that they have been revoked. 472 2.3.2. Tighly Coupled Locking Model 474 When locking-related operations are requested, they are primarily 475 dealt with by the metadata server, which generates the appropriate 476 stateids. These stateids must be made known to the storage device 477 using control protocol facilities, the details of which are not 478 discussed in this document. 480 Given this basic structure, locking-related operations are handled as 481 follows: 483 o OPENs are dealt with primarily on the metadata server. Stateids 484 are selected by the metadata server and associated with the client 485 id describing the client's connection to the metadata server. The 486 metadata server needs to interact with the storage device to 487 locate the file to be opened, and to make the storage device aware 488 of the association between the metadata-sever-chosen stateid and 489 the client and openowner that it represents. 491 OPEN_DOWNGRADE and CLOSE are executed initially on the metadata 492 server but the state change made must be propagated to the storage 493 device. 495 o Advisory byte-range locks can be implemented locally on the 496 metadata server. As in the case of OPENs, the stateids associated 497 with byte-range locks, are assigned by the metadata server and are 498 available for use on the metadata server. Because I/O operations 499 are allowed to present lock stateids, the metadata server needs 500 the ability to make the storage device aware of the association 501 between the metadata-sever-chosen stateid and the corresponding 502 open stateid it is associated with. 504 o Mandatory byte-range locks can be supported when both the metadata 505 server and the storage devices have the appropriate support. As 506 in the case of advisory byte-range locks, these are assigned by 507 the metadata server and are available for use on the metadata 508 server. To enable mandatory lock enforcement on the storage 509 device, the metadata server needs the ability to make the storage 510 device aware of the association between the metadata-sever-chosen 511 stateid and the client, openowner, and lock (i.e., lockowner, 512 byte-range, lock-type) that it represents. Because I/O operations 513 are allowed to present lock stateids, this information needs to be 514 propagated to all storage devices to which I/O might be directed 515 rather than only to daya storage device that contain the locked 516 region. 518 o Delegations are assigned by the metadata server which initiates 519 recalls when conflicting OPENs are processed. Because I/O 520 operations are allowed to present delegation stateids, the 521 metadata server requires the ability to make the storage device 522 aware of the association between the metadata-server-chosen 523 stateid and the filehandle and delegation type it represents, and 524 to break such an association. 526 o TEST_STATEID is processed locally on the metadata server, without 527 storage device involvement. 529 o FREE_STATEID is processed on the metadata server but the metadata 530 server requires the ability to propagate the request to the 531 corresponding storage devices. 533 Because the client will possess and use stateids valid on the storage 534 device, there will be a client lease on the storage device and the 535 possibility of lease expiration does exist. The best approach for 536 the storage device is to retain these locks as a courtesy. However, 537 if it does not do so, control protocol facilities need to provide the 538 means to synchronize lock state between the metadata server and 539 storage device. 541 Clients will also have leases on the metadata server, which are 542 subject to expiration. In dealing with lease expiration, the 543 metadata server would be expected to use control protocol facilities 544 enabling it to invalidate revoked stateids on the storage device. In 545 the event the client is not responsive, the metadata server may need 546 to use fencing to prevent revoked stateids from being acted upon by 547 the storage device. 549 3. XDR Description of the Flexible File Layout Type 551 This document contains the external data representation (XDR) 552 [RFC4506] description of the flexible file layout type. The XDR 553 description is embedded in this document in a way that makes it 554 simple for the reader to extract into a ready-to-compile form. The 555 reader can feed this document into the following shell script to 556 produce the machine readable XDR description of the flexible file 557 layout type: 559 561 #!/bin/sh 562 grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' 564 566 That is, if the above script is stored in a file called "extract.sh", 567 and this document is in a file called "spec.txt", then the reader can 568 do: 570 sh extract.sh < spec.txt > flex_files_prot.x 572 The effect of the script is to remove leading white space from each 573 line, plus a sentinel sequence of "///". 575 The embedded XDR file header follows. Subsequent XDR descriptions, 576 with the sentinel sequence are embedded throughout the document. 578 Note that the XDR code contained in this document depends on types 579 from the NFSv4.1 nfs4_prot.x file [RFC5662]. This includes both nfs 580 types that end with a 4, such as offset4, length4, etc., as well as 581 more generic types such as uint32_t and uint64_t. 583 3.1. Code Components Licensing Notice 585 Both the XDR description and the scripts used for extracting the XDR 586 description are Code Components as described in Section 4 of "Legal 587 Provisions Relating to IETF Documents" [LEGAL]. These Code 588 Components are licensed according to the terms of that document. 590 592 /// /* 593 /// * Copyright (c) 2012 IETF Trust and the persons identified 594 /// * as authors of the code. All rights reserved. 595 /// * 596 /// * Redistribution and use in source and binary forms, with 597 /// * or without modification, are permitted provided that the 598 /// * following conditions are met: 599 /// * 600 /// * o Redistributions of source code must retain the above 601 /// * copyright notice, this list of conditions and the 602 /// * following disclaimer. 603 /// * 604 /// * o Redistributions in binary form must reproduce the above 605 /// * copyright notice, this list of conditions and the 606 /// * following disclaimer in the documentation and/or other 607 /// * materials provided with the distribution. 608 /// * 609 /// * o Neither the name of Internet Society, IETF or IETF 610 /// * Trust, nor the names of specific contributors, may be 611 /// * used to endorse or promote products derived from this 612 /// * software without specific prior written permission. 613 /// * 614 /// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS 615 /// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED 616 /// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 617 /// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 618 /// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 619 /// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 620 /// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 621 /// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 622 /// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 623 /// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 624 /// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 625 /// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 626 /// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 627 /// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 628 /// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 629 /// * 630 /// * This code was derived from RFCTBD10. 631 /// * Please reproduce this note if possible. 632 /// */ 633 /// 634 /// /* 635 /// * flex_files_prot.x 636 /// */ 637 /// 638 /// /* 639 /// * The following include statements are for example only. 640 /// * The actual XDR definition files are generated separately 641 /// * and independently and are likely to have a different name. 642 /// * %#include 643 /// * %#include 644 /// */ 645 /// 647 649 4. Device Addressing and Discovery 651 Data operations to a storage device require the client to know the 652 network address of the storage device. The NFSv4.1+ GETDEVICEINFO 653 operation (Section 18.40 of [RFC5661]) is used by the client to 654 retrieve that information. 656 4.1. ff_device_addr4 658 The ff_device_addr4 data structure is returned by the server as the 659 storage protocol specific opaque field da_addr_body in the 660 device_addr4 structure by a successful GETDEVICEINFO operation. 662 664 /// struct ff_device_versions4 { 665 /// uint32_t ffdv_version; 666 /// uint32_t ffdv_minorversion; 667 /// uint32_t ffdv_rsize; 668 /// uint32_t ffdv_wsize; 669 /// bool ffdv_tightly_coupled; 670 /// }; 671 /// 672 /// struct ff_device_addr4 { 673 /// multipath_list4 ffda_netaddrs; 674 /// ff_device_versions4 ffda_versions<>; 675 /// }; 676 /// 678 680 The ffda_netaddrs field is used to locate the storage device. It 681 MUST be set by the server to a list holding one or more of the device 682 network addresses. 684 The ffda_versions array allows the metadata server to present choices 685 as to NFS version, minor version, and coupling strength to the 686 client. The ffdv_version and ffdv_minorversion represent the NFS 687 protocol to be used to access the storage device. This layout 688 specification defines the semantics for ffdv_versions 3 and 4. If 689 ffdv_version equals 3 then the server MUST set ffdv_minorversion to 0 690 and ffdv_tightly_coupled to false. The client MUST then access the 691 storage device using the NFSv3 protocol [RFC1813]. If ffdv_version 692 equals 4 then the server MUST set ffdv_minorversion to one of the 693 NFSv4 minor version numbers and the client MUST access the storage 694 device using NFSv4 with the specified minor version. 696 Note that while the client might determine that it cannot use any of 697 the configured combinations of ffdv_version, ffdv_minorversion, and 698 ffdv_tightly_coupled, when it gets the device list from the metadata 699 server, there is no way to indicate to the metadata server as to 700 which device it is version incompatible. If however, the client 701 waits until it retrieves the layout from the metadata server, it can 702 at that time clearly identify the storage device in question (see 703 Section 5.3). 705 The ffdv_rsize and ffdv_wsize are used to communicate the maximum 706 rsize and wsize supported by the storage device. As the storage 707 device can have a different rsize or wsize than the metadata server, 708 the ffdv_rsize and ffdv_wsize allow the metadata server to 709 communicate that information on behalf of the storage device. 711 ffdv_tightly_coupled informs the client as to whether the metadata 712 server is tightly coupled with the storage devices or not. Note that 713 even if the data protocol is at least NFSv4.1, it may still be the 714 case that there is loose coupling is in effect. If 715 ffdv_tightly_coupled is not set, then the client MUST commit writes 716 to the storage devices for the file before sending a LAYOUTCOMMIT to 717 the metadata server. I.e., the writes MUST be committed by the 718 client to stable storage via issuing WRITEs with stable_how == 719 FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how != 720 FILE_SYNC (see Section 3.3.7 of [RFC1813]). 722 4.2. Storage Device Multipathing 724 The flexible file layout type supports multipathing to multiple 725 storage device addresses. Storage device level multipathing is used 726 for bandwidth scaling via trunking and for higher availability of use 727 in the event of a storage device failure. Multipathing allows the 728 client to switch to another storage device address which may be that 729 of another storage device that is exporting the same data stripe 730 unit, without having to contact the metadata server for a new layout. 732 To support storage device multipathing, ffda_netaddrs contains an 733 array of one or more storage device network addresses. This array 734 (data type multipath_list4) represents a list of storage devices 735 (each identified by a network address), with the possibility that 736 some storage device will appear in the list multiple times. 738 The client is free to use any of the network addresses as a 739 destination to send storage device requests. If some network 740 addresses are less desirable paths to the data than others, then the 741 MDS SHOULD NOT include those network addresses in ffda_netaddrs. If 742 less desirable network addresses exist to provide failover, the 743 RECOMMENDED method to offer the addresses is to provide them in a 744 replacement device-ID-to-device-address mapping, or a replacement 745 device ID. When a client finds no response from the storage device 746 using all addresses available in ffda_netaddrs, it SHOULD send a 747 GETDEVICEINFO to attempt to replace the existing device-ID-to-device- 748 address mappings. If the MDS detects that all network paths 749 represented by ffda_netaddrs are unavailable, the MDS SHOULD send a 750 CB_NOTIFY_DEVICEID (if the client has indicated it wants device ID 751 notifications for changed device IDs) to change the device-ID-to- 752 device-address mappings to the available addresses. If the device ID 753 itself will be replaced, the MDS SHOULD recall all layouts with the 754 device ID, and thus force the client to get new layouts and device ID 755 mappings via LAYOUTGET and GETDEVICEINFO. 757 Generally, if two network addresses appear in ffda_netaddrs, they 758 will designate the same storage device. When the storage device is 759 accessed over NFSv4.1 or a higher minor version, the two storage 760 device addresses will support the implementation of client ID or 761 session trunking (the latter is RECOMMENDED) as defined in [RFC5661]. 762 The two storage device addresses will share the same server owner or 763 major ID of the server owner. It is not always necessary for the two 764 storage device addresses to designate the same storage device with 765 trunking being used. For example, the data could be read-only, and 766 the data consist of exact replicas. 768 5. Flexible File Layout Type 770 The layout4 type is defined in [RFC5662] as follows: 772 774 enum layouttype4 { 775 LAYOUT4_NFSV4_1_FILES = 1, 776 LAYOUT4_OSD2_OBJECTS = 2, 777 LAYOUT4_BLOCK_VOLUME = 3, 778 LAYOUT4_FLEX_FILES = 4 779 [[RFC Editor: please modify the LAYOUT4_FLEX_FILES 780 to be the layouttype assigned by IANA]] 781 }; 783 struct layout_content4 { 784 layouttype4 loc_type; 785 opaque loc_body<>; 786 }; 788 struct layout4 { 789 offset4 lo_offset; 790 length4 lo_length; 791 layoutiomode4 lo_iomode; 792 layout_content4 lo_content; 793 }; 795 797 This document defines structure associated with the layouttype4 value 798 LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an 799 XDR type "opaque". The opaque layout is uninterpreted by the generic 800 pNFS client layers, but is interpreted by the flexible file layout 801 type implementation. This section defines the structure of this 802 otherwise opaque value, ff_layout4. 804 5.1. ff_layout4 806 808 /// const FF_FLAGS_NO_LAYOUTCOMMIT = 0x00000001; 809 /// const FF_FLAGS_NO_IO_THRU_MDS = 0x00000002; 810 /// const FF_FLAGS_NO_READ_IO = 0x00000004; 812 /// typedef uint32_t ff_flags4; 813 /// 814 /// struct ff_data_server4 { 815 /// deviceid4 ffds_deviceid; 816 /// uint32_t ffds_efficiency; 817 /// stateid4 ffds_stateid; 818 /// nfs_fh4 ffds_fh_vers<>; 819 /// fattr4_owner ffds_user; 820 /// fattr4_owner_group ffds_group; 821 /// }; 822 /// 824 /// struct ff_mirror4 { 825 /// ff_data_server4 ffm_data_servers<>; 826 /// }; 827 /// 829 /// struct ff_layout4 { 830 /// length4 ffl_stripe_unit; 831 /// ff_mirror4 ffl_mirrors<>; 832 /// ff_flags4 ffl_flags; 833 /// uint32_t ffl_stats_collect_hint; 834 /// }; 835 /// 837 839 The ff_layout4 structure specifies a layout over a set of mirrored 840 copies of that portion of the data file described in the current 841 layout segment. This mirroring protects against loss of data in 842 layout segments. Note that while not explicitly shown in the above 843 XDR, each layout4 element returned in the logr_layout array of 844 LAYOUTGET4res (see Section 18.43.1 of [RFC5661]) describes a layout 845 segment. Hence each ff_layout4 also describes a layout segment. 847 It is possible that the file is concatenated from more than one 848 layout segment. Each layout segment MAY represent different striping 849 parameters, applying respectively only to the layout segment byte 850 range. 852 The ffl_stripe_unit field is the stripe unit size in use for the 853 current layout segment. The number of stripes is given inside each 854 mirror by the number of elements in ffm_data_servers. If the number 855 of stripes is one, then the value for ffl_stripe_unit MUST default to 856 zero. The only supported mapping scheme is sparse and is detailed in 857 Section 6. Note that there is an assumption here that both the 858 stripe unit size and the number of stripes is the same across all 859 mirrors. 861 The ffl_mirrors field is the array of mirrored storage devices which 862 provide the storage for the current stripe, see Figure 1. 864 The ffl_stats_collect_hint field provides a hint to the client on how 865 often the server wants it to report LAYOUTSTATS for a file. The time 866 is in seconds. 868 +-----------+ 869 | | 870 | | 871 | File | 872 | | 873 | | 874 +-----+-----+ 875 | 876 +------------+------------+ 877 | | 878 +----+-----+ +-----+----+ 879 | Mirror 1 | | Mirror 2 | 880 +----+-----+ +-----+----+ 881 | | 882 +-----------+ +-----------+ 883 |+-----------+ |+-----------+ 884 ||+-----------+ ||+-----------+ 885 +|| Storage | +|| Storage | 886 +| Devices | +| Devices | 887 +-----------+ +-----------+ 889 Figure 1 891 The ffs_mirrors field represents an array of state information for 892 each mirrored copy of the current layout segment. Each element is 893 described by a ff_mirror4 type. 895 ffds_deviceid provides the deviceid of the storage device holding the 896 data file. 898 ffds_fh_vers is an array of filehandles of the data file matching to 899 the available NFS versions on the given storage device. There MUST 900 be exactly as many elements in ffds_fh_vers as there are in 901 ffda_versions. Each element of the array corresponds to a particular 902 combination of ffdv_version, ffdv_minorversion, and 903 ffdv_tightly_coupled provided for the device. The array allows for 904 server implementations which have different filehandles for different 905 combinations of version, minor version, and coupling strength. See 906 Section 5.3 for how to handle versioning issues between the client 907 and storage devices. 909 For tight coupling, ffds_stateid provides the stateid to be used by 910 the client to access the file. For loose coupling and a NFSv4 911 storage device, the client may use an anonymous stateid to perform I/ 912 O on the storage device as there is no use for the metadata server 913 stateid (no control protocol). In such a scenario, the server MUST 914 set the ffds_stateid to be the anonymous stateid. 916 This specification of the ffds_stateid is mostly broken for the 917 tightly coupled model. There needs to exist a one to one mapping 918 from ffds_stateid to ffds_fh_vers - each open file on the storage 919 device might need an open stateid. As there are established loosely 920 coupled implementations of this version of the protocol, the only 921 viable approaches for a tightly coupled implementation would be to 922 either use an anonymous stateid for the ffds_stateid or restrict the 923 size of the ffds_fh_vers to be one. Fixing this issue will require a 924 new version of the protocol. 926 [[AI14: One reviewer points out for loosely coupled, we can use the 927 anon stateid and for tightly coupled we can use the "global stateid". 928 These make it appear that the bug in the spec was actually a feature. 929 The intent here is to own up to the bug and shipping code. Can it be 930 said nicer? --TH]] 932 For loosely coupled storage devices, ffds_user and ffds_group provide 933 the synthetic user and group to be used in the RPC credentials that 934 the client presents to the storage device to access the data files. 935 For tightly coupled storage devices, the user and group on the 936 storage device will be the same as on the metadata server. I.e., if 937 ffdv_tightly_coupled (see Section 4.1) is set, then the client MUST 938 ignore both ffds_user and ffds_group. 940 The allowed values for both ffds_user and ffds_group are specified in 941 Section 5.9 of [RFC5661]. For NFSv3 compatibility, user and group 942 strings that consist of decimal numeric values with no leading zeros 943 can be given a special interpretation by clients and servers that 944 choose to provide such support. The receiver may treat such a user 945 or group string as representing the same user as would be represented 946 by an NFSv3 uid or gid having the corresponding numeric value. Note 947 that if using Kerberos for security, the expectation is that these 948 values will be a name@domain string. 950 ffds_efficiency describes the metadata server's evaluation as to the 951 effectiveness of each mirror. Note that this is per layout and not 952 per device as the metric may change due to perceived load, 953 availability to the metadata server, etc. Higher values denote 954 higher perceived utility. The way the client can select the best 955 mirror to access is discussed in Section 8.1. 957 ffl_flags is a bitmap that allows the metadata server to inform the 958 client of particular conditions that may result from the more or less 959 tight coupling of the storage devices. 961 FF_FLAGS_NO_LAYOUTCOMMIT: can be set to indicate that the client is 962 not required to send LAYOUTCOMMIT to the metadata server. 964 F_FLAGS_NO_IO_THRU_MDS: can be set to indicate that the client 965 SHOULD not send I/O operations to the metadata server. I.e., even 966 if the client could determine that there was a network diconnect 967 to a storage device, the client SHOULD not try to proxy the I/O 968 through the metadata server. 970 FF_FLAGS_NO_READ_IO: can be set to indicate that the client SHOULD 971 not send READ requests with the layouts of iomode 972 LAYOUTIOMODE4_RW. Instead, it should request a layout of iomode 973 LAYOUTIOMODE4_READ from the metadata server. 975 5.1.1. Error Codes from LAYOUTGET 977 [RFC5661] provides little guidance as to how the client is to proceed 978 with a LAYOUTEGT which returns an error of either 979 NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, and NFS4ERR_DELAY. 980 Within the context of this document: 982 NFS4ERR_LAYOUTUNAVAILABLE: there is no layout available and the I/O 983 is to go to the metadata server. Note that it is possible to have 984 had a layout before a recall and not after. 986 NFS4ERR_LAYOUTTRYLATER: there is some issue preventing the layout 987 from being granted. If the client already has an appropriate 988 layout, it should continue with I/O to the storage devices. 990 NFS4ERR_DELAY: there is some issue preventing the layout from being 991 granted. If the client already has an appropriate layout, it 992 should not continue with I/O to the storage devices. 994 5.1.2. Client Interactions with FF_FLAGS_NO_IO_THRU_MDS 996 Even if the metadata server provides the FF_FLAGS_NO_IO_THRU_MDS, 997 flag, the client can still perform I/O to the metadata server. The 998 flag is at best a hint. The flag is indicating to the client that 999 the metadata server most likely wants to separate the metadata I/O 1000 from the data I/O to increase the performance of the metadata 1001 operations. If the metadata server detects that the client is 1002 performing I/O against it despite the use of the 1003 FF_FLAGS_NO_IO_THRU_MDS flag, it can recall the layout and either not 1004 set the flag on the new layout or not provide a layout (perhaps the 1005 intent was for the server to temporarily prevent data I/O to meet 1006 some goal). The client's I/O would then proceed according to the 1007 status codes as outlined in Section 5.1.1. 1009 5.2. Interactions Between Devices and Layouts 1011 In [RFC5661], the file layout type is defined such that the 1012 relationship between multipathing and filehandles can result in 1013 either 0, 1, or N filehandles (see Section 13.3). Some rationals for 1014 this are clustered servers which share the same filehandle or 1015 allowing for multiple read-only copies of the file on the same 1016 storage device. In the flexible file layout type, while there is an 1017 array of filehandles, they are independent of the multipathing being 1018 used. If the metadata server wants to provide multiple read-only 1019 copies of the same file on the same storage device, then it should 1020 provide multiple ff_device_addr4, each as a mirror. The client can 1021 then determine that since the ffds_fh_vers are different, then there 1022 are multiple copies of the file for the current layout segment 1023 available. 1025 5.3. Handling Version Errors 1027 When the metadata server provides the ffda_versions array in the 1028 ff_device_addr4 (see Section 4.1), the client is able to determine if 1029 it can not access a storage device with any of the supplied 1030 combinations of ffdv_version, ffdv_minorversion, and 1031 ffdv_tightly_coupled. However, due to the limitations of reporting 1032 errors in GETDEVICEINFO (see Section 18.40 in [RFC5661], the client 1033 is not able to specify which specific device it can not communicate 1034 with over one of the provided ffdv_version and ffdv_minorversion 1035 combinations. Using ff_ioerr4 (see Section 9.1.1 inside either the 1036 LAYOUTRETURN (see Section 18.44 of [RFC5661]) or the LAYOUTERROR (see 1037 Section 15.6 of [RFC7862] and Section 10 of this document), the 1038 client can isolate the problematic storage device. 1040 The error code to return for LAYOUTRETURN and/or LAYOUTERROR is 1041 NFS4ERR_MINOR_VERS_MISMATCH. It does not matter whether the mismatch 1042 is a major version (e.g., client can use NFSv3 but not NFSv4) or 1043 minor version (e.g., client can use NFSv4.1 but not NFSv4.2), the 1044 error indicates that for all the supplied combinations for 1045 ffdv_version and ffdv_minorversion, the client can not communicate 1046 with the storage device. The client can retry the GETDEVICEINFO to 1047 see if the metadata server can provide a different combination or it 1048 can fall back to doing the I/O through the metadata server. 1050 6. Striping via Sparse Mapping 1052 While other layout types support both dense and sparse mapping of 1053 logical offsets to physical offsets within a file (see for example 1054 Section 13.4 of [RFC5661]), the flexible file layout type only 1055 supports a sparse mapping. 1057 With sparse mappings, the logical offset within a file (L) is also 1058 the physical offset on the storage device. As detailed in 1059 Section 13.4.4 of [RFC5661], this results in holes across each 1060 storage device which does not contain the current stripe index. 1062 L: logical offset into the file 1064 W: stripe width 1065 W = number of elements in ffm_data_servers 1067 S: number of bytes in a stripe 1068 S = W * ffl_stripe_unit 1070 N: stripe number 1071 N = L / S 1073 7. Recovering from Client I/O Errors 1075 The pNFS client may encounter errors when directly accessing the 1076 storage devices. However, it is the responsibility of the metadata 1077 server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES 1078 layout type is used, the client MUST report the I/O errors to the 1079 server at LAYOUTRETURN time using the ff_ioerr4 structure (see 1080 Section 9.1.1). 1082 The metadata server analyzes the error and determines the required 1083 recovery operations such as recovering media failures or 1084 reconstructing missing data files. 1086 The metadata server SHOULD recall any outstanding layouts to allow it 1087 exclusive write access to the stripes being recovered and to prevent 1088 other clients from hitting the same error condition. In these cases, 1089 the server MUST complete recovery before handing out any new layouts 1090 to the affected byte ranges. 1092 Although the client implementation has the option to propagate a 1093 corresponding error to the application that initiated the I/O 1094 operation and drop any unwritten data, the client should attempt to 1095 retry the original I/O operation by either requesting a new layout or 1096 sending the I/O via regular NFSv4.1+ READ or WRITE operations to the 1097 metadata server. The client SHOULD attempt to retrieve a new layout 1098 and retry the I/O operation using the storage device first and only 1099 if the error persists, retry the I/O operation via the metadata 1100 server. 1102 8. Mirroring 1104 The flexible file layout type has a simple model in place for the 1105 mirroring of the file data constrained by a layout segment. There is 1106 no assumption that each copy of the mirror is stored identically on 1107 the storage devices. For example, one device might employ 1108 compression or deduplication on the data. However, the over the wire 1109 transfer of the file contents MUST appear identical. Note, this is a 1110 constraint of the selected XDR representation in which each mirrored 1111 copy of the layout segment has the same striping pattern (see 1112 Figure 1). 1114 The metadata server is responsible for determining the number of 1115 mirrored copies and the location of each mirror. While the client 1116 may provide a hint to how many copies it wants (see Section 12), the 1117 metadata server can ignore that hint and in any event, the client has 1118 no means to dictate neither the storage device (which also means the 1119 coupling and/or protocol levels to access the layout segments) nor 1120 the location of said storage device. 1122 The updating of mirrored layout segments is done via client-side 1123 mirroring. With this approach, the client is responsible for making 1124 sure modifications are made on all copies of the layout segments it 1125 is informed of via the layout. If a layout segment is being 1126 resilvered to a storage device, that mirrored copy will not be in the 1127 layout. Thus the metadata server MUST update that copy until the 1128 client is presented it in a layout. If the client is writing to the 1129 layout segments via the metadata server, then the metadata server 1130 MUST update all copies of the mirror. As seen in Section 8.3, during 1131 the resilvering, the layout is recalled, and the client has to make 1132 modifications via the metadata server. 1134 8.1. Selecting a Mirror 1136 When the metadata server grants a layout to a client, it MAY let the 1137 client know how fast it expects each mirror to be once the request 1138 arrives at the storage devices via the ffds_efficiency member. While 1139 the algorithms to calculate that value are left to the metadata 1140 server implementations, factors that could contribute to that 1141 calculation include speed of the storage device, physical memory 1142 available to the device, operating system version, current load, etc. 1144 However, what should not be involved in that calculation is a 1145 perceived network distance between the client and the storage device. 1147 The client is better situated for making that determination based on 1148 past interaction with the storage device over the different available 1149 network interfaces between the two. I.e., the metadata server might 1150 not know about a transient outage between the client and storage 1151 device because it has no presence on the given subnet. 1153 As such, it is the client which decides which mirror to access for 1154 reading the file. The requirements for writing to a mirrored layout 1155 segments are presented below. 1157 8.2. Writing to Mirrors 1159 The client is responsible for updating all mirrored copies of the 1160 layout segments that it is given in the layout. A single failed 1161 update is sufficient to fail the entire operation. If all but one 1162 copy is updated successfully and the last one provides an error, then 1163 the client needs to inform the metadata server about the error via 1164 either LAYOUTRETURN or LAYOUTERROR that the update failed to that 1165 storage device. If the client is updating the mirrors serially, then 1166 it SHOULD stop at the first error encountered and report that to the 1167 metadata server. If the client is updating the mirrors in parallel, 1168 then it SHOULD wait until all storage devices respond such that it 1169 can report all errors encountered during the update. 1171 The metadata server is then responsible for determining if it wants 1172 to remove the errant mirror from the layout, if the mirror has 1173 recovered from some transient error, etc. When the client tries to 1174 get a new layout, the metadata server informs it of the decision by 1175 the contents of the layout. The client MUST NOT make any assumptions 1176 that the contents of the previous layout will match those of the new 1177 one. If it has updates that were not committed to all mirrors, then 1178 it MUST resend those updates to all mirrors. 1180 There is no provision in the protocol for the metadata server to 1181 directly determine that the client has or has not recovered from an 1182 error. I.e., assume that the storage device was network partitioned 1183 from the client and all of the copies are successfully updated after 1184 the error was reported. There is no mechanism for the client to 1185 report that fact and the metadata server is forced to repair the file 1186 across the mirror. 1188 If the client supports NFSv4.2, it can use LAYOUTERROR and 1189 LAYOUTRETURN to provide hints to the metadata server about the 1190 recovery efforts. A LAYOUTERROR on a file is for a non-fatal error. 1191 A subsequent LAYOUTRETURN without a ff_ioerr4 indicates that the 1192 client successfully replayed the I/O to all mirrors. Any 1193 LAYOUTRETURN with a ff_ioerr4 is an error that the metadata server 1194 needs to repair. The client MUST be prepared for the LAYOUTERROR to 1195 trigger a CB_LAYOUTRECALL if the metadata server determines it needs 1196 to start repairing the file. 1198 8.3. Metadata Server Resilvering of the File 1200 The metadata server may elect to create a new mirror of the layout 1201 segments at any time. This might be to resilver a copy on a storage 1202 device which was down for servicing, to provide a copy of the layout 1203 segments on storage with different storage performance 1204 characteristics, etc. As the client will not be aware of the new 1205 mirror and the metadata server will not be aware of updates that the 1206 client is making to the layout segments, the metadata server MUST 1207 recall the writable layout segment(s) that it is resilvering. If the 1208 client issues a LAYOUTGET for a writable layout segment which is in 1209 the process of being resilvered, then the metadata server can deny 1210 that request with a NFS4ERR_LAYOUTUNAVAILABLE. The client would then 1211 have to perform the I/O through the metadata server. 1213 9. Flexible Files Layout Type Return 1215 layoutreturn_file4 is used in the LAYOUTRETURN operation to convey 1216 layout-type specific information to the server. It is defined in 1217 Section 18.44.1 of [RFC5661] as follows: 1219 1221 /* Constants used for LAYOUTRETURN and CB_LAYOUTRECALL */ 1222 const LAYOUT4_RET_REC_FILE = 1; 1223 const LAYOUT4_RET_REC_FSID = 2; 1224 const LAYOUT4_RET_REC_ALL = 3; 1226 enum layoutreturn_type4 { 1227 LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE, 1228 LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID, 1229 LAYOUTRETURN4_ALL = LAYOUT4_RET_REC_ALL 1230 }; 1232 struct layoutreturn_file4 { 1233 offset4 lrf_offset; 1234 length4 lrf_length; 1235 stateid4 lrf_stateid; 1236 /* layouttype4 specific data */ 1237 opaque lrf_body<>; 1238 }; 1239 union layoutreturn4 switch(layoutreturn_type4 lr_returntype) { 1240 case LAYOUTRETURN4_FILE: 1241 layoutreturn_file4 lr_layout; 1242 default: 1243 void; 1244 }; 1246 struct LAYOUTRETURN4args { 1247 /* CURRENT_FH: file */ 1248 bool lora_reclaim; 1249 layoutreturn_stateid lora_recallstateid; 1250 layouttype4 lora_layout_type; 1251 layoutiomode4 lora_iomode; 1252 layoutreturn4 lora_layoutreturn; 1253 }; 1255 1257 If the lora_layout_type layout type is LAYOUT4_FLEX_FILES and the 1258 lr_returntype is LAYOUTRETURN4_FILE, then the lrf_body opaque value 1259 is defined by ff_layoutreturn4 (See Section 9.3). It allows the 1260 client to report I/O error information or layout usage statistics 1261 back to the metadata server as defined below. Note that while the 1262 data strucures are built on concepts introduced in NFSv4.2, the 1263 effective discriminated union (lora_layout_type combined with 1264 ff_layoutreturn4) allows for a NFSv4.1 metadata server to utilize the 1265 data. 1267 9.1. I/O Error Reporting 1269 9.1.1. ff_ioerr4 1271 1273 /// struct ff_ioerr4 { 1274 /// offset4 ffie_offset; 1275 /// length4 ffie_length; 1276 /// stateid4 ffie_stateid; 1277 /// device_error4 ffie_errors<>; 1278 /// }; 1279 /// 1281 1283 Recall that [RFC7862] defines device_error4 as: 1285 1286 struct device_error4 { 1287 deviceid4 de_deviceid; 1288 nfsstat4 de_status; 1289 nfs_opnum4 de_opnum; 1290 }; 1292 1294 The ff_ioerr4 structure is used to return error indications for data 1295 files that generated errors during data transfers. These are hints 1296 to the metadata server that there are problems with that file. For 1297 each error, ffie_errors.de_deviceid, ffie_offset, and ffie_length 1298 represent the storage device and byte range within the file in which 1299 the error occurred; ffie_errors represents the operation and type of 1300 error. The use of device_error4 is described in Section 15.6 of 1301 [RFC7862]. 1303 Even though the storage device might be accessed via NFSv3 and 1304 reports back NFSv3 errors to the client, the client is responsible 1305 for mapping these to appropriate NFSv4 status codes as de_status. 1306 Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 1307 operations. 1309 9.2. Layout Usage Statistics 1311 9.2.1. ff_io_latency4 1313 1315 /// struct ff_io_latency4 { 1316 /// uint64_t ffil_ops_requested; 1317 /// uint64_t ffil_bytes_requested; 1318 /// uint64_t ffil_ops_completed; 1319 /// uint64_t ffil_bytes_completed; 1320 /// uint64_t ffil_bytes_not_delivered; 1321 /// nfstime4 ffil_total_busy_time; 1322 /// nfstime4 ffil_aggregate_completion_time; 1323 /// }; 1324 /// 1326 1328 Both operation counts and bytes transferred are kept in the 1329 ff_io_latency4. READ operations are used for read latencies. Both 1330 WRITE and COMMIT operations are used for write latencies. 1331 "Requested" counters track what the client is attempting to do and 1332 "completed" counters track what was done. Note that there is no 1333 requirement that the client only report completed results that have 1334 matching requested results from the reported period. 1336 ffil_bytes_not_delivered is used to track the aggregate number of 1337 bytes requested by not fulfilled due to error conditions. 1338 ffil_total_busy_time is the aggregate time spent with outstanding RPC 1339 calls, ffil_aggregate_completion_time is the sum of all latencies for 1340 completed RPC calls. 1342 Note that LAYOUTSTATS are cumulative, i.e., not reset each time the 1343 operation is sent. If two LAYOUTSTATS ops for the same file, layout 1344 stateid, and originating from the same NFS client are processed at 1345 the same time by the metadata server, then the one containing the 1346 larger values contains the most recent time series data. 1348 9.2.2. ff_layoutupdate4 1350 1352 /// struct ff_layoutupdate4 { 1353 /// netaddr4 ffl_addr; 1354 /// nfs_fh4 ffl_fhandle; 1355 /// ff_io_latency4 ffl_read; 1356 /// ff_io_latency4 ffl_write; 1357 /// nfstime4 ffl_duration; 1358 /// bool ffl_local; 1359 /// }; 1360 /// 1362 1364 ffl_addr differentiates which network address the client connected to 1365 on the storage device. In the case of multipathing, ffl_fhandle 1366 indicates which read-only copy was selected. ffl_read and ffl_write 1367 convey the latencies respectively for both read and write operations. 1368 ffl_duration is used to indicate the time period over which the 1369 statistics were collected. ffl_local if true indicates that the I/O 1370 was serviced by the client's cache. This flag allows the client to 1371 inform the metadata server about "hot" access to a file it would not 1372 normally be allowed to report on. 1374 9.2.3. ff_iostats4 1376 1377 /// struct ff_iostats4 { 1378 /// offset4 ffis_offset; 1379 /// length4 ffis_length; 1380 /// stateid4 ffis_stateid; 1381 /// io_info4 ffis_read; 1382 /// io_info4 ffis_write; 1383 /// deviceid4 ffis_deviceid; 1384 /// ff_layoutupdate4 ffis_layoutupdate; 1385 /// }; 1386 /// 1388 1390 Recall that [RFC7862] defines io_info4 as: 1392 1394 struct io_info4 { 1395 uint64_t ii_count; 1396 uint64_t ii_bytes; 1397 }; 1399 1401 With pNFS, data transfers are performed directly between the pNFS 1402 client and the storage devices. Therefore, the metadata server has 1403 no direct knowledge to the I/O operations being done and thus can not 1404 create on its own statistical information about client I/O to 1405 optimize data storage location. ff_iostats4 MAY be used by the 1406 client to report I/O statistics back to the metadata server upon 1407 returning the layout. 1409 Since it is not feasible for the client to report every I/O that used 1410 the layout, the client MAY identify "hot" byte ranges for which to 1411 report I/O statistics. The definition and/or configuration mechanism 1412 of what is considered "hot" and the size of the reported byte range 1413 is out of the scope of this document. It is suggested for client 1414 implementation to provide reasonable default values and an optional 1415 run-time management interface to control these parameters. For 1416 example, a client can define the default byte range resolution to be 1417 1 MB in size and the thresholds for reporting to be 1 MB/second or 10 1418 I/O operations per second. 1420 For each byte range, ffis_offset and ffis_length represent the 1421 starting offset of the range and the range length in bytes. 1422 ffis_read.ii_count, ffis_read.ii_bytes, ffis_write.ii_count, and 1423 ffis_write.ii_bytes represent, respectively, the number of contiguous 1424 read and write I/Os and the respective aggregate number of bytes 1425 transferred within the reported byte range. 1427 The combination of ffis_deviceid and ffl_addr uniquely identifies 1428 both the storage path and the network route to it. Finally, the 1429 ffl_fhandle allows the metadata server to differentiate between 1430 multiple read-only copies of the file on the same storage device. 1432 9.3. ff_layoutreturn4 1434 1436 /// struct ff_layoutreturn4 { 1437 /// ff_ioerr4 fflr_ioerr_report<>; 1438 /// ff_iostats4 fflr_iostats_report<>; 1439 /// }; 1440 /// 1442 1444 When data file I/O operations fail, fflr_ioerr_report<> is used to 1445 report these errors to the metadata server as an array of elements of 1446 type ff_ioerr4. Each element in the array represents an error that 1447 occurred on the data file identified by ffie_errors.de_deviceid. If 1448 no errors are to be reported, the size of the fflr_ioerr_report<> 1449 array is set to zero. The client MAY also use fflr_iostats_report<> 1450 to report a list of I/O statistics as an array of elements of type 1451 ff_iostats4. Each element in the array represents statistics for a 1452 particular byte range. Byte ranges are not guaranteed to be disjoint 1453 and MAY repeat or intersect. 1455 10. Flexible Files Layout Type LAYOUTERROR 1457 If the client is using NFSv4.2 to communicate with the metadata 1458 server, then instead of waiting for a LAYOUTRETURN to send error 1459 information to the metadata server (see Section 9.1), it MAY use 1460 LAYOUTERROR (see Section 15.6 of [RFC7862]) to communicate that 1461 information. For the flexible files layout type, this means that 1462 LAYOUTERROR4args is treated the same as ff_ioerr4. 1464 11. Flexible Files Layout Type LAYOUTSTATS 1466 If the client is using NFSv4.2 to communicate with the metadata 1467 server, then instead of waiting for a LAYOUTRETURN to send I/O 1468 statistics to the metadata server (see Section 9.2), it MAY use 1469 LAYOUTSTATS (see Section 15.7 of [RFC7862]) to communicate that 1470 information. For the flexible files layout type, this means that 1471 LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same 1472 contents as in ffis_layoutupdate. 1474 12. Flexible File Layout Type Creation Hint 1476 The layouthint4 type is defined in the [RFC5661] as follows: 1478 1480 struct layouthint4 { 1481 layouttype4 loh_type; 1482 opaque loh_body<>; 1483 }; 1485 1487 The layouthint4 structure is used by the client to pass a hint about 1488 the type of layout it would like created for a particular file. If 1489 the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body 1490 opaque value is defined by the ff_layouthint4 type. 1492 12.1. ff_layouthint4 1494 1496 /// union ff_mirrors_hint switch (bool ffmc_valid) { 1497 /// case TRUE: 1498 /// uint32_t ffmc_mirrors; 1499 /// case FALSE: 1500 /// void; 1501 /// }; 1502 /// 1504 /// struct ff_layouthint4 { 1505 /// ff_mirrors_hint fflh_mirrors_hint; 1506 /// }; 1507 /// 1509 1511 This type conveys hints for the desired data map. All parameters are 1512 optional so the client can give values for only the parameter it 1513 cares about. 1515 13. Recalling a Layout 1517 While Section 12.5.5 of [RFC5661] discusses layout type independent 1518 reasons for recalling a layout, the flexible file layout type 1519 metadata server should recall outstanding layouts in the following 1520 cases: 1522 o When the file's security policy changes, i.e., Access Control 1523 Lists (ACLs) or permission mode bits are set. 1525 o When the file's layout changes, rendering outstanding layouts 1526 invalid. 1528 o When existing layouts are inconsistent with the need to enforce 1529 locking constraints. 1531 o When existing layouts are inconsistent with the requirements 1532 regarding resilvering as described in Section 8.3. 1534 13.1. CB_RECALL_ANY 1536 The metadata server can use the CB_RECALL_ANY callback operation to 1537 notify the client to return some or all of its layouts. [RFC5661] 1538 defines the allowed types, but makes no provision to expand them. It 1539 does hint that "storage protocols" can expand the range, but does not 1540 define such a process. If we put the values under IANA control, then 1541 we could define the following types: 1543 1545 const RCA4_TYPE_MASK_FF_LAYOUT_MIN = -2; 1546 const RCA4_TYPE_MASK_FF_LAYOUT_MAX = -1; 1547 [[RFC Editor: please insert assigned constants]] 1549 struct CB_RECALL_ANY4args { 1550 uint32_t craa_layouts_to_keep; 1551 bitmap4 craa_type_mask; 1552 }; 1554 1556 Typically, CB_RECALL_ANY will be used to recall client state when the 1557 server needs to reclaim resources. The craa_type_mask bitmap 1558 specifies the type of resources that are recalled and the 1559 craa_layouts_to_keep value specifies how many of the recalled 1560 flexible file layouts the client is allowed to keep. The flexible 1561 file layout type mask flags are defined as follows: 1563 1565 /// enum ff_cb_recall_any_mask { 1566 /// FF_RCA4_TYPE_MASK_READ = -2, 1567 /// FF_RCA4_TYPE_MASK_RW = -1 1568 [[RFC Editor: please insert assigned constants]] 1569 /// }; 1570 /// 1572 1574 They represent the iomode of the recalled layouts. In response, the 1575 client SHOULD return layouts of the recalled iomode that it needs the 1576 least, keeping at most craa_layouts_to_keep Flexible File Layouts. 1578 The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return 1579 layouts of iomode LAYOUTIOMODE4_READ. Similarly, the 1580 PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts 1581 of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client 1582 is notified to return layouts of either iomode. 1584 14. Client Fencing 1586 In cases where clients are uncommunicative and their lease has 1587 expired or when clients fail to return recalled layouts within a 1588 lease period, at the least the server MAY revoke client layouts and 1589 reassign these resources to other clients (see Section 12.5.5 in 1590 [RFC5661]). To avoid data corruption, the metadata server MUST fence 1591 off the revoked clients from the respective data files as described 1592 in Section 2.2. 1594 15. Security Considerations 1596 The pNFS extension partitions the NFSv4.1+ file system protocol into 1597 two parts, the control path and the data path (storage protocol). 1598 The control path contains all the new operations described by this 1599 extension; all existing NFSv4 security mechanisms and features apply 1600 to the control path. The combination of components in a pNFS system 1601 is required to preserve the security properties of NFSv4.1+ with 1602 respect to an entity accessing data via a client, including security 1603 countermeasures to defend against threats that NFSv4.1+ provides 1604 defenses for in environments where these threats are considered 1605 significant. 1607 The metadata server enforces the file access-control policy at 1608 LAYOUTGET time. The client should use RPC authorization credentials 1609 (uid/gid for AUTH_SYS or tickets for Kerberos) for getting the layout 1610 for the requested iomode (READ or RW) and the server verifies the 1611 permissions and ACL for these credentials, possibly returning 1612 NFS4ERR_ACCESS if the client is not allowed the requested iomode. If 1613 the LAYOUTGET operation succeeds the client receives, as part of the 1614 layout, a set of credentials allowing it I/O access to the specified 1615 data files corresponding to the requested iomode. When the client 1616 acts on I/O operations on behalf of its local users, it MUST 1617 authenticate and authorize the user by issuing respective OPEN and 1618 ACCESS calls to the metadata server, similar to having NFSv4 data 1619 delegations. 1621 If access is allowed, the client uses the corresponding (READ or RW) 1622 credentials to perform the I/O operations at the data file's storage 1623 devices. When the metadata server receives a request to change a 1624 file's permissions or ACL, it SHOULD recall all layouts for that file 1625 and then MUST fence off any clients still holding outstanding layouts 1626 for the respective files by implicitly invalidating the previously 1627 distributed credential on all data file comprising the file in 1628 question. It is REQUIRED that this be done before committing to the 1629 new permissions and/or ACL. By requesting new layouts, the clients 1630 will reauthorize access against the modified access control metadata. 1631 Recalling the layouts in this case is intended to prevent clients 1632 from getting an error on I/Os done after the client was fenced off. 1634 15.1. Kerberized File Access 1636 15.1.1. Loosely Coupled 1638 RPCSEC_GSS version 3 (RPCSEC_GSSv3) [rpcsec_gssv3] could be used to 1639 authorize the client to the storage device on behalf of the metadata 1640 server. This would require that each of the metadata server, storage 1641 device, and client would have to implement RPCSEC_GSSv3. The second 1642 requirement does not match the intent of the loosely coupled model 1643 that the storage device need not be modified. 1645 Under this coupling model, the principal used to authenticate the 1646 metadata file is different than that used to authenticate the data 1647 file. For the metadata server, the user credentials would be 1648 generated by the same Kerberos server as the client. However, for 1649 the data storage access, the metadata server would generate the 1650 ticket granting tickets and provide them to the client. Fencing 1651 would then be controlled either by expiring the ticket or by 1652 modifying the syntethic uid or gid on the data file. 1654 15.1.2. Tightly Coupled 1656 With tight coupling, the principal used to access the metadata file 1657 is exactly the same as used to access the data file. As a result 1658 there are no security issues related to using Kerberos with a tightly 1659 coupled system. 1661 16. IANA Considerations 1663 As described in [RFC5661], new layout type numbers have been assigned 1664 by IANA. This document defines the protocol associated with the 1665 existing layout type number, LAYOUT4_FLEX_FILES. 1667 17. References 1669 17.1. Normative References 1671 [LEGAL] IETF Trust, "Legal Provisions Relating to IETF Documents", 1672 November 2008, . 1675 [RFC1813] IETF, "NFS Version 3 Protocol Specification", RFC 1813, 1676 June 1995. 1678 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1679 Requirement Levels", BCP 14, RFC 2119, March 1997. 1681 [RFC4506] Eisler, M., "XDR: External Data Representation Standard", 1682 STD 67, RFC 4506, May 2006. 1684 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 1685 Specification Version 2", RFC 5531, May 2009. 1687 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1688 "Network File System (NFS) Version 4 Minor Version 1 1689 Protocol", RFC 5661, January 2010. 1691 [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1692 "Network File System (NFS) Version 4 Minor Version 1 1693 External Data Representation Standard (XDR) Description", 1694 RFC 5662, January 2010. 1696 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 1697 version 4 Protocol", RFC 7530, March 2015. 1699 [RFC7862] Haynes, T., "NFS Version 4 Minor Version 2", RFC 7862, 1700 November 2016. 1702 [pNFSLayouts] 1703 Haynes, T., "Requirements for pNFS Layout Types", draft- 1704 ietf-nfsv4-layout-types-04 (Work In Progress), January 1705 2016. 1707 17.2. Informative References 1709 [RFC4519] Sciberras, A., Ed., "Lightweight Directory Access Protocol 1710 (LDAP): Schema for User Applications", RFC 4519, DOI 1711 10.17487/RFC4519, June 2006, 1712 . 1714 [rpcsec_gssv3] 1715 Adamson, W. and N. Williams, "Remote Procedure Call (RPC) 1716 Security Version 3", November 2014. 1718 Appendix A. Acknowledgments 1720 Those who provided miscellaneous comments to early drafts of this 1721 document include: Matt W. Benjamin, Adam Emerson, J. Bruce Fields, 1722 and Lev Solomonov. 1724 Those who provided miscellaneous comments to the final drafts of this 1725 document include: Anand Ganesh, Robert Wipfel, Gobikrishnan 1726 Sundharraj, and Trond Myklebust. 1728 Idan Kedar caught a nasty bug in the interaction of client side 1729 mirroring and the minor versioning of devices. 1731 Dave Noveck provided comprehensive reviews of the document during the 1732 working group last calls. 1734 Olga Kornievskaiaa made a convincing case against the use of a 1735 credential versus a principal in the fencing approach. Andy Adamson 1736 and Benjamin Kaduk helped to sharpen the focus. 1738 Tigran Mkrtchyan provided the use case for not allowing the client to 1739 proxy the I/O through the data server. 1741 Appendix B. RFC Editor Notes 1743 [RFC Editor: please remove this section prior to publishing this 1744 document as an RFC] 1746 [RFC Editor: prior to publishing this document as an RFC, please 1747 replace all occurrences of RFCTBD10 with RFCxxxx where xxxx is the 1748 RFC number of this document] 1750 Authors' Addresses 1752 Benny Halevy 1754 Email: bhalevy@gmail.com 1755 Thomas Haynes 1756 Primary Data, Inc. 1757 4300 El Camino Real Ste 100 1758 Los Altos, CA 94022 1759 USA 1761 Phone: +1 408 215 1519 1762 Email: thomas.haynes@primarydata.com