idnits 2.17.1 draft-ietf-nfsv4-migration-issues-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 2, 2015) is 3336 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 3530 (Obsoleted by RFC 7530) ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) == Outdated reference: A later version (-04) exists of draft-dnoveck-nfs-extension-01 Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 D. Noveck, Ed. 3 Internet-Draft Dell 4 Intended status: Informational P. Shivam 5 Expires: September 3, 2015 C. Lever 6 B. Baker 7 ORACLE 8 March 2, 2015 10 NFSv4 migration: Implementation experience and spec issues to resolve 11 draft-ietf-nfsv4-migration-issues-07 13 Abstract 15 The migration feature of NFSv4 provides for moving responsibility for 16 a single filesystem from one server to another, without disruption to 17 clients. Recent implementation experience has shown problems in the 18 existing specification for this feature. This document discusses the 19 issues which have arisen, explores the options available for curing 20 the issues, and explains the choices made in updating the NFSv4.0 and 21 NFSv4.1 specifications, to address migration. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on September 3, 2015. 40 Copyright Notice 42 Copyright (c) 2015 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 59 3. NFSv4.0 Implementation Experience . . . . . . . . . . . . . . 4 60 3.1. Implementation issues . . . . . . . . . . . . . . . . . . 4 61 3.1.1. Failure to free migrated state on client reboot . . . 4 62 3.1.2. Server reboots resulting in a confused lease 63 situation . . . . . . . . . . . . . . . . . . . . . . 5 64 3.1.3. Client complexity issues . . . . . . . . . . . . . . 6 65 3.2. Sources of Protocol difficulties . . . . . . . . . . . . 8 66 3.2.1. Issues with nfs_client_id4 generation and use . . . . 8 67 3.2.2. Issues with lease proliferation . . . . . . . . . . . 10 68 4. Issues to be resolved in NFSv4.0 . . . . . . . . . . . . . . 10 69 4.1. Possible changes to nfs_client_id4 client-string . . . . 10 70 4.2. Possible changes to handle differing nfs_client_id4 71 string values . . . . . . . . . . . . . . . . . . . . . . 12 72 4.3. Possible changes to add a new operation . . . . . . . . . 12 73 4.4. Other issues within migration-state sections . . . . . . 12 74 4.5. Issues within other sections . . . . . . . . . . . . . . 13 75 5. Proposed resolution of NFSv4.0 protocol difficulties . . . . 13 76 5.1. Proposed changes: nfs_client_id4 client-string . . . . . 14 77 5.2. Proposed changes: merged (vs. synchronized) leases . . . 14 78 5.3. Other proposed changes to migration-state sections . . . 16 79 5.3.1. Proposed changes: Client ID migration . . . . . . . . 16 80 5.3.2. Proposed changes: Callback re-establishment . . . . . 17 81 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework . . . . 17 82 5.4. Proposed changes to other sections . . . . . . . . . . . 17 83 5.4.1. Proposed changes: callback update . . . . . . . . . . 17 84 5.4.2. Proposed changes: clientid4 handling . . . . . . . . 18 85 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE . . . . . . . . 19 86 6. Results of proposed changes for NFSv4.0 . . . . . . . . . . . 20 87 6.1. Results: Failure to free migrated state on client reboot 21 88 6.2. Results: Server reboots resulting in confused lease 89 situation . . . . . . . . . . . . . . . . . . . . . . . . 21 90 6.3. Results: Client complexity issues . . . . . . . . . . . . 22 91 6.4. Result summary . . . . . . . . . . . . . . . . . . . . . 23 92 7. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . 24 93 7.1. Addressing state merger in NFSv4.1 . . . . . . . . . . . 24 94 7.2. Addressing pNFS relationship with migration . . . . . . . 25 95 7.3. Addressing server owner changes in NFSv4.1 . . . . . . . 25 96 8. Security Considerations . . . . . . . . . . . . . . . . . . . 26 97 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 98 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 27 99 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 27 100 11.1. Normative References . . . . . . . . . . . . . . . . . . 27 101 11.2. Informative References . . . . . . . . . . . . . . . . . 27 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 104 1. Introduction 106 This document is in the informational category, and while the facts 107 it reports may have normative implications, any such normative 108 significance reflects the readers' preferences. For example, we may 109 report that the reboot of a client with migrated state results in 110 state not being promptly cleared and that this will prevent granting 111 of conflicting lock requests at least for the lease time, which is a 112 fact. While it is to be expected that client and server implementers 113 will judge this to be a situation that is best avoided, the judgment 114 as to how pressing this issue should be considered is a judgment for 115 the reader, and eventually the nfsv4 working group to make. 117 We do explore possible ways in which such issues can be avoided, with 118 minimal negative effects, given that the working group has decided to 119 address these issues, but the choice of exactly how to address these 120 is best given effect in one or more standards-track documents and/or 121 errata. 123 This document focuses on NFSv4.0, since that is where the majority of 124 implementation experience has been. Nevertheless, there is 125 discussion of the implications of the NFSv4.0 experience for 126 migration in NFSv4.1, as well as discussion of other issues with 127 regard to the treatment of migration in NFSv4.1. 129 2. Conventions 131 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 132 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 133 document are to be interpreted as described in [RFC2119]. 135 In the context of this informational document, these normative 136 keywords will always occur in the context of a quotation, most often 137 direct but sometimes indirect. The context will make it clear 138 whether the quotation is from: 140 o The current definitive definition of the NFSv4.0 protocol, whether 141 that is the original NFSv4.0 specification [RFC3530], or its 142 expected successor [RFC3530bis]. 144 As the identity of that document may change during the lifetime of 145 this document, we will often refer to the current or pending 146 definition of NFSv4.0 and quote from portions of the documents 147 that are identical among all existing drafts. Given that RFC3530 148 and all RFC3530bis drafts agree as to the issues under discussion, 149 this should not cause undue difficulty. Note that to simplify 150 document maintenance, section names rather than section numbers 151 are used when referring to sections in existing documents so that 152 only minimal changes will be necessary as the identity of the 153 document defining NFSv4.0 changes. 155 o The current definitive definition of the NFSv4.1 protocol 156 [RFC5661]. 158 o A proposed or possible text to serve as a replacement for the 159 current definitive document text. Sometimes, a number of possible 160 alternative texts may be listed and benefits and detriments of 161 each examined in turn. 163 3. NFSv4.0 Implementation Experience 165 3.1. Implementation issues 167 Note that the examples below reflect current experience which arises 168 from clients implementing the recommendation to use different 169 nfs_client_id4 id strings for different server addresses, i.e. using 170 what is later referred to herein as the "non-uniform client-string 171 approach." 173 This is simply because that is the experience implementers have had. 174 The reader should not assume that in all cases, this practice is the 175 source of the difficulty. It may be so in some cases but clearly it 176 is not in all cases. 178 3.1.1. Failure to free migrated state on client reboot 180 The following sort of situation has proved troublesome: 182 o A client C establishes a clientid4 C1 with server ABC specifying 183 an nfs_client_id4 with id string value "C-ABC" and boot verifier 184 0x111. 186 o The client begins to access files in filesystem F on server ABC, 187 resulting in generating stateids S1, S2, etc. under the lease for 188 clientid C1. It may also access files on other filesystems on the 189 same server. 191 o The filesystem is migrated from server ABC to server XYZ. When 192 transparent state migration is in effect, stateids S1 and S2 and 193 clientid4 C1 are now available for use by client C at server XYZ. 195 o Client C reboots and attempts to access data on server XYZ, 196 whether in filesystem F or another. It does a SETCLIENTID with an 197 nfs_client_id4 with id string value "C-XYZ" and boot verifier 198 0x112. There is thus no occasion to free stateids S1 and S2 since 199 they are associated with a different client name and so lease 200 expiration is the only way that they can be gotten rid of. 202 Note here that while it seems clear to us in this example that C-XYZ 203 and C-ABC are from the same client, the server has no way to 204 determine the structure of the "opaque" id string. In the protocol, 205 it really is treated as opaque. Only the client knows which 206 nfs_client_id4 values designate the same client on a different 207 server. 209 3.1.2. Server reboots resulting in a confused lease situation 211 Further problems arise from scenarios like the following. 213 o Client C talks to server ABC using an nfs_client_id4 id string 214 such as "C-ABC" and a boot verifier v1. As a result, a lease with 215 clientid4 c.i is established: {v1, "C-ABC", c.i}. 217 o fs_a1 migrates from server ABC to server XYZ along with its state. 218 Now server XYZ also has a lease: {v1, "C-ABC", c.i}. 220 o Server ABC reboots. 222 o Client C talks to server ABC using an nfs_client_id4 id string 223 such as "C-ABC" and a boot verifier v1. As a result, a lease with 224 clientid4 c.j is established: {v1, "C-ABC", c.j}. 226 o fs_a2 migrates from server ABC to server XYZ. Now server XYZ also 227 has a lease: {v1, "C-ABC", c.j}. 229 o Now server XYZ has two leases that match {v1, "C-ABC", *}, when 230 the protocol clearly assumes there can be only one. 232 Note that if the client used "C" (rather than "C-ABC") as the 233 nfs_client_id4 id string, the exact same situation would arise. 235 One of the first cases in which this sort of situation has resulted 236 in difficulties is in connection with doing a SETCLIENTID for 237 callback update. 239 The SETCLIENTID for callback update only includes the nfs_client_id4, 240 assuming there can only be one such with a given nfs_client_id4 241 value. If there were multiple, confirmed client records with 242 identical nfs_client_id4 id string values, there would be no way to 243 map the callback update request to the correct client record. Apart 244 from the migration handling specified in [RFC3530] and [RFC3530bis], 245 such a situation cannot arise. 247 One possible accommodation for this particular issue that has been 248 used is to add a RENEW operation along with SETCLIENTID (on a 249 callback update) to disambiguate the client. 251 When the client updates the callback info to the destination, the 252 client would, by convention, send a compound like this: 254 { RENEW clientid4, SETCLIENTID nfs_client_id4,verf,cb } 256 The presence of the clientid4 in the compound would allow the server 257 to differentiate among the various leases that it knows of, all with 258 the same nfs_client_id4 value. 260 While this would be a reasonable patch for an isolated protocol 261 weakness, interoperable clients and servers would require that the 262 protocol truly be updated to allow such a situation, specifically 263 that of multiple clientid4's with the same nfs_client_id4 value. The 264 protocol is currently designed and implemented assuming this cannot 265 happen. We need to either prevent the situation from happening, or 266 fully adapt to the possibilities which can arise. See Section 4 for 267 a discussion of such issues. 269 3.1.3. Client complexity issues 271 Consider the following situation: 273 o There are a set of clients C1 through Cn accessing servers S1 274 through Sm. Each server manages some significant number of 275 filesystems with the filesystem count L being significantly 276 greater than m. 278 o Each client Cx will access a subset of the servers and so will 279 have up to m clientids, which we will call Cxy for server Sy. 281 o Now assume that for load-balancing or other operational reasons, 282 numbers of filesystems are migrated among the servers. As a 283 result, each client-server pair will have up to m clientids and 284 each client will have up to m**2 clientids. If we add the 285 possibility of server reboot, the only bound on a client's 286 clientid count is L. 288 Now, instead of a clientid4 identifying a client-server pair, we have 289 many more entities for the client to deal with. In addition, it 290 isn't clear how new state is to be incorporated in this structure. 292 The limitations of the migrated state (inability to be freed on 293 reboot) would argue against adding more such state but trying to 294 avoid that would run into its own difficulties. For example, a 295 single lockowner string presented under two different clientids would 296 appear as two different entities. 298 Thus we have to choose between: 300 o indefinite prolongation of foreign clientids even after all 301 transferred state is gone. 303 o having multiple requests for the same lockowner-string-named 304 entity carried on in parallel by separate identically named 305 lockowners under different clientid4's 307 o Adding serialization at the lock-owner string level, in addition 308 to that at the lockowner level. 310 In any case, we have gone (in adding migration as it was described) 311 from a situation in which 313 o Each client has a single clientid4/lease for each server it talks 314 to. 316 o Each client has a single nfs_client_id4 for each server it talks 317 to. 319 o Every state id can be mapped to an associated lease based on the 320 server it was obtained from. 322 To one in which 324 o Each client may have multiple clientid4's for a single server. 326 o For each stateid, the client must separately record the clientid4 327 that it is assigned to, or it must manage separate "state blobs" 328 for each fsid and map those to clientid4's. 330 o Before doing an operation that can result in a stateid, the client 331 must either find a "state blob" based on fsid or create a new one, 332 possibly with a new clientid4. 334 o There may be multiple clientid4's all connected to the same server 335 and using the same nfs_clientid4. 337 This sort of additional client complexity is troublesome and needs to 338 be eliminated. 340 3.2. Sources of Protocol difficulties 342 3.2.1. Issues with nfs_client_id4 generation and use 344 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 345 and [RFC3530bis] both agree. The section entitled "Client ID" says: 347 The second field, id is a variable length string that uniquely 348 defines the client. 350 There are two possible interpretations of the phrase "uniquely 351 defines" in the above: 353 o The relation between strings and clients is a function from such 354 strings to clients so that each string designates a single client. 356 o The relation between strings and clients is a bijection between 357 such strings and clients so that each string designates a single 358 client and each client is named by a single string. 360 The first interpretation would make these client-strings like phone 361 numbers (a single person can have several) while the second would 362 make them like social security numbers. 364 Endless debate about the true meaning of "uniquely defines" in this 365 context is quite possible but not very helpful. The following points 366 should be noted though: 368 o The second interpretation is more consistent with the way 369 "uniquely defines" is used elsewhere in the spec. 371 o The spec as now written intends the first interpretation (or is 372 internally inconsistent). In fact, it recommends, although it 373 doesn't "RECOMMEND" that a single client have at least as many 374 client-strings as server addresses that it interacts with. It 375 says, in the third bullet point regarding construction of the 376 string (which we shall henceforth refer to as client-string-BP3): 378 The string should be different for each server network address 379 that the client accesses, rather than common to all server 380 network addresses. 382 o If internode interactions are limited to those between a client 383 and its servers, there is no occasion for servers to be concerned 384 with the question of whether two client-strings designate the same 385 client, so that there is no occasion for the difference in 386 interpretation to matter. 388 o When transparent migration of client state occurs between two 389 servers, it becomes important to determine when state on two 390 different servers is for the same client or not, and this 391 distinction becomes very important. 393 Given the need for the server to be aware of client identity with 394 regard to migrated state, either client-string construction rules 395 will have to change or there will be a need to get around current 396 issues, or perhaps a combination of these two will be required. 397 Later sections will examine the options and propose a solution. 399 One consideration that may indicate that this cannot remain exactly 400 as it is today has to do with the fact that the current explanation 401 for this behavior is not correct. The current definitive definitions 402 of the NFSv4.0 protocol, [RFC3530] and [RFC3530bis] both agree. The 403 section entitled "Client ID" says: 405 The reason is that it may not be possible for the client to tell 406 if the same server is listening on multiple network addresses. If 407 the client issues SETCLIENTID with the same id string to each 408 network address of such a server, the server will think it is the 409 same client, and each successive SETCLIENTID will cause the server 410 to begin the process of removing the client's previous leased 411 state. 413 In point of fact, a "SETCLIENTID with the same id string" sent to 414 multiple network addresses will be treated as all from the same 415 client but will not "cause the server to begin the process of 416 removing the client's previous leased state" unless the server 417 believes it is a different instance of the same client, i.e. if the 418 id string is the same and there is a different boot verifier. If the 419 client does not reboot, the verifier should not change. If it does 420 reboot, the verifier will change, and the server should "begin the 421 process of removing the client's previous leased state. 423 The situation of multiple SETCLIENTID requests received by a server 424 on multiple network addresses is exactly the same, from the protocol 425 design point of view, as when multiple (i.e. duplicate) SETCLIENTID 426 requests are received by the server on a single network address. The 427 same protocol mechanisms that prevent erroneous state deletion in the 428 latter case prevent it in the former case. There is no reason for 429 special handling of the multiple-network-appearance case, in this 430 regard. 432 3.2.2. Issues with lease proliferation 434 It is often felt that this is a consequence of the client-string 435 construction issues, and it is certainly the case that the two are 436 closely connected in that non-uniform client-strings make it 437 impossible for the server to appropriately combine leases from the 438 same client. 440 However, even where the server could combine leases from the same 441 client, it needs to be clear how and when it will do so, so that the 442 client will be prepared. These issues will have to be addressed at 443 various places in the spec. 445 This could be enough only if we are prepared to do away with the 446 "should" recommending non-uniform client-strings and replace it with 447 a "should not" or even a "SHOULD NOT". Current client implementation 448 patterns make this an unpalatable choice for use as a general 449 solution, but it is reasonable to "RECOMMEND" this choice for a well- 450 defined subset of clients. One alternative would be to create a way 451 for the server to infer from client behavior which leases are held by 452 the same client and use this information to do appropriate lease 453 mergers. Prototyping and detailed specification work has shown that 454 this could be done but the resulting complexity is such that a better 455 choice is to "RECOMMEND" use of the uniform client-string approach 456 for clients supporting the migration feature. 458 Because of the discussion of client-string construction in [RFC3530] 459 and [RFC3530bis], most existing clients implement the non-uniform 460 client-string approach. As a result, existing servers may not have 461 been tested with clients implementing uniform client-strings. As a 462 consequence, care must be taken to preserve interoperability between 463 UCS-capable clients and servers that don't tolerate uniform client 464 strings for one reason or another. 466 4. Issues to be resolved in NFSv4.0 468 4.1. Possible changes to nfs_client_id4 client-string 470 The fact that the reason given in client-string-BP3 is not valid 471 makes the existing "should" insupportable. We can't either 473 o Keep a reason we know is invalid. 475 o Keep saying "should" without giving a reason. 477 What are often presented as reasons that motivate use of the non- 478 uniform approach always turn out to be cases in which, if the uniform 479 approach were used, the server will treat a client which accesses 480 that server via two different IP addresses as part of a single 481 client, as it in fact is. This may be disconcerting to a client 482 unaware that the two IP addresses connect to the same server. This 483 is not a reason to use the non-uniform approach but is better thought 484 of as an illustration of the fact that those using the uniform 485 approach need to be aware of the possibility of server trunking and 486 its effect on server behavior. 488 If it is possible to reliably infer the existence of trunking of 489 server IP addresses from observed server behavior, use of the uniform 490 approach would be more desirable, although compatibility issues would 491 have to be dealt with. 493 An alternative to having the client infer the existence of trunking 494 of IP server addresses, is to make this information available to the 495 client directly. See Section 4.3 for details. 497 It is always possible that a valid new reason will be found, but so 498 far none has been proposed. Given the history, the burden of proof 499 should be on those asserting the validity of a proposed new reason. 501 So we will assume for now that the "should" will have to go. The 502 question is what to replace it with. 504 o We can't say "MUST NOT", despite the problems this raises for 505 migration since this is pretty late in the day for such a change. 506 Many currently operating clients obey the existing "should". 507 Similar considerations would apply for "SHOULD NOT" or "should 508 not". 510 o Dropping client-string-BP3 entirely is a possibility but, given 511 the context and history, it would just be a confusing version of 512 "SHOULD NOT". 514 o Using "MAY" would clearly specify that both ways of doing this are 515 valid choices for clients and that servers will have to deal with 516 clients that make either choice. 518 o This might be modified by a "SHOULD" (or even a "MUST") for 519 particular groups of clients. 521 o There will have to be some text explaining why a client might make 522 either choice but, except for the particular cases referred to 523 above, we will have to make sure that it is truly descriptive, and 524 not slanted in either direction. 526 4.2. Possible changes to handle differing nfs_client_id4 string values 528 Given the difficulties caused by having different nfs_client_id4 529 client-string values for the same client, we have two choices: 531 o Deprecate the existing treatment and basically say the client is 532 on its own doing migration, if it follows it. 534 o Introduce a way of having the client provide client identity 535 information to the server, if it can be done compatibly while 536 staying within the bounds of v4.0. 538 4.3. Possible changes to add a new operation 540 It might be possible to return server-identity information to the 541 client, just as is done in NFSv4.1 by the response to the EXCHANGE_ID 542 operation. This could be done by a SETCLIENTID_PLUS optional 543 operation, which acts like SETCLIENTID, except that it returns server 544 identity information. Such information could be used by clients, 545 making it possible to for them to be aware of server trunking 546 relationships, rather than having to infer them from server behavior. 548 It has been generally thought that protocol extensions such as this 549 are not appropriate in bis documents and other documents updating 550 NFSv4 protocol definition RFC's. However, it is argued in [NFS-ext] 551 that protocol extensions, similar to those allowed between minor 552 versions, should be acceptable to correct mistakes within a minor 553 version. 555 A decision to adopt this approach will require considerable nfsv4 556 working group discussion and would probably best be effected by means 557 of a standards-track document laying out a modified NFSv4 extension/ 558 versioning model applying to all minor versions, as has been 559 proposed. 561 In view of the time to effect such changes, this approach is not 562 likely to be adopted in an RFC updating [RFC3530] or [RFC3530bis], 563 such as [migr-v4.0-update]. Still, it is worth keeping in mind, if 564 implementers have difficulties inferring trunking relationships using 565 the techniques discussed there. 567 4.4. Other issues within migration-state sections 569 There are a number of issues where the existing text is unclear and/ 570 or wrong and needs to be fixed in some way. 572 o Lack of clarity in the discussion of moving clientids (as well as 573 stateids) as part of moving state for migration. 575 o The discussion of synchronized leases is wrong in that there is no 576 way to determine (in the current spec) when leases are for the 577 same client and also wrong in suggesting a benefit from leases 578 synchronized at the point of transfer. What is needed is merger 579 of leases, which is necessary to keep client complexity 580 requirements from getting out of hand. 582 o Lack of clarity in the discussion of LEASE_MOVED handling, 583 including failure to fully address situations in which transparent 584 state migration did not occur. 586 4.5. Issues within other sections 588 There are a number of cases in which certain sections, not 589 specifically related to migration, require additional clarification. 590 This is generally because text that is clear in a context in which 591 leases and clientids are created in one place and live there forever 592 may need further refinement in the more dynamic environment that 593 arises as part of migration. 595 Some examples: 597 o Some people are under the impression that updating callback 598 endpoint information for an existing client, as used during 599 migration, may cause the destination server to free existing 600 state. There need to be additions to clarify the situation. 602 o The handling of the sets of clientid4's maintained by each server 603 needs to be clarified. In particular, the issue of how the client 604 adapts to the presumably independent and uncoordinated clientid4 605 sets needs to be clearly addressed 607 o Statements regarding handling of invalid clientid4's need to be 608 clarified and/or refined in light of the possibilities that arise 609 due to lease motion and merger. 611 o Confusion and lack of clarity about NFS4ERR_CLID_INUSE. 613 5. Proposed resolution of NFSv4.0 protocol difficulties 615 This section lists the changes which we believe are necessary to 616 resolve the difficulties mentioned above. Such change, along with 617 other clarifications found to be desirable during drafting and review 618 are contained in [migr-v4.0-update]. 620 5.1. Proposed changes: nfs_client_id4 client-string 622 We propose replacing client-string-BP3 with the following text and 623 adding the following proposed to provide implementation guidance. 625 The string MAY be different for each server network address that 626 the client accesses, rather than common to all server network 627 addresses. 629 In addition, given the importance of the issue of client identity and 630 the fact that both client string-approaches are to be considered 631 valid, a greatly expanded treatment of client identity desirable. It 632 should have the following major elements. 634 o It should fully describe the consequences of making the string 635 different for each network address (the non-uniform client-string 636 approach) and of making it the same for all network addresses (the 637 uniform client string approach). 639 o It should give helpful guidance about the factors that might 640 affect client implementation choice between these approaches. 642 o It should describe the compatibility issues that might cause 643 servers to be incompatible with the uniform approach and give 644 guidance about dealing with these. 646 o It should describe how a client using the uniform approach might 647 use server behavior to determine server address trunking patterns. 649 o It should present a clearer and more complete set of 650 recommendations to guide client string construction. 652 5.2. Proposed changes: merged (vs. synchronized) leases 654 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 655 and [RFC3530bis] both agree. The section entitled "Migration and 656 State" says: 658 As part of the transfer of information between servers, leases 659 would be transferred as well. The leases being transferred to the 660 new server will typically have a different expiration time from 661 those for the same client, previously on the old server. To 662 maintain the property that all leases on a given server for a 663 given client expire at the same time, the server should advance 664 the expiration time to the later of the leases being transferred 665 or the leases already present. This allows the client to maintain 666 lease renewal of both classes without special effort: 668 There are a number of problems with this and any resolution of our 669 difficulties must address them somehow. 671 o The current v4.0 spec recommends that the client make it 672 essentially impossible to determine when two leases are from "the 673 same client". 675 o It is not appropriate to speak of "maintain[ing] the property that 676 all leases on a given server for a given client expire at the same 677 time", since this is not a property that holds even in the absence 678 of migration. A server listening on multiple network addresses 679 may have the same client appear as multiple clients with no way to 680 recognize the client as the same. 682 o Even if the client identity issue could be resolved, advancing the 683 lease time at the point of migration would not maintain the 684 desired synchronization property. The leases would be 685 synchronized until one of them was renewed, after which they would 686 be unsynchronized again. 688 To avoid client complexity, we need to have no more than one lease 689 between a single client and a single server. This requires merger of 690 leases since there is no real help from synchronizing them at a 691 single instant. 693 For the uniform approach, the destination server would simply merge 694 leases as part of state transfer, since two leases with the same 695 nfs_client_id4 values must be for the same client. 697 We have made the following decisions as far as proposed normative 698 statements regarding for state merger. They reflect the facts that 699 we want to support fully migration support in the simplest way 700 possible and that we can't say MUST since we have older clients and 701 servers to deal with. 703 o Clients SHOULD use the uniform client-string approach in order to 704 get good migration support. 706 o Servers SHOULD provide automatic lease merger during state 707 migration so that clients using the uniform id approach get the 708 support automatically. 710 If the clients and the servers obey the SHOULD's, having more than a 711 single lease for a given client-server pair will be a transient 712 situation, cleaned up as part of adapting to use of migrated state. 714 Since clients and servers will be a mixture of old and new and 715 because nothing is a MUST we have to ensure that no combination will 716 show worse behavior than is exhibited by current (i.e. old) clients 717 and servers. 719 5.3. Other proposed changes to migration-state sections 721 5.3.1. Proposed changes: Client ID migration 723 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 724 and [RFC3530bis] both agree. The section entitled "Migration and 725 State" says: 727 In the case of migration, the servers involved in the migration of 728 a filesystem SHOULD transfer all server state from the original to 729 the new server. This must be done in a way that is transparent to 730 the client. This state transfer will ease the client's transition 731 when a filesystem migration occurs. If the servers are successful 732 in transferring all state, the client will continue to use 733 stateids assigned by the original server. Therefore the new 734 server must recognize these stateids as valid. This holds true 735 for the client ID as well. Since responsibility for an entire 736 filesystem is transferred with a migration event, there is no 737 possibility that conflicts will arise on the new server as a 738 result of the transfer of locks. 740 This poses some difficulties, mostly because the part about "client 741 ID" is not clear: 743 o It isn't clear what part of the paragraph the "this" in the 744 statement "this holds true ..." is meant to signify. 746 o The phrase "the client ID" is ambiguous, possibly indicating the 747 clientid4 and possibly indicating the nfs_client_id4. 749 o If the text means to suggest that the same clientid4 must be used, 750 the logic is not clear since the issue is not the same as for 751 stateids of which there might be many. Adapting to the change of 752 a single clientid, as might happen as a part of lease migration, 753 is relatively easy for the client. 755 We have decided that it is best to address this issue as follows: 757 o Make it clear that both clientid4 and nfs_client_id4 (including 758 both id string and boot verifier) are to be transferred. 760 o Indicate that the initial transfer will result in the same 761 clientid4 after transfer but this is not guaranteed since there 762 may conflict with an existing clientid4 on the destination server 763 and because lease merger can result in a change of the clientid4. 765 5.3.2. Proposed changes: Callback re-establishment 767 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 768 and [RFC3530bis] both agree. The section entitled "Migration and 769 State" says: 771 A client SHOULD re-establish new callback information with the new 772 server as soon as possible, according to sequences described in 773 sections "Operation 35: SETCLIENTID - Negotiate Client ID" and 774 "Operation 36: SETCLIENTID_CONFIRM - Confirm Client ID". This 775 ensures that server operations are not blocked by the inability to 776 recall delegations. 778 The above will need to be fixed to reflect the possibility of merging 779 of leases, 781 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework 783 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 784 and [RFC3530bis] both agree. The section entitled "Notification of 785 Migrated Lease" says: 787 Upon receiving the NFS4ERR_LEASE_MOVED error, a client that 788 supports filesystem migration MUST probe all filesystems from that 789 server on which it holds open state. Once the client has 790 successfully probed all those filesystems which are migrated, the 791 server MUST resume normal handling of stateful requests from that 792 client. 794 There is a lack of clarity that is prompted by ambiguity about what 795 exactly probing is and what the interlock between client and server 796 must be. This has led to some worry about the scalability of the 797 probing process, and although the time required does scale linearly 798 with the number of filesystems that the client may have state for 799 with respect to a given server, the actual process can be done 800 efficiently. 802 To address these issues we propose rewriting the above to be more 803 clear and to give suggestions about how to do the required scanning 804 efficiently. 806 5.4. Proposed changes to other sections 808 5.4.1. Proposed changes: callback update 810 Some changes are necessary to reduce confusion about the process of 811 callback information update and in particular to make it clear that 812 no state is freed as a result: 814 o Make it clear that after migration there are confirmed entries for 815 transferred clientid4/nfs_client_id4 pairs. 817 o Be explicit in the sections headed "otherwise," in the 818 descriptions of SETCLIENTID and SETCLIENTID_CONFIRM, that these 819 don't apply in the cases we are concerned about. 821 5.4.2. Proposed changes: clientid4 handling 823 To address both of the clientid4-related issues mentioned in 824 Section 4.5, we propose replacing the last three paragraphs of the 825 section entitled "Client ID" with the following: 827 Once a SETCLIENTID and SETCLIENTID_CONFIRM sequence has 828 successfully completed, the client uses the shorthand client 829 identifier, of type clientid4, instead of the longer and less 830 compact nfs_client_id4 structure. This shorthand client 831 identifier (a client ID) is assigned by the server and should be 832 chosen so that it will not conflict with a client ID previously 833 assigned by same server. This applies across server restarts or 834 reboots. 836 Distinct servers MAY assign clientid4's independently, and will 837 generally do so. Therefore, a client has to be prepared to deal 838 with multiple instances of the same clientid4 value received on 839 distinct IP addresses, denoting separate entities. When trunking 840 of server IP addresses is not a consideration, a client should 841 keep track of (IP-address, clientid4) pairs, so that each pair is 842 distinct. In the face of possible trunking of server IP 843 addresses, the client will use the receipt of the same clientid4 844 from multiple IP-addresses, as an indication that the two IP- 845 addresses may be trunked and proceed to determine, from the 846 observed server behavior whether the two addresses are in fact 847 trunked. 849 When a clientid4 is presented to a server and that clientid4 is 850 not recognized, the server will reject the request with the error 851 NFS4ERR_STALE_CLIENTID. This can occur for a number of reasons: 853 * A server reboot causing loss of the server's knowledge of the 854 client 856 * Client error sending an incorrect clientid4 or a valid 857 clientid4 to the wrong server. 859 * Loss of lease state due to lease expiration. 861 * Client or server error causing the server to believe that the 862 client has rebooted (i.e. receiving a SETCLIENTID with an 863 nfs_client_id4 which has a matching id string and a non- 864 matching boot verifier). 866 * Migration of all state under the associated lease causes its 867 non-existence to be recognized on the source server. 869 * Merger of state under the associated lease with another lease 870 under a different clientid causes the clientid4 serving as the 871 source of the merge to cease being recognized on its server. 873 In the event of a server reboot, or loss of lease state due to 874 lease expiration, the client must obtain a new clientid4 by use of 875 the SETCLIENTID operation and then proceed to any other necessary 876 recovery for the server reboot case (See the section entitled 877 "Server Failure and Recovery"). In cases of server or client 878 error resulting in this error, use of SETCLIENTID to establish a 879 new lease is desirable as well. 881 In the last two cases, different recovery procedures are required. 882 Note that in cases in which there is any uncertainty about which 883 sort of handling is applicable, the distinguishing characteristic 884 is that in reboot-like cases, the clientid4 and all associated 885 stateids cease to exist while in migration-related cases, the 886 clientid4 ceases to exist while the stateids are still valid. 888 The client must also employ the SETCLIENTID operation when it 889 receives a NFS4ERR_STALE_STATEID error using a stateid derived 890 from its current clientid4, since this indicates a situation, such 891 as server reboot which has invalidated the existing clientid4 and 892 associated stateids (see the section entitled "lock-owner" for 893 details). 895 See the detailed descriptions of SETCLIENTID and 896 SETCLIENTID_CONFIRM for a complete specification of the 897 operations. 899 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE 901 It appears to be the intention that only a single principal be used 902 for client establishment between any client-server pair. However: 904 o There is no explicit statement to this effect. 906 o The error that indicates a principal conflict has a name which 907 does not clarify this issue: NFS4ERR_CLID_INUSE. 909 o The definition of the error is also not very helpful: "The 910 SETCLIENTID operation has found that a client id is already in use 911 by another client". 913 As a result, servers exist which reject a SETCLIENTID simply because 914 there already exists a clientid for the same client, established 915 using a different IP address. Although this is generally understood 916 to be erroneous, such servers still exist and the spec should make 917 the correct behavior clear. 919 Although the error name cannot be changed, the following changes 920 should be made to avoid confusion: 922 o The definition of the error should be changed to read as follows: 924 The SETCLIENTID operation has found that the specified 925 nfs_client_id4 was previously presented with a different 926 principal and that client instance currently holds an active 927 lease. A server MAY return this error if the same principal is 928 used but a change in authentication flavor gives good reason to 929 reject the new SETCLIENTID operation as not bona fide. 931 o In the description of SETCLIENTID, the phrase "then the server 932 returns a NFS4ERR_CLID_INUSE error" should be expanded to read 933 "then the server returns a NFS4ERR_CLID_INUSE error, since use of 934 a single client with multiple principals is not allowed." 936 6. Results of proposed changes for NFSv4.0 938 The purpose of this section is to examine the troubling results 939 reported in Section 3.1. We will look at the scenarios as they would 940 be handled within the proposal. 942 Because the choice of uniform vs. non-uniform nfs_client_id4 id 943 strings is a "SHOULD" in these cases, we will designate clients that 944 follow this recommendation by SHOULD-UF-CID. 946 We will also have to take account of any merger-related "SHOULD" 947 clauses to better understand how they have addressed the issues seen. 948 We abbreviate as follows: 950 o SHOULD-SVR-AM refers to the server obeying the SHOULD which 951 RECOMMENDS that they merge leases with identical nfs_client_id4 id 952 strings and boot verifiers. 954 6.1. Results: Failure to free migrated state on client reboot 956 Let's look at the troublesome situation cited in Section 3.1.1. We 957 have already seen what happens when SHOULD-UF-CID does not hold. Now 958 let's look at the situation in which SHOULD-UF-CID holds, whether 959 SHOULD-SVR-AM is in effect or not. 961 o A client C establishes a clientid4 C1 with server ABC specifying 962 an nfs_client_id4 with id string value "C" and boot verifier 963 0x111. 965 o The client begins to access files in filesystem F on server ABC, 966 resulting in generating stateids S1, S2, etc. under the lease for 967 clientid C1. It may also access files on other filesystems on the 968 same server. 970 o The filesystem is migrated from ABC to server XYZ. When 971 transparent state migration is in effect, stateids S1 and S2 and 972 lease {0x111, "C", C1} are now available for use by client C at 973 server XYZ. 975 o Client C reboots and attempts to access data on server XYZ, 976 whether in filesystem F or another. It does a SETCLIENTID with an 977 nfs_client_id4 with id string value "C" and boot verifier 0x112. 978 The state associated with lease {0x111, "C", C1} is deleted as 979 part of creating {0x112, "C", C2}. No problem. 981 The correctness signature for this issue is 983 SHOULD-UF-CID 985 so if you have clients and servers that obey the SHOULD clauses, the 986 problem is gone regardless of the choice on the MAY. 988 6.2. Results: Server reboots resulting in confused lease situation 990 Now let's consider the scenario given in Section 3.1.2. We have 991 already seen what happens when SHOULD-UF-CID does not hold . Now 992 let's look at the situation in which SHOULD-UF-CID holds and SHOULD- 993 SVR-AM holds as well. 995 o Client C talks to server ABC using an nfs_client_id4 id string 996 such as "C-ABC" and boot verifier v1. As a result a lease with 997 clientid4 c.i established: {v1, "C-ABC", c.i}. 999 o Filesystem fs_a1 migrates from server ABC to server XYZ along with 1000 its state. Now server XYZ also has a lease: {v1, "C-ABC", c.i} 1002 o Server ABC reboots. 1004 o Client C talks to server ABC using an nfs_client_id4 id string 1005 such as "C-ABC" and boot verifier v1. As a result a lease with 1006 clientid4 c.j established: {v1, "C-ABC", c.j}. 1008 o fs_a2 migrates from server ABC to server XYZ. As part of 1009 migration the incoming lease is seen to denote same nfs_client_id4 1010 and so is merged with {v1, "C-ABC, c.i}. 1012 o Now server XYZ has only one lease that matches {v1, "C_ABC", *}, 1013 so the problem is solved 1015 Now let's consider the same scenario in the situation in which 1016 SHOULD-UF-CID holds and SHOULD-SVR-AM holds as well. 1018 o Client C talks to server ABC using an nfs_client_id4 id string "C" 1019 and boot verifier v1. As a result a lease with clientid4 c.i is 1020 established: {v1, "C", c.i}. 1022 o fs_a1 migrates from server ABC to server XYZ along with its state. 1023 Now XYZ also has a lease: {v1, "C", c.i} 1025 o Server ABC reboots. 1027 o Client C talks to server ABC using an nfs_client_id4 id string "C" 1028 and boot verifier v1. As a result a lease with clientid4 c.j is 1029 established: {v1, "C", c.j}. 1031 o fs_a2 migrates from server ABC to server XYZ. As part of 1032 migration the incoming lease is seen to denote the same 1033 nfs_client_id4 and so is merged with {v1, "C", c.i}. 1035 o Now server XYZ has only one lease that matches {v1, "C", *}, so 1036 the problem is solved 1038 The correctness signature for this issue is 1040 SHOULD-SVR-AM 1042 so if you have clients and servers that obey the SHOULD clauses, the 1043 problem is gone regardless of the choice on the MAY. 1045 6.3. Results: Client complexity issues 1047 Consider the following situation: 1049 o There are a set of clients C1 through Cn accessing servers S1 1050 through Sm. Each server manages some significant number of 1051 filesystems with the filesystem count L being significantly 1052 greater than m. 1054 o Each client Cx will access a subset of the servers and so will 1055 have up to m clientids, which we will call Cxy for server Sy. 1057 o Now assume that for load-balancing or other operational reasons, 1058 numbers of filesystems are migrated among the servers. As a 1059 result, depending on how this handled, the number of clientids may 1060 explode. See below. 1062 Now look what will happen under various scenarios: 1064 o We have previously (in Section 3.1.3) looked at this in case of 1065 client following the non-uniform client-string approach. In that 1066 case, each client-server pair could have up to m clientids and 1067 each client will have up to m**2 clientids. If we add the 1068 possibility of server reboot, the only bound on a client's 1069 clientid count is L. 1071 o If we look at this in the SHOULD-UF-CID case in which the SHOULD- 1072 SVR_AM condition holds, the situation is no different. Although 1073 the server has the client identity information that could enable 1074 same-client-same-server leases to be combined, it does not do so. 1075 We still have up to L clientids per client. 1077 o On the other hand, if we look at the SHOULD-UF-CID case in which 1078 SHOULD-SVR-AM holds, the problem is gone. There can be no more 1079 than m clientids per client, and n clientids per server. 1081 The correctness signature for this issue is 1083 (SHOULD-UF-CID & SHOULD-SVR-AM) 1085 so if you have clients and servers that obey the SHOULD clauses, the 1086 problem is gone regardless of the choice on the MAY. 1088 6.4. Result summary 1090 We have seen that (SHOULD-SVR-AM & SHOULD-UF-CID) are sufficient to 1091 solve the problems people have experienced. 1093 7. Issues for NFSv4.1 1095 Because NFSv4.1 embraces the uniform client-string approach, 1096 addressing migration issues is simpler. In the terms of Section 6, 1097 we already have SHOULD-UF-CID, for NFSv4.1, as advised by section 2.4 1098 of [RFC5661], simplifying the work to be done. 1100 Nevertheless, there are some issues that will have to be addressed. 1101 Some examples: 1103 o The other necessary part of addressing migration issues, which we 1104 call above SHOULD-SVR-AM, is not currently addressed by NFSv4.1 1105 and changes need to be made to make it clear that state needs to 1106 be appropriately merged as part of migration, to avoid multiple 1107 clientids between a client-server pair. 1109 o There needs to be some clarification of how migration, and 1110 particularly transparent state migration, should interact with 1111 pNFS layouts. 1113 o The current discussion (in [RFC5661]), of the possibility of 1114 server_owner changes is incomplete and confusing. 1116 Discussion of how to resolve these issues will appear in the sections 1117 below. 1119 7.1. Addressing state merger in NFSv4.1 1121 The existing treatment of state transfer in [RFC5661], has similar 1122 problems to that in [RFC3530] and [RFC3530bis] in that it assumes 1123 that the state for multiple filesystems on different servers will not 1124 be merged to so that it appears under a single common clientid. 1125 We've already seen the reasons that this is a problem, with regard to 1126 NFSv4.0. 1128 Although we don't have the problems stemming from the non-uniform 1129 client-string approach, there are a number of complexities in the 1130 existing treatment of state management in the section entitled "Lock 1131 State and File System Transitions" in [RFC5661] that make this non- 1132 trivial to address: 1134 o Migration is currently treated together with other sorts of 1135 filesystem transitions including transitioning between replicas 1136 without any NFS4ERR_MOVED errors. 1138 o There is separate handling and discussion of the cases of matching 1139 and non-matching server scopes. 1141 o In the case of matching server scopes, the text calls for an 1142 impossible degree of transparency. 1144 o In the case of non-matching server scopes, the text does not 1145 mention transparent state migration at all, resulting in a 1146 functional regression from NFSV4.0 1148 7.2. Addressing pNFS relationship with migration 1150 This is made difficult because, within the PNFS framework, migration 1151 might mean any of several things: 1153 o Transfer of the MDS, leaving DS's alone. 1155 This would be minimally disruptive to those using layouts but 1156 would require the pNFS control protocol to support the DS being 1157 directed to a new MDS. 1159 o Transfer of a DS, leaving everything else in place. 1161 Such a transfer can be handled without using migration at all. 1162 The server can recall/revoke layouts, as appropriate. 1164 o Transfer of the filesystem to a new filesystem with both MDS and 1165 DS's moving. 1167 In such a transfer, an entirely different set of DS's will be at 1168 the target location. There may even be no pNFS support on the 1169 destination filesystem at all. 1171 Migration needs to support both the first and last of these models. 1173 7.3. Addressing server owner changes in NFSv4.1 1175 Section 2.10.5 of [RFC5661] states the following. 1177 The client should be prepared for the possibility that 1178 eir_server_owner values may be different on subsequent EXCHANGE_ID 1179 requests made to the same network address, as a result of various 1180 sorts of reconfiguration events. When this happens and the 1181 changes result in the invalidation of previously valid forms of 1182 trunking, the client should cease to use those forms, either by 1183 dropping connections or by adding sessions. For a discussion of 1184 lock reclaim as it relates to such reconfiguration events, see 1185 Section 8.4.2.1. 1187 While this paragraph is literally true in that such reconfiguration 1188 events can happen and clients have to deal with them, it is confusing 1189 in that it can be read as suggesting that clients have to deal with 1190 them without disruption, which in general is impossible. 1192 A clearer alternative would be: 1194 It is always possible that, as a result of various sorts of 1195 reconfiguration events, eir_server_scope and eir_server_owner 1196 values may be different on subsequent EXCHANGE_ID requests made to 1197 the same network address. 1199 In most cases such reconfiguration events will be disruptive and 1200 indicate that an IP address formerly connected to one server is 1201 now connected to an entirely different one. 1203 Some guidelines on client handling of such situations follow: 1205 * When eir_server_scope changes, the client has no assurance that 1206 any id's it obtained previously (e.g. file handles) can be 1207 validly used on the new server, and, even if the new server 1208 accepts them, there is no assurance that this is not due to 1209 accident. Thus it is best to treat all such state as lost/ 1210 stale although a client may assume that the probability of 1211 inadvertent acceptance is low and treat this situation as 1212 within the next case. 1214 * When eir_server_scope remains the same and 1215 eir_server_owner.so_major_id changes, the client can use 1216 filehandles it has and attempt reclaims. It may find that 1217 these are now stale but if NFS4ERR_STALE is not received, he 1218 can proceed to reclaim his opens. 1220 * When eir_server_scope and eir_server_owner.so_major_id remain 1221 the same, the client has to use the now-current values of 1222 eir_server-owner.so_minor_id in deciding on appropriate forms 1223 of trunking. 1225 8. Security Considerations 1227 With regard to NFSv4.0, the current definitive definitions of the 1228 protocol, [RFC3530] and [RFC3530bis] both agree. The section 1229 entitled "Security Considerations" encourages that clients protect 1230 the integrity of the SECINFO operation, any GETATTR operation for the 1231 fs_locations attribute, and the operations SETCLIENTID/ 1232 SETCLIENTID_CONFIRM. A migration recovery event can use any or all 1233 of these operations. We do not recommend any change here. 1235 With regard to NFSv4.1, the Security Considerations section of 1236 [RFC5661] takes proper care of migration-related issues. No change 1237 is needed. 1239 9. IANA Considerations 1241 This document does not require actions by IANA. 1243 10. Acknowledgements 1245 The editor and authors of this document gratefully acknowledge the 1246 contributions of Trond Myklebust of NetApp and Robert Thurlow of 1247 Oracle. We also thank Tom Haynes of NetApp and Spencer Shepler of 1248 Microsoft for their guidance and suggestions. 1250 Special thanks go to members of the Oracle Solaris NFS team, 1251 especially Rick Mesta and James Wahlig, for their work implementing 1252 an NFSv4.0 migration prototype and identifying many of the issues 1253 documented here. 1255 11. References 1257 11.1. Normative References 1259 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1260 Requirement Levels", BCP 14, RFC 2119, March 1997. 1262 [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., 1263 Beame, C., Eisler, M., and D. Noveck, "Network File System 1264 (NFS) version 4 Protocol", RFC 3530, April 2003. 1266 [RFC3530bis] 1267 Haynes, T., Ed. and D. Noveck, Ed., "Network File System 1268 (NFS) Version 4 Protocol", 2014, . 1271 Work in progress. 1273 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 1274 System (NFS) Version 4 Minor Version 1 Protocol", RFC 1275 5661, January 2010. 1277 11.2. Informative References 1279 [NFS-ext] Noveck, D., "NFS Protocol Extension: Retrospect and 1280 Prospect", 2014, . 1283 Work in progress. 1285 [migr-v4.0-update] 1286 Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, 1287 "NFSv4.0 migration: Specification Update", 2015, 1288 . 1291 Work in progress. 1293 Authors' Addresses 1295 David Noveck (editor) 1296 Dell 1297 300 Innovative Way 1298 Nashua, NH 03062 1299 US 1301 Phone: +1 781 572 8038 1302 Email: dave_noveck@dell.com 1304 Piyush Shivam 1305 Oracle Corporation 1306 5300 Riata Park Ct. 1307 Austin, TX 78727 1308 US 1310 Phone: +1 512 401 1019 1311 Email: piyush.shivam@oracle.com 1313 Charles Lever 1314 Oracle Corporation 1315 1015 Granger Avenue 1316 Ann Arbor, MI 48104 1317 US 1319 Phone: +1 248 614 5091 1320 Email: chuck.lever@oracle.com 1321 Bill Baker 1322 Oracle Corporation 1323 5300 Riata Park Ct. 1324 Austin, TX 78727 1325 US 1327 Phone: +1 512 401 1081 1328 Email: bill.baker@oracle.com