idnits 2.17.1 draft-ietf-nfsv4-migration-issues-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 16, 2013) is 3847 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 3530 (Obsoleted by RFC 7530) == Outdated reference: A later version (-35) exists of draft-ietf-nfsv4-rfc3530bis-27 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 D. Noveck, Ed. 3 Internet-Draft EMC 4 Intended status: Informational P. Shivam 5 Expires: March 20, 2014 C. Lever 6 B. Baker 7 ORACLE 8 September 16, 2013 10 NFSv4 migration: Implementation experience and spec issues to resolve 11 draft-ietf-nfsv4-migration-issues-04 13 Abstract 15 The migration feature of NFSv4 provides for moving responsibility for 16 a single filesystem from one server to another, without disruption to 17 clients. Recent implementation experience has shown problems in the 18 existing specification for this feature. This document discusses the 19 issues which have arisen, explores the options available for curing 20 the issues, and explains the choices made in updating the NFSv4.0 and 21 NFSv4.1 specifications, to address migration. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on March 20, 2014. 40 Copyright Notice 42 Copyright (c) 2013 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 59 3. NFSv4.0 Implementation Experience . . . . . . . . . . . . . . 4 60 3.1. Implementation issues . . . . . . . . . . . . . . . . . . 4 61 3.1.1. Failure to free migrated state on client reboot . . . 4 62 3.1.2. Server reboots resulting in a confused lease 63 situation . . . . . . . . . . . . . . . . . . . . . . 5 64 3.1.3. Client complexity issues . . . . . . . . . . . . . . 6 65 3.2. Sources of Protocol difficulties . . . . . . . . . . . . 8 66 3.2.1. Issues with nfs_client_id4 generation and use . . . . 8 67 3.2.2. Issues with lease proliferation . . . . . . . . . . . 10 68 4. Issues to be resolved in NFSv4.0 . . . . . . . . . . . . . . 10 69 4.1. Possible changes to nfs_client_id4 client-string . . . . 10 70 4.2. Possible changes to handle differing nfs_client_id4 71 string values . . . . . . . . . . . . . . . . . . . . . . 11 72 4.3. Possible changes to add a new operation . . . . . . . . . 12 73 4.4. Other issues within migration-state sections . . . . . . 12 74 4.5. Issues within other sections . . . . . . . . . . . . . . 13 75 5. Proposed resolution of NFSv4.0 protocol difficulties . . . . 13 76 5.1. Proposed changes: nfs_client_id4 client-string . . . . . 13 77 5.2. Proposed changes: merged (vs. synchronized) leases . . . 14 78 5.3. Other proposed changes to migration-state sections . . . 16 79 5.3.1. Proposed changes: Client ID migration . . . . . . . . 16 80 5.3.2. Proposed changes: Callback re-establishment . . . . . 16 81 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework . . . . 17 82 5.4. Proposed changes to other sections . . . . . . . . . . . 17 83 5.4.1. Proposed changes: callback update . . . . . . . . . . 17 84 5.4.2. Proposed changes: clientid4 handling . . . . . . . . 18 85 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE . . . . . . . . 19 86 6. Results of proposed changes for NFSv4.0 . . . . . . . . . . . 20 87 6.1. Results: Failure to free migrated state on client reboot 20 88 6.2. Results: Server reboots resulting in confused lease 89 situation . . . . . . . . . . . . . . . . . . . . . . . . 21 90 6.3. Results: Client complexity issues . . . . . . . . . . . . 22 91 6.4. Result summary . . . . . . . . . . . . . . . . . . . . . 23 92 7. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . 23 93 7.1. Addressing state merger in NFSv4.1 . . . . . . . . . . . 24 94 7.2. Addressing pNFS relationship with migration . . . . . . . 25 95 7.3. Addressing server owner changes in NFSv4.1 . . . . . . . 25 96 8. Security Considerations . . . . . . . . . . . . . . . . . . . 26 97 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 98 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 27 99 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 27 100 11.1. Normative References . . . . . . . . . . . . . . . . . . 27 101 11.2. Informative References . . . . . . . . . . . . . . . . . 27 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 104 1. Introduction 106 This document is in the informational category, and while the facts 107 it reports may have normative implications, any such normative 108 significance reflects the readers' preferences. For example, we may 109 report that the reboot of a client with migrated state results in 110 state not being promptly cleared and that this will prevent granting 111 of conflicting lock requests at least for the lease time, which is a 112 fact. While it is to be expected that client and server implementers 113 will judge this to be a situation that is best avoided, the judgment 114 as to how pressing this issue should be considered is a judgment for 115 the reader, and eventually the nfsv4 working group to make. 117 We do explore possible ways in which such issues can be avoided, with 118 minimal negative effects, given that the working group has decided to 119 address these issues, but the choice of exactly how to address these 120 is best given effect in one or more standards-track documents and/or 121 errata. 123 This document focuses on NFSv4.0, since that is where the majority of 124 implementation experience has been. Nevertheless, there is 125 discussion of the implications of the NFSv4.0 experience for 126 migration in NFSv4.1, as well as discussion of other issues with 127 regard to the treatment of migration in NFSv4.1. 129 2. Conventions 131 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 132 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 133 document are to be interpreted as described in [RFC2119]. 135 In the context of this informational document, these normative 136 keywords will always occur in the context of a quotation, most often 137 direct but sometimes indirect. The context will make it clear 138 whether the quotation is from: 140 o The current definitive definition of the NFSv4.0 protocol, whether 141 that is the original NFSv4.0 specification [RFC3530], or its 142 expected successor [RFC3530bis]. 144 As the identity of that document may change during the lifetime of 145 this document, we will often refer to the current or pending 146 definition of NFSv4.0 and quote from portions of the documents 147 that are identical among all existing drafts. Given that RFC3530 148 and all RFC3530bis drafts agree as to the issues under discussion, 149 this should not cause undue difficulty. Note that to simplify 150 document maintenance, section names rather than section numbers 151 are used when referring to sections in existing documents so that 152 only minimal changes will be necessary as the identity of the 153 document defining NFSv4.0 changes. 155 o The current definitive definition of the NFSv4.1 protocol 156 [RFC5661]. 158 o A proposed or possible text to serve as a replacement for the 159 current definitive document text. Sometimes, a number of possible 160 alternative texts may be listed and benefits and detriments of 161 each examined in turn. 163 3. NFSv4.0 Implementation Experience 165 3.1. Implementation issues 167 Note that the examples below reflect current experience which arises 168 from clients implementing the recommendation to use different 169 nfs_client_id4 id strings for different server addresses, i.e. using 170 what is later referred to herein as the "non-uniform client-string 171 approach." 173 This is simply because that is the experience implementers have had. 174 The reader should not assume that in all cases, this practice is the 175 source of the difficulty. It may be so in some cases but clearly it 176 is not in all cases. 178 3.1.1. Failure to free migrated state on client reboot 180 The following sort of situation has proved troublesome: 182 o A client C establishes a clientid4 C1 with server ABC specifying 183 an nfs_client_id4 with id string value "C-ABC" and boot verifier 184 0x111. 186 o The client begins to access files in filesystem F on server ABC, 187 resulting in generating stateids S1, S2, etc. under the lease for 188 clientid C1. It may also access files on other filesystems on the 189 same server. 191 o The filesystem is migrated from server ABC to server XYZ. When 192 transparent state migration is in effect, stateids S1 and S2 and 193 clientid4 C1 are now available for use by client C at server XYZ. 195 o Client C reboots and attempts to access data on server XYZ, 196 whether in filesystem F or another. It does a SETCLIENTID with an 197 nfs_client_id4 with id string value "C-XYZ" and boot verifier 198 0x112. There is thus no occasion to free stateids S1 and S2 since 199 they are associated with a different client name and so lease 200 expiration is the only way that they can be gotten rid of. 202 Note here that while it seems clear to us in this example that C-XYZ 203 and C-ABC are from the same client, the server has no way to 204 determine the structure of the "opaque" id string. In the protocol, 205 it really is treated as opaque. Only the client knows which 206 nfs_client_id4 values designate the same client on a different 207 server. 209 3.1.2. Server reboots resulting in a confused lease situation 211 Further problems arise from scenarios like the following. 213 o Client C talks to server ABC using an nfs_client_id4 id string 214 such as "C-ABC" and a boot verifier v1. As a result, a lease with 215 clientid4 c.i is established: {v1, "C-ABC", c.i}. 217 o fs_a1 migrates from server ABC to server XYZ along with its state. 218 Now server XYZ also has a lease: {v1, "C-ABC", c.i}. 220 o Server ABC reboots. 222 o Client C talks to server ABC using an nfs_client_id4 id string 223 such as "C-ABC" and a boot verifier v1. As a result, a lease with 224 clientid4 c.j is established: {v1, "C-ABC", c.j}. 226 o fs_a2 migrates from server ABC to server XYZ. Now server XYZ also 227 has a lease: {v1, "C-ABC", c.j}. 229 o Now server XYZ has two leases that match {v1, "C-ABC", *}, when 230 the protocol clearly assumes there can be only one. 232 Note that if the client used "C" (rather than "C-ABC") as the 233 nfs_client_id4 id string, the exact same situation would arise. 235 One of the first cases in which this sort of situation has resulted 236 in difficulties is in connection with doing a SETCLIENTID for 237 callback update. 239 The SETCLIENTID for callback update only includes the nfs_client_id4, 240 assuming there can only be one such with a given nfs_client_id4 241 value. If there were multiple, confirmed client records with 242 identical nfs_client_id4 id string values, there would be no way to 243 map the callback update request to the correct client record. Apart 244 from the migration handling specified in [RFC3530] and [RFC3530bis], 245 such a situation cannot arise. 247 One possible accommodation for this particular issue that has been 248 used is to add a RENEW operation along with SETCLIENTID (on a 249 callback update) to disambiguate the client. 251 When the client updates the callback info to the destination, the 252 client would, by convention, send a compound like this: 254 { RENEW clientid4, SETCLIENTID nfs_client_id4,verf,cb } 256 The presence of the clientid4 in the compound would allow the server 257 to differentiate among the various leases that it knows of, all with 258 the same nfs_client_id4 value. 260 While this would be a reasonable patch for an isolated protocol 261 weakness, interoperable clients and servers would require that the 262 protocol truly be updated to allow such a situation, specifically 263 that of multiple clientid4's with the same nfs_client_id4 value. The 264 protocol is currently designed and implemented assuming this cannot 265 happen. We need to either prevent the situation from happening, or 266 fully adapt to the possibilities which can arise. See Section 4 for 267 a discussion of such issues. 269 3.1.3. Client complexity issues 271 Consider the following situation: 273 o There are a set of clients C1 through Cn accessing servers S1 274 through Sm. Each server manages some significant number of 275 filesystems with the filesystem count L being significantly 276 greater than m. 278 o Each client Cx will access a subset of the servers and so will 279 have up to m clientids, which we will call Cxy for server Sy. 281 o Now assume that for load-balancing or other operational reasons, 282 numbers of filesystems are migrated among the servers. As a 283 result, each client-server pair will have up to m clientids and 284 each client will have up to m**2 clientids. If we add the 285 possibility of server reboot, the only bound on a client's 286 clientid count is L. 288 Now, instead of a clientid4 identifying a client-server pair, we have 289 many more entities for the client to deal with. In addition, it 290 isn't clear how new state is to be incorporated in this structure. 292 The limitations of the migrated state (inability to be freed on 293 reboot) would argue against adding more such state but trying to 294 avoid that would run into its own difficulties. For example, a 295 single lockowner string presented under two different clientids would 296 appear as two different entities. 298 Thus we have to choose between: 300 o indefinite prolongation of foreign clientids even after all 301 transferred state is gone. 303 o having multiple requests for the same lockowner-string-named 304 entity carried on in parallel by separate identically named 305 lockowners under different clientid4's 307 o Adding serialization at the lock-owner string level, in addition 308 to that at the lockowner level. 310 In any case, we have gone (in adding migration as it was described) 311 from a situation in which 313 o Each client has a single clientid4/lease for each server it talks 314 to. 316 o Each client has a single nfs_client_id4 for each server it talks 317 to. 319 o Every state id can be mapped to an associated lease based on the 320 server it was obtained from. 322 To one in which 324 o Each client may have multiple clientid4's for a single server. 326 o For each stateid, the client must separately record the clientid4 327 that it is assigned to, or it must manage separate "state blobs" 328 for each fsid and map those to clientid4's. 330 o Before doing an operation that can result in a stateid, the client 331 must either find a "state blob" based on fsid or create a new one, 332 possibly with a new clientid4. 334 o There may be multiple clientid4's all connected to the same server 335 and using the same nfs_clientid4. 337 This sort of additional client complexity is troublesome and needs to 338 be eliminated. 340 3.2. Sources of Protocol difficulties 342 3.2.1. Issues with nfs_client_id4 generation and use 344 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 345 and [RFC3530bis] both agree. The section entitled "Client ID" says: 347 The second field, id is a variable length string that uniquely 348 defines the client. 350 There are two possible interpretations of the phrase "uniquely 351 defines" in the above: 353 o The relation between strings and clients is a function from such 354 strings to clients so that each string designates a single client. 356 o The relation between strings and clients is a bijection between 357 such strings and clients so that each string designates a single 358 client and each client is named by a single string. 360 The first interpretation would make these client-strings like phone 361 numbers (a single person can have several) while the second would 362 make them like social security numbers. 364 Endless debate about the true meaning of "uniquely defines" in this 365 context is quite possible but not very helpful. The following points 366 should be noted though: 368 o The second interpretation is more consistent with the way 369 "uniquely defines" is used elsewhere in the spec. 371 o The spec as now written intends the first interpretation (or is 372 internally inconsistent). In fact, it recommends, although it 373 doesn't "RECOMMEND" that a single client have at least as many 374 client-strings as server addresses that it interacts with. It 375 says, in the third bullet point regarding construction of the 376 string (which we shall henceforth refer to as client-string-BP3): 378 The string should be different for each server network address 379 that the client accesses, rather than common to all server 380 network addresses. 382 o If internode interactions are limited to those between a client 383 and its servers, there is no occasion for servers to be concerned 384 with the question of whether two client-strings designate the same 385 client, so that there is no occasion for the difference in 386 interpretation to matter. 388 o When transparent migration of client state occurs between two 389 servers, it becomes important to determine when state on two 390 different servers is for the same client or not, and this 391 distinction becomes very important. 393 Given the need for the server to be aware of client identity with 394 regard to migrated state, either client-string construction rules 395 will have to change or there will be a need to get around current 396 issues, or perhaps a combination of these two will be required. 397 Later sections will examine the options and propose a solution. 399 One consideration that may indicate that this cannot remain exactly 400 as it is today has to do with the fact that the current explanation 401 for this behavior is not correct. The current definitive definitions 402 of the NFSv4.0 protocol, [RFC3530] and [RFC3530bis] both agree. The 403 section entitled "Client ID" says: 405 The reason is that it may not be possible for the client to tell 406 if the same server is listening on multiple network addresses. If 407 the client issues SETCLIENTID with the same id string to each 408 network address of such a server, the server will think it is the 409 same client, and each successive SETCLIENTID will cause the server 410 to begin the process of removing the client's previous leased 411 state. 413 In point of fact, a "SETCLIENTID with the same id string" sent to 414 multiple network addresses will be treated as all from the same 415 client but will not "cause the server to begin the process of 416 removing the client's previous leased state" unless the server 417 believes it is a different instance of the same client, i.e. if the 418 id string is the same and there is a different boot verifier. If the 419 client does not reboot, the verifier should not change. If it does 420 reboot, the verifier will change, and the server should "begin the 421 process of removing the client's previous leased state. 423 The situation of multiple SETCLIENTID requests received by a server 424 on multiple network addresses is exactly the same, from the protocol 425 design point of view, as when multiple (i.e. duplicate) SETCLIENTID 426 requests are received by the server on a single network address. The 427 same protocol mechanisms that prevent erroneous state deletion in the 428 latter case prevent it in the former case. There is no reason for 429 special handling of the multiple-network-appearance case, in this 430 regard. 432 3.2.2. Issues with lease proliferation 434 It is often felt that this is a consequence of the client-string 435 construction issues, and it is certainly the case that the two are 436 closely connected in that non-uniform client-strings make it 437 impossible for the server to appropriately combine leases from the 438 same client. 440 However, even where the server could combine leases from the same 441 client, it needs to be clear how and when it will do so, so that the 442 client will be prepared. These issues will have to be addressed at 443 various places in the spec. 445 This could be enough only if we are prepared to do away with the 446 "should" recommending non-uniform client-strings and replace it with 447 a "should not" or even a "SHOULD NOT". Current client implementation 448 patterns make this an unpalatable choice for use as a general 449 solution, but it is reasonable to "RECOMMEND" this choice for a well- 450 defined subset of clients. One alternative would be to create a way 451 for the server to infer from client behavior which leases are held by 452 the same client and use this information to do appropriate lease 453 mergers. Prototyping and detailed specification work has shown that 454 this could be done but the resulting complexity is such that a better 455 choice is to "RECOMMEND" use of the uniform client-string approach 456 for clients supporting the migration feature. 458 Because of the discussion of client-string construction in [RFC3530] 459 and [RFC3530bis], most existing clients implement the non-uniform 460 client-string approach. As a result, existing servers may not have 461 been tested with clients implementing uniform client-strings. As a 462 consequence, care must be taken to preserve interoperability between 463 UCS-capable clients and servers that don't tolerate uniform client 464 strings for one reason or another. 466 4. Issues to be resolved in NFSv4.0 468 4.1. Possible changes to nfs_client_id4 client-string 470 The fact that the reason given in client-string-BP3 is not valid 471 makes the existing "should" insupportable. We can't either 473 o Keep a reason we know is invalid. 475 o Keep saying "should" without giving a reason. 477 What are often presented as reasons that motivate use of the non- 478 uniform approach always turn out to be cases in which, if the uniform 479 approach were used, the server will treat a client which accesses 480 that server via two different IP addresses as part of a single 481 client, as it in fact is. This may be disconcerting to a client 482 unaware that the two IP addresses connect to the same server. This 483 is not a reason to use the non-uniform approach but is better thought 484 of as an illustration of the fact that those using the uniform 485 approach need to be aware of the possibility of server trunking and 486 its effect on server behavior. 488 If it is possible to reliably infer the existence of trunking of 489 server IP addresses from observed server behavior, use of the uniform 490 approach would be more desirable, although compatibility issues would 491 have to be dealt with. 493 An alternative to having the client infer the existence of trunking 494 of IP server addresses, is to make this information available to the 495 client directly. See Section 4.3 for details. 497 It is always possible that a valid new reason will be found, but so 498 far none has been proposed. Given the history, the burden of proof 499 should be on those asserting the validity of a proposed new reason. 501 So we will assume for now that the "should" will have to go. The 502 question is what to replace it with. 504 o We can't say "MUST NOT", despite the problems this raises for 505 migration since this is pretty late in the day for such a change. 506 Many currently operating clients obey the existing "should". 507 Similar considerations would apply for "SHOULD NOT" or "should 508 not". 510 o Dropping client-string-BP3 entirely is a possibility but, given 511 the context and history, it would just be a confusing version of 512 "SHOULD NOT". 514 o Using "MAY" would clearly specify that both ways of doing this are 515 valid choices for clients and that servers will have to deal with 516 clients that make either choice. 518 o This might be modified by a "SHOULD" (or even a "MUST") for 519 particular groups of clients. 521 o There will have to be some text explaining why a client might make 522 either choice but, except for the particular cases referred to 523 above, we will have to make sure that it is truly descriptive, and 524 not slanted in either direction. 526 4.2. Possible changes to handle differing nfs_client_id4 string values 527 Given the difficulties caused by having different nfs_client_id4 528 client-string values for the same client, we have two choices: 530 o Deprecate the existing treatment and basically say the client is 531 on its own doing migration, if it follows it. 533 o Introduce a way of having the client provide client identity 534 information to the server, if it can be done compatibly while 535 staying within the bounds of v4.0. 537 4.3. Possible changes to add a new operation 539 It might be possible to return server-identity information to the 540 client, just as is done in NFSv4.1 by the response to the EXCHANGE_ID 541 operation. This could be done by a SETCLIENTID_PLUS optional 542 operation, which acts like SETCLIENTID, except that it returns server 543 identity information. Such information could be used by clients, 544 making it possible to for them to be aware of server trunking 545 relationships, rather than having to infer them from server behavior. 547 It had always been thought that protocol extensions such as this were 548 not appropriate to bis documents and other documents updating NFSv4 549 protocol definition RFC's. However, it is argued in [NFS-ext] that 550 protocol extensions, similar to those allowed between minor versions, 551 should be acceptable to correct mistakes within a minor version. 553 A decision to adopt this approach will depend on nfsv4 working group 554 discussion and would probably best be effected by means of a 555 standards-track document laying out a modified NFSv4 extension/ 556 versioning model for all minor versions. 558 In view of the time to effect such changes, this approach is not 559 likely to be adopted in an RFC updating [RFC3530] or [RFC3530bis], 560 such as [migr-v4.0-update]. Still, it is worth keeping in mind, if 561 implementers have difficulties inferring trunking relationships using 562 the techniques discussed there. 564 4.4. Other issues within migration-state sections 566 There are a number of issues where the existing text is unclear and/ 567 or wrong and needs to be fixed in some way. 569 o Lack of clarity in the discussion of moving clientids (as well as 570 stateids) as part of moving state for migration. 572 o The discussion of synchronized leases is wrong in that there is no 573 way to determine (in the current spec) when leases are for the 574 same client and also wrong in suggesting a benefit from leases 575 synchronized at the point of transfer. What is needed is merger 576 of leases, which is necessary to keep client complexity 577 requirements from getting out of hand. 579 o Lack of clarity in the discussion of LEASE_MOVED handling, 580 including failure to fully address situations in which transparent 581 state migration did not occur. 583 4.5. Issues within other sections 585 There are a number of cases in which certain sections, not 586 specifically related to migration, require additional clarification. 587 This is generally because text that is clear in a context in which 588 leases and clientids are created in one place and live there forever 589 may need further refinement in the more dynamic environment that 590 arises as part of migration. 592 Some examples: 594 o Some people are under the impression that updating callback 595 endpoint information for an existing client, as used during 596 migration, may cause the destination server to free existing 597 state. There need to be additions to clarify the situation. 599 o The handling of the sets of clientid4's maintained by each server 600 needs to be clarified. In particular, the issue of how the client 601 adapts to the presumably independent and uncoordinated clientid4 602 sets needs to be clearly addressed 604 o Statements regarding handling of invalid clientid4's need to be 605 clarified and/or refined in light of the possibilities that arise 606 due to lease motion and merger. 608 o Confusion and lack of clarity about NFS4ERR_CLID_INUSE. 610 5. Proposed resolution of NFSv4.0 protocol difficulties 612 This section lists the changes which we believe are necessary to 613 resolve the difficulties mentioned above. Such change, along with 614 other clarifications found to be desirable during drafting and review 615 are contained in [migr-v4.0-update]. 617 5.1. Proposed changes: nfs_client_id4 client-string 619 We propose replacing client-string-BP3 with the following text and 620 adding the following proposed to provide implementation guidance. 622 The string MAY be different for each server network address that 623 the client accesses, rather than common to all server network 624 addresses. 626 In addition, given the importance of the issue of client identity and 627 the fact that both client string-approaches are to be considered 628 valid, a greatly expanded treatment of client identity desirable. It 629 should have the following major elements. 631 o It should fully describe the consequences of making the string 632 different for each network address (the non-uniform client-string 633 approach) and of making it the same for all network addresses (the 634 uniform client string approach). 636 o It should give helpful guidance about the factors that might 637 affect client implementation choice between these approaches. 639 o It should describe the compatibility issues that might cause 640 servers to be incompatible with the uniform approach and give 641 guidance about dealing with these. 643 o It should describe how a client using the uniform approach might 644 use server behavior to determine server address trunking patterns. 646 o It should present a clearer and more complete set of 647 recommendations to guide client string construction. 649 5.2. Proposed changes: merged (vs. synchronized) leases 651 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 652 and [RFC3530bis] both agree. The section entitled "Migration and 653 State" says: 655 As part of the transfer of information between servers, leases 656 would be transferred as well. The leases being transferred to the 657 new server will typically have a different expiration time from 658 those for the same client, previously on the old server. To 659 maintain the property that all leases on a given server for a 660 given client expire at the same time, the server should advance 661 the expiration time to the later of the leases being transferred 662 or the leases already present. This allows the client to maintain 663 lease renewal of both classes without special effort: 665 There are a number of problems with this and any resolution of our 666 difficulties must address them somehow. 668 o The current v4.0 spec recommends that the client make it 669 essentially impossible to determine when two leases are from "the 670 same client". 672 o It is not appropriate to speak of "maintain[ing] the property that 673 all leases on a given server for a given client expire at the same 674 time", since this is not a property that holds even in the absence 675 of migration. A server listening on multiple network addresses 676 may have the same client appear as multiple clients with no way to 677 recognize the client as the same. 679 o Even if the client identity issue could be resolved, advancing the 680 lease time at the point of migration would not maintain the 681 desired synchronization property. The leases would be 682 synchronized until one of them was renewed, after which they would 683 be unsynchronized again. 685 To avoid client complexity, we need to have no more than one lease 686 between a single client and a single server. This requires merger of 687 leases since there is no real help from synchronizing them at a 688 single instant. 690 For the uniform approach, the destination server would simply merge 691 leases as part of state transfer, since two leases with the same 692 nfs_client_id4 values must be for the same client. 694 We have made the following decisions as far as proposed normative 695 statements regarding for state merger. They reflect the facts that 696 we want to support fully migration support in the simplest way 697 possible and that we can't say MUST since we have older clients and 698 servers to deal with. 700 o Clients SHOULD use the uniform client-string approach in order to 701 get good migration support. 703 o Servers SHOULD provide automatic lease merger during state 704 migration so that clients using the uniform id approach get the 705 support automatically. 707 If the clients and the servers obey the SHOULD's, having more than a 708 single lease for a given client-server pair will be a transient 709 situation, cleaned up as part of adapting to use of migrated state. 711 Since clients and servers will be a mixture of old and new and 712 because nothing is a MUST we have to ensure that no combination will 713 show worse behavior than is exhibited by current (i.e. old) clients 714 and servers. 716 5.3. Other proposed changes to migration-state sections 718 5.3.1. Proposed changes: Client ID migration 720 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 721 and [RFC3530bis] both agree. The section entitled "Migration and 722 State" says: 724 In the case of migration, the servers involved in the migration of 725 a filesystem SHOULD transfer all server state from the original to 726 the new server. This must be done in a way that is transparent to 727 the client. This state transfer will ease the client's transition 728 when a filesystem migration occurs. If the servers are successful 729 in transferring all state, the client will continue to use 730 stateids assigned by the original server. Therefore the new 731 server must recognize these stateids as valid. This holds true 732 for the client ID as well. Since responsibility for an entire 733 filesystem is transferred with a migration event, there is no 734 possibility that conflicts will arise on the new server as a 735 result of the transfer of locks. 737 This poses some difficulties, mostly because the part about "client 738 ID" is not clear: 740 o It isn't clear what part of the paragraph the "this" in the 741 statement "this holds true ..." is meant to signify. 743 o The phrase "the client ID" is ambiguous, possibly indicating the 744 clientid4 and possibly indicating the nfs_client_id4. 746 o If the text means to suggest that the same clientid4 must be used, 747 the logic is not clear since the issue is not the same as for 748 stateids of which there might be many. Adapting to the change of 749 a single clientid, as might happen as a part of lease migration, 750 is relatively easy for the client. 752 We have decided that it is best to address this issue as follows: 754 o Make it clear that both clientid4 and nfs_client_id4 (including 755 both id string and boot verifier) are to be transferred. 757 o Indicate that the initial transfer will result in the same 758 clientid4 after transfer but this is not guaranteed since there 759 may conflict with an existing clientid4 on the destination server 760 and because lease merger can result in a change of the clientid4. 762 5.3.2. Proposed changes: Callback re-establishment 763 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 764 and [RFC3530bis] both agree. The section entitled "Migration and 765 State" says: 767 A client SHOULD re-establish new callback information with the new 768 server as soon as possible, according to sequences described in 769 sections "Operation 35: SETCLIENTID - Negotiate Client ID" and 770 "Operation 36: SETCLIENTID_CONFIRM - Confirm Client ID". This 771 ensures that server operations are not blocked by the inability to 772 recall delegations. 774 The above will need to be fixed to reflect the possibility of merging 775 of leases, 777 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework 779 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 780 and [RFC3530bis] both agree. The section entitled "Notification of 781 Migrated Lease" says: 783 Upon receiving the NFS4ERR_LEASE_MOVED error, a client that 784 supports filesystem migration MUST probe all filesystems from that 785 server on which it holds open state. Once the client has 786 successfully probed all those filesystems which are migrated, the 787 server MUST resume normal handling of stateful requests from that 788 client. 790 There is a lack of clarity that is prompted by ambiguity about what 791 exactly probing is and what the interlock between client and server 792 must be. This has led to some worry about the scalability of the 793 probing process, and although the time required does scale linearly 794 with the number of filesystems that the client may have state for 795 with respect to a given server, the actual process can be done 796 efficiently. 798 To address these issues we propose rewriting the above to be more 799 clear and to give suggestions about how to do the required scanning 800 efficiently. 802 5.4. Proposed changes to other sections 804 5.4.1. Proposed changes: callback update 806 Some changes are necessary to reduce confusion about the process of 807 callback information update and in particular to make it clear that 808 no state is freed as a result: 810 o Make it clear that after migration there are confirmed entries for 811 transferred clientid4/nfs_client_id4 pairs. 813 o Be explicit in the sections headed "otherwise," in the 814 descriptions of SETCLIENTID and SETCLIENTID_CONFIRM, that these 815 don't apply in the cases we are concerned about. 817 5.4.2. Proposed changes: clientid4 handling 819 To address both of the clientid4-related issues mentioned in 820 Section 4.5, we propose replacing the last three paragraphs of the 821 section entitled "Client ID" with the following: 823 Once a SETCLIENTID and SETCLIENTID_CONFIRM sequence has 824 successfully completed, the client uses the shorthand client 825 identifier, of type clientid4, instead of the longer and less 826 compact nfs_client_id4 structure. This shorthand client 827 identifier (a client ID) is assigned by the server and should be 828 chosen so that it will not conflict with a client ID previously 829 assigned by same server. This applies across server restarts or 830 reboots. 832 Distinct servers MAY assign clientid4's independently, and will 833 generally do so. Therefore, a client has to be prepared to deal 834 with multiple instances of the same clientid4 value received on 835 distinct IP addresses, denoting separate entities. When trunking 836 of server IP addresses is not a consideration, a client should 837 keep track of (IP-address, clientid4) pairs, so that each pair is 838 distinct. In the face of possible trunking of server IP 839 addresses, the client will use the receipt of the same clientid4 840 from multiple IP-addresses, as an indication that the two IP- 841 addresses may be trunked and proceed to determine, from the 842 observed server behavior whether the two addresses are in fact 843 trunked. 845 When a clientid4 is presented to a server and that clientid4 is 846 not recognized, the server will reject the request with the error 847 NFS4ERR_STALE_CLIENTID. This can occur for a number of reasons: 849 * A server reboot causing loss of the server's knowledge of the 850 client 852 * Client error sending an incorrect clientid4 or a valid 853 clientid4 to the wrong server. 855 * Loss of lease state due to lease expiration. 857 * Client or server error causing the server to believe that the 858 client has rebooted (i.e. receiving a SETCLIENTID with an 859 nfs_client_id4 which has a matching id string and a non- 860 matching boot verifier). 862 * Migration of all state under the associated lease causes its 863 non-existence to be recognized on the source server. 865 * Merger of state under the associated lease with another lease 866 under a different clientid causes the clientid4 serving as the 867 source of the merge to cease being recognized on its server. 869 In the event of a server reboot, or loss of lease state due to 870 lease expiration, the client must obtain a new clientid4 by use of 871 the SETCLIENTID operation and then proceed to any other necessary 872 recovery for the server reboot case (See the section entitled 873 "Server Failure and Recovery"). In cases of server or client 874 error resulting in this error, use of SETCLIENTID to establish a 875 new lease is desirable as well. 877 In the last two cases, different recovery procedures are required. 878 Note that in cases in which there is any uncertainty about which 879 sort of handling is applicable, the distinguishing characteristic 880 is that in reboot-like cases, the clientid4 and all associated 881 stateids cease to exist while in migration-related cases, the 882 clientid4 ceases to exist while the stateids are still valid. 884 The client must also employ the SETCLIENTID operation when it 885 receives a NFS4ERR_STALE_STATEID error using a stateid derived 886 from its current clientid4, since this indicates a situation, such 887 as server reboot which has invalidated the existing clientid4 and 888 associated stateids (see the section entitled "lock-owner" for 889 details). 891 See the detailed descriptions of SETCLIENTID and 892 SETCLIENTID_CONFIRM for a complete specification of the 893 operations. 895 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE 897 It appears to be the intention that only a single principal be used 898 for client establishment between any client-server pair. However: 900 o There is no explicit statement to this effect. 902 o The error that indicates a principal conflict has a name which 903 does not clarify this issue: NFS4ERR_CLID_INUSE. 905 o The definition of the error is also not very helpful: "The 906 SETCLIENTID operation has found that a client id is already in use 907 by another client". 909 As a result, servers exist which reject a SETCLIENTID simply because 910 there already exists a clientid for the same client, established 911 using a different IP address. Although this is generally understood 912 to be erroneous, such servers still exist and the spec should make 913 the correct behavior clear. 915 Although the error name cannot be changed, the following changes 916 should be made to avoid confusion: 918 o The definition of the error should be changed to read as follows: 920 The SETCLIENTID operation has found that the specified 921 nfs_client_id4 was previously presented with a different 922 principal and that client instance currently holds an active 923 lease. A server MAY return this error if the same principal is 924 used but a change in authentication flavor gives good reason to 925 reject the new SETCLIENTID operation as not bona fide. 927 o In the description of SETCLIENTID, the phrase "then the server 928 returns a NFS4ERR_CLID_INUSE error" should be expanded to read 929 "then the server returns a NFS4ERR_CLID_INUSE error, since use of 930 a single client with multiple principals is not allowed." 932 6. Results of proposed changes for NFSv4.0 934 The purpose of this section is to examine the troubling results 935 reported in Section 3.1. We will look at the scenarios as they would 936 be handled within the proposal. 938 Because the choice of uniform vs. non-uniform nfs_client_id4 id 939 strings is a "SHOULD" in these cases, we will designate clients that 940 follow this recommendation by SHOULD-UF-CID. 942 We will also have to take account of any merger-related "SHOULD" 943 clauses to better understand how they have addressed the issues seen. 944 We abbreviate as follows: 946 o SHOULD-SVR-AM refers to the server obeying the SHOULD which 947 RECOMMENDS that they merge leases with identical nfs_client_id4 id 948 strings and boot verifiers. 950 6.1. Results: Failure to free migrated state on client reboot 951 Let's look at the troublesome situation cited in Section 3.1.1. We 952 have already seen what happens when SHOULD-UF-CID does not hold. Now 953 let's look at the situation in which SHOULD-UF-CID holds, whether 954 SHOULD-SVR-AM is in effect or not. 956 o A client C establishes a clientid4 C1 with server ABC specifying 957 an nfs_client_id4 with id string value "C" and boot verifier 958 0x111. 960 o The client begins to access files in filesystem F on server ABC, 961 resulting in generating stateids S1, S2, etc. under the lease for 962 clientid C1. It may also access files on other filesystems on the 963 same server. 965 o The filesystem is migrated from ABC to server XYZ. When 966 transparent state migration is in effect, stateids S1 and S2 and 967 lease {0x111, "C", C1} are now available for use by client C at 968 server XYZ. 970 o Client C reboots and attempts to access data on server XYZ, 971 whether in filesystem F or another. It does a SETCLIENTID with an 972 nfs_client_id4 with id string value "C" and boot verifier 0x112. 973 The state associated with lease {0x111, "C", C1} is deleted as 974 part of creating {0x112, "C", C2}. No problem. 976 The correctness signature for this issue is 978 SHOULD-UF-CID 980 so if you have clients and servers that obey the SHOULD clauses, the 981 problem is gone regardless of the choice on the MAY. 983 6.2. Results: Server reboots resulting in confused lease situation 985 Now let's consider the scenario given in Section 3.1.2. We have 986 already seen what happens when SHOULD-UF-CID does not hold . Now 987 let's look at the situation in which SHOULD-UF-CID holds and SHOULD- 988 SVR-AM holds as well. 990 o Client C talks to server ABC using an nfs_client_id4 id string 991 such as "C-ABC" and boot verifier v1. As a result a lease with 992 clientid4 c.i established: {v1, "C-ABC", c.i}. 994 o Filesystem fs_a1 migrates from server ABC to server XYZ along with 995 its state. Now server XYZ also has a lease: {v1, "C-ABC", c.i} 997 o Server ABC reboots. 999 o Client C talks to server ABC using an nfs_client_id4 id string 1000 such as "C-ABC" and boot verifier v1. As a result a lease with 1001 clientid4 c.j established: {v1, "C-ABC", c.j}. 1003 o fs_a2 migrates from server ABC to server XYZ. As part of 1004 migration the incoming lease is seen to denote same nfs_client_id4 1005 and so is merged with {v1, "C-ABC, c.i}. 1007 o Now server XYZ has only one lease that matches {v1, "C_ABC", *}, 1008 so the problem is solved 1010 Now let's consider the same scenario in the situation in which 1011 SHOULD-UF-CID holds and SHOULD-SVR-AM holds as well. 1013 o Client C talks to server ABC using an nfs_client_id4 id string "C" 1014 and boot verifier v1. As a result a lease with clientid4 c.i is 1015 established: {v1, "C", c.i}. 1017 o fs_a1 migrates from server ABC to server XYZ along with its state. 1018 Now XYZ also has a lease: {v1, "C", c.i} 1020 o Server ABC reboots. 1022 o Client C talks to server ABC using an nfs_client_id4 id string "C" 1023 and boot verifier v1. As a result a lease with clientid4 c.j is 1024 established: {v1, "C", c.j}. 1026 o fs_a2 migrates from server ABC to server XYZ. As part of 1027 migration the incoming lease is seen to denote the same 1028 nfs_client_id4 and so is merged with {v1, "C", c.i}. 1030 o Now server XYZ has only one lease that matches {v1, "C", *}, so 1031 the problem is solved 1033 The correctness signature for this issue is 1035 SHOULD-SVR-AM 1037 so if you have clients and servers that obey the SHOULD clauses, the 1038 problem is gone regardless of the choice on the MAY. 1040 6.3. Results: Client complexity issues 1042 Consider the following situation: 1044 o There are a set of clients C1 through Cn accessing servers S1 1045 through Sm. Each server manages some significant number of 1046 filesystems with the filesystem count L being significantly 1047 greater than m. 1049 o Each client Cx will access a subset of the servers and so will 1050 have up to m clientids, which we will call Cxy for server Sy. 1052 o Now assume that for load-balancing or other operational reasons, 1053 numbers of filesystems are migrated among the servers. As a 1054 result, depending on how this handled, the number of clientids may 1055 explode. See below. 1057 Now look what will happen under various scenarios: 1059 o We have previously (in Section 3.1.3) looked at this in case of 1060 client following the non-uniform client-string approach. In that 1061 case, each client-server pair could have up to m clientids and 1062 each client will have up to m**2 clientids. If we add the 1063 possibility of server reboot, the only bound on a client's 1064 clientid count is L. 1066 o If we look at this in the SHOULD-UF-CID case in which the SHOULD- 1067 SVR_AM condition holds, the situation is no different. Although 1068 the server has the client identity information that could enable 1069 same-client-same-server leases to be combined, it does not do so. 1070 We still have up to L clientids per client. 1072 o On the other hand, if we look at the SHOULD-UF-CID case in which 1073 SHOULD-SVR-AM holds, the problem is gone. There can be no more 1074 than m clientids per client, and n clientids per server. 1076 The correctness signature for this issue is 1078 (SHOULD-UF-CID & SHOULD-SVR-AM) 1080 so if you have clients and servers that obey the SHOULD clauses, the 1081 problem is gone regardless of the choice on the MAY. 1083 6.4. Result summary 1085 We have seen that (SHOULD-SVR-AM & SHOULD-UF-CID) are sufficient to 1086 solve the problems people have experienced. 1088 7. Issues for NFSv4.1 1090 Because NFSv4.1 embraces the uniform client-string approach, 1091 addressing migration issues is simpler. In the terms of Section 6, 1092 we already have SHOULD-UF-CID, for NFSv4.1, as advised by section 2.4 1093 of [RFC5661], simplifying the work to be done. 1095 Nevertheless, there are some issues that will have to be addressed. 1096 Some examples: 1098 o The other necessary part of addressing migration issues, which we 1099 call above SHOULD-SVR-AM, is not currently addressed by NFSv4.1 1100 and changes need to be made to make it clear that state needs to 1101 be appropriately merged as part of migration, to avoid multiple 1102 clientids between a client-server pair. 1104 o There needs to be some clarification of how migration, and 1105 particularly transparent state migration, should interact with 1106 pNFS layouts. 1108 o The current discussion (in [RFC5661]), of the possibility of 1109 server_owner changes is incomplete and confusing. 1111 Discussion of how to resolve these issues will appear in the sections 1112 below. 1114 7.1. Addressing state merger in NFSv4.1 1116 The existing treatment of state transfer in [RFC5661], has similar 1117 problems to that in [RFC3530] and [RFC3530bis] in that it assumes 1118 that the state for multiple filesystems on different servers will not 1119 be merged to so that it appears under a single common clientid. 1120 We've already seen the reasons that this is a problem, with regard to 1121 NFSv4.0. 1123 Although we don't have the problems stemming from the non-uniform 1124 client-string approach, there are a number of complexities in the 1125 existing treatment of state management in the section entitled "Lock 1126 State and File System Transitions" in [RFC5661] that make this non- 1127 trivial to address: 1129 o Migration is currently treated together with other sorts of 1130 filesystem transitions including transitioning between replicas 1131 without any NFS4ERR_MOVED errors. 1133 o There is separate handling and discussion of the cases of matching 1134 and non-matching server scopes. 1136 o In the case of matching server scopes, the text calls for an 1137 impossible degree of transparency. 1139 o In the case of non-matching server scopes, the text does not 1140 mention transparent state migration at all, resulting in a 1141 functional regression from NFSV4.0 1143 7.2. Addressing pNFS relationship with migration 1145 This is made difficult because, within the PNFS framework, migration 1146 might mean any of several things: 1148 o Transfer of the MDS, leaving DS's alone. 1150 This would be minimally disruptive to those using layouts but 1151 would require the pNFS control protocol to support the DS being 1152 directed to a new MDS. 1154 o Transfer of a DS, leaving everything else in place. 1156 Such a transfer can be handled without using migration at all. 1157 The server can recall/revoke layouts, as appropriate. 1159 o Transfer of the filesystem to a new filesystem with both MDS and 1160 DS's moving. 1162 In such a transfer, an entirely different set of DS's will be at 1163 the target location. There may even be no pNFS support on the 1164 destination filesystem at all. 1166 Migration needs to support both the first and last of these models. 1168 7.3. Addressing server owner changes in NFSv4.1 1170 Section 2.10.5 of [RFC5661] states the following. 1172 The client should be prepared for the possibility that 1173 eir_server_owner values may be different on subsequent EXCHANGE_ID 1174 requests made to the same network address, as a result of various 1175 sorts of reconfiguration events. When this happens and the 1176 changes result in the invalidation of previously valid forms of 1177 trunking, the client should cease to use those forms, either by 1178 dropping connections or by adding sessions. For a discussion of 1179 lock reclaim as it relates to such reconfiguration events, see 1180 Section 8.4.2.1. 1182 While this paragraph is literally true in that such reconfiguration 1183 events can happen and clients have to deal with them, it is confusing 1184 in that it can be read as suggesting that clients have to deal with 1185 them without disruption, which in general is impossible. 1187 A clearer alternative would be: 1189 It is always possible that, as a result of various sorts of 1190 reconfiguration events, eir_server_scope and eir_server_owner 1191 values may be different on subsequent EXCHANGE_ID requests made to 1192 the same network address. 1194 In most cases such reconfiguration events will be disruptive and 1195 indicate that an IP address formerly connected to one server is 1196 now connected to an entirely different one. 1198 Some guidelines on client handling of such situations follow: 1200 * When eir_server_scope changes, the client has no assurance that 1201 any id's it obtained previously (e.g. file handles) can be 1202 validly used on the new server, and, even if the new server 1203 accepts them, there is no assurance that this is not due to 1204 accident. Thus it is best to treat all such state as lost/ 1205 stale although a client may assume that the probability of 1206 inadvertent acceptance is low and treat this situation as 1207 within the next case. 1209 * When eir_server_scope remains the same and 1210 eir_server_owner.so_major_id changes, the client can use 1211 filehandles it has and attempt reclaims. It may find that 1212 these are now stale but if NFS4ERR_STALE is not received, he 1213 can proceed to reclaim his opens. 1215 * When eir_server_scope and eir_server_owner.so_major_id remain 1216 the same, the client has to use the now-current values of 1217 eir_server-owner.so_minor_id in deciding on appropriate forms 1218 of trunking. 1220 8. Security Considerations 1222 The current definitive definitions of the NFSv4.0 protocol, [RFC3530] 1223 and [RFC3530bis] both agree. The section entitled "Security 1224 Considerations" encourages that clients protect the integrity of the 1225 SECINFO operation, any GETATTR operation for the fs_locations 1226 attribute, and the operations SETCLIENTID/SETCLIENTID_CONFIRM. A 1227 migration recovery event can use any or all of these operations. We 1228 do not recommend any change here. 1230 9. IANA Considerations 1232 This document does not require actions by IANA. 1234 10. Acknowledgements 1236 The editor and authors of this document gratefully acknowledge the 1237 contributions of Trond Myklebust of NetApp and Robert Thurlow of 1238 Oracle. We also thank Tom Haynes of NetApp and Spencer Shepler of 1239 Microsoft for their guidance and suggestions. 1241 Special thanks go to members of the Oracle Solaris NFS team, 1242 especially Rick Mesta and James Wahlig, for their work implementing 1243 an NFSv4.0 migration prototype and identifying many of the issues 1244 documented here. 1246 11. References 1248 11.1. Normative References 1250 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1251 Requirement Levels", BCP 14, RFC 2119, March 1997. 1253 [RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., 1254 Beame, C., Eisler, M., and D. Noveck, "Network File System 1255 (NFS) version 4 Protocol", RFC 3530, April 2003. 1257 [RFC3530bis] 1258 Haynes, T., Ed. and D. Noveck, Ed., "Network File System 1259 (NFS) Version 4 Protocol", 2013, . 1262 Work in progress. 1264 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 1265 System (NFS) Version 4 Minor Version 1 Protocol", RFC 1266 5661, January 2010. 1268 11.2. Informative References 1270 [NFS-ext] Noveck, D., "NFS Protocol Extension: Retrospect and 1271 Prospect", 2013, . 1274 Work in progress. 1276 [migr-v4.0-update] 1277 Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, 1278 "NFSv4.0 migration: Specification Update", 2013, . 1282 Work in progress. 1284 Authors' Addresses 1286 David Noveck (editor) 1287 EMC Corporation 1288 228 South Street 1289 Hopkinton, MA 01748 1290 US 1292 Phone: +1 508 249 5748 1293 Email: david.noveck@emc.com 1295 Piyush Shivam 1296 Oracle Corporation 1297 5300 Riata Park Ct. 1298 Austin, TX 78727 1299 US 1301 Phone: +1 512 401 1019 1302 Email: piyush.shivam@oracle.com 1304 Charles Lever 1305 Oracle Corporation 1306 1015 Granger Avenue 1307 Ann Arbor, MI 48104 1308 US 1310 Phone: +1 248 614 5091 1311 Email: chuck.lever@oracle.com 1313 Bill Baker 1314 Oracle Corporation 1315 5300 Riata Park Ct. 1316 Austin, TX 78727 1317 US 1319 Phone: +1 512 401 1081 1320 Email: bill.baker@oracle.com