idnits 2.17.1 draft-ietf-nfsv4-migration-issues-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 24, 2015) is 3169 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) == Outdated reference: A later version (-11) exists of draft-ietf-nfsv4-versioning-01 Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 D. Noveck, Ed. 3 Internet-Draft HP 4 Intended status: Informational P. Shivam 5 Expires: February 25, 2016 C. Lever 6 B. Baker 7 ORACLE 8 August 24, 2015 10 NFSv4 migration: Implementation experience and spec issues to resolve 11 draft-ietf-nfsv4-migration-issues-08 13 Abstract 15 The migration feature of NFSv4 provides for moving responsibility for 16 a single filesystem from one server to another, without disruption to 17 clients. Recent implementation experience has shown problems in the 18 existing specification for this feature. This document discusses the 19 issues which have arisen, explores the options available for curing 20 the issues, and explains the choices to be made in updating the 21 NFSv4.0 and NFSv4.1 specifications, to properly address migration. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on February 25, 2016. 40 Copyright Notice 42 Copyright (c) 2015 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 59 3. NFSv4.0 Implementation Experience . . . . . . . . . . . . . . 4 60 3.1. Implementation issues . . . . . . . . . . . . . . . . . . 4 61 3.1.1. Failure to free migrated state on client reboot . . . 4 62 3.1.2. Server reboots resulting in a confused lease 63 situation . . . . . . . . . . . . . . . . . . . . . . 5 64 3.1.3. Client complexity issues . . . . . . . . . . . . . . 6 65 3.2. Sources of Protocol difficulties . . . . . . . . . . . . 7 66 3.2.1. Issues with nfs_client_id4 generation and use . . . . 7 67 3.2.2. Issues with lease proliferation . . . . . . . . . . . 9 68 4. Issues to be resolved in NFSv4.0 . . . . . . . . . . . . . . 10 69 4.1. Possible changes to nfs_client_id4 client-string . . . . 10 70 4.2. Possible changes to handle differing nfs_client_id4 71 string values . . . . . . . . . . . . . . . . . . . . . . 11 72 4.3. Possible changes to add a new operation . . . . . . . . . 12 73 4.4. Other issues within migration-state sections . . . . . . 12 74 4.5. Issues within other sections . . . . . . . . . . . . . . 12 75 5. Proposed resolution of NFSv4.0 protocol difficulties . . . . 13 76 5.1. Proposed changes: nfs_client_id4 client-string . . . . . 13 77 5.2. Proposed changes: merged (vs. synchronized) leases . . . 14 78 5.3. Other proposed changes to migration-state sections . . . 15 79 5.3.1. Proposed changes: Client ID migration . . . . . . . . 15 80 5.3.2. Proposed changes: Callback re-establishment . . . . . 16 81 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework . . . . 16 82 5.4. Proposed changes to other sections . . . . . . . . . . . 17 83 5.4.1. Proposed changes: callback update . . . . . . . . . . 17 84 5.4.2. Proposed changes: clientid4 handling . . . . . . . . 17 85 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE . . . . . . . . 19 86 6. Results of proposed changes for NFSv4.0 . . . . . . . . . . . 20 87 6.1. Results: Failure to free migrated state on client reboot 20 88 6.2. Results: Server reboots resulting in confused lease 89 situation . . . . . . . . . . . . . . . . . . . . . . . . 21 90 6.3. Results: Client complexity issues . . . . . . . . . . . . 22 91 6.4. Result summary . . . . . . . . . . . . . . . . . . . . . 23 92 7. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . 23 93 7.1. Addressing state merger in NFSv4.1 . . . . . . . . . . . 24 94 7.2. Addressing pNFS relationship with migration . . . . . . . 24 95 7.3. Addressing server owner changes in NFSv4.1 . . . . . . . 25 96 8. Security Considerations . . . . . . . . . . . . . . . . . . . 26 97 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 98 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 99 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 26 100 11.1. Normative References . . . . . . . . . . . . . . . . . . 26 101 11.2. Informative References . . . . . . . . . . . . . . . . . 27 102 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 27 104 1. Introduction 106 This document is in the informational category, and while the facts 107 it reports may have normative implications, any such normative 108 significance reflects the readers' preferences. For example, we may 109 report that the reboot of a client with migrated state results in 110 state not being promptly cleared and that this will prevent granting 111 of conflicting lock requests at least for the lease time, which is a 112 fact. While it is to be expected that client and server implementers 113 will judge this to be a situation that is best avoided, the judgment 114 as to how pressing this issue should be considered is a judgment for 115 the reader, and eventually the nfsv4 working group to make. 117 We do explore possible ways in which such issues can be avoided, with 118 minimal negative effects, given that the working group has decided to 119 address these issues, but the choice of exactly how to address these 120 is best given effect in one or more standards-track documents and/or 121 errata. 123 This document focuses on NFSv4.0, since that is where the majority of 124 implementation experience has been. Nevertheless, there is 125 discussion of the implications of the NFSv4.0 experience for 126 migration in NFSv4.1, as well as discussion of other issues with 127 regard to the treatment of migration in NFSv4.1. 129 2. Conventions 131 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 132 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 133 document are to be interpreted as described in [RFC2119]. 135 In the context of this informational document, these normative 136 keywords will always occur in the context of a quotation, most often 137 direct but sometimes indirect. The context will make it clear 138 whether the quotation is from: 140 o The current definitive definition of the NFSv4.0 protocol 141 [RFC7530]. 143 o The current definitive definition of the NFSv4.1 protocol 144 [RFC5661]. 146 o A proposed or possible text to serve as a replacement for the 147 current definitive document text. Sometimes, a number of possible 148 alternative texts may be listed and benefits and detriments of 149 each examined in turn. 151 3. NFSv4.0 Implementation Experience 153 3.1. Implementation issues 155 Note that the examples below reflect current experience which arises 156 from clients implementing the recommendation to use different 157 nfs_client_id4 id strings for different server addresses, i.e. using 158 what is later referred to herein as the "non-uniform client-string 159 approach." 161 This is simply because that is the experience implementers have had. 162 The reader should not assume that in all cases, this practice is the 163 source of the difficulty. It may be so in some cases but clearly it 164 is not in all cases. 166 3.1.1. Failure to free migrated state on client reboot 168 The following sort of situation has proved troublesome: 170 o A client C establishes a clientid4 C1 with server ABC specifying 171 an nfs_client_id4 with id string value "C-ABC" and boot verifier 172 0x111. 174 o The client begins to access files in filesystem F on server ABC, 175 resulting in generating stateids S1, S2, etc. under the lease for 176 clientid C1. It may also access files on other filesystems on the 177 same server. 179 o The filesystem is migrated from server ABC to server XYZ. When 180 transparent state migration is in effect, stateids S1 and S2 and 181 clientid4 C1 are now available for use by client C at server XYZ. 183 o Client C reboots and attempts to access data on server XYZ, 184 whether in filesystem F or another. It does a SETCLIENTID with an 185 nfs_client_id4 with id string value "C-XYZ" and boot verifier 186 0x112. There is thus no occasion to free stateids S1 and S2 since 187 they are associated with a different client name and so lease 188 expiration is the only way that they can be gotten rid of. 190 Note here that while it seems clear to us in this example that C-XYZ 191 and C-ABC are from the same client, the server has no way to 192 determine the structure of the "opaque" id string. In the protocol, 193 it really is treated as opaque. Only the client knows which 194 nfs_client_id4 values designate the same client on a different 195 server. 197 3.1.2. Server reboots resulting in a confused lease situation 199 Further problems arise from scenarios like the following. 201 o Client C talks to server ABC using an nfs_client_id4 id string 202 such as "C-ABC" and a boot verifier v1. As a result, a lease with 203 clientid4 c.i is established: {v1, "C-ABC", c.i}. 205 o fs_a1 migrates from server ABC to server XYZ along with its state. 206 Now server XYZ also has a lease: {v1, "C-ABC", c.i}. 208 o Server ABC reboots. 210 o Client C talks to server ABC using an nfs_client_id4 id string 211 such as "C-ABC" and a boot verifier v1. As a result, a lease with 212 clientid4 c.j is established: {v1, "C-ABC", c.j}. 214 o fs_a2 migrates from server ABC to server XYZ. Now server XYZ also 215 has a lease: {v1, "C-ABC", c.j}. 217 o Now server XYZ has two leases that match {v1, "C-ABC", *}, when 218 the protocol clearly assumes there can be only one. 220 Note that if the client used "C" (rather than "C-ABC") as the 221 nfs_client_id4 id string, the exact same situation would arise. 223 One of the first cases in which this sort of situation has resulted 224 in difficulties is in connection with doing a SETCLIENTID for 225 callback update. 227 The SETCLIENTID for callback update only includes the nfs_client_id4, 228 assuming there can only be one such with a given nfs_client_id4 229 value. If there were multiple, confirmed client records with 230 identical nfs_client_id4 id string values, there would be no way to 231 map the callback update request to the correct client record. Apart 232 from the migration handling specified in [RFC7530], such a situation 233 cannot arise. 235 One possible accommodation for this particular issue that has been 236 used is to add a RENEW operation along with SETCLIENTID (on a 237 callback update) to disambiguate the client. 239 When the client updates the callback info to the destination, the 240 client would, by convention, send a compound like this: 242 { RENEW clientid4, SETCLIENTID nfs_client_id4,verf,cb } 244 The presence of the clientid4 in the compound would allow the server 245 to differentiate among the various leases that it knows of, all with 246 the same nfs_client_id4 value. 248 While this would be a reasonable patch for an isolated protocol 249 weakness, interoperable clients and servers would require that the 250 protocol truly be updated to allow such a situation, specifically 251 that of multiple clientid4's with the same nfs_client_id4 value. The 252 protocol is currently designed and implemented assuming this cannot 253 happen. We need to either prevent the situation from happening, or 254 fully adapt to the possibilities which can arise. See Section 4 for 255 a discussion of such issues. 257 3.1.3. Client complexity issues 259 Consider the following situation: 261 o There are a set of clients C1 through Cn accessing servers S1 262 through Sm. Each server manages some significant number of 263 filesystems with the filesystem count L being significantly 264 greater than m. 266 o Each client Cx will access a subset of the servers and so will 267 have up to m clientids, which we will call Cxy for server Sy. 269 o Now assume that for load-balancing or other operational reasons, 270 numbers of filesystems are migrated among the servers. As a 271 result, each client-server pair will have up to m clientids and 272 each client will have up to m**2 clientids. If we add the 273 possibility of server reboot, the only bound on a client's 274 clientid count is L. 276 Now, instead of a clientid4 identifying a client-server pair, we have 277 many more entities for the client to deal with. In addition, it 278 isn't clear how new state is to be incorporated in this structure. 280 The limitations of the migrated state (inability to be freed on 281 reboot) would argue against adding more such state but trying to 282 avoid that would run into its own difficulties. For example, a 283 single lockowner string presented under two different clientids would 284 appear as two different entities. 286 Thus we have to choose between: 288 o indefinite prolongation of foreign clientids even after all 289 transferred state is gone. 291 o having multiple requests for the same lockowner-string-named 292 entity carried on in parallel by separate identically named 293 lockowners under different clientid4's 295 o Adding serialization at the lock-owner string level, in addition 296 to that at the lockowner level. 298 In any case, we have gone (in adding migration as it was described) 299 from a situation in which 301 o Each client has a single clientid4/lease for each server it talks 302 to. 304 o Each client has a single nfs_client_id4 for each server it talks 305 to. 307 o Every state id can be mapped to an associated lease based on the 308 server it was obtained from. 310 To one in which 312 o Each client may have multiple clientid4's for a single server. 314 o For each stateid, the client must separately record the clientid4 315 that it is assigned to, or it must manage separate "state blobs" 316 for each fsid and map those to clientid4's. 318 o Before doing an operation that can result in a stateid, the client 319 must either find a "state blob" based on fsid or create a new one, 320 possibly with a new clientid4. 322 o There may be multiple clientid4's all connected to the same server 323 and using the same nfs_clientid4. 325 This sort of additional client complexity is troublesome and needs to 326 be eliminated. 328 3.2. Sources of Protocol difficulties 330 3.2.1. Issues with nfs_client_id4 generation and use 332 In the current definitive definition of the NFSv4.0 protocol, 333 [RFC7530], the section entitled "Client ID" says: 335 The second field, id is a variable length string that uniquely 336 defines the client. 338 There are two possible interpretations of the phrase "uniquely 339 defines" in the above: 341 o The relation between strings and clients is a function from such 342 strings to clients so that each string designates a single client. 344 o The relation between strings and clients is a bijection between 345 such strings and clients so that each string designates a single 346 client and each client is named by a single string. 348 The first interpretation would make these client-strings like phone 349 numbers (a single person can have several) while the second would 350 make them like social security numbers. 352 Endless debate about the true meaning of "uniquely defines" in this 353 context is quite possible but not very helpful. The following points 354 should be noted though: 356 o The second interpretation is more consistent with the way 357 "uniquely defines" is used elsewhere in the spec. 359 o The spec as now written intends the first interpretation (or is 360 internally inconsistent). In fact, it recommends, although it 361 doesn't "RECOMMEND" that a single client have at least as many 362 client-strings as server addresses that it interacts with. It 363 says, in the third bullet point regarding construction of the 364 string (which we shall henceforth refer to as client-string-BP3): 366 The string should be different for each server network address 367 that the client accesses, rather than common to all server 368 network addresses. 370 o If internode interactions are limited to those between a client 371 and its servers, there is no occasion for servers to be concerned 372 with the question of whether two client-strings designate the same 373 client, so that there is no occasion for the difference in 374 interpretation to matter. 376 o When transparent migration of client state occurs between two 377 servers, it becomes important to determine when state on two 378 different servers is for the same client or not, and this 379 distinction becomes very important. 381 Given the need for the server to be aware of client identity with 382 regard to migrated state, either client-string construction rules 383 will have to change or there will be a need to get around current 384 issues, or perhaps a combination of these two will be required. 385 Later sections will examine the options and propose a solution. 387 One consideration that may indicate that this cannot remain exactly 388 as it is today has to do with the fact that the current explanation 389 for this behavior is not correct. In the current definitive 390 definition of the NFSv4.0 protocol [RFC7530], the section entitled 391 "Client ID" says: 393 The reason is that it may not be possible for the client to tell 394 if the same server is listening on multiple network addresses. If 395 the client issues SETCLIENTID with the same id string to each 396 network address of such a server, the server will think it is the 397 same client, and each successive SETCLIENTID will cause the server 398 to begin the process of removing the client's previous leased 399 state. 401 In point of fact, a "SETCLIENTID with the same id string" sent to 402 multiple network addresses will be treated as all from the same 403 client but will not "cause the server to begin the process of 404 removing the client's previous leased state" unless the server 405 believes it is a different instance of the same client, i.e. if the 406 id string is the same and there is a different boot verifier. If the 407 client does not reboot, the verifier should not change. If it does 408 reboot, the verifier will change, and the server should "begin the 409 process of removing the client's previous leased state. 411 The situation of multiple SETCLIENTID requests received by a server 412 on multiple network addresses is exactly the same, from the protocol 413 design point of view, as when multiple (i.e. duplicate) SETCLIENTID 414 requests are received by the server on a single network address. The 415 same protocol mechanisms that prevent erroneous state deletion in the 416 latter case prevent it in the former case. There is no reason for 417 special handling of the multiple-network-appearance case, in this 418 regard. 420 3.2.2. Issues with lease proliferation 422 It is often felt that this is a consequence of the client-string 423 construction issues, and it is certainly the case that the two are 424 closely connected in that non-uniform client-strings make it 425 impossible for the server to appropriately combine leases from the 426 same client. 428 However, even where the server could combine leases from the same 429 client, it needs to be clear how and when it will do so, so that the 430 client will be prepared. These issues will have to be addressed at 431 various places in the spec. 433 This could be enough only if we are prepared to do away with the 434 "should" recommending non-uniform client-strings and replace it with 435 a "should not" or even a "SHOULD NOT". Current client implementation 436 patterns make this an unpalatable choice for use as a general 437 solution, but it is reasonable to "RECOMMEND" this choice for a well- 438 defined subset of clients. One alternative would be to create a way 439 for the server to infer from client behavior which leases are held by 440 the same client and use this information to do appropriate lease 441 mergers. Prototyping and detailed specification work has shown that 442 this could be done but the resulting complexity is such that a better 443 choice is to "RECOMMEND" use of the uniform client-string approach 444 for clients supporting the migration feature. 446 Because of the discussion of client-string construction in [RFC7530], 447 most existing clients implement the non-uniform client-string 448 approach. As a result, existing servers may not have been tested 449 with clients implementing uniform client-strings. As a consequence, 450 care must be taken to preserve interoperability between UCS-capable 451 clients and servers that don't tolerate uniform client strings for 452 one reason or another. 454 4. Issues to be resolved in NFSv4.0 456 4.1. Possible changes to nfs_client_id4 client-string 458 The fact that the reason given in client-string-BP3 is not valid 459 makes the existing "should" insupportable. We can't either 461 o Keep a reason we know is invalid. 463 o Keep saying "should" without giving a reason. 465 What are often presented as reasons that motivate use of the non- 466 uniform approach always turn out to be cases in which, if the uniform 467 approach were used, the server will treat a client which accesses 468 that server via two different IP addresses as part of a single 469 client, as it in fact is. This may be disconcerting to a client 470 unaware that the two IP addresses connect to the same server. This 471 is not a reason to use the non-uniform approach but is better thought 472 of as an illustration of the fact that those using the uniform 473 approach need to be aware of the possibility of server trunking and 474 its effect on server behavior. 476 If it is possible to reliably infer the existence of trunking of 477 server IP addresses from observed server behavior, use of the uniform 478 approach would be more desirable, although compatibility issues would 479 have to be dealt with. 481 An alternative to having the client infer the existence of trunking 482 of IP server addresses, is to make this information available to the 483 client directly. See Section 4.3 for details. 485 It is always possible that a valid new reason will be found, but so 486 far none has been proposed. Given the history, the burden of proof 487 should be on those asserting the validity of a proposed new reason. 489 So we will assume for now that the "should" will have to go. The 490 question is what to replace it with. 492 o We can't say "MUST NOT", despite the problems this raises for 493 migration since this is pretty late in the day for such a change. 494 Many currently operating clients obey the existing "should". 495 Similar considerations would apply for "SHOULD NOT" or "should 496 not". 498 o Dropping client-string-BP3 entirely is a possibility but, given 499 the context and history, it would just be a confusing version of 500 "SHOULD NOT". 502 o Using "MAY" would clearly specify that both ways of doing this are 503 valid choices for clients and that servers will have to deal with 504 clients that make either choice. 506 o This might be modified by a "SHOULD" (or even a "MUST") for 507 particular groups of clients. 509 o There will have to be some text explaining why a client might make 510 either choice but, except for the particular cases referred to 511 above, we will have to make sure that it is truly descriptive, and 512 not slanted in either direction. 514 4.2. Possible changes to handle differing nfs_client_id4 string values 516 Given the difficulties caused by having different nfs_client_id4 517 client-string values for the same client, we have two choices: 519 o Deprecate the existing treatment and basically say the client is 520 on its own doing migration, if it follows it. 522 o Introduce a way of having the client provide client identity 523 information to the server, if it can be done compatibly while 524 staying within the bounds of v4.0. 526 4.3. Possible changes to add a new operation 528 It might be possible to return server-identity information to the 529 client, just as is done in NFSv4.1 by the response to the EXCHANGE_ID 530 operation. This could be done by a SETCLIENTID_PLUS optional 531 operation, which acts like SETCLIENTID, except that it returns server 532 identity information. Such information could be used by clients, 533 making it possible to for them to be aware of server trunking 534 relationships, rather than having to infer them from server behavior. 536 It has been generally thought that protocol extensions such as this 537 are not appropriate in bis documents and other documents updating 538 NFSv4 protocol definition RFC's. However, [NFSv4-vers] discusses 539 means by which protocol extensions, similar to those allowed between 540 minor versions, can be used to correct protocol mistakes. 542 A decision to adopt this approach would require waiting for 543 [NFSv4-vers] to become a Proposed Standard. In view of the time 544 necessary for that to happen, this approach is not expected to be 545 adopted in an RFC updating [RFC7530], such as [migr-v4.0-update]. 546 Still, it is worth keeping in mind, if implementers have difficulties 547 inferring trunking relationships using the techniques discussed 548 there. 550 4.4. Other issues within migration-state sections 552 There are a number of issues where the existing text is unclear and/ 553 or wrong and needs to be fixed in some way. 555 o Lack of clarity in the discussion of moving clientids (as well as 556 stateids) as part of moving state for migration. 558 o The discussion of synchronized leases is wrong in that there is no 559 way to determine (in the current spec) when leases are for the 560 same client and also wrong in suggesting a benefit from leases 561 synchronized at the point of transfer. What is needed is merger 562 of leases, which is necessary to keep client complexity 563 requirements from getting out of hand. 565 o Lack of clarity in the discussion of LEASE_MOVED handling, 566 including failure to fully address situations in which transparent 567 state migration did not occur. 569 4.5. Issues within other sections 571 There are a number of cases in which certain sections, not 572 specifically related to migration, require additional clarification. 573 This is generally because text that is clear in a context in which 574 leases and clientids are created in one place and live there forever 575 may need further refinement in the more dynamic environment that 576 arises as part of migration. 578 Some examples: 580 o Some people are under the impression that updating callback 581 endpoint information for an existing client, as used during 582 migration, may cause the destination server to free existing 583 state. There need to be additions to clarify the situation. 585 o The handling of the sets of clientid4's maintained by each server 586 needs to be clarified. In particular, the issue of how the client 587 adapts to the presumably independent and uncoordinated clientid4 588 sets needs to be clearly addressed 590 o Statements regarding handling of invalid clientid4's need to be 591 clarified and/or refined in light of the possibilities that arise 592 due to lease motion and merger. 594 o Confusion and lack of clarity about NFS4ERR_CLID_INUSE. 596 5. Proposed resolution of NFSv4.0 protocol difficulties 598 This section lists the changes which we believe are necessary to 599 resolve the difficulties mentioned above. Such change, along with 600 other clarifications found to be desirable during drafting and review 601 are contained in [migr-v4.0-update]. 603 5.1. Proposed changes: nfs_client_id4 client-string 605 We propose replacing client-string-BP3 with the following text and 606 adding the following proposed to provide implementation guidance. 608 The string MAY be different for each server network address that 609 the client accesses, rather than common to all server network 610 addresses. 612 In addition, given the importance of the issue of client identity and 613 the fact that both client string-approaches are to be considered 614 valid, a greatly expanded treatment of client identity desirable. It 615 should have the following major elements. 617 o It should fully describe the consequences of making the string 618 different for each network address (the non-uniform client-string 619 approach) and of making it the same for all network addresses (the 620 uniform client string approach). 622 o It should give helpful guidance about the factors that might 623 affect client implementation choice between these approaches. 625 o It should describe the compatibility issues that might cause 626 servers to be incompatible with the uniform approach and give 627 guidance about dealing with these. 629 o It should describe how a client using the uniform approach might 630 use server behavior to determine server address trunking patterns. 632 o It should present a clearer and more complete set of 633 recommendations to guide client string construction. 635 5.2. Proposed changes: merged (vs. synchronized) leases 637 In the current definitive definition of the NFSv4.0 protocol, 638 [RFC7530], the section entitled "Migration and State" says: 640 As part of the transfer of information between servers, leases 641 would be transferred as well. The leases being transferred to the 642 new server will typically have a different expiration time from 643 those for the same client, previously on the old server. To 644 maintain the property that all leases on a given server for a 645 given client expire at the same time, the server should advance 646 the expiration time to the later of the leases being transferred 647 or the leases already present. This allows the client to maintain 648 lease renewal of both classes without special effort: 650 There are a number of problems with this and any resolution of our 651 difficulties must address them somehow. 653 o The current v4.0 spec recommends that the client make it 654 essentially impossible to determine when two leases are from "the 655 same client". 657 o It is not appropriate to speak of "maintain[ing] the property that 658 all leases on a given server for a given client expire at the same 659 time", since this is not a property that holds even in the absence 660 of migration. A server listening on multiple network addresses 661 may have the same client appear as multiple clients with no way to 662 recognize the client as the same. 664 o Even if the client identity issue could be resolved, advancing the 665 lease time at the point of migration would not maintain the 666 desired synchronization property. The leases would be 667 synchronized until one of them was renewed, after which they would 668 be unsynchronized again. 670 To avoid client complexity, we need to have no more than one lease 671 between a single client and a single server. This requires merger of 672 leases since there is no real help from synchronizing them at a 673 single instant. 675 For the uniform approach, the destination server would simply merge 676 leases as part of state transfer, since two leases with the same 677 nfs_client_id4 values must be for the same client. 679 We have made the following decisions as far as proposed normative 680 statements regarding for state merger. They reflect the facts that 681 we want to support fully migration support in the simplest way 682 possible and that we can't say MUST since we have older clients and 683 servers to deal with. 685 o Clients SHOULD use the uniform client-string approach in order to 686 get good migration support. 688 o Servers SHOULD provide automatic lease merger during state 689 migration so that clients using the uniform id approach get the 690 support automatically. 692 If the clients and the servers obey the SHOULD's, having more than a 693 single lease for a given client-server pair will be a transient 694 situation, cleaned up as part of adapting to use of migrated state. 696 Since clients and servers will be a mixture of old and new and 697 because nothing is a MUST we have to ensure that no combination will 698 show worse behavior than is exhibited by current (i.e. old) clients 699 and servers. 701 5.3. Other proposed changes to migration-state sections 703 5.3.1. Proposed changes: Client ID migration 705 In the current definitive definition of the NFSv4.0 protocol 706 [RFC7530], the section entitled "Migration and State" says: 708 In the case of migration, the servers involved in the migration of 709 a filesystem SHOULD transfer all server state from the original to 710 the new server. This must be done in a way that is transparent to 711 the client. This state transfer will ease the client's transition 712 when a filesystem migration occurs. If the servers are successful 713 in transferring all state, the client will continue to use 714 stateids assigned by the original server. Therefore the new 715 server must recognize these stateids as valid. This holds true 716 for the client ID as well. Since responsibility for an entire 717 filesystem is transferred with a migration event, there is no 718 possibility that conflicts will arise on the new server as a 719 result of the transfer of locks. 721 This poses some difficulties, mostly because the part about "client 722 ID" is not clear: 724 o It isn't clear what part of the paragraph the "this" in the 725 statement "this holds true ..." is meant to signify. 727 o The phrase "the client ID" is ambiguous, possibly indicating the 728 clientid4 and possibly indicating the nfs_client_id4. 730 o If the text means to suggest that the same clientid4 must be used, 731 the logic is not clear since the issue is not the same as for 732 stateids of which there might be many. Adapting to the change of 733 a single clientid, as might happen as a part of lease migration, 734 is relatively easy for the client. 736 We have decided that it is best to address this issue as follows: 738 o Make it clear that both clientid4 and nfs_client_id4 (including 739 both id string and boot verifier) are to be transferred. 741 o Indicate that the initial transfer will result in the same 742 clientid4 after transfer but this is not guaranteed since there 743 may conflict with an existing clientid4 on the destination server 744 and because lease merger can result in a change of the clientid4. 746 5.3.2. Proposed changes: Callback re-establishment 748 In the current definitive definition of the NFSv4.0 protocol 749 [RFC7530], the section entitled "Migration and State" says: 751 A client SHOULD re-establish new callback information with the new 752 server as soon as possible, according to sequences described in 753 sections "Operation 35: SETCLIENTID - Negotiate Client ID" and 754 "Operation 36: SETCLIENTID_CONFIRM - Confirm Client ID". This 755 ensures that server operations are not blocked by the inability to 756 recall delegations. 758 The above will need to be fixed to reflect the possibility of merging 759 of leases, 761 5.3.3. Proposed changes: NFS4ERR_LEASE_MOVED rework 763 In the current definitive definition of the NFSv4.0 protocol 764 [RFC7530], the section entitled "Notification of Migrated Lease" 765 says: 767 Upon receiving the NFS4ERR_LEASE_MOVED error, a client that 768 supports filesystem migration MUST probe all filesystems from that 769 server on which it holds open state. Once the client has 770 successfully probed all those filesystems which are migrated, the 771 server MUST resume normal handling of stateful requests from that 772 client. 774 There is a lack of clarity that is prompted by ambiguity about what 775 exactly probing is and what the interlock between client and server 776 must be. This has led to some worry about the scalability of the 777 probing process, and although the time required does scale linearly 778 with the number of filesystems that the client may have state for 779 with respect to a given server, the actual process can be done 780 efficiently. 782 To address these issues we propose rewriting the above to be more 783 clear and to give suggestions about how to do the required scanning 784 efficiently. 786 5.4. Proposed changes to other sections 788 5.4.1. Proposed changes: callback update 790 Some changes are necessary to reduce confusion about the process of 791 callback information update and in particular to make it clear that 792 no state is freed as a result: 794 o Make it clear that after migration there are confirmed entries for 795 transferred clientid4/nfs_client_id4 pairs. 797 o Be explicit in the sections headed "otherwise," in the 798 descriptions of SETCLIENTID and SETCLIENTID_CONFIRM, that these 799 don't apply in the cases we are concerned about. 801 5.4.2. Proposed changes: clientid4 handling 803 To address both of the clientid4-related issues mentioned in 804 Section 4.5, we propose replacing the last three paragraphs of the 805 section entitled "Client ID" with the following: 807 Once a SETCLIENTID and SETCLIENTID_CONFIRM sequence has 808 successfully completed, the client uses the shorthand client 809 identifier, of type clientid4, instead of the longer and less 810 compact nfs_client_id4 structure. This shorthand client 811 identifier (a client ID) is assigned by the server and should be 812 chosen so that it will not conflict with a client ID previously 813 assigned by same server. This applies across server restarts or 814 reboots. 816 Distinct servers MAY assign clientid4's independently, and will 817 generally do so. Therefore, a client has to be prepared to deal 818 with multiple instances of the same clientid4 value received on 819 distinct IP addresses, denoting separate entities. When trunking 820 of server IP addresses is not a consideration, a client should 821 keep track of (IP-address, clientid4) pairs, so that each pair is 822 distinct. In the face of possible trunking of server IP 823 addresses, the client will use the receipt of the same clientid4 824 from multiple IP-addresses, as an indication that the two IP- 825 addresses may be trunked and proceed to determine, from the 826 observed server behavior whether the two addresses are in fact 827 trunked. 829 When a clientid4 is presented to a server and that clientid4 is 830 not recognized, the server will reject the request with the error 831 NFS4ERR_STALE_CLIENTID. This can occur for a number of reasons: 833 * A server reboot causing loss of the server's knowledge of the 834 client 836 * Client error sending an incorrect clientid4 or a valid 837 clientid4 to the wrong server. 839 * Loss of lease state due to lease expiration. 841 * Client or server error causing the server to believe that the 842 client has rebooted (i.e. receiving a SETCLIENTID with an 843 nfs_client_id4 which has a matching id string and a non- 844 matching boot verifier). 846 * Migration of all state under the associated lease causes its 847 non-existence to be recognized on the source server. 849 * Merger of state under the associated lease with another lease 850 under a different clientid causes the clientid4 serving as the 851 source of the merge to cease being recognized on its server. 853 In the event of a server reboot, or loss of lease state due to 854 lease expiration, the client must obtain a new clientid4 by use of 855 the SETCLIENTID operation and then proceed to any other necessary 856 recovery for the server reboot case (See the section entitled 857 "Server Failure and Recovery"). In cases of server or client 858 error resulting in this error, use of SETCLIENTID to establish a 859 new lease is desirable as well. 861 In the last two cases, different recovery procedures are required. 862 Note that in cases in which there is any uncertainty about which 863 sort of handling is applicable, the distinguishing characteristic 864 is that in reboot-like cases, the clientid4 and all associated 865 stateids cease to exist while in migration-related cases, the 866 clientid4 ceases to exist while the stateids are still valid. 868 The client must also employ the SETCLIENTID operation when it 869 receives a NFS4ERR_STALE_STATEID error using a stateid derived 870 from its current clientid4, since this indicates a situation, such 871 as server reboot which has invalidated the existing clientid4 and 872 associated stateids (see the section entitled "lock-owner" for 873 details). 875 See the detailed descriptions of SETCLIENTID and 876 SETCLIENTID_CONFIRM for a complete specification of the 877 operations. 879 5.4.3. Proposed changes: NFS4ERR_CLID_INUSE 881 It appears to be the intention that only a single principal be used 882 for client establishment between any client-server pair. However: 884 o There is no explicit statement to this effect. 886 o The error that indicates a principal conflict has a name which 887 does not clarify this issue: NFS4ERR_CLID_INUSE. 889 o The definition of the error is also not very helpful: "The 890 SETCLIENTID operation has found that a client id is already in use 891 by another client". 893 As a result, servers exist which reject a SETCLIENTID simply because 894 there already exists a clientid for the same client, established 895 using a different IP address. Although this is generally understood 896 to be erroneous, such servers still exist and the spec should make 897 the correct behavior clear. 899 Although the error name cannot be changed, the following changes 900 should be made to avoid confusion: 902 o The definition of the error should be changed to read as follows: 904 The SETCLIENTID operation has found that the specified 905 nfs_client_id4 was previously presented with a different 906 principal and that client instance currently holds an active 907 lease. A server MAY return this error if the same principal is 908 used but a change in authentication flavor gives good reason to 909 reject the new SETCLIENTID operation as not bona fide. 911 o In the description of SETCLIENTID, the phrase "then the server 912 returns a NFS4ERR_CLID_INUSE error" should be expanded to read 913 "then the server returns a NFS4ERR_CLID_INUSE error, since use of 914 a single client with multiple principals is not allowed." 916 6. Results of proposed changes for NFSv4.0 918 The purpose of this section is to examine the troubling results 919 reported in Section 3.1. We will look at the scenarios as they would 920 be handled within the proposal. 922 Because the choice of uniform vs. non-uniform nfs_client_id4 id 923 strings is a "SHOULD" in these cases, we will designate clients that 924 follow this recommendation by SHOULD-UF-CID. 926 We will also have to take account of any merger-related "SHOULD" 927 clauses to better understand how they have addressed the issues seen. 928 We abbreviate as follows: 930 o SHOULD-SVR-AM refers to the server obeying the SHOULD which 931 RECOMMENDS that they merge leases with identical nfs_client_id4 id 932 strings and boot verifiers. 934 6.1. Results: Failure to free migrated state on client reboot 936 Let's look at the troublesome situation cited in Section 3.1.1. We 937 have already seen what happens when SHOULD-UF-CID does not hold. Now 938 let's look at the situation in which SHOULD-UF-CID holds, whether 939 SHOULD-SVR-AM is in effect or not. 941 o A client C establishes a clientid4 C1 with server ABC specifying 942 an nfs_client_id4 with id string value "C" and boot verifier 943 0x111. 945 o The client begins to access files in filesystem F on server ABC, 946 resulting in generating stateids S1, S2, etc. under the lease for 947 clientid C1. It may also access files on other filesystems on the 948 same server. 950 o The filesystem is migrated from ABC to server XYZ. When 951 transparent state migration is in effect, stateids S1 and S2 and 952 lease {0x111, "C", C1} are now available for use by client C at 953 server XYZ. 955 o Client C reboots and attempts to access data on server XYZ, 956 whether in filesystem F or another. It does a SETCLIENTID with an 957 nfs_client_id4 with id string value "C" and boot verifier 0x112. 959 The state associated with lease {0x111, "C", C1} is deleted as 960 part of creating {0x112, "C", C2}. No problem. 962 The correctness signature for this issue is 964 SHOULD-UF-CID 966 so if you have clients and servers that obey the SHOULD clauses, the 967 problem is gone regardless of the choice on the MAY. 969 6.2. Results: Server reboots resulting in confused lease situation 971 Now let's consider the scenario given in Section 3.1.2. We have 972 already seen what happens when SHOULD-UF-CID does not hold . Now 973 let's look at the situation in which SHOULD-UF-CID holds and SHOULD- 974 SVR-AM holds as well. 976 o Client C talks to server ABC using an nfs_client_id4 id string 977 such as "C-ABC" and boot verifier v1. As a result a lease with 978 clientid4 c.i established: {v1, "C-ABC", c.i}. 980 o Filesystem fs_a1 migrates from server ABC to server XYZ along with 981 its state. Now server XYZ also has a lease: {v1, "C-ABC", c.i} 983 o Server ABC reboots. 985 o Client C talks to server ABC using an nfs_client_id4 id string 986 such as "C-ABC" and boot verifier v1. As a result a lease with 987 clientid4 c.j established: {v1, "C-ABC", c.j}. 989 o fs_a2 migrates from server ABC to server XYZ. As part of 990 migration the incoming lease is seen to denote same nfs_client_id4 991 and so is merged with {v1, "C-ABC, c.i}. 993 o Now server XYZ has only one lease that matches {v1, "C_ABC", *}, 994 so the problem is solved 996 Now let's consider the same scenario in the situation in which 997 SHOULD-UF-CID holds and SHOULD-SVR-AM holds as well. 999 o Client C talks to server ABC using an nfs_client_id4 id string "C" 1000 and boot verifier v1. As a result a lease with clientid4 c.i is 1001 established: {v1, "C", c.i}. 1003 o fs_a1 migrates from server ABC to server XYZ along with its state. 1004 Now XYZ also has a lease: {v1, "C", c.i} 1006 o Server ABC reboots. 1008 o Client C talks to server ABC using an nfs_client_id4 id string "C" 1009 and boot verifier v1. As a result a lease with clientid4 c.j is 1010 established: {v1, "C", c.j}. 1012 o fs_a2 migrates from server ABC to server XYZ. As part of 1013 migration the incoming lease is seen to denote the same 1014 nfs_client_id4 and so is merged with {v1, "C", c.i}. 1016 o Now server XYZ has only one lease that matches {v1, "C", *}, so 1017 the problem is solved 1019 The correctness signature for this issue is 1021 SHOULD-SVR-AM 1023 so if you have clients and servers that obey the SHOULD clauses, the 1024 problem is gone regardless of the choice on the MAY. 1026 6.3. Results: Client complexity issues 1028 Consider the following situation: 1030 o There are a set of clients C1 through Cn accessing servers S1 1031 through Sm. Each server manages some significant number of 1032 filesystems with the filesystem count L being significantly 1033 greater than m. 1035 o Each client Cx will access a subset of the servers and so will 1036 have up to m clientids, which we will call Cxy for server Sy. 1038 o Now assume that for load-balancing or other operational reasons, 1039 numbers of filesystems are migrated among the servers. As a 1040 result, depending on how this handled, the number of clientids may 1041 explode. See below. 1043 Now look what will happen under various scenarios: 1045 o We have previously (in Section 3.1.3) looked at this in case of 1046 client following the non-uniform client-string approach. In that 1047 case, each client-server pair could have up to m clientids and 1048 each client will have up to m**2 clientids. If we add the 1049 possibility of server reboot, the only bound on a client's 1050 clientid count is L. 1052 o If we look at this in the SHOULD-UF-CID case in which the SHOULD- 1053 SVR_AM condition holds, the situation is no different. Although 1054 the server has the client identity information that could enable 1055 same-client-same-server leases to be combined, it does not do so. 1056 We still have up to L clientids per client. 1058 o On the other hand, if we look at the SHOULD-UF-CID case in which 1059 SHOULD-SVR-AM holds, the problem is gone. There can be no more 1060 than m clientids per client, and n clientids per server. 1062 The correctness signature for this issue is 1064 (SHOULD-UF-CID & SHOULD-SVR-AM) 1066 so if you have clients and servers that obey the SHOULD clauses, the 1067 problem is gone regardless of the choice on the MAY. 1069 6.4. Result summary 1071 We have seen that (SHOULD-SVR-AM & SHOULD-UF-CID) are sufficient to 1072 solve the problems people have experienced. 1074 7. Issues for NFSv4.1 1076 Because NFSv4.1 embraces the uniform client-string approach, 1077 addressing migration issues is simpler. In the terms of Section 6, 1078 we already have SHOULD-UF-CID, for NFSv4.1, as advised by section 2.4 1079 of [RFC5661], simplifying the work to be done. 1081 Nevertheless, there are some issues that will have to be addressed. 1082 Some examples: 1084 o The other necessary part of addressing migration issues, which we 1085 call above SHOULD-SVR-AM, is not currently addressed by NFSv4.1 1086 and changes need to be made to make it clear that state needs to 1087 be appropriately merged as part of migration, to avoid multiple 1088 clientids between a client-server pair. 1090 o There needs to be some clarification of how migration, and 1091 particularly transparent state migration, should interact with 1092 pNFS layouts. 1094 o The current discussion (in [RFC5661]), of the possibility of 1095 server_owner changes is incomplete and confusing. 1097 Discussion of how to resolve these issues will appear in the sections 1098 below. 1100 7.1. Addressing state merger in NFSv4.1 1102 The existing treatment of state transfer in [RFC5661], has similar 1103 problems to that in [RFC7530] in that it assumes that the state for 1104 multiple filesystems on different servers will not be merged to so 1105 that it appears under a single common clientid. We've already seen 1106 the reasons that this is a problem, with regard to NFSv4.0. 1108 Although we don't have the problems stemming from the non-uniform 1109 client-string approach, there are a number of complexities in the 1110 existing treatment of state management in the section entitled "Lock 1111 State and File System Transitions" in [RFC5661] that make this non- 1112 trivial to address: 1114 o Migration is currently treated together with other sorts of 1115 filesystem transitions including transitioning between replicas 1116 without any NFS4ERR_MOVED errors. 1118 o There is separate handling and discussion of the cases of matching 1119 and non-matching server scopes. 1121 o In the case of matching server scopes, the text calls for an 1122 impossible degree of transparency. 1124 o In the case of non-matching server scopes, the text does not 1125 mention transparent state migration at all, resulting in a 1126 functional regression from NFSV4.0 1128 7.2. Addressing pNFS relationship with migration 1130 This is made difficult because, within the PNFS framework, migration 1131 might mean any of several things: 1133 o Transfer of the MDS, leaving DS's alone. 1135 This would be minimally disruptive to those using layouts but 1136 would require the pNFS control protocol to support the DS being 1137 directed to a new MDS. 1139 o Transfer of a DS, leaving everything else in place. 1141 Such a transfer can be handled without using migration at all. 1142 The server can recall/revoke layouts, as appropriate. 1144 o Transfer of the filesystem to a new filesystem with both MDS and 1145 DS's moving. 1147 In such a transfer, an entirely different set of DS's will be at 1148 the target location. There may even be no pNFS support on the 1149 destination filesystem at all. 1151 Migration needs to support both the first and last of these models. 1153 7.3. Addressing server owner changes in NFSv4.1 1155 Section 2.10.5 of [RFC5661] states the following. 1157 The client should be prepared for the possibility that 1158 eir_server_owner values may be different on subsequent EXCHANGE_ID 1159 requests made to the same network address, as a result of various 1160 sorts of reconfiguration events. When this happens and the 1161 changes result in the invalidation of previously valid forms of 1162 trunking, the client should cease to use those forms, either by 1163 dropping connections or by adding sessions. For a discussion of 1164 lock reclaim as it relates to such reconfiguration events, see 1165 Section 8.4.2.1. 1167 While this paragraph is literally true in that such reconfiguration 1168 events can happen and clients have to deal with them, it is confusing 1169 in that it can be read as suggesting that clients have to deal with 1170 them without disruption, which in general is impossible. 1172 A clearer alternative would be: 1174 It is always possible that, as a result of various sorts of 1175 reconfiguration events, eir_server_scope and eir_server_owner 1176 values may be different on subsequent EXCHANGE_ID requests made to 1177 the same network address. 1179 In most cases such reconfiguration events will be disruptive and 1180 indicate that an IP address formerly connected to one server is 1181 now connected to an entirely different one. 1183 Some guidelines on client handling of such situations follow: 1185 * When eir_server_scope changes, the client has no assurance that 1186 any id's it obtained previously (e.g. file handles) can be 1187 validly used on the new server, and, even if the new server 1188 accepts them, there is no assurance that this is not due to 1189 accident. Thus it is best to treat all such state as lost/ 1190 stale although a client may assume that the probability of 1191 inadvertent acceptance is low and treat this situation as 1192 within the next case. 1194 * When eir_server_scope remains the same and 1195 eir_server_owner.so_major_id changes, the client can use 1196 filehandles it has and attempt reclaims. It may find that 1197 these are now stale but if NFS4ERR_STALE is not received, he 1198 can proceed to reclaim his opens. 1200 * When eir_server_scope and eir_server_owner.so_major_id remain 1201 the same, the client has to use the now-current values of 1202 eir_server-owner.so_minor_id in deciding on appropriate forms 1203 of trunking. 1205 8. Security Considerations 1207 With regard to NFSv4.0, the Security Considerations section of 1208 [RFC7530] encourages clients to protect the integrity of the SECINFO 1209 operation, any GETATTR operation for the fs_locations attribute, and 1210 the operations SETCLIENTID/SETCLIENTID_CONFIRM. A migration recovery 1211 event can use any or all of these operations. We do not recommend 1212 any change here. 1214 With regard to NFSv4.1, the Security Considerations section of 1215 [RFC5661] takes proper care of migration-related issues. No change 1216 is needed. 1218 9. IANA Considerations 1220 This document does not require actions by IANA. 1222 10. Acknowledgements 1224 The editor and authors of this document gratefully acknowledge the 1225 contributions of Trond Myklebust of NetApp and Robert Thurlow of 1226 Oracle. We also thank Tom Haynes of NetApp and Spencer Shepler of 1227 Microsoft for their guidance and suggestions. 1229 Special thanks go to members of the Oracle Solaris NFS team, 1230 especially Rick Mesta and James Wahlig, for their work implementing 1231 an NFSv4.0 migration prototype and identifying many of the issues 1232 documented here. 1234 11. References 1236 11.1. Normative References 1238 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1239 Requirement Levels", BCP 14, RFC 2119, 1240 DOI 10.17487/RFC2119, March 1997, 1241 . 1243 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 1244 "Network File System (NFS) Version 4 Minor Version 1 1245 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, 1246 . 1248 [RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System 1249 (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, 1250 March 2015, . 1252 11.2. Informative References 1254 [migr-v4.0-update] 1255 Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, 1256 "NFSv4.0 migration: Specification Update", 2015, 1257 . 1260 Work in progress. 1262 [NFSv4-vers] 1263 Noveck, D., "NFSv4 Version Management", 2015, 1264 . 1267 Work in progress. 1269 Authors' Addresses 1271 David Noveck (editor) 1272 Hewlett-Packard 1273 165 Dascomb Road 1274 Andover, MA 01810 1275 US 1277 Phone: +1 978 474 2011 1278 Email: davenoveck@gmail.com 1280 Piyush Shivam 1281 Oracle Corporation 1282 5300 Riata Park Ct. 1283 Austin, TX 78727 1284 US 1286 Phone: +1 512 401 1019 1287 Email: piyush.shivam@oracle.com 1288 Charles Lever 1289 Oracle Corporation 1290 1015 Granger Avenue 1291 Ann Arbor, MI 48104 1292 US 1294 Phone: +1 248 614 5091 1295 Email: chuck.lever@oracle.com 1297 Bill Baker 1298 Oracle Corporation 1299 5300 Riata Park Ct. 1300 Austin, TX 78727 1301 US 1303 Phone: +1 512 401 1081 1304 Email: bill.baker@oracle.com