idnits 2.17.1 draft-ietf-nfsv4-migration-issues-14.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1905 has weird spacing: '... pieces of th...' -- The document date (November 16, 2017) is 2351 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 D. Noveck, Ed. 3 Internet-Draft NetApp 4 Intended status: Informational P. Shivam 5 Expires: May 20, 2018 C. Lever 6 B. Baker 7 ORACLE 8 November 16, 2017 10 NFSv4 Migration and Trunking: Implementation and Specification Issues 11 draft-ietf-nfsv4-migration-issues-14 13 Abstract 15 This document discusses a range of implementation and specification 16 issues concerning features related to the use of location-related 17 attributes in NFSv4. These include migration, which transfers 18 responsibility for a file system from one server to another, and 19 trunking which deals with the discovery and control of the set of 20 network addresses to use to access a file system. The focus of the 21 discussion, which relates to multiple minor versions, is on defining 22 the appropriate clarifications and corrections for existing 23 specifications. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on May 20, 2018. 42 Copyright Notice 44 Copyright (c) 2017 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. Language . . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 2.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 62 2.2. Use of Normative Terms . . . . . . . . . . . . . . . . . 4 63 2.3. Terminology Used in this Document . . . . . . . . . . . . 5 64 3. Multi-version Issues and Their Resolution . . . . . . . . . . 6 65 3.1. Multi-Version Issues . . . . . . . . . . . . . . . . . . 6 66 3.2. Resolution of Multi-Version Issues . . . . . . . . . . . 7 67 3.2.1. Providing Trunking Discovery . . . . . . . . . . . . 8 68 3.2.2. Interaction of Trunking and Migration . . . . . . . . 9 69 4. NFSv4.0 Issues . . . . . . . . . . . . . . . . . . . . . . . 11 70 4.1. Core NFSv4.0 Migration Issues . . . . . . . . . . . . . . 11 71 4.2. Resolution of Core Migration Protocol Difficulties in 72 NFSv4.0 . . . . . . . . . . . . . . . . . . . . . . . . . 12 73 4.3. Additional NFSv4.0 Issues . . . . . . . . . . . . . . . . 12 74 4.4. Resolution of Additional NFSv4.0 Issues . . . . . . . . . 13 75 5. Issues for NFSv4.1 . . . . . . . . . . . . . . . . . . . . . 14 76 5.1. Issues to Address for NFSv4.1 . . . . . . . . . . . . . . 14 77 5.1.1. Addressing State Merger in NFSv4.1 . . . . . . . . . 15 78 5.1.2. Addressing pNFS Relationship with Migration . . . . . 15 79 5.1.3. Addressing Server_owner Changes in NFSv4.1 . . . . . 16 80 5.1.4. Addressing Confirmation Status of Migrated 81 Client IDs in NFSv4.1 . . . . . . . . . . . . . . . . 17 82 5.1.5. Addressing Changes in Trunking Configuration . . . . 18 83 5.1.6. Addressing Session Migration in NFSv4.1 . . . . . . . 19 84 5.2. Possible Resolutions for NFSv4.1 Protocol Issues . . . . 19 85 5.2.1. Client ID Confirmation Issues . . . . . . . . . . . . 19 86 5.2.2. Dealing with Multiple Location Entries . . . . . . . 20 87 5.2.3. Migration and pNFS . . . . . . . . . . . . . . . . . 23 88 5.3. Defining Server Responsibilities for NFSv4.1 . . . . . . 24 89 5.3.1. Server Responsibilities in Effecting Transparent 90 State Migration . . . . . . . . . . . . . . . . . . . 24 91 5.3.2. Synchronizing Session Transfer . . . . . . . . . . . 25 92 5.4. Defining Client Responsibilities for NFSv4.1 . . . . . . 28 93 5.4.1. Client Recovery from Migration Events . . . . . . . . 28 94 5.4.2. The Migration Discovery Process . . . . . . . . . . . 30 95 5.4.3. Overview of Client Response to NFS4ERR_MOVED . . . . 31 96 5.4.4. Obtaining Access to Sessions and State after 97 Migration . . . . . . . . . . . . . . . . . . . . . . 33 98 5.4.5. Obtaining Access to Sessions and State after Network 99 Address Transfer . . . . . . . . . . . . . . . . . . 35 100 5.5. Resolution of NFSv4.1 Issues . . . . . . . . . . . . . . 35 101 6. Evolution of Issue Handling . . . . . . . . . . . . . . . . . 38 102 6.1. History of this Document . . . . . . . . . . . . . . . . 38 103 6.2. Further Work Needed . . . . . . . . . . . . . . . . . . . 39 104 7. Security Considerations . . . . . . . . . . . . . . . . . . . 42 105 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 43 106 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 43 107 9.1. Normative References . . . . . . . . . . . . . . . . . . 43 108 9.2. Informative References . . . . . . . . . . . . . . . . . 44 109 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 44 110 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 45 112 1. Introduction 114 This is an informational document that discusses a number of related 115 issues in multiple versions of NFSv4. 117 Many of these relate to the migration feature of NFSv4, which 118 provides for moving responsibility for a single filesystem from one 119 server to another, without disruption to clients. A number of 120 problems in the specification of this feature in NFSv4.0 were 121 resolved by the publication of [RFC7931], which added trunking 122 detection to NFSV4.0. However, NFSv4.0 remains without an 123 appropriate discussion of trunking discovery, which has many 124 important connections with migration. As a result, NFSv4.0 requires 125 clarification of how the client is to respond to changes in the 126 trunking arrangements to use, both when migration occurs and when it 127 does not. 129 In addition, there are specification issues to be resolved with 130 regard to the NFSv4.1 version of these features which are discussed 131 in this document. 133 All of the issues discussed relate to the handling and interpretation 134 of the location-related attributes fs_locations and fs_locations_info 135 and to the proper client and server handling of changes in the values 136 of these attributes 138 These issues are all related to the protocol features for effecting 139 file system migration, or to trunking discovery but it is not 140 possible to treat each of these features in isolation. These 141 features are inherently linked because migration needs to deal with 142 the possibility of multiple server addresses in location attributes 143 and because location attributes, which provide trunking-related 144 information, may change, which might or might not involve migration. 146 2. Language 148 2.1. Requirements Language 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 152 document are to be interpreted as described in [RFC2119]. 154 2.2. Use of Normative Terms 156 This document, which deals with existing issues/problems in 157 standards-track documents, is in the informational category, and 158 while the facts it reports may have normative implications, any such 159 normative significance reflects the readers' preferences. For 160 example, we may report that the existing definition of migration for 161 NFSv4.1 does not properly describe how migrating state is to be 162 merged with existing state for the destination server. While it is 163 to be expected that client and server implementers will judge this to 164 be a situation that it would be appropriate to resolve, the judgment 165 as to how pressing this issue should be considered is a judgment for 166 the reader, and eventually the nfsv4 working group to make. 168 We do explore possible ways in which such issues can be dealt with, 169 with minimal negative effects, given that the working group has 170 decided to address these issues, but the choice of exactly how to 171 address these is best given effect in one or more standards-track 172 documents and/or errata. 174 In the context of this informational document, these normative 175 keywords will generally occur in the context of a quotation, most 176 often direct but sometimes indirect. The context will make it clear 177 whether the quotation is from: 179 o The base definition of the NFSv4.0 protocol [RFC7530]. 181 o The document updating the handling of migration in NFSv4.0 182 [RFC7931]. 184 o The current definition of the NFSv4.1 protocol [RFC5661]. 186 An additional possibility is that these terms may appear in a 187 proposed or possible text to serve as a replacement for a current 188 protocol specification. Sometimes, a number of possible alternative 189 texts may be listed and benefits and detriments of each examined in 190 turn. 192 2.3. Terminology Used in this Document 194 In this document the phrase "client ID" always refers to the 64-bit 195 shorthand identifier assigned by the server (a clientid4) and never 196 to the structure which the client uses to identify itself to the 197 server (called an nfs_client_id4 or client_owner in NFSv4.0 and 198 NFSv4.1 respectively). The opaque identifier within those structures 199 is referred to as a client id string". 201 Regarding trunking of network addresses, we use the following 202 terminology: 204 o Trunking detection refers to ways of deciding whether two specific 205 network addresses are connected to the same NFSv4 server. The 206 means available to make this determination depends on the protocol 207 version, and, in some cases, on the client implementation. 209 o Two network addresses connected to the same server are said to be 210 server-trunkable. 212 o Two network addresses connected to the same server such that those 213 addresses can be used to support a single common session are 214 referred to as session-trunkable. Note NFSv4.1 allows two 215 addresses to be server-trunkable without being session-trunkable, 216 while in NFSv4.0 no addresses are session-trunkable, since there 217 are no sessions. 219 o Trunking discovery is a process by which a client using one 220 network address can obtain other addresses that are trunkable 221 (either server-trunkable or session-trunkable) with it. 223 Regarding terminology relating to attributes used in trunking 224 discovery and other multi-server namespace features: 226 o Location attributes include the fs_locations and fs_locations_info 227 attributes. 229 o Location entries are the individual file system locations in the 230 location attributes. 232 o Location elements are derived from location entries. If a 233 location entry specifies an IP address there is only a single 234 corresponding location element. Location entries that contain a 235 host name, are resolved using DNS, and may result in one or more 236 location elements. All location elements consist of a location 237 address which is the IP address of an interface to a server and an 238 fs name which is the location of the file system within the 239 server's pseudo-fs. The fs name is empty if the server has no 240 pseudo-fs and only a single exported file system at the root 241 filehandle. 243 o Two location elements are said to be server-trunkable if they 244 specify the same fs name and the location addresses are such that 245 the location addresses are server-trunkable. 247 o Two location elements are said to be session-trunkable if they 248 specify the same fs name and the location addresses are such that 249 the location addresses are session-trunkable. 251 Each set of server-trunkable location elements defines the available 252 access paths to a particular file system. When there are multiple 253 such file systems, each of these, which contains the same data, is a 254 replica of the others. Logically, such replication is symmetric, 255 since the fs currently in use and an alternate fs are replicas of 256 each other. Often, in other documents, the term "replica" is not 257 applied to the fs currently in use, despite the fact that the 258 replication relation is inherently symmetric. 260 3. Multi-version Issues and Their Resolution 262 3.1. Multi-Version Issues 264 The source of these issues rises from a lack of clarity regarding the 265 meaning of and proper handling for location attributes that specify 266 more than a single server address. Such situations can arise as a 267 result of multiple entries in the same attribute or because a single 268 entry has a server name which, when processed by DNS, is mapped to 269 multiple server addresses. 271 Both [RFC7530] and [RFC5661] indicate that multiple addresses may be 272 present and that these addresses may be different paths to the same 273 server as well as different copies of the same data. However, the 274 following issues have, for both protocols, interfered with the 275 recognition of the existing location attributes as a way of providing 276 a trunking discovery function: 278 o There is no discussion of the use of these attributes when a file 279 system is first accessed, giving the impression that they are only 280 to be used as a way of overcoming access difficulties. 282 o The treatment of migration (and in the case of NFSv4.1 of file 283 system transitions in general) is written as if only a single 284 server address will be accessed. 286 o Although location attributes can contain the addresses of 287 migration targets and of additional replicas as well, the issues 288 that arise when both of these are specified are not clearly 289 discussed. 291 In addition, there are factors that relates to specific protocol 292 versions and documents: 294 o In NFSv4.0, as described solely by [RFC7530], trunking is treated 295 as a problem to be avoided, making the whole matter moot. 297 o In NFSv4.0, as described by [RFC7530] together with [RFC7931], the 298 situation is different. There is a means of trunking detection 299 suggested in [RFC7931] but it is a suggestion only valid when the 300 client chooses to use the uniform client id string model. 302 o For NFSv4.1, as described by [RFC5661], there is a standard method 303 of trunking detection, which can be relied upon. 305 The issues that need to be resolved for both versions are: 307 o Provision of a trunking discovery facility to allow a client to 308 find out about other addresses that may be used to access the 309 current server. 311 o Better integration of migration with trunking changes, including 312 situations in which the set of addresses to access the same server 313 changes (without migration) and those in which there is a shift to 314 a different server, but trunking of addresses on either the source 315 or destination is involved 317 3.2. Resolution of Multi-Version Issues 319 Although the specifics of addressing these issues will be different 320 for different versions, there are some common aspects discussed in 321 the subsections below: 323 o The trunking discovery function, except for the additional 324 location attribute fs_locations_info, is to be addressed in 325 substantially the same way in both versions, as explained in 326 Section 3.2.1. 328 o The interaction of trunking and migration is discussed in general 329 terms in Section 3.2.2. However, the specifics of the NFSv4.1 330 client's response to NFS4ERR_MOVED are discussed in Sections 331 5.4.3, 5.4.4, and 5.4.5. 333 3.2.1. Providing Trunking Discovery 335 A client can discover a set network addresses to use to access a file 336 system using an NFSv4 server in a number of ways: 338 o If the client is accessing a server using its name, that name can 339 be mapped to a set of IP addresses using DNS and if multiple 340 addresses are available, those addresses can generally be used 341 together to access the server. 343 o A client connected to a server without knowledge of its name can 344 obtain the value of a location attribute (e.g. fs_locations). 345 Where an entry within that attribute specifies a server name, DNS 346 can be used to obtain one or more network addresses corresponding 347 to that name. In cases in which one of those is the address being 348 used, the others that corresponding to that name can also be used 349 to access the server. 351 o A client can obtain the value of a location attribute (e.g. 352 fs_locations) and use location entries that specify network 353 addresses. When there is a means of trunking detection available 354 (see below), all of addresses that are determined to correspond to 355 the same server can be used to access the server. 357 Note that the last two of these are usable in situations in which 358 NFS4ERR_MOVED was returned. Note that this does not necessarily mean 359 that migration has occurred since there may be a shift in the set of 360 network addresses to be used without changing to a different server. 361 See Section 3.2.2 for further discussion. 363 Which of the above means of providing trunking information is 364 appropriate to use in a given environment will depend on security 365 considerations, the possible need for the server to direct different 366 clients to different sets of addresses, and the availability of 367 trunking detection facilities on the clients. 369 With regard to security, the possibility that requests to determine 370 the set of network addresses corresponding to a given server might be 371 interfered with or have their responses corrupted needs to be taken 372 account of. As a result, when use of DNSSEC is not available, it 373 might not be advisable to present server names in location attributes 374 and present the network addresses directly, eliminating the need to 375 use DNS to effect this translation. Fetching of location attributes 376 should be done with integrity protection. 378 In some situations, the server will want to direct clients to use 379 specific sets of network addresses, to effect load balancing, or to 380 meet quality-of-service goals. In such environments, presentation of 381 network addresses directly in the location attribute can help give 382 the server the necessary control over the paths to be used when 383 accessing particular file systems. When such techniques are used, 384 servers typically present their own network addresses in the location 385 attribute while adding the names of other servers, such as those used 386 to access replicas. 388 Trunking detection allows the client to determine whether two network 389 addresses can be used to access the same server. The availability of 390 trunking detection depends on the protocol version, and, in some 391 case, on client implementation choices: 393 o For NFSv4.0, a means by which it can be determined if two network 394 addresses correspond to the same server is suggested in [RFC7931]. 395 However is it is optional and only available to clients using the 396 uniform client id string approach. 398 o For NFSv4.1, the client can compare the server_owner returned in 399 the response to EXCHANGE_ID to determine if two network addresses 400 correspond to the same server. 402 As a result, direct presentation of network addresses in location 403 entries may be problematic for NFSv4.0, since some clients might not 404 have the trunking detection facilities that allow them to take 405 advantage of this information. For further discussion of issues 406 related to NFSv4.0, see Section 4.4. 408 3.2.2. Interaction of Trunking and Migration 410 When the set of network addresses designated by a location attribute 411 changes, NFS4ERR_MOVED may or may not result, and in some of the 412 cases in which it is returned migration will occur, while in others 413 there a shift in the network addresses used to access a particular 414 file system with no migration. 416 o When the list of networks addresses is a superset of that 417 previously in effect, there is no need for migration or any other 418 sort of client adjustment. Nevertheless, the client is free to 419 use an additional address if it provides another path to the same 420 server. If, on the other hand, it does not do so, the client may 421 treat it as it does a replica, to be used if the current server 422 addresses become unavailable. 424 o When the list of networks addresses is a subset of that previously 425 in effect, immediate action is not needed if the address was not 426 being used. The client should avoid using it in the future, 427 whether the address is for a replica or a potential additional 428 path to the server being used. 430 o When an address being removed is one of a number of paths to the 431 current server, the client can cease to use it but it can continue 432 to use it until NFS4ERR_MOVED is received. This is not considered 433 a migration event, unless it is the last available path to the 434 server that has become unusable. 436 When migration does occur, multiple addresses may be in use on the 437 server previous to migration and multiple addresses may be available 438 for use on the destination server. 440 With regard to the server in use, it may be that return of 441 NFS4ERR_MOVED indicates that a particular network address is no 442 longer to be used, without implying that migration of the file system 443 to a different server is needed In light of this possibility, clients 444 are best off not concluding that migration has occurred until 445 concluding that all the network addresses known to be associated with 446 the server are not usable. 448 It should be noted that the need to defer this determination is not 449 absolute. If a client is not aware of all network addresses for any 450 reason, if may conclude that migration has occurred when it has not 451 and treat a switch to a different server address as if it were a 452 migration event. This is generally harmless since the use of the 453 same server via a new address will appear as a successful Transparent 454 State Migration. 456 While significant harm will not arise from this misapprehension, it 457 can give rise to disconcerting situations. For example, if a lock 458 has been revoked during the address shift, it will appear to the 459 client as if the lock has been lost during migration, normally 460 calling for it to be recoverable via an fs-specific grace period 461 associated with the migration event. 463 With regard to the destination server, it is desirable for the client 464 to be aware of all the valid network addresses that can be used to 465 access the destination server. However, there is no need for this to 466 be done immediately. Implementations can process the additional 467 location elements in parallel with normal use of the first valid 468 location entry found to access the destination. 470 Because a location attribute may include entries relating to the 471 current server, the migration destination and possible replicas to 472 use, scanning for available network addresses could potentially be a 473 long process. The following list of helpful practices, here 474 presented as suggestions, could become RECOMMENDATIONs or 475 REQUIREMENTs in future standards-track documents 476 o Servers are well advised to place location entries that represent 477 addresses usable with the current server or a migration target 478 before those associated with replicas. 480 o A client can cease scanning for trunkable location entries once it 481 encounters one whose fs_name differs from the current fs name. 483 o A client can cease scanning for trunkable location entries once it 484 encounters a location element whose address in not server- 485 trunkable with the one it is using. 487 4. NFSv4.0 Issues 489 4.1. Core NFSv4.0 Migration Issues 491 Many of the problems seen with Transparent State Migration derived 492 from the inability of NFSv4.0servers to determine whether two client 493 IDs, issued on different servers, corresponded to the same client. 494 This difficulty derived in turn from the common practice, recommended 495 by [RFC7530], in which each client presented different client 496 identification strings to different servers, rather than presenting 497 the same identification string to all servers. 499 This practice, later referred to as the "non-uniform" client id 500 string approach, derived from concern that, since NFSv4.0 provided no 501 means to determine whether two IP addresses correspond to the server, 502 a single client connected to both might be confused by the fact that 503 state changes made via one IP address might unexpectedly affect the 504 state maintained with respect to the second IP address, thought of as 505 a separate server 507 To avoid this unexpected behavior, clients used the non-uniform 508 client id string approach. By doing so, a client connected to two 509 different servers (or to two IP addresses connected to the same 510 server) appeared to be two different servers. Since the server is 511 under the impression that two different clients are involved, state 512 changes made on each distinct IP address cannot be reflected on 513 another. 515 However, by doing things in that way, state migrated from server to 516 server cannot be referred to the actual client which generated it, 517 leading to confusion. 519 In addition to this core problem, the following issues with regard to 520 Transparent State Migration needed to be addressed: 522 o Clarification regarding the ability to merge state from different 523 leases even though their expiration times might not be precisely 524 synchronized. 526 o Clarifying the treatment of client IDs since it is not always 527 clear when clientid4 and when nfs_client_id4 was intended. 529 o Clarifying the logic of returning NFS4ERR_LEASE_MOVED. 531 o Clarifying the handling NFS4ERR_CLID_INUSE. 533 4.2. Resolution of Core Migration Protocol Difficulties in NFSv4.0 535 The client string identification issue was addressed in [RFC7931] as 536 follows: 538 o Defining both the uniform and non-uniform client id string 539 approaches as valid choices but indicating that the latter posed 540 difficulties for Transparent Stare Migration. 542 o Providing a way that clients using the uniform approach could use 543 to determine whether two IP addresses are connected to the same 544 server. 546 o Allowing clients using the uniform approach to avoid negative 547 consequences due to otherwise unexpected behavior since behavior 548 that is a consequence of known trunking relationships is not 549 unexpected. 551 o As a result, servers migrating state can be aware of the fact that 552 the same client is associated with two different items of state 553 even when that state was originally created on two different 554 servers. 556 Since all of the other issues noted in Section 4.1 were also 557 addressed by [RFC7931], publication of that document updating 558 [RFC7530] addressed all issues with Transparent State Migration in 559 NFSv4.0 known at that time. 561 4.3. Additional NFSv4.0 Issues 563 In light of the fact that a large set of migration-specific issues 564 were addressed by the publication of [RFC7931], the remaining issues 565 derive from those mentioned in Section 3.1. These include: 567 o Introducing facilities for trunking discovery. 569 o Clarifying the relationship between migration and trunking. 571 4.4. Resolution of Additional NFSv4.0 Issues 573 One possible approach to addressing these issues would entail 574 publication of an additional standards-track document updating 575 [RFC7530]. 577 Fortunately, it appears that all of the material to be updated 578 appears in Section 8 of that document, whether it concerns the 579 provision of trunking discovery or the interaction of trunking and 580 migration. It also appears that none of the material to be updated 581 is in sections updated by [RFC7931]. 583 A review of the existing Section 8 of [RFC7530], shows the following 584 sections as requiring significant attention: 586 o The existing Section 8.1 requires a considerable expansion to 587 explain the various uses of the fs_locations and the possible 588 interactions among them. 590 o The existing Section 8.4 may require substantial re-organization 591 to reflect the facts that fs_locations has multiple functions and 592 may be referenced on multiple occasions. 594 o The existing Section 8.5 follows the previous approach for NFSv4.0 595 in assuming that trunking simply cannot and should not happen. 596 For example, the last paragraph says: 598 If a single location entry designates multiple server IP 599 addresses, the client should choose a single one to use. When 600 two server addresses are designated by a single location entry 601 and they correspond to different servers, this normally 602 indicates some sort of misconfiguration, and so the client 603 should avoid using such location entries when alternatives are 604 available. When they are not, clients should pick one of the 605 IP addresses and use it, without using others that are not 606 directed to the same server. 608 As written, this seems to foreclose any use of trunking in 609 connection with migration. In retrospect, it appears that this 610 section should have been revised as part of [RFC7931], but since 611 that was not done then, the issue needs to be addressed now. 613 Overall, it appears that, in addition to the revision of Section 8.1, 614 Sections 8.4 and 8.5 need to be reorganized. One likely approach is 615 to divide the material into three main sections based on the 616 circumstances in which the attribute is accessed: 618 A section dealing with initial use and how it is trunking 619 discovery and preparation for replication. 621 A section dealing with subsequent attribute change and how it 622 affects trunking discovery and preparation for replication. 624 A section dealing with how the attribute is used to deal with 625 communication problems including receiving NFS4ERR_MOVED but also 626 including other communication difficulties driving selection of a 627 new replica. 629 5. Issues for NFSv4.1 631 5.1. Issues to Address for NFSv4.1 633 Because NFSv4.1 embraces the uniform client-string approach, as 634 advised by section 2.4 of [RFC5661], addressing migration issues is 635 simpler, in that a shift in client id string models is not required. 636 Instead, NFSv4 returns information in the EXCHANGE_ID response to 637 enable trunking relationships to be determined by the client. 639 Despite this simplification, there are substantial issues that need 640 to be dealt with: 642 o The other necessary part of addressing migration issues, providing 643 for the server's merger of leases that relate to the same client, 644 is not currently addressed by [RFC5661] and changes need to be 645 made to make it clear that state needs to be appropriately merged 646 as part of migration, to avoid multiple client IDs between a 647 client-server pair. 649 o The current discussion (in [RFC5661]), of the possibility of 650 server_owner changes is incomplete and confusing. 652 o As with NFSV4.0, the interaction of trunking with migration and 653 other aspects of multi-server namespace needs to be clarified. 655 Addressing migration in NFSv41 will also require adaptation of the 656 approaches used in [RFC7931] to the NFSv4.1 environment including: 658 o The use of EXCHANGE_ID needs to be accommodated including issues 659 associated with the expected confirmation status of client IDs 660 transferred by Transparent State Migration. 662 o The use of sessions needs to be addressed including discussion of 663 the proper use of the status bits returned by the SEQUENCE 664 operation. 666 In addition, there are a number of new features within NFSv4.1 whose 667 relationship with migration needs to be clarified. Some examples: 669 o There needs to be some clarification of how migration, and 670 particularly Transparent State Migration, should interact with 671 pNFS layouts. 673 o There are a number of issues related to the migration of sessions 674 that need to be addressed. 676 Discussion of how to resolve these issues will appear in the sections 677 below. 679 5.1.1. Addressing State Merger in NFSv4.1 681 The existing treatment of state transfer in [RFC5661], has similar 682 problems to that in [RFC7530] in that it assumes that the state for 683 multiple filesystems formerly on different servers will not be merged 684 so that it appears under a single common client ID. We've already 685 seen the reasons that this is a problem with regard to NFSv4.0. 687 Although we don't have the problems stemming from the non-uniform 688 client-string approach, there are a number of complexities in the 689 existing treatment of state management in the section entitled "Lock 690 State and File System Transitions" in [RFC5661] that make this non- 691 trivial to address: 693 o Migration is currently treated together with other sorts of 694 filesystem transitions including transitioning between replicas 695 without any NFS4ERR_MOVED errors. 697 o There is separate handling and discussion of the cases of matching 698 and non-matching server scopes. 700 o In the case of matching server scopes, the text calls for an 701 unrealistic degree of transparency, suggesting that the source and 702 destination servers need to cooperate in stateid and client ID 703 assignment. 705 o In the case of non-matching server scopes, the text does not 706 mention the possibility of the transparent migration of state at 707 all, resulting in a functional regression from NFSV4.0 709 5.1.2. Addressing pNFS Relationship with Migration 711 This is made difficult because, within the pNFS framework, migration 712 might mean any of several things: 714 o Transfer of the MDS, leaving DS's as they are. 716 This would be minimally disruptive to those using layouts but 717 would require the pNFS control protocol being used to support the 718 DS being directed to a new MDS. 720 o Transfer of a DS, leaving everything else in place. 722 Such a transfer can be handled without using migration at all. 723 The server can recall/revoke layouts, and issue new ones, as 724 appropriate. 726 o Transfer of the filesystem to a new filesystem with both MDS and 727 DS's moving. 729 In such a transfer, an entirely different set of DS's will be at 730 the target location. There may even be no pNFS support on the 731 destination filesystem at all. 733 Migration needs to support both the first and last of these models. 735 5.1.3. Addressing Server_owner Changes in NFSv4.1 737 Section 2.10.5 of [RFC5661] states the following. 739 The client should be prepared for the possibility that 740 eir_server_owner values may be different on subsequent EXCHANGE_ID 741 requests made to the same network address, as a result of various 742 sorts of reconfiguration events. When this happens and the 743 changes result in the invalidation of previously valid forms of 744 trunking, the client should cease to use those forms, either by 745 dropping connections or by adding sessions. For a discussion of 746 lock reclaim as it relates to such reconfiguration events, see 747 Section 8.4.2.1. 749 While this paragraph is literally true in that such reconfiguration 750 events can happen and clients have to deal with them, it is confusing 751 in that it can be read as suggesting that clients have to deal with 752 them without disruption, which in general is impossible. 754 A clearer alternative would be: 756 It is always possible that, as a result of various sorts of 757 reconfiguration events, eir_server_scope and eir_server_owner 758 values may be different on subsequent EXCHANGE_ID requests made to 759 the same network address. 761 In most cases such reconfiguration events will be disruptive and 762 indicate that an IP address formerly connected to one server is 763 now connected to an entirely different one. 765 Some guidelines on client handling of such situations follow: 767 o When eir_server_scope changes, the client has no assurance that 768 any id's it obtained previously (e.g. file handles) can be 769 validly used on the new server, and, even if the new server 770 accepts them, there is no assurance that this is not due to 771 accident. Thus it is best to treat all such state as lost/ 772 stale although a client may assume that the probability of 773 inadvertent acceptance is low and treat this situation as 774 within the next case. 776 o When eir_server_scope remains the same and 777 eir_server_owner.so_major_id changes, the client can use 778 filehandles it has and attempt reclaims. It may find that 779 these are now stale but if NFS4ERR_STALE is not received, he 780 can proceed to reclaim his opens. 782 o When eir_server_scope and eir_server_owner.so_major_id remain 783 the same, the client has to use the now-current values of 784 eir_server-owner.so_minor_id in deciding on appropriate forms 785 of trunking. 787 5.1.4. Addressing Confirmation Status of Migrated Client IDs in NFSv4.1 789 When a client ID is transferred between systems as a part of 790 migration, it has never been clear whether it should be considered 791 confirmed or unconfirmed on the target server. In the case in which 792 an associated session is transferred together with the client ID, it 793 is clear that the transferred client ID needs to be considered 794 confirmed, as the existence of an associated session is incompatible 795 with an unconfirmed client ID. 797 The case in which a client ID is transferred without an associated 798 session is less clear-cut, particularly since the treatment of 799 EXCHANGE_ID in [RFC5661] assumes that CREATE_SESSION is the only 800 means by which a client id may be confirmed. While this assumption 801 is valid in the absence of Transparent State Migration, 802 implementation of migration means that if this assumption is 803 maintained, it is not clear how migrated client IDs can be a 804 accommodated. If this assumption were maintained, we would have to 805 choose between the following two alternatives, regarding whether the 806 client ID to be reported as confirmed when EXCHANGE_ID is used to 807 register an already-known client_owner with the server. 809 o Report the client ID unconfirmed, because of the lack of an 810 associated session. This makes it simpler for the client to 811 determine whether there is an associated session transferred at 812 the same time. However, it is inconsistent with the fact there 813 are stateids which have been transferred with the client ID. 815 o Report the client ID as confirmed, because it was confirmed on the 816 source server and the transfer is not considered to have affected 817 that. Given the current description of EXCHANGE_ID in [RFC5661], 818 some modification in the treatment of client id confirmation is 819 called for. In particular, provision would have to be made to 820 enable the client id slot sequence id to be used by the client to 821 be determined. 823 Although the first approach makes it simpler for the client to 824 determine whether there is an associated session transferred at the 825 same time, it makes it more difficult to determine whether 826 Transparent State Migration has occurred. Section 5.1.6. 828 In any case, adjustments will be required to deal with the fact that 829 [RFC5661] currently assumes that a client id can only be confirmed by 830 issuing a CREATE_SESSION. In order to properly deal with the status 831 of migrated client ids, we have to distinguish among: 833 o The confirmation status as reported by EXCHANGE_ID. 835 o Whether the client id is considered confirmed as that term is used 836 in the many other cases in which the confirmation status of a 837 client ID affects how requests are handled. 839 o How the client is to determine the initial sequence id to be used 840 when doing operations such as CREATE_SESSION. 842 In [RFC5661] as it currently stands all of these are tied together 843 and it is not obvious how migrated client IDs could be accommodated 844 in this structure, and what changes are necessary to make this 845 possible. For more discussion of this issue, see Section 5.2.1. 847 5.1.5. Addressing Changes in Trunking Configuration 849 When the client us capable of finding out a set of network addresses 850 to use in accessing a server, it is always possible for that set to 851 change. 853 This sometimes requires that a network address previously used to 854 access a server becomes invalid for that purpose. This requires a 855 way of notifying the client and a way for the client to adapt to this 856 change by using a new set of network addresses to access the server. 858 his will involve recovery much like hat for migration although the 859 same server and file system is used throughout. 861 5.1.6. Addressing Session Migration in NFSv4.1 863 Some issues that need to be addressed regard the migration of 864 sessions, in addition to client IDs and stateids 866 o It needs to be made clearer how the client can deal with the 867 possibility that sessions might or might not be transferred as 868 part of Transparent State Migration. 870 o Rules need to be clarified regarding possible transfer of sessions 871 when either the source session is being used to access other file 872 systems on source server or there is already a session connecting 873 the client to the destination server. 875 o There needs to be more detail regarding how the protocol avoids 876 situations in which the same session is subject to concurrent 877 changes on two different servers at the same time. 879 5.2. Possible Resolutions for NFSv4.1 Protocol Issues 881 The subsections below explore some ways of dealing with clarifying 882 the protocol to address issues discussed in Section 5.1 884 5.2.1. Client ID Confirmation Issues 886 As mentioned previously [RFC5661], makes no provision for client IDs 887 that are confirmed other than through the use of CREATE_SESSION. For 888 example Section 18.35 of [RFC5661] states: 890 The client uses the EXCHANGE_ID operation to register a particular 891 client owner with the server. The client ID returned from this 892 operation will be necessary for requests that create state on the 893 server and will serve as a parent object to sessions created by 894 the client. In order to confirm the client ID it must first be 895 used, along with the returned eir_sequenceid, as arguments to 896 CREATE_SESSION. If the flag EXCHGID4_FLAG_CONFIRMED_R is set in 897 the result, eir_flags, then eir_sequenceid MUST be ignored, as it 898 has no relevancy. 900 In deciding how to address the status of migrated client IDs in the 901 case of Transparent State Migration, we should avoid giving undue 902 weight to the last sentence of the above simply because it is stated 903 in the form of a normative requirement. We should instead focus on 904 the reasons such terms (i.e. those defined by [RFC2119]) are to be 905 used, to state interoperability constraints. In this case, the 906 "MUST" applies to a conclusion based on the premise that a 907 CREATE_SESSION must have been done to assure that the client ID is 908 reliably known to the server. 910 In that light, let us consider a possible replacement, that treats 911 confirmation by means of CREATE_SESSION as one of a number of 912 possible means and avoids some the undesirable consequences of 913 adherence to the current approach, originally conceived without 914 taking state migration into account. 916 The client uses the EXCHANGE_ID operation to register a particular 917 client_owner with the server. However, when the client_owner has 918 been already been registered by other means (e.g. Transparent 919 State Migration), the client may still use EXCHANGE_ID to obtain 920 the client ID assigned previously. 922 The client ID returned from this operation will be associated with 923 the connection on which the EXHANGE_ID is received and will serve 924 as a parent object for sessions created by the client on this 925 connection or to which the connection is bound. As a result of 926 using those sessions to make requests involving the creation of 927 state, that state will become associated with the client ID 928 returned. 930 In situations in which the registration of the client_owner has 931 not occurred previously, the client ID must first be used, along 932 with the returned eir_sequenceid, in creating an associated 933 session using CREATE_SESSION. 935 If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the result, 936 eir_flags, then it is an indication that the registration of the 937 client_owner has already occurred and that a further 938 CREATE_SESSION is not needed to confirm it. Of course, subsequent 939 CREATE_SESSION operations may be needed for other reasons. 941 The value eir_seqenceid is used to establish an initial sequence 942 value associate with the client ID returned. In cases in which a 943 CREATE_SESSION has already been done, there is no need for this 944 value, since sequencing of such request has already been 945 established and the client has no need for this value and will 946 ignore it 948 5.2.2. Dealing with Multiple Location Entries 950 The possibility that more than one server address may be present in 951 location attributes requires further clarification. This is 952 particularly the case, given the potential role of trunking for 953 NFSv4.1, whose connection to migration needs to be clarified. 955 The description of the location attributes in [RFC5661], while it 956 indicates that multiple address entries in these attributes may be 957 used to indicate alternate paths to the file system, does so mainly 958 in the context of replication and does so without mentioning 959 trunking. The discussion of migration does not discuss the 960 possibility of multiple location entries or trunking, which we will 961 explore here. 963 We will cover cases in which multiple addresses appear directly in 964 the attributes as well as those in which the multiple addresses 965 result because a single location entry is expanded into multiple 966 location elements using addresses provided by DNS. 968 When the set of valid location elements by which a file system may be 969 accessed changes, migration need not be involved. Some cases to 970 consider: 972 o When the set of location elements expands, migration is not 973 involved. In the case in which the additional elements are not 974 trunkable with ones previously being used, the new elements serve 975 as additional access locations, available in case of the failure 976 of server addresses being used. When additional elements are 977 trunkable with those currently being used the client may use the 978 additional addresses just as they might have if they had been 979 available when use of the file system began. 981 There is no current mechanism by which the client can be notified 982 of a change in the set of available location for an fs. Given the 983 client has at least one IP address available to access the 984 filesystem in question, periodic polling is an adequate mechanism 985 for the client to find additional server addresses to use to 986 access the file system. 988 o When the set of location elements contracts but none of the 989 elements no longer usable were in fact being used by the client, 990 then no migration is involved and no change in network addresses 991 is needed. Only if the client were to start using one of the 992 unavailable elements would the client be notified (via 993 NFS4ERR_MOVED) of the need to not use those elements and to use 994 others provided by a location attribute. 996 When a specific server address being used becomes unavailable to 997 service a particular file system, NFS4ERR_MOVED will be returned, and 998 the client will respond based on the available locations. Whether 999 continuity of locking state will be available depends on a number of 1000 factors: 1002 o If there are still elements in use trunkable with the element that 1003 has become unavailable, there will still be a continuity of 1004 locking state, even though Transparent State Migration per se has 1005 not occurred. If the in-use addresses are session-trunkable with 1006 the address becoming unavailable, only one connection is lost and 1007 all existing sessions will remain available. If, on the other 1008 hand, the in-use addresses are only clientid-trunkable with the 1009 address becoming unavailable, a session can be lost. However, 1010 that session can be made available on those other nodes, just as 1011 they it would have been if Transparent State Migration were in 1012 effect, even though no migration has occurred. 1014 o Otherwise, if there are available addresses trunkable with the one 1015 that has become unavailable, the client has access to existing 1016 locking state once it establishes a connection with the new 1017 addresses, using a new or existing session depending on the type 1018 of trunking in effect. This is also similar to the case in which 1019 Transparent State Migration has occurred, even though there is no 1020 migration, with the state remaining on the existing server. 1022 Note that this case, as well as the previous one, can be expected 1023 in the case in which the server seeks to direct traffic with 1024 regard to particular file systems to choose addresses, in the 1025 interest of load balancing, to adjust to hardware availability 1026 constraints, or for other reasons. 1028 o In other cases, migration has occurred and the client can 1029 determine whether Transparent State Migration occurred and whether 1030 any locking state was lost during the transfer. 1032 Whether migration has occurred or not, the client can use the 1033 procedure described in Section 5.4.3 to recover access to existing 1034 locking state and, in some cases, sessions. 1036 One should note the following differences between migration with 1037 Transparent State Migration and the similar cases in which there is a 1038 continuity of locking state with no change in the server. 1040 o When locks are lost (as indicated when using them or via the 1041 SEQ4_STATUS flags) and migration has not been done, they are not 1042 to be reclaimed, except when SEQ4_STATUS_RESTART_RECLAIM_NEEDED is 1043 set. Instead such losses are treated as lock revocations and 1044 acknowledged using FREE_STATEID. 1046 o When migration has not been done, there is no need for a 1047 RECLAIM_COMPLETE (with rca_one_fs set to true). 1049 5.2.3. Migration and pNFS 1051 When pNFS is involved, the protocol is capable of supporting: 1053 o Migration of the MDS, leaving DS's in place. 1055 o Migration of the file system as a whole, including the MDS and 1056 associated DS's. 1058 o Replacement of one DS by another. 1060 o Migration of a pNFS file system to one in which pNFS is not used. 1062 o Migration of a file system not using pNFS to one in which layouts 1063 are available. 1065 Migration of the MDS function is directly supported by Transparent 1066 State Migration. Layout state will normally be transparently 1067 transferred, just as other state is. As a result, Transparent State 1068 Migration provides a framework in which, given appropriate inter-MDS 1069 data transfer, one MDS can be substituted for another. 1071 Migration of the file system function as a whole can be accomplished 1072 by recalling all layouts as part of the initial phase of the 1073 migration process. As a result, IO will be done through the MDS 1074 during the migration process, and new layouts can be granted once the 1075 client is interacting with the new MDS. An MDS can also effect this 1076 sort of transition by revoking all layouts as part of Transparent 1077 State Migration, as long as the client is notified about the loss of 1078 state. 1080 In order to allow migration to a file system on which pNFS is not 1081 supported, clients need to be prepared for a situation in which 1082 layouts are not available or supported on the destination file system 1083 and so direct IO requests to the destination server, rather than 1084 depending on layouts being available. 1086 Replacement of one DS by another is not addressed by migration as 1087 such but can be effected by an MDS recalling layouts for the DS to be 1088 replaced and issuing new ones to be served by the successor DS. 1090 Migration may transfer a file system from a server which does not 1091 support pNFS to one which does. In order to properly adapt to this 1092 situation, clients which support pNFS, but function adequately in its 1093 absence, should check for pNFS support when a file system is migrated 1094 and be prepared to use pNFS when support is available. 1096 5.3. Defining Server Responsibilities for NFSv4.1 1098 The subsections below discuss the responsibilities of source and 1099 destination servers in effecting the necessary transfer of 1100 information to support Transparent State Migration. 1102 5.3.1. Server Responsibilities in Effecting Transparent State Migration 1104 The basic responsibility of the source server in effecting 1105 Transparent State Migration is to make available to the destination 1106 server a description of each piece of locking state associated with 1107 the file system being migrated. In addition to client id string and 1108 verifier, the source server needs to provide, for each stateid: 1110 o The stateid including the current sequence value. 1112 o The associated client ID. 1114 o The handle of the associated file. 1116 o The type of the lock, such as open, byte-range lock, delegation, 1117 layout. 1119 o For locks such as opens and byte-range locks, there will be 1120 information about the owner(s) of the lock. 1122 o For recallable/revocable lock types, the current recall status 1123 needs to be included. 1125 o For each lock type there will by type-specific information, such 1126 as share and deny modes for opens and type and byte ranges for 1127 byte-range locks and layouts. 1129 A further server responsibility concerns locks that are revoked or 1130 otherwise lost during the process of file system migration. Because 1131 locks that appear to be lost during the process of migration will be 1132 reclaimed by the client, the servers have to take steps to ensure 1133 that locks revoked soon before or soon after migration are not 1134 inadvertently allowed to be reclaimed in situations in which the 1135 continuity of lock possession cannot be assured. 1137 o For locks lost on the source but whose loss has not yet been 1138 acknowledged by the client (by using FREE_STATEID), the 1139 destination must be aware of this loss so that it can deny a 1140 request to reclaim them. 1142 o For locks lost on the destination after the state transfer but 1143 before the client's RECLAIM_COMPLTE is done, the destination 1144 server should note these and not allow them to be reclaimed. 1146 An additional responsibility of the cooperating servers concerns 1147 situations in which a stateid cannot be transferred transparently 1148 because it conflicts with an existing stateid held by the client and 1149 associated with a different file system. In this case there are two 1150 valid choices: 1152 o Treat the transfer, as in NFSv4.0, as one without Transparent 1153 State Migration. In this case, conflicting locks cannot be 1154 granted until the client does a RECLAIM_COMPLETE, after reclaiming 1155 the locks it had, with the exception of reclaims denied because 1156 they were attempts to reclaim locks that had been lost. 1158 o Implement Transparent State Migration, except for the lock with 1159 the conflicting stateid. In this case, the client will be aware 1160 of a lost lock (through the SEQ4_STATUS flags) and be allowed to 1161 reclaim it. 1163 5.3.2. Synchronizing Session Transfer 1165 When transferring state between the source and destination, the 1166 issues discussed in Section 7.2 of [RFC7931] must still be attended 1167 to. In this case, the use of NFS4ERR_DELAY is still necessary in 1168 NFSv4.1, as it was in NFSv4.0, to prevent locking state changing 1169 while it is being transferred. 1171 There are a number of important differences in the NFS4.1 context: 1173 o The absence of RELEASE_LOCKOWNER means that the one case in which 1174 an operation could not be deferred by use of NFS4ERR_DELAY no 1175 longer exists. 1177 o Sequencing of operations is no longer done using owner-based 1178 operation sequences numbers. Instead, sequencing is session- 1179 based 1181 As a result, when sessions are not transferred, the techniques 1182 discussed in [RFC7931] are adequate and will not be further 1183 discussed. 1185 When sessions are transferred, there are a number of issues that pose 1186 challenges since, 1188 o A single session may be used to access multiple file systems, not 1189 all of which are being transferred. 1191 o Requests made on a session may, even if rejected, affect the state 1192 of the session by advancing the sequence number associated with 1193 the slot used. 1195 As a result, when the filesystem state might otherwise be considered 1196 unmodifiable, the client might have any number of in-flight requests, 1197 each of which is capable of changing session state, which may be of a 1198 number of types: 1200 1. Those requests that were processed on the migrating file system, 1201 before migration began. 1203 2. Those requests which got the error NFS4ERR_DELAY because the file 1204 system being accessed was in the process of being migrated. 1206 3. Those requests which got the error NFS4ERR_MOVED because the file 1207 system being accessed had been migrated. 1209 4. Those requests that accessed the migrating file system, in order 1210 to obtain location or status information. 1212 5. Those requests that did not reference the migrating file system. 1214 It should be noted that the history of any particular slot is likely 1215 to include a number of these request classes. In the case in which a 1216 session which is migrated is used by filesystems other than the one 1217 migrated, requests of class 5 may be common and be the last request 1218 processed, for many slots. 1220 Since session state can change even after the locking state has been 1221 fixed as part of the migration process, the session state known to 1222 the client could be different from that on the destination server, 1223 which necessarily reflects the session state on the source server, at 1224 an earlier time. In deciding how to deal with this situation, it is 1225 helpful to distinguish between two sorts of behavioral consequences 1226 of the choice of initial sequence ID values. 1228 o The error NFS4ERR_SEQ_MISORDERED is returned when the sequence ID 1229 in a request is neither equal to the last one seen for the current 1230 slot nor the next greater one. 1232 In view of the difficulty of arriving at a mutually acceptable 1233 value for the correct last sequence value at the point of 1234 migration, it may be necessary for the server to show some degree 1235 of forbearance, when the sequence ID is one that would be 1236 considered unacceptable if session migration were not involved. 1238 o Returning the cached reply for a previously executed request when 1239 the sequence ID in the request matches the last value recorded for 1240 the slot. 1242 In the cases in which an error is returned and there is no 1243 possibility of any non-idempotent operation having been executed, 1244 it may not be necessary to adhere to this as strictly as might be 1245 proper if session migration were not involved. For example, the 1246 fact that the error NFS4ERR_DELAY was returned may not assist the 1247 client in any material way, while the fact that NFS4ERR_MOVED was 1248 returned by the source server may not be relevant when the request 1249 was reissued, directed to the destination server. 1251 One part of adapting to these sorts of issues would restrict 1252 enforcement of normal slot sequence enforcement semantics until the 1253 client itself, by issuing a request using a particular slot on the 1254 destination server, established the new starting sequence for that 1255 slot on the migrated session. 1257 An important issue is that the specification needs to take note of 1258 all potential COMPOUNDs, even if they might be unlikely in practice. 1259 For example, a COMPOUND is allowed to access multiple file systems 1260 and might perform non-idempotent operations in some of them before 1261 accessing a file system being migrated. Also, a COMPOUND may return 1262 considerable data in the response, before being rejected with 1263 NFS4ERR_DELAY or NFS4ERR_MOVED, and may in addition be marked as 1264 sa_cachethis. 1266 Some possibilities that need to be considered to address the issues: 1268 o Do not enforce any sequencing semantics for a particular slot 1269 until the client has established the starting sequence for that 1270 slot on the destination server. 1272 o For each slot, do not return a cached reply returning 1273 NFS4ERR_DELAY or NFS4ERR_MOVED until the client has established 1274 the starting sequence for that slot on the destination server. 1276 o Until the client has established the starting sequence for a 1277 particular slot on the destination server, do not report 1278 NFS4ERR_SEQ_MISORDERED or return a cached reply returning 1279 NFS4ERR_DELAY or NFS4ERR_MOVED, where the reply consists solely of 1280 a series of operations where the response is NFS4_OK until the 1281 final error. 1283 5.4. Defining Client Responsibilities for NFSv4.1 1285 The subsections below discuss the responsibilities of the client in 1286 dealing with transition to a new server (migration) and to use of new 1287 network addresses in accessing existing servers. 1289 5.4.1. Client Recovery from Migration Events 1291 When a file system is migrated, there a number of migration-related 1292 status indications with which clients need to deal: 1294 o If an attempt is made to use or return a filehandle within a file 1295 system that has been migrated away from the server on which it was 1296 previously available, the error NFS4ERR_MOVED is returned. 1298 This condition continues on subsequent attempts to access the file 1299 system in question. The only way the client can avoid the error 1300 is to cease accessing the filesystem in question at its old server 1301 location and access it instead on the server to which it has been 1302 migrated. 1304 o Whenever a SEQUENCE operation is sent by a client to a server 1305 which generated state held on that client which is associated with 1306 a file system that has been migrated away from the server on which 1307 it was previously available, the status bit 1308 SEQ4_STATUS_LEASE_MOVED is set in the response. 1310 This condition continues until the client acknowledges the 1311 notification by fetching a location attribute for the migrated 1312 file system. When there are multiple migrated file systems, a 1313 location attribute for each such migrated file system needs to be 1314 fetched, in order to clear the condition. Even after the 1315 condition is cleared, the client needs to respond by using the 1316 location information to access the destination server to ensure 1317 that leases are not needlessly expired. 1319 Unlike the case of NFSv4.0 in which the corresponding conditions are 1320 distinct errors and thus mutually exclusive, in NFSv4.1 the client 1321 can, and often will, receive both indications on the same request. 1322 As a result, implementations need to address the question of how to 1323 co-ordinate the necessary recovery actions when both indications 1324 arrive simultaneously. It should be noted that when the server 1325 decides whether SEQ4_STATUS_LEASE_MOVED is to be set, it has no way 1326 of knowing which file system will be referenced or whether 1327 NFS4ERR_MOVED will be returned. 1329 While it is true that, when only a single migrated file system is 1330 involved, a single set of actions will clear both indications, the 1331 possibility of multiple migrated file systems calls for an approach 1332 in which there are separate recovery actions for each indication. In 1333 general, the response to neither indication can be subsumed within 1334 the other since: 1336 o If the client were to respond only to the MOVED indication, there 1337 would be no effective client response to a situation in which a 1338 file system was not being actively accessed at the time migration 1339 occurred. As a result, leases on the destination server might be 1340 needlessly expired. 1342 o If the client were to respond only to the LEASE_MOVED indication, 1343 recovery for migrated file systems in active use could be deferred 1344 in order to accomplish recovery for others not being actively 1345 accessed. The consequences of this choice can pose particular 1346 problems when there are a large number of file systems supported 1347 by a particular server, or when it happens that some servers, 1348 after receiving migrated file systems have periods of 1349 unavailability, such as occur as a result of server reboot. This 1350 can result in recovery for actively accessed migrated file systems 1351 being unnecessarily delayed for long periods of time. 1353 Similar considerations apply to other arrangements in which one of 1354 the indications, while not ignored per se, is subsumed within a 1355 single recovery process focused on recovery for the other indication. 1357 Although clients are free to decide on their own approaches to 1358 recovery, we will explore below an approach with the following 1359 characteristics: 1361 o All instances of the MOVED indication, whether they involve 1362 migration or not, should be dealt with promptly, either by doing 1363 the necessary recovery directly, providing that it be done 1364 asynchronously, or ensuring that it is already under way. 1366 o All instances of the LEASE_MOVED indication should be dealt with 1367 asynchronously, in a migration discovery thread whose job is to 1368 clear that indication by fetching the appropriate location 1369 attribute. Because this thread will only be fetching a location 1370 attribute and the fs_status attribute for the file systems 1371 referenced by the client, it cannot receive MOVED indications. 1372 Some useful guidance regarding possible implementation of a 1373 migration discovery thread can be found in Section 5.4.2. 1375 o When a migration discovery thread happens upon a migrated file 1376 system (i.e. not present and not a referral), the thread is likely 1377 to have cleared one (out of an unknown number) of file systems 1378 whose migration needs to be responded to. The discovery thread 1379 needs to schedule the appropriate migration recovery (as described 1380 in Section 5.4.3). This is necessary to ensure that migrated file 1381 systems will be referenced on the destination server in order to 1382 avoid unnecessary lease expiration. 1384 For many of the migrated file systems discovered in this way, the 1385 client has not received any MOVED indication. In such cases, 1386 lease recovery needs to be scheduled but it should not interfere 1387 with continuation of the migration discovery function. 1389 o When a migration discovery thread receives a LEASE_MOVED 1390 indication, it takes no special action but continues its normal 1391 operation. On the other hand, if a LEASE_MOVED indication is not 1392 received, it indicates that the thread has completed its work 1393 successfully. 1395 5.4.2. The Migration Discovery Process 1397 As noted above, LEASE_MOVED indications are best dealt with in a 1398 migration discovery thread. Because of this structure, 1400 o No action needs to be taken for such indications received by the 1401 migration discovery threads, since continuation of that thread's 1402 work will address the issue. 1404 o For such indications received in other contexts, the generally 1405 appropriate response is to initiate or otherwise provide for the 1406 execution of a migration discovery thread for file systems 1407 associated with the server IP address returning the indication. 1409 o In all cases in which the appropriate migration discovery thread 1410 is running, nothing further needs to be done to respond to 1411 LEASE_MOVED indications. 1413 This leaves a potential difficulty in situations in which the 1414 migration discovery thread is near to completion but is still 1415 operating. One should not ignore a LEASE_MOVED indication if the 1416 discovery thread is not able to respond to migrated file system 1417 without additional aid. A further difficulty in addressing such 1418 situation is that a LEASE_MOVED indication may reflect the server's 1419 state at the time the SEQUENCE operation was processed, which may be 1420 different from that in effect at the time the response is received. 1422 A useful approach to this issue involves the use of separate 1423 externally-visible discovery thread states representing non- 1424 operation, normal operation, and completion/verification of migration 1425 discovery processing. 1427 Within that framework, discovery thread processing would proceed as 1428 follows. 1430 o While in the normal-operation state, the thread would fetch, for 1431 successive file systems known to the client on the server being 1432 worked on, a location attribute plus the fs_status attribute. 1434 o If the fs_status attribute indicates that the file system is a 1435 migrated one (i.e. fss_absent is true and fss_type != 1436 STATUS4_REFERRAL) and thus that it is likely that the fetch of the 1437 location attribute has cleared one the file systems contributing 1438 to the LEASE_MOVED indication. 1440 o In cases in which that happened, the thread cannot know whether 1441 the LEASE_MOVED indication has been cleared and so it enters the 1442 completion/verification state and proceeds to issue a COMPOUND to 1443 see if the LEASE_MOVED indication has been cleared. 1445 o When the discovery thread is in the completion/verification state, 1446 if others get a LEASE_MOVED indication they note this fact and it 1447 is used when the request completes, as described below. 1449 When the request used in the completion/verification state completes: 1451 o If a LEASE_MOVED indication is returned, the discovery thread 1452 resumes its normal work. 1454 o Otherwise, if there is any record that other requests saw a 1455 LEASE_MOVED indication, that record is cleared and the 1456 verification request retried. The discovery thread remains in 1457 completion/verification state. 1459 o If there has been no LEASE_MOVED indication, the work of the 1460 discovery thread is considered completed and it enters the non- 1461 operating state. 1463 5.4.3. Overview of Client Response to NFS4ERR_MOVED 1465 This section outlines a way in which a client that receives 1466 NFS4ERR_MOVED can respond by using a new server or network address if 1467 one is available. As part of that process, it will determine: 1469 o Whether the NFS4ERR_MOVED indicates migration has occurred, or 1470 whether it indicates another sort of file system transition as 1471 discussed in Section 5.2.2. 1473 o In the case of migration, whether Transparent State Migration has 1474 occurred. 1476 o Whether any state has been lost during the process of Transparent 1477 State Migration. 1479 o Whether sessions have been transferred as part of Transparent 1480 State Migration. 1482 During the first phase of this process, the client proceeds to 1483 examine location entries to find the initial network address it will 1484 use to continue access to the file system or its replacement. For 1485 each location entry that the client examines, the process consists of 1486 five steps: 1488 1. Performing an EXCHANGE_ID is directed at the location address. 1489 This operation is used to register the client-owner with the 1490 server, to obtain a client ID to be use subsequently to 1491 communicate with it, to obtain tat client ID's confirmation 1492 status and, to determine server_owner and scope for the purpose 1493 of determining if the entry is trunkable with that previously 1494 being used to access the file system (i.e. that it represents 1495 another path to the same file system and can share locking state 1496 with it). 1498 2. Making an initial determination of whether migration has 1499 occurred. The initial determination will be based on whether the 1500 EXCHANGE_ID results indicate that the current location element is 1501 server-trunkable with that used to access the file system when 1502 access was terminated by receiving NFS4ERR_MOVED. If it is, then 1503 migration has not occurred and the transition is dealt with, at 1504 least initially, as one involving continued access to the same 1505 file system on the same server through a new network address. 1507 3. Obtaining access to existing session state or creating new 1508 sessions. How this is done depends on the initial determination 1509 of whether migration has occurred and can be done as described in 1510 Section 5.4.4 in the case of migration or as described in 1511 Section 5.4.5 in the case of a network address transfer without 1512 migration. 1514 4. Verification of the trunking relationship assumed in step 2 as 1515 discussed in Section 2.10.5.1 of [RFC5661]. Although this step 1516 will generally confirm the initial determination, it is possible 1517 for verification to fail with the result that an initial 1518 determination that a network address shift (without migration) 1519 has occurred may be invalidated and migration determined to have 1520 occurred. There is no need to redo step 3 above, since it will 1521 be possible to continue use of the session established already. 1523 5. Obtaining access to existing locking state and/or reobtaining it. 1524 How this is done depends on the final determination of whether 1525 migration has occurred and can be done as described in 1526 Section 5.4.4 in the case of migration or as described in 1527 Section 5.4.5 in the case of a network address transfer without 1528 migration. 1530 Once the initial address has been determined, clients are free to 1531 apply an abbreviated process to find additional addresses trunkable 1532 with it (clients may seek session-trunkable or server trunkable 1533 addresses depending on whether they support clientid trunking). 1534 During this later phase of the process, further location entries are 1535 examined using the abbreviated procedure specified below: 1537 1. Before the EXCHANGE_ID, the fs_name field is examined and if it 1538 does not match that currently being used, the entry is ignored. 1539 otherwise, one proceeds as specified by step 1 above,. 1541 2. In the case that the network address is session-trunkable with 1542 one used previously a BIND_CONN_TO_SESSION is used to access that 1543 session using new network address. Otherwise, or if the bind 1544 operation fails, a CREATE_SESSION is done. 1546 3. The verification procedure referred to in step 4 above is used. 1547 However, if it fails, the entry is ignored and the next available 1548 entry is used. 1550 5.4.4. Obtaining Access to Sessions and State after Migration 1552 In the event that migration has occurred, the determination of 1553 whether Transparent State Migration has occurred is driven by the 1554 client ID returned by the EXCHANGE_ID and the reported confirmation 1555 status. 1557 o If the client ID is an unconfirmed client ID not previously known 1558 to the client, then Transparent State Migration has not occurred. 1560 o If the client ID is a confirmed client ID previously known to the 1561 client, then any transferred state would have been merged with an 1562 existing client ID representing the client to the destination 1563 server. In this state merger case, Transparent State Migration 1564 might or might not have occurred and a determination as to whether 1565 it has occurred is deferred until sessions are established and we 1566 are ready to begin state recovery. 1568 o If the client ID is a confirmed client ID not previously known to 1569 the client, then the client can conclude that the client ID was 1570 transferred as part of Transparent State Migration. In this 1571 transferred client ID case, Transparent State Migration has 1572 occurred although some state may have been lost. 1574 Once the client ID has been obtained, it is necessary to obtain 1575 access to sessions to continue communication with the new server. In 1576 any of the cases in which Transparent State Migration has occurred, 1577 it is possible that a session was transferred as well. To deal with 1578 that possibility, clients can, after doing the EXCHANGE_ID, issue a 1579 BIND_CONN_TO_SESSION to connect the transferred session to a 1580 connection to the new server. If that fails, it is an indication 1581 that the session was not transferred and that a new session needs to 1582 be created to take its place. 1584 In some situations, it is possible for a BIND_CONN_TO_SESSION to 1585 succeed without session migration having occurred. If state merger 1586 has taken place then the associated client ID may have already had a 1587 set of existing sessions, with it being possible that the sessionid 1588 of a given session is the same as one that might have been migrated. 1589 In that event, a BIND_CONN_TO_SESSION might succeed, even though 1590 there could have been no migration of the session with that 1591 sessionid. 1593 Once the client has determined the initial migration status, and 1594 determined that there was a shift to a new server, it needs to re- 1595 establish its lock state, if possible. To enable this to happen 1596 without loss of the guarantees normally provided by locking, the 1597 destination server needs to implement a per-fs grace period in all 1598 cases in which lock state was lost, including those in which 1599 Transparent State Migration was not implemented. 1601 Clients need to be deal with the following cases: 1603 o In the state merger case, it is possible that the server has not 1604 attempted Transparent State Migration, in which case state may 1605 have been lost without it being reflected in the SEQ4_STATUS bits. 1606 To determine whether this has happened, the client can use 1607 TEST_STATEID to check whether the stateids created on the source 1608 server are still accessible on the destination server. Once a 1609 single stateid is found to have been successfully transferred, the 1610 client can conclude that Transparent State Migration was begun and 1611 any failure to transport all of the stateids will be reflected in 1612 the SEQ4_STATUS bits. Otherwise. Transparent State Migration has 1613 not occurred. 1615 o In a case in which Transparent State Migration has not occurred, 1616 the client can use the per-fs grace period provided by the 1617 destination server to reclaim locks that were held on the source 1618 server. 1620 o In a case in which Transparent State Migration has occurred, and 1621 no lock state was lost (as shown by SEQ4_STATUS flags), no lock 1622 reclaim is necessary. 1624 o In a case in which Transparent State Migration has occurred, and 1625 some lock state was lost (as shown by SEQ4_STATUS flags), existing 1626 stateids need to be checked for validity using TEST_STATEID, and 1627 reclaim used to re-establish any that were not transferred. 1629 For all of the cases above, RECLAIM_COMPLETE with an rca_one_fs value 1630 of true should be done before normal use of the file system including 1631 obtaining new locks for the file system. This applies even if no 1632 locks were lost and needed to be reclaimed. 1634 5.4.5. Obtaining Access to Sessions and State after Network Address 1635 Transfer 1637 The case in which there is a transfer to a new network address 1638 without migration is similar to that described in Section 5.4.4 in 1639 that there is a need to obtain access to needed sessions and locking 1640 state. However, the details are simpler and will vary depending on 1641 the type of trunking between the address receiving NFS4ERR_MOVED and 1642 that to which the transfer is to be made 1644 To make a session available for use, a BIND_CONN_TO_SESSION should be 1645 used to obtain access to the session previously in use. Only if this 1646 fails, should a CREATE_SESSION be done. While this procedure mirrors 1647 that in Section 5.4.4, there is an important difference in that 1648 preservation of the session is not purely optional but depends on the 1649 type of trunking. 1651 Access to appropriate locking state should need no actions beyond 1652 access to the session. However. the SEQ4_STATUS bits should be 1653 checked for lost locking state, including the need to reclaim locks 1654 after a server reboot. 1656 5.5. Resolution of NFSv4.1 Issues 1658 One possibility is that addressing all of the NFSv4.1 issues would 1659 entail publication of a standards-track document updating [RFC5661]. 1661 Such a document would have three major elements: 1663 o A considerable expansion of the existing Section 11.4 explaining 1664 the various uses of the location attribute and the possible 1665 interactions among these various uses. This, like the 1666 corresponding replacement section for NFSv4.0 would be based on 1667 our Section 3.2 above. Information regarding the specifics of 1668 trunking discovery might appear in this section, in a new sub- 1669 section. As part of this revision, the existing Section 11.4.2 1670 would need to be revised to explain all the possible results of 1671 NFS4ERR_MOVED including migration and a possible transparent 1672 transition in which the network address changes but the server 1673 does not. 1675 o A revision of the existing section 18.35 (dealing with 1676 EXCHANGE_ID) addressing the issues discussed in Section 5.2.1. 1678 o A major replacement of the existing Section 11.7, entitled 1679 "Effecting File System Transitions", as discussed below. 1681 In addition, there is a set of smaller changes necessary 1683 o Update the existing Section 2.10.5 to clarify the proper response 1684 to server_owner changes, as described in our Section 5.1.3. 1686 o Replacement of the existing Section 15.1.2.4 to reflect the fact 1687 that NFS4ERR_MOVED can occur when a file system is now accessible 1688 at a different network address. A possible replacement text might 1689 be: 1691 The file system that contains the current filehandle object is 1692 not accessible using the network address which has been used. 1693 It may have been relocated, migrated to another server, be 1694 accessible using another network address on the current server, 1695 or it may have never been present. The client may obtain the 1696 new file system location by obtaining the "fs_locations" or 1697 "fs_locations_info" attribute for the current filehandle. For 1698 further discussion, refer to Section 11.4.2 1700 The replacement for the existing section 11.7 would maintain most 1701 sections essentially as they are, only making minor changes to 1702 include server-trunking in the discussion. However, in some cases 1703 involving more significant changes to existing sub-sections, and 1704 potential new sub-sections are listed below: 1706 o The existing Section 11.7.1 needs to be modified to refer 1707 explicitly to the previous discussion of trunking discovery. 1709 In addition, the term "multi-home single-server namespace", used 1710 nowhere else in [RFC5661], poses difficulties. From the 1711 description given it appears that the case being referred to in 1712 one in which two network addresses return server_owners with the 1713 same major_id and different minor_id values, making the network 1714 addresses server-trunkable without being session trunkable. 1716 A better approach would be to refer to "server-trunking" as used 1717 elsewhere in this document and use the replacement for the 1718 existing Section 18.35 to identify clientid trunking as the means 1719 to adapt to network addresses which are server-trunkable without 1720 being session-trunkable and session trunking as the means to adapt 1721 to network addresses which are session-trunkable. 1723 o The existing Section 11.7.2 needs to be better connected to 1724 trunking discovery. By calling these "transparent" transitions, 1725 it obscures the fact that some (or all) of the "transitions" it is 1726 discussing are not in fact transitions between servers or file 1727 systems but merely changes the set of communication paths in use. 1729 o The existing Section 11.7.2.1, needs to address more clearly the 1730 case of server-trunkable addresses which are not session- 1731 trunkable. As it is, it mentions the related concept of 1732 clustering, but only deals explicitly with the case in which two 1733 distinct servers share access to one or more file systems and does 1734 not mention the case in which the network addresses can be used to 1735 access a shared stateid space without being session-trunkable. 1737 o The existing Section 11.7.2.2, while correct, needs to be part of 1738 a general re-organization since the characteristics it lists as 1739 necessary for a transparent transition will be of use in other 1740 contexts, particularly as they apply to Transparent State 1741 Migration as well. It make sense to move these to a new sub- 1742 section within the equivalent of the Existing Section 11.7. 1744 o The existing Section 11.7.7, needs the a major rework to deal with 1745 its basic assumption, that existing state can only be made 1746 available on the destination server if the source and destination 1747 co-operate in state management and maintain a common client id 1748 space. It is not clear how this can be done, other than for 1749 servers working together so as to provide clientid trunking, a 1750 case that is probably considered as a "transparent transition". 1751 The section needs to modified to allow something along the lines 1752 of NFSv4.0-style Transparent State Migration with the details 1753 provided by a later section (see below). 1755 A related issues concerns the sentence, "In the case of migration, 1756 the servers involved in the migration of a file system SHOULD 1757 transfer all server state from the original to the new server. It 1758 is unclear why this is a "SHOULD" as the rest of the paragraph 1759 essentially tells the client that it needs to be prepared for the 1760 server not to do this. The equivalent is a "should" in [RFC7931], 1761 and there is no reason to add to confusion by making a "SHOULD" in 1762 NFSv4.1. also, there is no mention of the need to provide a fs- 1763 specific grace period in the cases in which Transparent State 1764 Migration is not made available. 1766 o Adding a new section (at level of the existing Section 11.7.7) 1767 about state transfer during migration. Although the phrase 1768 "Transparent State Migration" is well established in the context 1769 of NFSv4.0, the word "transparent" could cause confusion given the 1770 existing use of the phrase "transparent transitions". A possible 1771 title for the new section is "State Transfer during Migration" 1773 The new section would present the NFSv4.1-equivalent of Transparent 1774 State Migration as described in [RFC7931]. This would address the 1775 issues presented in Section 5.1 along the lines suggested in Sections 1776 5.2, 5.3, and 5.4. 1778 6. Evolution of Issue Handling 1780 6.1. History of this Document 1782 The contents of successive versions of this document have changed 1783 because new issues have been discovered, because there have been 1784 changes in our understanding of how these features should interact, 1785 and because some of the issues have been adequately addressed with 1786 regard to certain protocol versions. 1788 As a result, it may be helpful to understand the history of these 1789 issues, which is complicated because multiple NFSv4 protocols have 1790 been involved. 1792 This history can be summarized as follows 1794 o Initially, the focus was on the difficulties seen in NFSv4.0 1795 implementations of Transparent State Migration, and on identifying 1796 possible corrections to [RFC7530] that might address these issues. 1798 At this point, treatment of NFSv4.1 was minimal. 1800 o As examination of the issues continued, it became clear that the 1801 use of the non-uniform client string model was a critical element 1802 of the problem and further work proceeded on that basis. 1804 During the period, treatment of NFSv4.1 was expanded but the fact 1805 that NFSv4.1 had existing facilities for trunking detection was 1806 taken as an indication that the problems would not be difficult to 1807 address.. 1809 o As work proceeded on a standards-track document addressing the 1810 NFSv4.0 issues, material that proposed changes to address the 1811 issues became less relevant, since the effective vehicle for 1812 addressing these issues became the standards-track document 1813 eventually published as [RFC7931]. 1815 During this period, and subsequently, treatment of NFSv4.1 1816 remained essentially unchanged. 1818 o With the publication of [RFC7931], material regarding fixes for 1819 the NSV4.0 became vestigial but the material was retained for a 1820 while together with a shift from proposing those changes to 1821 reporting that they had been made. 1823 o Later, in response to experiences testing existing NFSv4.1 1824 implementations of migration, the focus of the document shifted 1825 decisively to NFSv4.1. As part of the analysis of migration 1826 within NFSv4.1, it was realized that issues related to the 1827 appearance of multiple addresses were fundamental to clearly 1828 describing how migration would work and that changes in the set of 1829 such addresses might or might not involve migration. 1831 At this point, discussion of NFSV4.0 issues was further limited. 1832 The issues seen were noted but the discussion of the resolution 1833 was limited to explaining that they had been addressed by the 1834 publication of [RFC7931]. 1836 o Finally, based on the results of work to provide NFSv4 with 1837 trunking discovery facilities, a decision was made that this work 1838 was most appropriately dealt with together with migration, for 1839 reasons noted previously. 1841 Since the trunking discovery facilities apply to all NFSv4 minor 1842 versions, work was needed to define those for NFSv4.0as well, 1843 together with the necessary interactions with migration. 1845 Although there is a need for further working group discussion and 1846 review, it appears that the issues to be dealt with have been 1847 identified and that most work to address these issues need to take 1848 place as part of the construction of one or more standards-track 1849 documents. See Section 6.2 for further information about possible 1850 approaches to providing the necessary documents. 1852 6.2. Further Work Needed 1854 The following table classifies issues in this area and indicates 1855 which are currently adequately addressed and where the protocol 1856 specifications need further correction or clarification. Where the 1857 topic is adequately addressed, a reference is given to the RFC 1858 providing support for the issue. In other cases, an area name 1859 (explained below) is given. 1861 +---------+-----------+-----------+---------------+-----------------+ 1862 | Version | Trunking | Trunking | Transparent | Interaction of | 1863 | | Detection | Discovery | State | Trunking and | 1864 | | | | Migration | Migration | 1865 +---------+-----------+-----------+---------------+-----------------+ 1866 | v4.0 | [RFC7931] | TrDisc-0 | [RFC7931] | Int-0 | 1867 | v4.1+ | [RFC5661] | TrDisc-1 | TSM-1 | Int-1 | 1868 +---------+-----------+-----------+---------------+-----------------+ 1870 The following table explains the work that needs to be done 1871 corresponding to each area name above. 1873 +----------+--------------------------------------------------------+ 1874 | Area | Description | 1875 | Name | | 1876 +----------+--------------------------------------------------------+ 1877 | TrDisc-0 | Although it is possible for there to be multiple | 1878 | | location entries for a given file system, the | 1879 | | possibility of using these to enable trunking | 1880 | | discovery was not addressed in [RFC7530], most likely | 1881 | | because trunking was considered a problem to be | 1882 | | avoided (rather than a helpful feature) at that time. | 1883 | | This situation could have been addressed by the | 1884 | | publication of [RFC7931] but unfortunately that did | 1885 | | not happen. | 1886 +----------+--------------------------------------------------------+ 1887 | TrDisc-1 | Despite the fact that [RFC5661] provides a means of | 1888 | | trunking detection, trunking discovery was not | 1889 | | addressed. This problem was compounded by confusion | 1890 | | regarding multiple file system replicas arising from | 1891 | | the fact that multiple network addresses connected to | 1892 | | the same server were treated as if they were referring | 1893 | | to distinct sets of replicas. | 1894 +----------+--------------------------------------------------------+ 1895 | TSM-1 | Unlike [RFC7530], which mishandled Transparent State | 1896 | | Migration because of confusion arising from the lack | 1897 | | of appropriate trunking support, [RFC5661] simply | 1898 | | neglected to provide any description of this feature. | 1899 | | It appears likely that confusion between the needs of | 1900 | | migration and those of dealing with shifts in | 1901 | | responsibility for clustered file system access had | 1902 | | significant role in allowing this issue to be ignored. | 1903 | | Rectifying this situation along the lines of [RFC7931] | 1904 | | is complicated by the need to rewrite significant | 1905 | | pieces of the section about multi-server namespace to | 1906 | | address this confusion. Beyond this, the necessary | 1907 | | treatment will need to reflect changes required by the | 1908 | | use of the sessions model and related changes in | 1909 | | NFSv.1 and also address migration-related issues | 1910 | | raised by optional features such as pNFS and the | 1911 | | fs_locations_info attribute. | 1912 +----------+--------------------------------------------------------+ 1913 | Int-0 | The need to provide trunking-related information puts | 1914 | | additional focus on the issue of dealing with changes | 1915 | | in the value of location-related attributes. This | 1916 | | applies when trunking configurations change and at | 1917 | | other times as well. In addition, the existence of | 1918 | | multiple network addresses connected to the same | 1919 | | server requires clarification when migration and | 1920 | | replication features are used. | 1921 +----------+--------------------------------------------------------+ 1922 | Int-1 | This requires similar handling to the case above. | 1923 | | However, further work is made necessary by the fact | 1924 | | that shifts between different sets of network | 1925 | | addresses are erroneously treated as instances of | 1926 | | migration in [RFC5661]. | 1927 +----------+--------------------------------------------------------+ 1929 There are number of possible ways of packaging the necessary changes 1930 into RFCs. Some of these are impractical for various reasons: 1932 o While it possible to treat each area in its own RFC, writing five 1933 RFCs would increase the work required, and delay needed 1934 corrections to both versions. Further, it would result in a 1935 situation in which in which someone needing to understand the 1936 specification of NFS version 4.0 or 4.1 would need to be familiar 1937 with a large set of RFCs. 1939 o One could have a document addressing all of the areas above Such a 1940 document would update both [RFC7530] and [RFC5661]. That would 1941 result in a confusing document given how different the v4.0 and 1942 v4.1 protocols are, since most readers will want a clear 1943 description of one or the other. 1945 o It is also possible to produce separate documents addressing 1946 Trdisc-*, TSM-1., an Int-*. This would be subject many of the 1947 difficulties of the two approaches above. 1949 The alternative, of organizing the changes by minor version, is being 1950 actively pursued. 1952 o [I-D.cel-nfsv4-mv0-trunking-update] is a personal draft with a 1953 Standards Track Intended Status. It addresses the issues within 1954 TrDisc-0 and Int-0 by providing updates to [RFC7530], the vast 1955 majority of which are within Section 8 of that document. 1957 o [I-D.dnoveck-nfsv4-mv1-msns-update] is a personal draft with a 1958 Standards Track Intended Status. It addresses the issues within 1959 TrDisc-1, TSM-1, and Int-1 providing updates to [RFC5661], the 1960 vast majority of which are within Section 11 of that document. 1962 These documents will require additional review and discussion before 1963 the working group considers making them working group documents and 1964 decides either to do so, or to address these issues in some other 1965 way. 1967 If the former path is taken, it will be necessary to consolidate the 1968 changes currently specified in these documents. Currently, these 1969 document replace individual sub-sections of Section 8 (of [RFC7530]) 1970 or Section 11 (of [RFC5661]). While this is helpful in explaining 1971 what is changing and why, things will be different when the eventual 1972 RFC is published. At that point, it is likely to be more important 1973 to have simply understood specifications of NFS versions 4.0 and 4.1. 1974 At that point, a full replacement section of the affected section may 1975 be more desirable as the basis of the RFC to be published. 1977 7. Security Considerations 1979 In general, the Security Considerations sections of existing 1980 specifications for NFS versions 4.0 and 4.1 provide recommendations 1981 for appropriate handling of requests obtaining location-related 1982 information. In particular, it is recommended that integrity 1983 protection be used when fetching location-related attributes: 1985 o With regard to NFSv4.01, this is done in Section 8.6 of [RFC7931] 1986 which updates the Security Considerations section of [RFC7530]. 1988 o With regard to NFSv4.1, this is done in the Security 1989 Considerations section of [RFC5661]. 1991 Despite this however, there is a need for further changes in the 1992 Security Considerations with regard to both minor versions dealt with 1993 here. The following issues need to be addressed: 1995 o Because of the potential use of DNS to convert server names to a 1996 set of server network addresses, such translations are subject to 1997 the same sorts of potential interference with trunking discovery 1998 that would occur when trunking discovery is provided using network 1999 addresses returned in the location-related attributes. 2001 To address this issue, specifications for both minor versions need 2002 to mention the issue and indicate that use of DNSSEC [RFC4033] is 2003 appropriate. When it is not available, the server should allow 2004 use of DNS for trunking discovery to be avoided by presenting 2005 network addresses in the location-related attributes, with these 2006 values subject to RPCSECGSS integrity protection. 2008 o Although use of RPCSEC_GSS ([RFC2203], [RFC5403], [RFC7861]) with 2009 integrity protection is RECOMMENDED and "implementations" are 2010 REQUIRED to provide support. However, the possibility that a 2011 particular client may be unable to use RPCSEC_GSS when accessing a 2012 particular server cannot be excluded. As a result, it is 2013 necessary to discuss how such situations affect trunking 2014 discovery, referral, replication, and migration. 2016 o In the case of replication, referral, and migration, it is 2017 necessary to discuss how RPCSEC_GSS mutual authentication on the 2018 destination can be used to make sure that the network addresses 2019 provided by trunking discovery have not been interfered with and 2020 correspond to the server names provided by the location attributes 2021 on the server to which the client was directed. 2023 8. IANA Considerations 2025 This document does not require actions by IANA. 2027 9. References 2029 9.1. Normative References 2031 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2032 Requirement Levels", BCP 14, RFC 2119, 2033 DOI 10.17487/RFC2119, March 1997, 2034 . 2036 [RFC2203] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol 2037 Specification", RFC 2203, DOI 10.17487/RFC2203, September 2038 1997, . 2040 [RFC4033] Arends, R., Austein, R., Larson, M., Massey, D., and S. 2041 Rose, "DNS Security Introduction and Requirements", 2042 RFC 4033, DOI 10.17487/RFC4033, March 2005, 2043 . 2045 [RFC5403] Eisler, M., "RPCSEC_GSS Version 2", RFC 5403, 2046 DOI 10.17487/RFC5403, February 2009, 2047 . 2049 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 2050 "Network File System (NFS) Version 4 Minor Version 1 2051 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, 2052 . 2054 [RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System 2055 (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, 2056 March 2015, . 2058 [RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC) 2059 Security Version 3", RFC 7861, DOI 10.17487/RFC7861, 2060 November 2016, . 2062 [RFC7931] Noveck, D., Ed., Shivam, P., Lever, C., and B. Baker, 2063 "NFSv4.0 Migration: Specification Update", RFC 7931, 2064 DOI 10.17487/RFC7931, July 2016, 2065 . 2067 9.2. Informative References 2069 [I-D.cel-nfsv4-mv0-trunking-update] 2070 Lever, C. and D. Noveck, "NFS version 4.0 Trunking 2071 Update", draft-cel-nfsv4-mv0-trunking-update-00 (work in 2072 progress), November 2017. 2074 [I-D.dnoveck-nfsv4-mv1-msns-update] 2075 Noveck, D. and C. Lever, "NFSv4.1 Update for Multi-Server 2076 Namespace", draft-dnoveck-nfsv4-mv1-msns-update-01 (work 2077 in progress), November 2017. 2079 Acknowledgements 2081 The editor and authors of this document gratefully acknowledge the 2082 contributions of Trond Myklebust of Primary Data, Robert Thurlow of 2083 Oracle, and Andy Adamson of NetApp. We also thank Tom Haynes of 2084 Primary Data and Spencer Shepler of Microsoft for their guidance and 2085 suggestions. 2087 Special thanks go to members of the Oracle Solaris NFS team, 2088 especially Rick Mesta and James Wahlig who were then part of that 2089 team, for their work implementing an NFSv4.0 migration prototype and 2090 identifying many of the issues documented here. Also, the work of 2091 Xuan Qi for Oracle using NFSv4.1 client and server prototypes was 2092 helpful. 2094 Authors' Addresses 2096 David Noveck (editor) 2097 NetApp 2098 26 Locust Avenue 2099 Lexington, MA 02421 2100 US 2102 Phone: +1 781 572 8038 2103 Email: davenoveck@gmail.com 2105 Piyush Shivam 2106 Oracle Corporation 2107 5300 Riata Park Ct. 2108 Austin, TX 78727 2109 US 2111 Phone: +1 512 401 1019 2112 Email: piyush.shivam@oracle.com 2114 Charles Lever 2115 Oracle Corporation 2116 1015 Granger Avenue 2117 Ann Arbor, MI 48104 2118 US 2120 Phone: +1 248 614 5091 2121 Email: chuck.lever@oracle.com 2123 Bill Baker 2124 Oracle Corporation 2125 5300 Riata Park Ct. 2126 Austin, TX 78727 2127 US 2129 Phone: +1 512 401 1081 2130 Email: bill.baker@oracle.com