idnits 2.17.1 draft-noveck-nfsv4-referrals-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 17. -- Found old boilerplate from RFC 3978, Section 5.5 on line 1051. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1062. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1069. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1075. ** Found boilerplate matching RFC 3978, Section 5.4, paragraph 1 (on line 1042), which is fine, but *also* found old RFC 2026, Section 10.4C, paragraph 1 text on line 37. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** The document seems to lack an RFC 3978 Section 5.4 Reference to BCP 78 -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == Line 139 has weird spacing: '...uggests other...' -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (February 2005) is 7002 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC3530' is defined on line 1015, but no explicit reference was found in the text ** Obsolete normative reference: RFC 3530 (Obsoleted by RFC 7530) Summary: 10 errors (**), 0 flaws (~~), 4 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 INTERNET-DRAFT David Noveck 3 Expires: August 2005 Network Appliance, Inc. 4 Rodney C. Burnett 5 IBM, Inc. 7 February 2005 9 Implementation Guide for Referrals in NFSv4 10 draft-noveck-nfsv4-referrals-00.txt 12 Status of this Memo 14 By submitting this Internet-Draft, I certify that any applicable 15 patent or other IPR claims of which I am aware have been disclosed, 16 or will be disclosed, and any of which I become aware will be 17 disclosed, in accordance with RFC 3668. 19 Internet-Drafts are working documents of the Internet Engineering 20 Task Force (IETF), its areas, and its working groups. Note that 21 other groups may also distribute working documents as Internet- 22 Drafts. 24 Internet-Drafts are draft documents valid for a maximum of six 25 months and may be updated, replaced, or obsoleted by other 26 documents at any time. It is inappropriate to use Internet-Drafts 27 as reference material or to cite them other than as "work in 28 progress." 30 The list of current Internet-Drafts can be accessed at 31 http://www.ietf.org/ietf/1id-abstracts.txt The list of 32 Internet-Draft Shadow Directories can be accessed at 33 http://www.ietf.org/shadow.html. 35 Copyright Notice 37 Copyright (C) The Internet Society (2005). All Rights Reserved. 39 Abstract 41 RFC3530 describes a migration feature using the NFS4ERR_MOVED error 42 code and the fs_locations attribute. The description focuses on 43 the case of migration (i.e. relocation) of a file system already 44 known to the client. The simpler limiting case in a which a file 45 system not previously known to the client was located elsewhere, 46 which we here call a referral, was not clearly described. Because 47 of this and also because of some inconsistencies and ambiguities in 48 the text of RFC3530, there has been confusion about how the client 49 and server should interact in performing a referral. This document 50 provides a guide to the implementation of referrals, and in so 51 doing, addresses the relevant problems in RFC3530. 53 Table Of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2 56 1.1. Referrals as a Limiting Case of Migrations . . . . . . . 3 57 1.2. Pure Referrals . . . . . . . . . . . . . . . . . . . . . 3 58 2. Issues as to When NFS4ERR_MOVED May/Should be Returned . . 4 59 2.1. PUTFH Returning NFS4ERR_MOVED . . . . . . . . . . . . . 5 60 2.2. GETFH Returning NFS4ERR_MOVED . . . . . . . . . . . . . 5 61 2.3. GETATTR Returning NFS4ERR_MOVED . . . . . . . . . . . . 6 62 2.4. Ops Not Allowed to Return NFS4ERR_MOVED . . . . . . . . 6 63 3. Attribute Issues . . . . . . . . . . . . . . . . . . . . . 7 64 3.1. General Issues of Incomplete Attribute Sets . . . . . . 7 65 3.2. Attributes Needed for Root of an Absent Fs . . . . . . . 10 66 3.3. Attributes Valid for Root of an Absent Fs . . . . . . . 12 67 3.4. Attributes Not Valid for Root of an Absent Fs . . . . . 13 68 4. Referral Examples . . . . . . . . . . . . . . . . . . . . 14 69 4.1. Referral Example (LOOKUP) . . . . . . . . . . . . . . . 15 70 4.2. Referral Example (READDIR) . . . . . . . . . . . . . . . 19 71 5. Additional Client-side Considerations . . . . . . . . . . 21 72 Acknowledgements . . . . . . . . . . . . . . . . . . . . . 22 73 Normative References . . . . . . . . . . . . . . . . . . . 22 74 Authors' Addresses . . . . . . . . . . . . . . . . . . . . 22 75 Full Copyright Statement . . . . . . . . . . . . . . . . . 23 77 1. Introduction 79 RFC3530 describes a migration feature using the NFS4ERR_MOVED error 80 code and the fs_locations attribute. The description focuses on 81 the case of migration of a file system already known to the client, 82 while the alternate case in a which a file system not previously 83 known to the client was located elsewhere, which we here call a 84 referral, was not clearly described. 86 Analysis of how this case would be dealt with has shown that the 87 protocol in RFC3530 is capable of dealing properly with this sub- 88 case. However, there are difficulties for implementers in 89 determining how precisely this is to be done, because the text of 90 RFC3530 does not address this case. In addition, there are a 91 number of ambiguities and inconsistencies in the text, that 92 increase the difficulties for implementers and that raise the 93 probability that clients and servers may not properly interact. 95 This document is an attempt to deal with these spec issues and to 96 provide a clear guide to follow in developing inter-operable 97 referral implementations. 99 1.1. Referrals as a Limiting Case of Migrations 101 If a server implements file system migration, individual clients, 102 not being synchronized with the migration event, will encounter 103 different situations from the client point of view. That is, even 104 though some clients will have received filehandles within the 105 migrated filesystem, others may at that time be first referencing 106 the root of the migrated filesystem and need to be redirected to 107 the new fs location, just as those clients that have already 108 accessed the filesystem (when it was present) need to be 109 redirected. 111 Thus referrals are a limiting sub-case of migration and when a 112 migration event occurs some clients will see an ordinary migration 113 in the case in which access (by that client) to the filesystem 114 being moved has already occurred, while others will see a referral 115 when they first attempt to access the absent filesystem. 117 1.2. Pure Referrals 119 Given that clients supporting migration need to be prepared for a 120 migration event, a server can establish a situation, which we refer 121 to as a pure referral, in which all clients see a referral. Since 122 no client can enter the absent filesystem, the presumptive 123 migration of the absent filesystem, assumes a purely conventional 124 character, in that for each client it appears to have happened 125 before all access by that client. This allows the server to 126 establish a situation in which referral to an absent filesystem may 127 occur for filesystems that were never present on the server. 129 This functionality may be used to allow construction of multi- 130 server namespaces and may serve as a building block for the 131 construction of wider global namespaces. 133 2. Issues as to When NFS4ERR_MOVED May/Should be Returned 135 RFC3530 contains the following statement in section 6.2: "The 136 NFS4ERR_MOVED error is returned for all operations except PUTFH and 137 GETATTR.", which is contradicted by the error lists in the detailed 138 operation descriptions. PUTFH and GETATTR, while the statement 139 above suggests otherwise, have NFS4ERR_MOVED listed as error 140 codes, whereas there are six other ops (PUTROOTFH, PUTPUBFH, RENEW, 141 SETCLIENTID, SETCLIENTID_CONFIRM, RELEASE_OWNER) which are not 142 allowed to return NFS4ERR_MOVED despite the "all operations except" 143 clause given above. 145 The individual ops will be discussed below but in order to 146 understand how these contradictions arose and how they are to be 147 dealt with, it is important to be clear on what point the current 148 filesystem (i.e. the filesystem associated with the current 149 filehandle) is to be tested for absence in deciding whether to 150 return NFS4ERR_MOVED. For operations which change the current 151 filehandle, (i.e. PUTFH, PUTROOTFH, PUTPUBFH, RESTOREFH, LOOKUP, 152 LOOKUPP), this distinction is critical. The spec is, however, not 153 very clear on this issue so we need to consider the alternatives. 155 Let's first consider a check on the current filehandle after the 156 operation. In part, that seems dubious because it returns an error 157 for an operation where the circumstances that give rise to the 158 error (i.e. changing the filehandle) depend on the successful 159 execution of the operation giving rise to the error. However, the 160 main reason such an interpretation is insupportable is that it 161 would make some situations, wherein fs_locations is needed, 162 impossible to deal with. For example, LOOKUP followed by 163 GETATTR(fs_locations) would not be possible if the LOOKUP returned 164 NFS4ERR_MOVED when the current filehandle at the start of the 165 LOOKUP was in a present filesystem while the current filehandle at 166 the end of the LOOKUP was in an absent filesystem. An 167 NFS4ERR_MOVED returned by LOOKUP would not allow execution of the 168 GETATTR, any exception for GETATTR of fs_locations notwithstanding. 170 On the other hand, making the check at the start of each operation 171 (except some GETATTR's) allows fs_locations to be determined 172 whenever necessary and is the best way to resolve the issue. 174 Except where RFC3530 has made this impossible by not listing 175 NFS4ERR_MOVED as an error code for certain ops, each op should 176 check at the beginning of the operation for a current filehandle 177 within an absent fs. If so, NFS4ERR_MOVED is returned. A partial 178 exception is GETATTR for which the return of NFS4ERR_MOVED (or not) 179 is dependent on the set of attributes requested. This is discussed 180 further below. 182 2.1. PUTFH Returning NFS4ERR_MOVED 184 As noted above, section 6.2 indicates that PUTFH will not return 185 NFS4ERR_MOVED while the detailed description for PUTFH (section 186 14.2.20), lists NFS4ERR_MOVED as a valid return. How are we to 187 resolve the contradiction? 189 It would seem that the exception for PUTFH in section 6.2, is 190 unnecessary and derives from a confusion about the logic of 191 NFS4ERR_MOVED. Clearly the sequence of PUTFH(fh for absent volume) 192 followed by GETATTR(fs_locations) must be allowed but it requires 193 no specific exception for PUTFH to do so, since only GETATTR, and 194 not PUTFH, has a filehandle within an absent filesystem as the 195 current handle at the start of the operation. Thus no exception 196 for PUTFH is required, as is recognized by section 14.2.20. 198 PUTFH will never return NFS4ERR_MOVED because the filehandle it is 199 establishing belongs to an absent filesystem. This is because the 200 current filehandle/filesystem is checked for absence at the start 201 of each op and the new filehandle for PUTFH has not been 202 established at the start of the PUTFH. This allows the sequence 203 PUTFH-GETATTR(fs_locations) to proceed without any specific 204 exception for PUTFH or any analysis of the op stream that 205 encompasses multiple ops (e.g. looking ahead to the GETATTR). 207 NFS4ERR_MOVED can happen as a result of a PUTFH, but only in rather 208 odd circumstances such as the following: 210 o PUTFH (fh within absent filesystem) 212 o PUTFH (fh within present filesystem) 214 In this case, NFS4ERR_MOVED would be returned on the second PUTFH 215 since the current filehandle at the start of that op is within an 216 absent filesystem. It may seem odd that the PUTFH which is 217 establishing a current filehandle within a present filesystem 218 should get an error while the one that established a current 219 filehandle within an absent filesystem does not but this behavior 220 is a consequence of a uniform set of rules for NFS4ERR_MOVED, the 221 basis of which is that the current filehandle is tested at the 222 start of an operation. 224 2.2. GETFH Returning NFS4ERR_MOVED 226 While RFC 3530 does not make any exception for GETFH when the 227 current filehandle is within an absent filesystem, the fact that 228 GETFH is such a passive, purely interrogative operation, may lead 229 readers to wrongly suppose that an NFSERR_MOVED error will not 230 arise in this situation. 232 In fact, GETFH will result in NFS4ERR_MOVED being returned if the 233 current file handle is for an absent filesystem. This is 234 particularly helpful in dealing with pure referral situations since 235 a filehandle that a client does not ever see can pose no 236 difficulties. For example, it is irrelevant whether the target 237 filesystem has persistent or volatile filehandles. An inherently 238 unobserved filehandle poses no expiration difficulties regardless 239 of whether the target filesystem (on the target server) has 240 persistent or volatile filehandles. Servers should never return 241 filehandles within absent filesystems for pure referral cases. 243 2.3. GETATTR Returning NFS4ERR_MOVED 245 As noted above, Section 6.2 indicates that NFS4ERR_MOVED is not 246 returned for a GETATTR operation, but NFS4ERR_MOVED is listed as an 247 error that can be returned by GETATTR (in section 14.2.7). Since 248 the purpose of the exception for GETATTR is to allow fs_locations 249 to be fetched, the best resolution of this contradiction is for 250 NFS4ERR_MOVED to be returned by GETATTR's that ask for some 251 unavailable attribute and that do not interrogate the fs_locations 252 attribute. This maintains the exception which allows GETATTR to be 253 used to get fs_locations information by establishing the rule that 254 GETATTR's which interrogate fs_locations (with or without 255 additional attributes) will not return NFS4ERR_MOVED. Neither will 256 GETATTR's that only request attributes that are available. These 257 considerations are further discussed in section 3. 259 Sometimes clients need a quickly checked indication of whether a 260 filesystenm is absent or not. In most cases, because fs_locations 261 is not requested, this indication will be the NFS4ERR_MOVED being 262 returned by the GETATTR. When fs_locations is requested, a 263 convenient way to provide this indication is to request the 264 filehandle attribute. Since this attribute will not be provided 265 for referrals (see section 3.4.2), examining the mask of returned 266 attributes for the presence of the filehandle attribute is quick 267 way to deterine whether directory in question is part of an absent 268 filesystem. 270 2.4. Ops Not Allowed to Return NFS4ERR_MOVED 272 As noted above, despite the fact that section 6.2 suggests 273 otherwise, a number of ops, all of which do not require a current 274 filehandle, are not listed as returning NFS4ERR_MOVED. These ops 275 are PUTROOTFH, PUTPUBFH, RENEW, SETCLIENTID, and RELEASE_LOCKOWNER. 277 Note also that DELEGPURGE, another op which does not require a 278 current filehandle, is listed as able to return NFS4ERR_MOVED. 280 Although this contradiction could be resolved by changing the op 281 description to match 6.2, it does not seem that there is any strong 282 reason to do so. The protocol will function adequately, if these 283 are treated as exceptions, and NFS4ERR_MOVED is returned by the 284 next op allowed to do so, as long as the current filehandle at the 285 start of that op is one within an absent fs. Note that PUTROOTFH 286 and PUTPUBFH could cause a pending absent fs indication not to 287 result in an error code, since they change the current filehandle 288 to be within a new, presumably present, filesystem. However, in 289 this case the client would not have issued any operations prior to 290 the PUTROOTFH or PUTPUBFH that actually operated on the absent fs. 292 3. Attribute Issues 294 RFC3530 anticipates that clients will request additional attributes 295 beyond fs_locations (see section 6.2 of that document) and allows 296 the server to return fs_locations only. The return of other 297 attributes, implicit in the statement "server may return 298 fs_locations only" is not clearly dealt with. While some of these, 299 such as fsid, are important for all forms of migration, this 300 omission is most critical in the case of referrals. There are a 301 number of attributes besides fs_locations that need to be provided 302 at the root of an absent filesystem for server and client to 303 properly collaborate to perform a referral. 305 There are also a large number of other attributes which the server 306 should not provide for absent filesystem since providing them is 307 the province of the referral's target server. Providing possibly 308 conflicting values on the server that is the source of the referral 309 can only contribute to confusion. 311 3.1. General Issues of Incomplete Attribute Sets 313 Migration or referral events naturally create situations in which 314 all of the attributes normally supported on a server are not 315 obtainable. RFC3530 is in places ambivalent and/or apparently 316 self-contradictory on such issues. 318 The first problem concerns the statement in the third paragraph of 319 section 6.2: "If the client requests more attributes than just 320 fs_locations, the server may return fs_locations only. This is to 321 be expected since the server has migrated the filesystem and may 322 not have a method of obtaining additional attribute data." 323 While the above seems quite reasonable, it is seemingly 324 contradicted by the following text from section 14.2.7 the second 325 paragraph of the DESCRIPTION for GETATTR: "The server must return a 326 value for each attribute that the client requests if the attribute 327 is supported by the server. If the server does not support an 328 attribute or cannot approximate a useful value then it must not 329 return the attribute value and must not set the attribute bit in 330 the result bitmap. The server must return an error if it supports 331 an attribute but cannot obtain its value. In that case no 332 attribute values will be returned." 334 The above is a helpful restriction in that it allows clients to 335 simplify their attribute interpretation code. It allows them to 336 assume that all of the attributes they request are present, often 337 making it possible to get successive attributes at fixed offsets 338 within the data stream. However, it seems to contradict what is 339 said in section 6.2, where it is clearly anticipated, at least when 340 fs_locations is requested, that fewer (often many fewer) attributes 341 will be available than are requested. It could be argued that you 342 could harmonize these two by being creative with the interpretation 343 of the phrase "if the attribute is supported by the server". One 344 could argue that many attributes are not supported by the server 345 for an absent fs even though the text talking about attributes 346 "supported by a server" seems to indicate that this is not allowed 347 to be different for different fs's (which is troublesome in itself 348 as one server might have some filesystems that do support ACLs and 349 some that don't support then, for example). 351 Note however that the following paragraph in the description says, 352 "All servers must support the mandatory attributes as specified in 353 the section 'File Attributes'". That's reasonable enough in 354 general, but for an absent fs it is not reasonable and so section 355 14.2.7 and section 6.2 are contradictory. 357 The text in section 14.2.7 would create great obstacles in the 358 implementation of migration given section 6.2. In order to get the 359 additional attributes mentioned, the client would have to guess at 360 the set to be provided and, if that guess were not valid, get an 361 error, presumably NFS4ERR_MOVED, returned. For the purposes of 362 this document, we will treat the specific statement regarding 363 fs_locations in section 6.2 as controlling. The contradictory text 364 in section 14.2.7 will be treated as an overgeneralization to which 365 an exception, for the case of migration and referrals, must be 366 understood. 368 When fs_locations is requested, then a subset of the requested 369 attributes may be provided by the server without generating the 370 error NFS4ERR_MOVED. The subset returned must always include 371 fs_locations. In addition, if all of the attributes requested can 372 be provided, then NFS4ERR_MOVED will also not be returned but in 373 this case, a subset of the requested attributes may not be 374 provided. The following sections give guidance on the attributes 375 the server should provide for filehandles within an absent 376 filesystem. 378 A related issue concerns attributes in a READDIR. There has been 379 discussion, not yet fully resolved, on the working group list about 380 whether all the attributes specified in the attribute mask for 381 READDIR must be provided for all readdir entries when those 382 attributes are supported by the associated filesystems. It can be 383 argued that that would be implied if one combined the words of 384 section 14.2.7 with text that (even though somewhat more loosely) 385 implies that the attributes in READDIR are to be provided via 386 GETATTR. It has also been argued that the role of READDIR with 387 attributes as a replacement for v3 READDIRPLUS makes it more 388 appropriate to treat the attribute mask parameters as a hint with 389 the server allowed to omit any attributes requested (possibly with 390 some exceptions) whenever inconvenient. In the discussion, there 391 have also been occasional suggestions that attributes's status (as 392 mandatory or recommended) should have role in deciding whether 393 servers are obliged to provide them when requested for READDIR. 395 Regardless of how this debate is to be resolved in the general 396 case, it is clear in the case of an absent filesystem, if there is 397 a general obligation to provide all requested attributes, that an 398 exception must be made for directory entries that represent the 399 root of an absent fs (a referral). For these entries, the full set 400 of requested attributes may simply not be available, as they 401 likewise would not be available for GETATTR. 403 Further, we will assume in our examples, that for the few 404 attributes necessary to perform the referral (see below for 405 details), the server will always provide these when requested as it 406 should be easy for the server to do. 408 So in this document, we will assume that regardless of the general 409 resolution of this issue, in the case of an absent filesystem some 410 flexibility must be provided, even if it is determined that making 411 the attribute mask for READDIR a hint in general is not justified. 413 So we will assume that for READDIR: 415 o When fs_locations is among the attributes requested, the 416 server may provide a subset of the other requested attributes 417 together with fs_locations for roots of absent fs's, without 418 causing any error for the READDIR as a whole. If rdattr_error 419 is also requested and there are attributes which are not 420 available, then rdattr_error will receive the value 421 NFS4ERR_MOVED. 423 o When fs_locations is not requested, but all of the attributes 424 requested can be provided, then they will be provided and no 425 NFS4ERR_MOVED will be generated. An example would be READDIR's 426 that request mounted_on_fileid either with or without fsid. 428 o When fs_locations is not requested, but rdattr_error is and 429 some attributes requested are not available because of the 430 absence of the filesystem, the server will return 431 NFS4ERR_MOVED for the rdattr_error attribute and, in addition, 432 the requested attributes that are valid for the root of an 433 absent filesystem. 435 o When neither fs_locations nor rdattr_error is requested and 436 there is a directory within an absent fs within the directory 437 being read, if some unavaailable attributes are requested, no 438 data will be returned and the READDIR will get an 439 NFS4ERR_MOVED error. (The examples will suppose this but 440 clients will not depend on this behavior and clients should 441 work with a more liberal interpretation if this is decided on. 443 3.2. Attributes Needed for Root of an Absent Fs 445 The following options need to be provided for referrals. While it 446 is true that some of these are not mandatory in NFSv4, and the spec 447 treats providing others as optional in the case of an absent 448 filesystem, they do need to be provided to do a referral as 449 envisioned herein. 451 3.2.1. fsid 453 The fsid attribute allows clients to recognize when fs boundaries 454 have been crossed. This applies also when one crosses into an 455 absent filesystem and servers should provide this information when 456 fs_locations is being requested for an absent filesystem. 458 To avoid misunderstanding, it should be noted that the fsid 459 provided in this case is solely so that the fs boundaries can be 460 properly noted and that the fsid returned will not necessarily be 461 valid after resolution of the referral or migration event. The 462 logic of fsid handling for NFSv4 is that fsid's are only unique 463 within a per-server context. This would seem to be a strong 464 indication that they need not be persistent when file systems are 465 moved from server to server, although RFC 3530 does not 466 specifically address the matter. 468 A key reason for always providing an fsid can be seen by 469 considering client behavior. When a client gets the error 470 NFS4ERR_MOVED, it has a number of key steps to get through. It 471 must determine if it has accessed the filesystem on the server that 472 the moved error involves. If so, then it represents a data 473 migration event in the strict sense, rather than a referral. The 474 client must recover state and properly use rootpath data from the 475 old and new server to graft the new server's namespace into the 476 client's local space. If a client gets an NFS4ERR_MOVED error and 477 it cannot get an fsid, then it must use fs_locations, fs_root, and 478 rootpath data to locally determine if it has actually accessed the 479 data or if this is a referral for this client. If it uses fs_root 480 and there have been renames along the path to the fsid, then the 481 client will not be able to distinguish a migration from a referral. 482 The best solution here is for the client to always get an fsid. All 483 a server has to do is generate and fsid value that cannot represent 484 actual exported data at the server. We want to have a model where 485 referrals can still function even if renames or other events change 486 the path to the referral point. Always providing fsid's improves 487 the usefulness of the NFS namespace since referral and migration 488 events can continue to be handled in the face of namespace 489 management. This also helps to be reduce client complexity. 491 3.2.2. mounted_on_fileid 493 The mounted_on_fileid attribute is of particular importance to many 494 clients, in that they need this information to form a proper 495 response to a readdir() call. When a readdir() call is done within 496 UNIX, the d_ino field of each of the entries needs to have a unique 497 value normally derived from the NFSv4 fileid attribute. It is in 498 the case in which a file system boundary is crossed that using the 499 fileid attribute for this purpose, particularly when crossing into 500 an absent fs, will pose problems. Note first that the fileid 501 attribute, since it is within a new fs and thus a new fileid space, 502 will not be unique within the directory. Also, since the fs, at 503 its new location, may arrange things differently, the fileid 504 decided on at the directing server may be overridden at the target 505 server, making it of little value. 507 Neither of these problems arise in the case of mounted_on_fileid 508 since that fileid is in the context of the mounted-on fs and unique 509 within it. Servers need to support and always return valid data 510 for mounted-on-fileid, even for the case of an absent filesystem. 511 The returned data represents a fileid for the entry in the 512 directory that represents the absent fs. It's not the fileid of the 513 root of the absent fs itself. Per the attribute handling rules 514 presented in section 3.1, the server may return mounted-on-fileid 515 as a subset of the requested attributes. The presence of 516 fs_locations and rdattr_error attributes in a READDIR request 517 determines if and how the server sets rdattr_error in the partial 518 return case. 520 Clients are strongly encouraged to always include mounted-on-fileid 521 and rdattr_error in READDIR requests. This increases the potential 522 that the client will receive information needed to satisfy the 523 request and avoid further READDIR requests with a more restricted 524 set of attributes. As a result, client performance is improved and 525 the network traffic is reduced. 527 3.2.3. rdattr_error attribute 529 This attribute needs to be provided for READDIR's or else a READDIR 530 of a directory that contains the root of absent fs's cannot be done 531 effectively. In order to avoid NFS4ERR_MOVED on the directory as a 532 whole, providing this error via the rdattr_error attribute allows 533 attributes for the entry in question, which can be provided to the 534 client, as well allowing the READDIR to provide information about 535 other entries within the directory. 537 3.3. Attributes Valid for Root of an Absent Fs 539 Beyond the essential attributes discussed above, there are several 540 other attributes that a server could decide to return for an absent 541 fs. These should pose little potential for inconsistencies or 542 client side problems. They are mentioned below, but clients should 543 not expect servers to return them. 545 3.3.1. type attribute 547 For the root of absent fs, providing the value NF4DIR poses no 548 difficulties as the target will provide the same value at the root 549 of the target filesystem. 551 Such a value isn't really needed as clients can assume that when 552 there is a change of fsid value, the root of new filesystem will be 553 a directory, as it must be. 555 3.3.2. change attribute 557 When a server implements pure referrals for a large number of 558 filesystems, it may be convenient for clients to cache the 559 destination fs_locations and interrogate the change attribute to 560 see if there has been any change in the locations attribute to 561 require a refetch. Therefore, the server must update the change 562 attribute when the target of the referral change. 564 If the server does provide this information, the client must 565 understand that there is no necessary continuity between the change 566 values on referring server and on the target, as there may not be 567 in the general migration case. Clients are best advised to 568 periodically refetch such values on the target server and assume 569 that it is possible that a change to the object may have occurred 570 before the fetch of the change attribute on the target, and that 571 this fact means that data and attribute caches on he client are 572 best flushed. 574 3.4. Attributes Not Valid for Root of an Absent Fs 576 Attributes not listed in the sections above should not be provided 577 in the case of the root of an absent filesystem (the referral 578 case). The general principle is that such information is 579 appropriately provided at the destination specified by the 580 fs_locations attribute and that specifying a value at the point of 581 referral will either be incorrect or superfluous. 583 We present a few examples of this principle below for particular 584 attributes but the same sort of logic applies to all of the other 585 attributes, to some degree or other. 587 3.4.1. supp_attr attribute 589 Since the referring server has no basis to determine what 590 attributes are supported on the target, it is best not to provide 591 this attribute on the referring server. 593 It could be argued a value limited to attributes actually provided 594 by the referring server could be provided, with the clear 595 understanding that the value on the target will be different. 596 However, there is very little value in doing that since there is 597 only a single accessible object on the referring server for each 598 fs, and the supported attributes are simply those that the server 599 returns together with fs_locations when requested. 601 3.4.2. filehandle attribute 603 Just as a filehandle for the absent fs cannot be provided to the 604 client via GETFH, it should similarly not be provided as the 605 filehandle attribute and for the same reasons. 607 A filehandle provided for an absent filesystem poses the difficulty 608 of possible expiration or remapping. This is an issue when 609 migration happens after the file system has been accessed, but in 610 the pure referral case, this issue can and should be avoided by 611 never allowing the client to see such a filehandle at all. 613 3.4.3. fh_expire_type attribute 615 In the pure referral case, the issue of filehandle expiration type 616 is rendered moot by the lack of visibility of filehandles within 617 the absent filesystem. Providing a value for this attribute would 618 limit the value on the target server to no purpose. 620 By not providing a value, the referring server allows the target 621 server flexibility to choose the value without fear of confusing 622 the client. 624 Note that the same logic applies to such attributes as 625 link_support, symlink_support, case_insensitive, case_preservng, 626 chown_restricted, and homogeneous. The referring server should 627 support any choice of such values on the target and the client 628 should have no interest in the guesses on the server. 630 3.4.4. fileid attribute 632 Specifying a value for the root of the absent filesystem would 633 serve no purpose as the value at the target would have to be 634 definitive and any value at the referring server might conflict 635 with a value at the target server. 637 4. Referral Examples 639 The details of how referrals proceed are implicit in the 640 specification of migration in RFC 3530. However, because the 641 details of handling of this case are so different from those in the 642 cases discussed therein, examples tailored to the referral 643 situation are needed to clarify matters and allow correct and 644 consistent implementations. 646 The examples given in the sections below are somewhat artificial in 647 that an actual client will not typically do a multi-component 648 lookup, but will have cached information regarding the upper levels 649 of the name hierarchy. However, these example are chosen to make 650 the required behavior clear and easy to put within the scope of a 651 small number of requests, without getting unduly into details of 652 how specific clients will choose to cache things. 654 4.1. Referral Example (LOOKUP) 656 Let us suppose that the following COMPOUND is issued in an 657 environment in which /src/linux/2.7/latest is absent from the 658 target server. This may be for a number of reasons. It may be the 659 case that the file system has moved, or, it may be the case that 660 the target server is functioning mainly, or solely, to refer 661 clients to the servers on which various file systems are located. 663 o PUTROOTFH 665 o LOOKUP "src" 667 o LOOKUP "linux" 669 o LOOKUP "2.7" 671 o LOOKUP "latest" 673 o GETFH 675 o GETATTR fsid,fileid,size,ctime 677 Under the given circumstances, the following will be the result. 679 o PUTROOTFH --> NFS_OK 681 Current fh is root of pseudo-fs. 683 o LOOKUP "src" --> NFS_OK 685 Current fh is for /src and is within pseudo-fs. 687 o LOOKUP "linux" --> NFS_OK 689 Current fh is for /src/linux and is within pseudo-fs. 691 o LOOKUP "2.7" --> NFS_OK 693 Current fh is for /src/linux/2.7 and is within pseudo-fs. 695 o LOOKUP "latest" --> NFS_OK 697 Current fh is for /src/linux/2.7/latest and is within a new, 698 absent fs, but ... 700 The client will never see the value of that fh. 702 o GETFH --> NFS4ERR_MOVED 704 Fails because current fh is in an absent fs at the start of 705 the operation and the spec makes no exception for GETFH. 707 o GETATTR fsid,fileid,size,ctime 709 Not executed because the failure of the GETFH stops processing 710 of the COMPOUND. 712 Given the failure of the GETFH, the client has the job of 713 determining the root of the absent file system and where to find 714 that file system, i.e. the server and path relative to that 715 server's root fh. Note here that in this example, the client did 716 not obtain filehandles and attribute information (e.g. fsid) for 717 the intermediate directories, so that he would not be sure where 718 the absent file system starts. It could be the case, for example, 719 that /src/linux/2.7 is the root of the moved filesystem and that 720 the reason that the lookup of "latest" succeeded is that the 721 filesystem was not absent on that op but was moved between the last 722 LOOKUP and the GETFH (since COMPOUND is not atomic). Even if we 723 had the fsid's for all of the intermediate directories, we could 724 have no way of knowing that /src/linux/2.7/latest was the root of a 725 new fs, since we don't yet have its fsid. 727 In order to get the necessary information, let us re-issue the 728 chain of lookup's with GETFH's and GETATTR's to at least get the 729 fsid's so we can be sure where the appropriate fs boundaries are. 730 The client could choose to get fs_locations at the same time but in 731 most cases the client will have a good guess as to where fs 732 boundaries are (because of where NFS4ERR_MOVED was gotten and where 733 not) making fetching of fs_location info unnecessary. 735 o PUTROOTFH --> NFS_OK 737 Current fh is root of pseudo-fs. 739 o GETATTR(fsid) --> NFS_OK 741 Just for completeness. Normally, clients will know the fsid 742 of the pseudo-fs as soon as they establish communication with 743 a server. 745 o LOOKUP "src" --> NFS_OK 746 o GETATTR(fsid) --> NFS_OK 748 Get current fsid to see where fs boundaries are. The fsid 749 will be that for the pseudo-fs in this example, so no 750 boundary. 752 o GETFH --> NFS_OK 754 Current fh is for /src and is within pseudo-fs. 756 o LOOKUP "linux" --> NFS_OK 758 Current fh is for /src/linux and is within pseudo-fs. 760 o GETATTR(fsid) --> NFS_OK 762 Get current fsid to see where fs boundaries are. The fsid 763 will be that for the pseudo-fs in this example, so no 764 boundary. 766 o GETFH --> NFS_OK 768 Current fh is for /src/linux and is within pseudo-fs. 770 o LOOKUP "2.7" --> NFS_OK 772 Current fh is for /src/linux/2.7 and is within pseudo-fs. 774 o GETATTR(fsid) --> NFS_OK 776 Get current fsid to see where fs boundaries are. The fsid 777 will be that for the pseudo-fs in this example, so no 778 boundary. 780 o GETFH --> NFS_OK 782 Current fh is for /src/linux/2.7 and is within pseudo-fs. 784 o LOOKUP "latest" --> NFS_OK 786 Current fh is for /src/linux/2.7/latest and is within a new, 787 absent fs, but ... 789 The client will never see the value of that fh 791 o GETATTR(fsid, fs_locations) --> NFS_OK 792 We are getting the fsid to know where the fs boundaries are. 793 While RFC 3530 does not oblige the server to give us anything 794 but fs_locations, we are assuming that the server is following 795 the implementation herein and providing it. Note that the 796 fsid we are given will not necessarily be preserved at the new 797 location. That fsid might be different and in fact the fsid 798 we have for this fs might a valid fsid of a different fs on 799 that new server. 801 In this particular case, we are pretty sure anyway that what 802 has moved is /src/linux/2.7/latest rather than /src/linux/2.7 803 since we have the fsid of the latter and it is that of the 804 pseudo-fs, which presumably cannot move. However, in other 805 examples, we might not have this kind of information to rely 806 on (e.g. /src/linux/2.7 might be a non-pseudo filesystem 807 separate from /src/linux/2.7/latest), so we need to have 808 another reliable source information on the boundary of the fs 809 which is moved. If, for example, the filesystem "/src/linux" 810 had moved we would have a case of migration rather than 811 referral and once the boundaries of the migrated filesystem 812 was clear we could fetch fs_locations. 814 We are fetching fs_location because the fact that we got an 815 NFS4ERR_MOVED at this point means that it most likely that 816 this is a referral and we need the destination. Even if it is 817 the case that "/src/linux/2.7" is a filesystem which has 818 migrated, we will still need the location information for that 819 file system. 821 o GETFH --> NFS4ERR_MOVED 823 Fails because current fh is in an absent fs at the start of 824 the operation and the spec makes no exception for GETFH. Note 825 that this has the happy consequence that we don't have to 826 worry about the volatility or lack thereof of the fh. If the 827 root of the fs on the new location is a persistent fh, then we 828 can assume that this fh, which we never saw is a persistent 829 fh, which, if we could see it, would exactly match the new fh. 830 At least, there is no evidence to disprove that. On the other 831 hand, if we find a volatile root at the new location, then the 832 filehandle which we never saw must have been volatile or at 833 least nobody can prove otherwise. 835 Given the above, the client knows where the root of the absent file 836 system is, by noting where the change of fsid occurred. The 837 fs_locations attribute also gives the client the actual location of 838 the absent file system, so that the referral can proceed. The 839 server gives the client the bare minimum of information about the 840 absent file system so that there will be very little scope for 841 problems of conflict between information sent by the referring 842 server and information of the file system's home. No filehandles 843 and very few attributes are present on the referring server and the 844 client can treat those it receives as basically transient 845 information with the function of enabling the referral. 847 4.2. Referral Example (READDIR) 849 Another context in which a client may encounter referrals is when 850 it does a READDIR on directory in which some of the sub-directories 851 are the roots of absent file systems. 853 Suppose such a directory is read as follows: 855 o PUTROOTFH 857 o LOOKUP "src" 859 o LOOKUP "linux" 861 o LOOKUP "2.7" 863 o READDIR (fsid, size, ctime, mounted_on_fileid) 865 In this case, because rdattr_error is not requested, fs_locations 866 is not requested, and some of attributes cannot be provided the 867 result will be an NFS4ERR_MOVED error on the READDIR, with the 868 detailed results as follows: 870 o PUTROOTFH --> NFS_OK 872 Current fh is root of pseudo-fs. 874 o LOOKUP "src" --> NFS_OK 876 Current fh is for /src and is within pseudo-fs. 878 o LOOKUP "linux" --> NFS_OK 880 Current fh is for /src/linux and is within pseudo-fs. 882 o LOOKUP "2.7" --> NFS_OK 884 Current fh is for /src/linux/2.7 and is within pseudo-fs. 886 o READDIR (fsid, size, ctime, mounted_on_fileid) --> 887 NFS4ERR_MOVED 888 Note that the same error would have been returned if 889 /src/linux/2.7 had migrated, when in fact it is because the 890 directory contains the root of an absent fs. 892 So now suppose that we reissue with rdattr_error: 894 o PUTROOTFH 896 o LOOKUP "src" 898 o LOOKUP "linux" 900 o LOOKUP "2.7" 902 o READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid) 904 The results will be: 906 o PUTROOTFH --> NFS_OK 908 Current fh is root of pseudo-fs. 910 o LOOKUP "src" --> NFS_OK 912 Current fh is for /src and is within pseudo-fs. 914 o LOOKUP "linux" --> NFS_OK 916 Current fh is for /src/linux and is within pseudo-fs. 918 o LOOKUP "2.7" --> NFS_OK 920 Current fh is for /src/linux/2.7 and is within pseudo-fs. 922 o READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid) 923 --> NFS_OK 925 The attributes for "latest" will only contain rdattr_error 926 with the value will be NFS4ERR_MOVED, together with an fsid 927 value and an a value for mounted_on_fileid. 929 So suppose we do another READDIR to get fs_locations info, although 930 we could have used a GETATTR directly, as in the previous section. 932 o PUTROOTFH 934 o LOOKUP "src" 935 o LOOKUP "linux" 937 o LOOKUP "2.7" 939 o READDIR (rdattr_error, fs_locations, mounted_on_fileid, fsid, 940 size, ctime) 942 The results would be: 944 o PUTROOTFH --> NFS_OK 946 Current fh is root of pseudo-fs. 948 o LOOKUP "src" --> NFS_OK 950 Current fh is for /src and is within pseudo-fs. 952 o LOOKUP "linux" --> NFS_OK 954 Current fh is for /src/linux and is within pseudo-fs. 956 o LOOKUP "2.7" --> NFS_OK 958 Current fh is for /src/linux/2.7 and is within pseudo-fs. 960 o READDIR (rdattr_error, fs_locations, mounted_on_fileid, fsid, 961 size, ctime) --> NFS_OK 963 The attributes for "latest" will only contain 965 + rdattr_error (value: NFS4ERR_MOVED) 967 + fs_locations (value: target:/path/on/target) 969 + mounted_on_fileid (value: unique fileid within referring 970 fs) 972 + fsid (value: unique value within referring server) 974 The attribute entry for "latest" will not contain size or 975 ctime. 977 5. Additional Client-side Considerations 979 When clients make use of servers that implement referrals and 980 migration, care should be taken so that a user who mounts a given 981 filesystem that includes a referral or a relocated filesystem 982 continue to see a coherent picture of that user-side filesystem 983 despite the fact that it contains a number of server-side 984 filesystems which may be on different servers. 986 One important issue is upward navigation from the root of a server- 987 side filesystem to its parent (specified as ".." in UNIX). The 988 client needs to determine when it hits an fsid root going up the 989 filetree. When at such a point, and needs to ascend to the parent, 990 it must do so locally instead of sending a LOOKUPP call to the 991 server. The LOOKUPP would normally return the ancestor of the 992 target filesystem on the target server, which not part of the space 993 that the client mounted. 995 Another issue concerns refresh of referral locations. When 996 referrals are used extensively, they may change as server 997 configurations change. It is expected that clients will cache 998 information related to traversing referrals so that future client 999 side requests are resolved locally without server communication. 1000 This is usually rooted in client-side name lookup caching. Clients 1001 should periodically purge this data for referral points in order to 1002 detect changes in location information. When the change attribute 1003 changes for directories that hold referral entries or for the 1004 referral entires themselves, clients should consider any associated 1005 cached referral information to be out of date. 1007 6. Acknowledgements 1009 The authors wish to thank Neil Brown for his helpful comments with 1010 regard to an earlier draft that has been the source of much of the 1011 material here. 1013 7. Normative References 1015 [RFC3530] 1016 S. Shepler, et. al., "NFS Version 4 Protocol", Standards Track 1017 RFC 1019 Authors' Addresses 1021 David Noveck 1022 Network Appliance, Inc. 1023 375 Totten Pond Road 1024 Waltham, MA 02451 USA 1026 Phone: +1 781 768 5347 1027 EMail: dnoveck@netapp.com 1029 Rodney C. Burnett 1030 IBM, Inc. 1031 13001 Trailwood Rd 1032 Austin, TX 78727 USA 1034 Phone: +1 512 838 8498 1035 EMail: cburnett@us.ibm.com 1037 Full Copyright Statement 1039 Copyright (C) The Internet Society (2005). This document is 1040 subject to the rights, licenses and restrictions contained in BCP 1041 78 and except as set forth therein, the authors retain all their 1042 rights. 1044 This document and the information contained herein are provided on 1045 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 1046 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND 1047 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, 1048 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT 1049 THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR 1050 ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A 1051 PARTICULAR PURPOSE. 1053 Intellectual Property 1055 The IETF takes no position regarding the validity or scope of any 1056 Intellectual Property Rights or other rights that might be claimed 1057 to pertain to the implementation or use of the technology described 1058 in this document or the extent to which any license under such 1059 rights might or might not be available; nor does it represent that 1060 it has made any independent effort to identify any such rights. 1061 Information on the procedures with respect to rights in RFC 1062 documents can be found in BCP 78 and BCP 79. 1064 Copies of IPR disclosures made to the IETF Secretariat and any 1065 assurances of licenses to be made available, or the result of an 1066 attempt made to obtain a general license or permission for the use 1067 of such proprietary rights by implementers or users of this 1068 specification can be obtained from the IETF on-line IPR repository 1069 at http://www.ietf.org/ipr. 1071 The IETF invites any interested party to bring to its attention any 1072 copyrights, patents or patent applications, or other proprietary 1073 rights that may cover technology that may be required to implement 1074 this standard. Please address the information to the IETF at ietf- 1075 ipr@ietf.org. 1077 Acknowledgement 1079 Funding for the RFC Editor function is currently provided by the 1080 Internet Society.