idnits 2.17.1 draft-ietf-nfsv4-referrals-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 18. -- Found old boilerplate from RFC 3978, Section 5.5 on line 1103. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1114. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1121. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1127. ** Found boilerplate matching RFC 3978, Section 5.4, paragraph 1 (on line 1094), which is fine, but *also* found old RFC 2026, Section 10.4C, paragraph 1 text on line 38. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == Line 150 has weird spacing: '...uggests other...' -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (July 2005) is 6860 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC3530' is defined on line 1067, but no explicit reference was found in the text ** Obsolete normative reference: RFC 3530 (Obsoleted by RFC 7530) Summary: 7 errors (**), 0 flaws (~~), 4 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 INTERNET-DRAFT David Noveck 3 Expires: January 2006 Network Appliance, Inc. 4 Rodney C. Burnett 5 IBM, Inc. 7 July 2005 9 Implementation Guide for Referrals in NFSv4 10 draft-ietf-nfsv4-referrals-00.txt 12 Status of this Memo 14 By submitting this Internet-Draft, each author represents 15 that any applicable patent or other IPR claims of which he 16 or she is aware have been or will be disclosed, and any of 17 which he or she becomes aware will be disclosed, in 18 accordance with Section 6 of BCP 79. 20 Internet-Drafts are working documents of the Internet Engineering 21 Task Force (IETF), its areas, and its working groups. Note that 22 other groups may also distribute working documents as Internet- 23 Drafts. 25 Internet-Drafts are draft documents valid for a maximum of six 26 months and may be updated, replaced, or obsoleted by other 27 documents at any time. It is inappropriate to use Internet-Drafts 28 as reference material or to cite them other than as "work in 29 progress." 31 The list of current Internet-Drafts can be accessed at 32 http://www.ietf.org/ietf/1id-abstracts.txt The list of 33 Internet-Draft Shadow Directories can be accessed at 34 http://www.ietf.org/shadow.html. 36 Copyright Notice 38 Copyright (C) The Internet Society (2005). All Rights Reserved. 40 Abstract 42 RFC3530 describes a migration feature using the NFS4ERR_MOVED error 43 code and the fs_locations attribute. The description focuses on 44 the case of migration (i.e. relocation) of a file system already 45 known to the client. The simpler limiting case in a which a file 46 system not previously known to the client was located elsewhere, 47 which we here call a referral, was not clearly described. Because 48 of this and also because of some inconsistencies and ambiguities in 49 the text of RFC3530, there has been confusion about how the client 50 and server should interact in performing a referral. This document 51 provides a guide to the implementation of referrals, and in so 52 doing, addresses the relevant problems in RFC3530. 54 Table Of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2 57 1.1. Referrals as a Limiting Case of Migrations . . . . . . . 3 58 1.2. Pure Referrals . . . . . . . . . . . . . . . . . . . . . 3 59 2. Issues as to When NFS4ERR_MOVED May/Should be Returned . . 4 60 2.1. PUTFH Returning NFS4ERR_MOVED . . . . . . . . . . . . . 5 61 2.2. GETFH Returning NFS4ERR_MOVED . . . . . . . . . . . . . 6 62 2.3. GETATTR Returning NFS4ERR_MOVED . . . . . . . . . . . . 6 63 2.4. Ops Not Allowed to Return NFS4ERR_MOVED . . . . . . . . 7 64 3. Attribute Issues . . . . . . . . . . . . . . . . . . . . . 7 65 3.1. Number of fs_locations . . . . . . . . . . . . . . . . . 7 66 3.2. General Issues of Incomplete Attribute Sets . . . . . . 8 67 3.3. Attributes Needed for Root of an Absent Fs . . . . . . . 11 68 3.4. Attributes Valid for Root of an Absent Fs . . . . . . . 13 69 3.5. Attributes Not Valid for Root of an Absent Fs . . . . . 14 70 4. Referral Examples . . . . . . . . . . . . . . . . . . . . 15 71 4.1. Referral Example (LOOKUP) . . . . . . . . . . . . . . . 15 72 4.2. Referral Example (READDIR) . . . . . . . . . . . . . . . 20 73 5. Additional Client-side Considerations . . . . . . . . . . 22 74 Acknowledgements . . . . . . . . . . . . . . . . . . . . . 23 75 Normative References . . . . . . . . . . . . . . . . . . . 23 76 Authors' Addresses . . . . . . . . . . . . . . . . . . . . 23 77 Full Copyright Statement . . . . . . . . . . . . . . . . . 23 79 1. Introduction 81 RFC3530 describes a migration feature using the NFS4ERR_MOVED error 82 code and the fs_locations attribute. The description focuses on 83 the case of migration of a file system already known to the client, 84 while the alternate case in a which a file system not previously 85 known to the client was located elsewhere, which we here call a 86 referral, was not clearly described. 88 Analysis of how this case would be dealt with has shown that the 89 protocol in RFC3530 is capable of dealing properly with this sub- 90 case. However, there are difficulties for implementers in 91 determining how precisely this is to be done, because the text of 92 RFC3530 does not address this case. In addition, there are a 93 number of ambiguities and inconsistencies in the text, that 94 increase the difficulties for implementers and that raise the 95 probability that clients and servers may not properly interact. 97 This document is an attempt to deal with these spec issues and to 98 provide a clear guide to follow in developing inter-operable 99 referral implementations. 101 This document may deal with some aspects of issues related to the 102 migration case where the data actually relocated from one server to 103 another while under client access. For example, clarification of 104 apparent contradictions in the spec may relate to both referrals 105 and the general migration case. However, the focus of this 106 document is on the case of referrals and other aspects of migration 107 will only be dealt with as necessary to accomplish that main 108 purpose. 110 1.1. Referrals as a Limiting Case of Migrations 112 If a server implements file system migration, individual clients, 113 not being synchronized with the migration event, will encounter 114 different situations from the client point of view. That is, even 115 though some clients will have received filehandles within the 116 migrated filesystem, others may at that time be first referencing 117 the root of the migrated filesystem and need to be redirected to 118 the new fs location, just as those clients that have already 119 accessed the filesystem (when it was present) need to be 120 redirected. 122 Thus referrals are a limiting sub-case of migration and when a 123 migration event occurs some clients will see an ordinary migration 124 in the case in which access (by that client) to the filesystem 125 being moved has already occurred, while others will see a referral 126 when they first attempt to access the absent filesystem. 128 1.2. Pure Referrals 130 Given that clients supporting migration need to be prepared for a 131 migration event, a server can establish a situation, which we refer 132 to as a pure referral, in which all clients see a referral. Since 133 no client can enter the absent filesystem, the presumptive 134 migration of the absent filesystem, assumes a purely conventional 135 character, in that for each client it appears to have happened 136 before all access by that client. This allows the server to 137 establish a situation in which referral to an absent filesystem may 138 occur for filesystems that were never present on the server. 140 This functionality may be used to allow construction of multi- 141 server namespaces and may serve as a building block for the 142 construction of wider global namespaces. 144 2. Issues as to When NFS4ERR_MOVED May/Should be Returned 146 RFC3530 contains the following statement in section 6.2: "The 147 NFS4ERR_MOVED error is returned for all operations except PUTFH and 148 GETATTR.", which is contradicted by the error lists in the detailed 149 operation descriptions. PUTFH and GETATTR, while the statement 150 above suggests otherwise, have NFS4ERR_MOVED listed as error 151 codes, whereas there are six other ops (PUTROOTFH, PUTPUBFH, RENEW, 152 SETCLIENTID, SETCLIENTID_CONFIRM, RELEASE_OWNER) which are not 153 allowed to return NFS4ERR_MOVED despite the "all operations except" 154 clause given above. 156 The individual ops will be discussed below but in order to 157 understand how these contradictions arose and how they are to be 158 dealt with, it is important to be clear on what point the current 159 filesystem (i.e. the filesystem associated with the current 160 filehandle) is to be tested for absence in deciding whether to 161 return NFS4ERR_MOVED. For operations which change the current 162 filehandle, (i.e. PUTFH, PUTROOTFH, PUTPUBFH, RESTOREFH, LOOKUP, 163 LOOKUPP), this distinction is critical. The spec is, however, not 164 very clear on this issue so we need to consider the alternatives. 166 Let's first consider a check on the current filehandle after the 167 operation. In part, that seems dubious because it returns an error 168 for an operation where the circumstances that give rise to the 169 error (i.e. changing the filehandle) depend on the successful 170 execution of the operation giving rise to the error. However, the 171 main reason such an interpretation is insupportable is that it 172 would make some situations, wherein fs_locations is needed, 173 impossible to deal with. For example, LOOKUP followed by 174 GETATTR(fs_locations) would not be possible if the LOOKUP returned 175 NFS4ERR_MOVED when the current filehandle at the start of the 176 LOOKUP was in a present filesystem while the current filehandle at 177 the end of the LOOKUP was at the root of an absent filesystem. An 178 NFS4ERR_MOVED returned by LOOKUP would not allow execution of the 179 GETATTR, any exception for GETATTR of fs_locations notwithstanding. 181 On the other hand, making the check at the start of each operation 182 (except some GETATTR's) allows fs_locations to be determined 183 whenever necessary and is the best way to resolve the issue. 185 Except where RFC3530 has made this impossible by not listing 186 NFS4ERR_MOVED as an error code for certain ops, each op should 187 check at the beginning of the operation for a current filehandle 188 within an absent fs. If so, NFS4ERR_MOVED is returned. A partial 189 exception is GETATTR for which the return of NFS4ERR_MOVED (or not) 190 is dependent on the set of attributes requested. This is discussed 191 further below. 193 2.1. PUTFH Returning NFS4ERR_MOVED 195 As noted above, section 6.2 indicates that PUTFH will not return 196 NFS4ERR_MOVED while the detailed description for PUTFH (section 197 14.2.20), lists NFS4ERR_MOVED as a valid return. How are we to 198 resolve the contradiction? 200 It would seem that the exception for PUTFH in section 6.2, is 201 unnecessary and derives from a confusion about the logic of 202 NFS4ERR_MOVED. Clearly the sequence of PUTFH(fh within absent 203 filesystem) followed by GETATTR(fs_locations) must be allowed but 204 it requires no specific exception for PUTFH to do so, since only 205 GETATTR, and not PUTFH, has a filehandle within an absent 206 filesystem as the current handle at the start of the operation. 207 Thus no exception for PUTFH is required, as is recognized by 208 section 14.2.20. 210 PUTFH will never return NFS4ERR_MOVED because the filehandle it is 211 establishing belongs to an absent filesystem. This is because the 212 current filehandle/filesystem is checked for absence at the start 213 of each op and the new filehandle for PUTFH has not been 214 established at the start of the PUTFH. This allows the sequence 215 PUTFH-GETATTR(fs_locations) to proceed without any specific 216 exception for PUTFH or any analysis of the op stream that 217 encompasses multiple ops (e.g. looking ahead to the GETATTR). 219 NFS4ERR_MOVED can happen as a result of a PUTFH, but only in rather 220 odd circumstances such as the following: 222 o PUTFH (fh within absent filesystem) 224 o PUTFH (fh within present filesystem) 226 In this case, NFS4ERR_MOVED would be returned on the second PUTFH 227 since the current filehandle at the start of that op is within an 228 absent filesystem. It may seem odd that the PUTFH which is 229 establishing a current filehandle within a present filesystem 230 should get an error while the one that established a current 231 filehandle within an absent filesystem does not but this behavior 232 is a consequence of a uniform set of rules for NFS4ERR_MOVED, the 233 basis of which is that the current filehandle is tested at the 234 start of an operation. 236 2.2. GETFH Returning NFS4ERR_MOVED 238 While RFC 3530 does not make any exception for GETFH when the 239 current filehandle is within an absent filesystem, the fact that 240 GETFH is such a passive, purely interrogative operation, may lead 241 readers to wrongly suppose that an NFSERR_MOVED error will not 242 arise in this situation. 244 In fact, GETFH will result in NFS4ERR_MOVED being returned if the 245 current file handle is for an absent filesystem. This is 246 particularly helpful in dealing with pure referral situations since 247 a filehandle that a client does not ever see can pose no 248 difficulties. For example, it is irrelevant whether the target 249 filesystem has persistent or volatile filehandles. An inherently 250 unobserved filehandle poses no expiration difficulties regardless 251 of whether the target filesystem (on the target server) has 252 persistent or volatile filehandles. Servers should never return 253 filehandles within absent filesystems for pure referral cases. 255 2.3. GETATTR Returning NFS4ERR_MOVED 257 As noted above, Section 6.2 indicates that NFS4ERR_MOVED is not 258 returned for a GETATTR operation, but NFS4ERR_MOVED is listed as an 259 error that can be returned by GETATTR (in section 14.2.7). Since 260 the purpose of the exception for GETATTR is to allow fs_locations 261 to be fetched, the best resolution of this contradiction is for 262 NFS4ERR_MOVED to be returned by GETATTR's that ask for some 263 unavailable attribute and that do not interrogate the fs_locations 264 attribute. This maintains the exception which allows GETATTR to be 265 used to get fs_locations information by establishing the rule that 266 GETATTR's which interrogate fs_locations (with or without 267 additional attributes) will not return NFS4ERR_MOVED. Neither will 268 GETATTR's that only request attributes that are available. These 269 considerations are further discussed in section 3. 271 Sometimes clients need a quickly checked indication of whether a 272 filesystem is absent or not. In most cases, because fs_locations 273 is not requested, this indication will be the NFS4ERR_MOVED being 274 returned by the GETATTR. When fs_locations is requested, a 275 convenient way to provide this indication is to request the 276 filehandle attribute. Since this attribute will not be provided 277 for referrals (see section 3.4.2), examining the mask of returned 278 attributes for the presence of the filehandle attribute is quick 279 way to determine whether the directory in question is part of an 280 absent filesystem. 282 2.4. Ops Not Allowed to Return NFS4ERR_MOVED 284 As noted above, despite the fact that section 6.2 suggests 285 otherwise, a number of ops, all of which do not require a current 286 filehandle, are not listed as returning NFS4ERR_MOVED. These ops 287 are PUTROOTFH, PUTPUBFH, RENEW, SETCLIENTID, and RELEASE_LOCKOWNER. 288 Note also that DELEGPURGE, another op which does not require a 289 current filehandle, is listed as able to return NFS4ERR_MOVED. 291 Although this contradiction could be resolved by changing the op 292 description to match 6.2, it does not seem that there is any strong 293 reason to do so. The protocol will function adequately, if these 294 are treated as exceptions, and NFS4ERR_MOVED is returned by the 295 next op allowed to do so, as long as the current filehandle at the 296 start of that op is one within an absent fs. Note that PUTROOTFH 297 and PUTPUBFH could cause a pending absent fs indication not to 298 result in an error code, since they change the current filehandle 299 to be within a new, presumably present, filesystem. However, in 300 this case the client would not have issued any operations prior to 301 the PUTROOTFH or PUTPUBFH that actually operated on the absent fs. 303 3. Attribute Issues 305 RFC3530 anticipates that clients will request additional attributes 306 beyond fs_locations (see section 6.2 of that document) and allows 307 the server to return fs_locations only. The return of other 308 attributes, implicit in the statement "server may return 309 fs_locations only" is not clearly dealt with. While some of these, 310 such as fsid, are important for all forms of migration, this 311 omission is most critical in the case of referrals. There are a 312 number of attributes besides fs_locations that need to be provided 313 at the root of an absent filesystem for server and client to 314 properly collaborate to perform a referral. 316 There are also a large number of other attributes which the server 317 should not provide for absent filesystem since providing them is 318 the province of the referral's target server. Providing possibly 319 conflicting values on the server that is the source of the referral 320 can only contribute to confusion. 322 3.1. Number of fs_locations 324 It should be noted that with referrals, it is valid for an 325 fs_locations response to describe multiple locations if the target 326 of the referral is a replicated filesystem. This allows the client 327 to find one of the replicas when one or more of the supplied 328 locations are unavailable. Once the client establishes contact with 329 a target server, it will then obtain fs_locations as appropriate 330 from the server where the referral target data resides. 332 Although it is valid for fs_locations to specify more than one 333 location in the case of referral, it must specify at least one. 334 This is in contrast to the case in which fs_locations is fetched 335 for a filesystem which is present in order to determine replica 336 locations in the case of failure. In that case, it is valid for a 337 server to return an empty fs_locations attribute, since the client 338 already has a valid location for the server and is only requesting 339 additional locations and when there aren't any, an empty 340 fs_locations gives him exactly the information being sought. When 341 handling the returned attribute, the client should consider the 342 replying server as the implied last entry in the list when it is 343 not present in the returned fs_locations. 345 In the referral case, the client is getting NFS4ERR_MOVED and has 346 no current valid location for the filesystem and thus at least one 347 location is required. In other cases, although a server may 348 include his own location for the filesystem in the fs_locations 349 attribute, he is not required to do so. 351 When the server validly returns an fs_locations which has a null 352 array of locations, it may choose to return an empty (zero 353 component) fs_root since the only purpose of fs_root is to aid in 354 the interpretation of the associated locations in the fs_locations 355 attribute. 357 3.2. General Issues of Incomplete Attribute Sets 359 Migration or referral events naturally create situations in which 360 all of the attributes normally supported on a server are not 361 obtainable. RFC3530 is in places ambivalent and/or apparently 362 self-contradictory on such issues. 364 The first problem concerns the statement in the third paragraph of 365 section 6.2: "If the client requests more attributes than just 366 fs_locations, the server may return fs_locations only. This is to 367 be expected since the server has migrated the filesystem and may 368 not have a method of obtaining additional attribute data." 370 While the above seems quite reasonable, it is seemingly 371 contradicted by the following text from section 14.2.7 in the 372 second paragraph of the DESCRIPTION for GETATTR: "The server must 373 return a value for each attribute that the client requests if the 374 attribute is supported by the server. If the server does not 375 support an attribute or cannot approximate a useful value then it 376 must not return the attribute value and must not set the attribute 377 bit in the result bitmap. The server must return an error if it 378 supports an attribute but cannot obtain its value. In that case no 379 attribute values will be returned." 381 The above is a helpful restriction in that it allows clients to 382 simplify their attribute interpretation code. It allows them to 383 assume that all of the attributes they request are present, often 384 making it possible to get successive attributes at fixed offsets 385 within the data stream. However, it seems to contradict what is 386 said in section 6.2, where it is clearly anticipated, at least when 387 fs_locations is requested, that fewer (often many fewer) attributes 388 will be available than are requested. It could be argued that you 389 could harmonize these two by being creative with the interpretation 390 of the phrase "if the attribute is supported by the server". One 391 could argue that many attributes are not supported by the server 392 for an absent fs even though the text talking about attributes 393 "supported by a server" seems to indicate that this is not allowed 394 to be different for different fs's (which is troublesome in itself 395 as one server might have some filesystems that do support ACLs and 396 some that don't support then, for example). 398 Note however that the following paragraph in the description says, 399 "All servers must support the mandatory attributes as specified in 400 the section 'File Attributes'". That's reasonable enough in 401 general, but for an absent fs it is not reasonable and so section 402 14.2.7 and section 6.2 are contradictory. 404 The text in section 14.2.7 would create great obstacles in the 405 implementation of migration given section 6.2. In order to get the 406 additional attributes mentioned, the client would have to guess at 407 the set to be provided and, if that guess were not valid, get an 408 error, presumably NFS4ERR_MOVED, returned. For the purposes of 409 this document, we will treat the specific statement regarding 410 fs_locations in section 6.2 as controlling. The contradictory text 411 in section 14.2.7 will be treated as an overgeneralization to which 412 an exception, for the case of migration and referrals, must be 413 understood. 415 When fs_locations is requested, then a subset of the requested 416 attributes may be provided by the server without generating the 417 error NFS4ERR_MOVED. The subset returned must always include 418 fs_locations. In addition, if all of the attributes requested can 419 be provided, then NFS4ERR_MOVED will also not be returned but in 420 this case, a subset of the requested attributes may not be 421 provided. The following sections give guidance on the attributes 422 the server should provide for filehandles within an absent 423 filesystem. 425 A related issue concerns attributes in a READDIR. There has been 426 discussion, not yet fully resolved, on the working group list about 427 whether all the attributes specified in the attribute mask for 428 READDIR must be provided for all readdir entries when those 429 attributes are supported by the associated filesystems. It can be 430 argued that that would be implied if one combined the words of 431 section 14.2.7 with text that (even though somewhat more loosely) 432 implies that the attributes in READDIR are to be provided via 433 GETATTR. It has also been argued that the role of READDIR with 434 attributes as a replacement for v3 READDIRPLUS makes it more 435 appropriate to treat the attribute mask parameters as a hint with 436 the server allowed to omit any attributes requested (possibly with 437 some exceptions) whenever inconvenient. In the discussion, there 438 have also been occasional suggestions that an attribute's status 439 (as mandatory or recommended) should have role in deciding whether 440 servers are obliged to provide them when requested for READDIR. 442 Regardless of how this debate is to be resolved in the general 443 case, it is clear in the case of an absent filesystem, if there is 444 a general obligation to provide all requested attributes, that an 445 exception must be made for directory entries that represent the 446 root of an absent fs (a referral). For these entries, the full set 447 of requested attributes may simply not be available, as they 448 likewise would not be available for GETATTR. 450 Further, we will assume in our examples, that for the few 451 attributes necessary to perform the referral (see below for 452 details), the server will always provide these when requested as it 453 should be easy for the server to do. 455 So in this document, we will assume that regardless of the general 456 resolution of this issue, in the case of an absent filesystem some 457 flexibility must be provided, even if it is determined that making 458 the attribute mask for READDIR a hint in general is not justified. 460 So we will assume that for READDIR: 462 o When fs_locations is among the attributes requested, the 463 server may provide a subset of the other requested attributes 464 together with fs_locations for roots of absent fs's, without 465 causing any error for the READDIR as a whole. If rdattr_error 466 is also requested and there are attributes which are not 467 available, then rdattr_error will receive the value 468 NFS4ERR_MOVED. 470 o When fs_locations is not requested, but all of the attributes 471 requested can be provided, then they will be provided and no 472 NFS4ERR_MOVED will be generated. An example would be READDIR's 473 that request mounted_on_fileid either with or without fsid. 475 o When fs_locations is not requested, but rdattr_error is and 476 some attributes requested are not available because of the 477 absence of the filesystem, the server will return 478 NFS4ERR_MOVED for the rdattr_error attribute and, in addition, 479 the requested attributes that are valid for the root of an 480 absent filesystem. 482 o When neither fs_locations nor rdattr_error is requested and 483 there is a directory within an absent fs within the directory 484 being read, if some unavailable attributes are requested, no 485 data will be returned and the READDIR will get an 486 NFS4ERR_MOVED error. (The examples will suppose this but 487 clients will not depend on this behavior and clients should 488 work with a more liberal interpretation if this is decided on. 490 3.3. Attributes Needed for Root of an Absent Fs 492 The following options need to be provided for referrals. While it 493 is true that some of these are not mandatory in NFSv4, and the spec 494 treats providing others as optional in the case of an absent 495 filesystem, they do need to be provided to do a referral as 496 envisioned herein. 498 3.3.1. fsid 500 The fsid attribute allows clients to recognize when fs boundaries 501 have been crossed. This applies also when one crosses into an 502 absent filesystem and servers should provide this information when 503 fs_locations is being requested for a filehandle within an absent 504 filesystem, most typically at the root of that absent filesystem. 506 To avoid misunderstanding, it should be noted that the fsid 507 provided in this case is solely so that the fs boundaries can be 508 properly noted and that the fsid returned will not necessarily be 509 valid after resolution of the referral or migration event. The 510 logic of fsid handling for NFSv4 is that fsid's are only unique 511 within a per-server context. This would seem to be a strong 512 indication that they need not be persistent when file systems are 513 moved from server to server, although RFC 3530 does not 514 specifically address the matter. 516 A key reason for always providing an fsid can be seen by 517 considering client behavior. When a client gets the error 518 NFS4ERR_MOVED, it has a number of key steps to get through. It 519 must determine if it has previously accessed the filesystem on the 520 server that the moved error involves. If so, then it represents a 521 data migration event in the strict sense, rather than a referral. 522 The client must recover state and properly use rootpath data from 523 the old and new server to graft the new server's namespace into the 524 client's local space. If a client gets an NFS4ERR_MOVED error and 525 it cannot get an fsid, then it must use fs_locations, fs_root, and 526 rootpath data to locally determine if it has been accessing the 527 filesystem or if the error applies to an fh in a different. 528 (absent) filesystem. If it uses fs_root and there have been 529 renames along the path to the fh, then the client will not be able 530 to distinguish a migration from a referral. The best solution here 531 is for the client to always get an fsid. All a server has to do is 532 generate an fsid value that cannot represent actual exported data 533 at the server. We want to have a model where referrals can still 534 function even if renames or other events change the path to the 535 referral point. Always providing fsid's improves the usefulness of 536 the NFS namespace since referral and migration events can continue 537 to be handled in the face of namespace management. This also helps 538 to reduce client complexity. 540 3.3.2. mounted_on_fileid 542 The mounted_on_fileid attribute is of particular importance to many 543 clients, in that they need this information to form a proper 544 response to a readdir() call. When a readdir() call is done within 545 UNIX, the d_ino field of each of the entries needs to have a unique 546 value normally derived from the NFSv4 fileid attribute. It is in 547 the case in which a file system boundary is crossed that using the 548 fileid attribute for this purpose, particularly when crossing into 549 an absent fs, will pose problems. Note first that the fileid 550 attribute, since it is within a new fs and thus a new fileid space, 551 need not be unique within the directory. Also, since the fs, at 552 its new location, may arrange things differently, the fileid 553 decided on at the directing server may be overridden at the target 554 server, making it of little value. 556 Neither of these problems arise in the case of mounted_on_fileid 557 since that fileid is in the context of the mounted-on fs and unique 558 within it. Servers need to support and always return valid data 559 for mounted-on-fileid, even for the case of an absent filesystem. 560 The returned data represents a fileid for the entry in the 561 directory that represents the absent fs. It's not the fileid of the 562 root of the absent fs itself. Per the attribute handling rules 563 presented in section 3.1, the server may return mounted-on-fileid 564 as a subset of the requested attributes. The presence of 565 fs_locations and rdattr_error attributes in a READDIR request 566 determines if and how the server sets rdattr_error in the partial 567 return case. 569 Clients are strongly encouraged to always include mounted-on-fileid 570 and rdattr_error in READDIR requests. This increases the potential 571 that the client will receive information needed to satisfy the 572 request and avoid further READDIR requests with a more restricted 573 set of attributes. As a result, client performance is improved and 574 network traffic is reduced. 576 3.3.3. rdattr_error attribute 578 This attribute needs to be provided for READDIR's or else a READDIR 579 of a directory that contains the root of absent fs's cannot be done 580 effectively. In order to avoid NFS4ERR_MOVED on the directory as a 581 whole, providing this error via the rdattr_error attribute allows 582 attributes for the entry in question, which can be provided to the 583 client, as well allowing the READDIR to provide information about 584 other entries within the directory. 586 3.4. Attributes Valid for Root of an Absent Fs 588 Beyond the essential attributes discussed above, there are several 589 other attributes that a server could decide to return for an absent 590 fs. These should pose little potential for inconsistencies or 591 client side problems. They are mentioned below, but clients should 592 not expect servers to return them. 594 3.4.1. type attribute 596 For the root of absent fs, providing the value NF4DIR poses no 597 difficulties as the target will provide the same value at the root 598 of the target filesystem. 600 Such a value isn't really needed as clients can assume that when 601 there is a change of fsid value, the root of new filesystem will be 602 a directory, as it must be. 604 3.4.2. change attribute 606 When a server implements pure referrals for a large number of 607 filesystems, it may be convenient for clients to cache the 608 destination fs_locations and interrogate the change attribute to 609 see if there has been any change in the locations attribute to 610 require a refetch. Therefore, if the server does return a value 611 for the change attribute when fetched for an absent filesystem, the 612 server must update the change attribute when the target of the 613 referral, i.e. the fs_locations value, changes. 615 If the server does provide this information, the client must 616 understand that there is no necessary continuity between the change 617 values on referring server and on the target, as there may not be 618 in the general migration case. Clients are best advised to refetch 619 such values on the target server and assume that it is possible 620 that a change to the object may have occurred before the fetch of 621 the change attribute on the target, and that this fact means that 622 data and attribute caches on he client are best flushed. 624 3.5. Attributes Not Valid for Root of an Absent Fs 626 Attributes not listed in the sections above should not be provided 627 in the case of the root of an absent filesystem (the referral 628 case). The general principle is that such information is 629 appropriately provided at the destination specified by the 630 fs_locations attribute and that specifying a value at the point of 631 referral will either be incorrect or superfluous. 633 We present a few examples of this principle below for particular 634 attributes but the same sort of logic applies to all of the other 635 attributes, to some degree or other. 637 3.5.1. supp_attr attribute 639 Since the referring server has no basis to determine what 640 attributes are supported on the target, it is best not to provide 641 this attribute on the referring server. 643 It could be argued a value limited to attributes actually provided 644 by the referring server could be provided, with the clear 645 understanding that the value on the target will be different. 646 However, there is very little value in doing that since there is 647 only a single accessible object on the referring server for each 648 fs, and the supported attributes are simply those that the server 649 returns together with fs_locations when requested. 651 3.5.2. filehandle attribute 653 Just as a filehandle for the absent fs cannot be provided to the 654 client via GETFH, it should similarly not be provided as the 655 filehandle attribute and for the same reasons. 657 A filehandle provided for an absent filesystem poses the difficulty 658 of possible expiration or remapping. This is an issue when 659 migration happens after the file system has been accessed, but in 660 the pure referral case, this issue can and should be avoided by 661 never allowing the client to see such a filehandle at all. 663 3.5.3. fh_expire_type attribute 665 In the pure referral case, the issue of filehandle expiration type 666 is rendered moot by the lack of visibility of filehandles within 667 the absent filesystem. Providing a value for this attribute would 668 limit the value on the target server to no purpose. 670 By not providing a value, the referring server allows the target 671 server flexibility to choose the value without fear of confusing 672 the client. 674 Note that the same logic applies to such attributes as 675 link_support, symlink_support, case_insensitive, case_preserving, 676 chown_restricted, and homogeneous. The referring server should 677 support any choice of such values on the target and the client 678 should have no interest in guesses on the part of the server. 680 3.5.4. fileid attribute 682 Specifying a value for the root of the absent filesystem would 683 serve no purpose as the value at the target would have to be 684 definitive and any value at the referring server might conflict 685 with a value at the target server. 687 4. Referral Examples 689 The details of how referrals proceed are implicit in the 690 specification of migration in RFC 3530. However, because the 691 details of handling of this case are so different from those in the 692 cases discussed therein, examples tailored to the referral 693 situation are needed to clarify matters and allow correct and 694 consistent implementations. 696 The examples given in the sections below are somewhat artificial in 697 that an actual client will not typically do a multi-component 698 lookup, but will have cached information regarding the upper levels 699 of the name hierarchy. However, these example are chosen to make 700 the required behavior clear and easy to put within the scope of a 701 small number of requests, without getting unduly into details of 702 how specific clients might choose to cache things. 704 4.1. Referral Example (LOOKUP) 706 Let us suppose that the following COMPOUND is issued in an 707 environment in which /src/linux/2.7/latest is absent from the 708 target server. This may be for a number of reasons. It may be the 709 case that the file system has moved, or, it may be the case that 710 the target server is functioning mainly, or solely, to refer 711 clients to the servers on which various file systems are located. 713 o PUTROOTFH 715 o LOOKUP "src" 717 o LOOKUP "linux" 719 o LOOKUP "2.7" 721 o LOOKUP "latest" 723 o GETFH 725 o GETATTR fsid,fileid,size,ctime 727 Under the given circumstances, the following will be the result. 729 o PUTROOTFH --> NFS_OK 731 Current fh is root of pseudo-fs. 733 o LOOKUP "src" --> NFS_OK 735 Current fh is for /src and is within pseudo-fs. 737 o LOOKUP "linux" --> NFS_OK 739 Current fh is for /src/linux and is within pseudo-fs. 741 o LOOKUP "2.7" --> NFS_OK 743 Current fh is for /src/linux/2.7 and is within pseudo-fs. 745 o LOOKUP "latest" --> NFS_OK 747 Current fh is for /src/linux/2.7/latest and is within a new, 748 absent fs, but ... 750 The client will never see the value of that fh. 752 o GETFH --> NFS4ERR_MOVED 754 Fails because current fh is in an absent fs at the start of 755 the operation and the spec makes no exception for GETFH. 757 o GETATTR fsid,fileid,size,ctime 759 Not executed because the failure of the GETFH stops processing 760 of the COMPOUND. 762 Given the failure of the GETFH, the client has the job of 763 determining the root of the absent file system and where to find 764 that file system, i.e. the server and path relative to that 765 server's root fh. Note here that in this example, the client did 766 not obtain filehandles and attribute information (e.g. fsid) for 767 the intermediate directories, so that he would not be sure where 768 the absent file system starts. It could be the case, for example, 769 that /src/linux/2.7 is the root of the moved filesystem and that 770 the reason that the lookup of "latest" succeeded is that the 771 filesystem was not absent on that op but was moved between the last 772 LOOKUP and the GETFH (since COMPOUND is not atomic). Even if we 773 had the fsid's for all of the intermediate directories, we could 774 have no way of knowing that /src/linux/2.7/latest was the root of a 775 new fs, since we don't yet have its fsid. 777 In order to get the necessary information, let us re-issue the 778 chain of lookup's with GETFH's and GETATTR's to at least get the 779 fsid's so we can be sure where the appropriate fs boundaries are. 780 The client could choose to get fs_locations at the same time but in 781 most cases the client will have a good guess as to where fs 782 boundaries are (because of where NFS4ERR_MOVED was gotten and where 783 not) making fetching of fs_location info unnecessary. 785 o PUTROOTFH --> NFS_OK 787 Current fh is root of pseudo-fs. 789 o GETATTR(fsid) --> NFS_OK 791 Just for completeness. Normally, clients will know the fsid 792 of the pseudo-fs as soon as they establish communication with 793 a server. 795 o LOOKUP "src" --> NFS_OK 797 o GETATTR(fsid) --> NFS_OK 799 Get current fsid to see where fs boundaries are. The fsid 800 will be that for the pseudo-fs in this example, so no 801 boundary. 803 o GETFH --> NFS_OK 804 Current fh is for /src and is within pseudo-fs. 806 o LOOKUP "linux" --> NFS_OK 808 Current fh is for /src/linux and is within pseudo-fs. 810 o GETATTR(fsid) --> NFS_OK 812 Get current fsid to see where fs boundaries are. The fsid 813 will be that for the pseudo-fs in this example, so no 814 boundary. 816 o GETFH --> NFS_OK 818 Current fh is for /src/linux and is within pseudo-fs. 820 o LOOKUP "2.7" --> NFS_OK 822 Current fh is for /src/linux/2.7 and is within pseudo-fs. 824 o GETATTR(fsid) --> NFS_OK 826 Get current fsid to see where fs boundaries are. The fsid 827 will be that for the pseudo-fs in this example, so no 828 boundary. 830 o GETFH --> NFS_OK 832 Current fh is for /src/linux/2.7 and is within pseudo-fs. 834 o LOOKUP "latest" --> NFS_OK 836 Current fh is for /src/linux/2.7/latest and is within a new, 837 absent fs, but ... 839 The client will never see the value of that fh 841 o GETATTR(fsid, fs_locations) --> NFS_OK 843 We are getting the fsid to know where the fs boundaries are. 844 While RFC 3530 does not oblige the server to give us anything 845 but fs_locations, we are assuming that the server is following 846 the recommendations herein and providing it. Note that the 847 fsid we are given will not necessarily be preserved at the new 848 location. That fsid might be different and in fact the fsid 849 we have for this fs might a valid fsid of a different fs on 850 that new server. 852 In this particular case, we are pretty sure anyway that what 853 has moved is /src/linux/2.7/latest rather than /src/linux/2.7 854 since we have the fsid of the latter and it is that of the 855 pseudo-fs, which presumably cannot move. However, in other 856 examples, we might not have this kind of information to rely 857 on (e.g. /src/linux/2.7 might be a non-pseudo filesystem 858 separate from /src/linux/2.7/latest), so we need to have 859 another reliable source information on the boundary of the fs 860 which is moved. If, for example, the filesystem "/src/linux" 861 had moved we would have a case of migration rather than 862 referral and once the boundaries of the migrated filesystem 863 was clear we could fetch fs_locations. 865 We are fetching fs_location because the fact that we got an 866 NFS4ERR_MOVED at this point means that it most likely that 867 this is a referral and we need the destination. Even if it is 868 the case that "/src/linux/2.7" is a filesystem which has 869 migrated, we will still need the location information for that 870 file system. 872 o GETFH --> NFS4ERR_MOVED 874 Fails because current fh is in an absent fs at the start of 875 the operation and the spec makes no exception for GETFH. Note 876 that this has the happy consequence that we don't have to 877 worry about the volatility or lack thereof of the fh. If the 878 root of the fs on the new location is a persistent fh, then we 879 can assume that this fh, which we never saw is a persistent 880 fh, which, if we could see it, would exactly match the new fh. 881 At least, there is no evidence to disprove that. On the other 882 hand, if we find a volatile root at the new location, then the 883 filehandle which we never saw must have been volatile or at 884 least nobody can prove otherwise. 886 Given the above, the client knows where the root of the absent file 887 system is, by noting where the change of fsid occurred. The 888 fs_locations attribute also gives the client the actual location of 889 the absent file system, so that the referral can proceed. The 890 server gives the client the bare minimum of information about the 891 absent file system so that there will be very little scope for 892 problems of conflict between information sent by the referring 893 server and information of the file system's home. No filehandles 894 and very few attributes are present on the referring server and the 895 client can treat those it receives as basically transient 896 information with the function of enabling the referral. 898 4.2. Referral Example (READDIR) 900 Another context in which a client may encounter referrals is when 901 it does a READDIR on directory in which some of the sub-directories 902 are the roots of absent file systems. 904 Suppose such a directory is read as follows: 906 o PUTROOTFH 908 o LOOKUP "src" 910 o LOOKUP "linux" 912 o LOOKUP "2.7" 914 o READDIR (fsid, size, ctime, mounted_on_fileid) 916 In this case, because rdattr_error is not requested, fs_locations 917 is not requested, and some of attributes cannot be provided the 918 result will be an NFS4ERR_MOVED error on the READDIR, with the 919 detailed results as follows: 921 o PUTROOTFH --> NFS_OK 923 Current fh is root of pseudo-fs. 925 o LOOKUP "src" --> NFS_OK 927 Current fh is for /src and is within pseudo-fs. 929 o LOOKUP "linux" --> NFS_OK 931 Current fh is for /src/linux and is within pseudo-fs. 933 o LOOKUP "2.7" --> NFS_OK 935 Current fh is for /src/linux/2.7 and is within pseudo-fs. 937 o READDIR (fsid, size, ctime, mounted_on_fileid) --> 938 NFS4ERR_MOVED 940 Note that the same error would have been returned if 941 /src/linux/2.7 had migrated, when in fact it is because the 942 directory contains the root of an absent fs. 944 So now suppose that we reissue with rdattr_error: 946 o PUTROOTFH 948 o LOOKUP "src" 950 o LOOKUP "linux" 952 o LOOKUP "2.7" 954 o READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid) 956 The results will be: 958 o PUTROOTFH --> NFS_OK 960 Current fh is root of pseudo-fs. 962 o LOOKUP "src" --> NFS_OK 964 Current fh is for /src and is within pseudo-fs. 966 o LOOKUP "linux" --> NFS_OK 968 Current fh is for /src/linux and is within pseudo-fs. 970 o LOOKUP "2.7" --> NFS_OK 972 Current fh is for /src/linux/2.7 and is within pseudo-fs. 974 o READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid) 975 --> NFS_OK 977 The attributes for "latest" will only contain rdattr_error 978 with the value will be NFS4ERR_MOVED, together with an fsid 979 value and an a value for mounted_on_fileid. 981 So suppose we do another READDIR to get fs_locations info, although 982 we could have used a GETATTR directly, as in the previous section. 984 o PUTROOTFH 986 o LOOKUP "src" 988 o LOOKUP "linux" 990 o LOOKUP "2.7" 992 o READDIR (rdattr_error, fs_locations, mounted_on_fileid, fsid, 993 size, ctime) 995 The results would be: 997 o PUTROOTFH --> NFS_OK 999 Current fh is root of pseudo-fs. 1001 o LOOKUP "src" --> NFS_OK 1003 Current fh is for /src and is within pseudo-fs. 1005 o LOOKUP "linux" --> NFS_OK 1007 Current fh is for /src/linux and is within pseudo-fs. 1009 o LOOKUP "2.7" --> NFS_OK 1011 Current fh is for /src/linux/2.7 and is within pseudo-fs. 1013 o READDIR (rdattr_error, fs_locations, mounted_on_fileid, fsid, 1014 size, ctime) --> NFS_OK 1016 The attributes for "latest" will only contain 1018 + rdattr_error (value: NFS4ERR_MOVED) 1020 + fs_locations (value: target:/path/on/target) 1022 + mounted_on_fileid (value: unique fileid within referring 1023 fs) 1025 + fsid (value: unique value within referring server) 1027 The attribute entry for "latest" will not contain size or 1028 ctime. 1030 5. Additional Client-side Considerations 1032 When clients make use of servers that implement referrals and 1033 migration, care should be taken so that a user who mounts a given 1034 filesystem that includes a referral or a relocated filesystem 1035 continue to see a coherent picture of that user-side filesystem 1036 despite the fact that it contains a number of server-side 1037 filesystems which may be on different servers. 1039 One important issue is upward navigation from the root of a server- 1040 side filesystem to its parent (specified as ".." in UNIX). The 1041 client needs to determine when it hits an fsid root going up the 1042 filetree. When at such a point, and needs to ascend to the parent, 1043 it must do so locally instead of sending a LOOKUPP call to the 1044 server. The LOOKUPP would normally return the ancestor of the 1045 target filesystem on the target server, which may not be part of 1046 the space that the client mounted. 1048 Another issue concerns refresh of referral locations. When 1049 referrals are used extensively, they may change as server 1050 configurations change. It is expected that clients will cache 1051 information related to traversing referrals so that future client 1052 side requests are resolved locally without server communication. 1053 This is usually rooted in client-side name lookup caching. Clients 1054 should periodically purge this data for referral points in order to 1055 detect changes in location information. When the change attribute 1056 changes for directories that hold referral entries or for the 1057 referral entries themselves, clients should consider any associated 1058 cached referral information to be out of date. 1060 6. Acknowledgements 1062 The authors wish to thank Neil Brown and Ted Anderson for their 1063 helpful comments on various drafts of the material presented here. 1065 7. Normative References 1067 [RFC3530] 1068 S. Shepler, et. al., "NFS Version 4 Protocol", Standards Track 1069 RFC 1071 Authors' Addresses 1073 David Noveck 1074 Network Appliance, Inc. 1075 375 Totten Pond Road 1076 Waltham, MA 02451 USA 1078 Phone: +1 781 768 5347 1079 EMail: dnoveck@netapp.com 1081 Rodney C. Burnett 1082 IBM, Inc. 1083 13001 Trailwood Rd 1084 Austin, TX 78727 USA 1086 Phone: +1 512 838 8498 1087 EMail: cburnett@us.ibm.com 1089 Full Copyright Statement 1091 Copyright (C) The Internet Society (2005). This document is 1092 subject to the rights, licenses and restrictions contained in BCP 1093 78, and except as set forth therein, the authors retain all their 1094 rights. 1096 This document and the information contained herein are provided on 1097 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 1098 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND 1099 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, 1100 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT 1101 THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR 1102 ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A 1103 PARTICULAR PURPOSE. 1105 Intellectual Property 1107 The IETF takes no position regarding the validity or scope of any 1108 Intellectual Property Rights or other rights that might be claimed 1109 to pertain to the implementation or use of the technology described 1110 in this document or the extent to which any license under such 1111 rights might or might not be available; nor does it represent that 1112 it has made any independent effort to identify any such rights. 1113 Information on the procedures with respect to rights in RFC 1114 documents can be found in BCP 78 and BCP 79. 1116 Copies of IPR disclosures made to the IETF Secretariat and any 1117 assurances of licenses to be made available, or the result of an 1118 attempt made to obtain a general license or permission for the use 1119 of such proprietary rights by implementers or users of this 1120 specification can be obtained from the IETF on-line IPR repository 1121 at http://www.ietf.org/ipr. 1123 The IETF invites any interested party to bring to its attention any 1124 copyrights, patents or patent applications, or other proprietary 1125 rights that may cover technology that may be required to implement 1126 this standard. Please address the information to the IETF at ietf- 1127 ipr@ietf.org. 1129 Acknowledgement 1131 Funding for the RFC Editor function is currently provided by the 1132 Internet Society.