NFSv4 Working Group D. Noveck Internet-Draft Network Appliance Expires: December 21, 2006 June 19, 2006 Chapters for Migration, Replication, and Referrals for v4.1 Draft draft-dnoveck-location-chapters-00 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on December 21, 2006. Copyright Notice Copyright (C) The Internet Society (2006). Abstract This includes material proposed for inclusion into the next v4.1 draft. It includes one revised chapter, one rewritten chapter, and a set of suggestions for changes in other part of the v4.1 document. Noveck Expires December 21, 2006 [Page 1] Internet-Draft Location Chapters for 4.1 June 2006 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Single-server Name Space . . . . . . . . . . . . . . . . . . . 5 2.1. Server Exports . . . . . . . . . . . . . . . . . . . . . . 5 2.2. Browsing Exports . . . . . . . . . . . . . . . . . . . . . 5 2.3. Server Pseudo Filesystem . . . . . . . . . . . . . . . . . 6 2.4. Multiple Roots . . . . . . . . . . . . . . . . . . . . . . 6 2.5. Filehandle Volatility . . . . . . . . . . . . . . . . . . 6 2.6. Exported Root . . . . . . . . . . . . . . . . . . . . . . 7 2.7. Mount Point Crossing . . . . . . . . . . . . . . . . . . . 7 2.8. Security Policy and Name Space Presentation . . . . . . . 7 3. Multi-server Name Space . . . . . . . . . . . . . . . . . . . 9 3.1. Location attributes . . . . . . . . . . . . . . . . . . . 9 3.2. File System Presence or Absence . . . . . . . . . . . . . 9 3.3. Getting Attributes for an Absent File System . . . . . . . 10 3.3.1. GETATTR Within an Absent File System . . . . . . . . . 11 3.3.2. READDIR and Absent File Systems . . . . . . . . . . . 12 3.4. Uses of Location Information . . . . . . . . . . . . . . . 12 3.4.1. File System Replication . . . . . . . . . . . . . . . 13 3.4.2. File System Migration . . . . . . . . . . . . . . . . 14 3.4.3. Referrals . . . . . . . . . . . . . . . . . . . . . . 14 3.5. Additional Client-side Considerations . . . . . . . . . . 15 3.6. Effecting File System Transitions . . . . . . . . . . . . 16 3.6.1. Transparent File System Transitions . . . . . . . . . 17 3.6.2. Filehandles and File System Transitions . . . . . . . 18 3.6.3. Fileid's and File System Transitions . . . . . . . . . 19 3.6.4. Fsid's and File System Transitions . . . . . . . . . . 20 3.6.5. The Change Attribute and File System Transitions . . . 20 3.6.6. Lock State and File System Transitions . . . . . . . . 20 3.6.7. Write Verifiers and File System Transitions . . . . . 24 3.7. Effecting File System Referrals . . . . . . . . . . . . . 24 3.7.1. Referral Example (LOOKUP) . . . . . . . . . . . . . . 25 3.7.2. Referral Example (READDIR) . . . . . . . . . . . . . . 28 3.8. The Attribute fs_absent . . . . . . . . . . . . . . . . . 31 3.9. The Attribute fs_locations . . . . . . . . . . . . . . . . 31 3.10. The Attribute fs_locations_info . . . . . . . . . . . . . 33 3.11. The Attribute fs_status . . . . . . . . . . . . . . . . . 42 4. Other Changes . . . . . . . . . . . . . . . . . . . . . . . . 46 5. References . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 48 Intellectual Property and Copyright Statements . . . . . . . . . . 49 Noveck Expires December 21, 2006 [Page 2] Internet-Draft Location Chapters for 4.1 June 2006 1. Introduction This draft consists of mainly of proposed material for the v4.1 draft, to replace the handling of server name space, migration, and replication in the current v4.1 draft, which has been inherited pretty much unchanged from RFC3530 [1]. It is based on the current working group document on ideas published in some previously published but now expired individual and working group drafts (Guide for Referrals in NFSv4, and Next Steps for NFSv4 Migration/Replication. Other than stylistic changes that reflect the difference in purpose between those earlier drafts and the v4.1 spec-to-be, the following changes have been made: o Deleted type as an attribute to be returned in an absent fs (sometimes). Too much additional complexity for no real value. o Added RMDA-capability flag to fs_locations_info as requested by Trond. o Added field telling how-current fs is to fs_status as requested by Craig E. o In light of "we don't know if this is the right set of stuff" comments I'd heard, restructured the fs_locations_info stuff for greater changeability/expandability. Also hope to make this easier to understand without compromising other goals. o Deleted fh-replacement stuff due to lack of interest. Could add back if people really want it. o Deleted VLCACHE bit. Lack of interest. Could come back. o Deleted option to transparently split an fs. Could come back if there is a groundswell of support. o Re-organized some of the continuity information in a way that I thinks adds clarity to this. Got rid of same-fs and now have explicitly server, endpoint, and sharing classes, which is a lot clearer. In doing this, I adopted a straight session orientation, although I have not had time to update this document to fully reflect the working group's decisions to make sessions mandatory. This document proposes the following changes relative to the current draft of the spec for NFSv4 Minor Version 1 (draft-ietf-nfsv4-minorversion1-02.txt [2]): Noveck Expires December 21, 2006 [Page 3] Internet-Draft Location Chapters for 4.1 June 2006 o Deletion of the current chapter 4 (Filesystem Migration and Replication) from that draft. o Replacement of the current chapter 5 (NFS Server Name Space) by the second chapter of this document (Single-server Name Space). o Deletion of section 6.14 (Migration, Replication and State) from the current v4.1 draft. o Addition to the draft of the third chapter of this document, (Multi-server Name Space) following the current chapter 10 of the v4.1 drafts (NFSv4.1 Sessions). This provides the replacement (rewritten to reflect referrals and other changes of the current draft's) chapter 4 and section 6.14. o Minor changes to other areas of the spec as set fourth in the fourth chapter of this document (Other Spec Changes Needed). Noveck Expires December 21, 2006 [Page 4] Internet-Draft Location Chapters for 4.1 June 2006 2. Single-server Name Space This chapter describes the NFSv4 single-server name space. Single- server namespaces may be presented directly to clients, or they may be used as a basis to form larger multi-server namespaces (e.g. site- wide or organization-wide) to be presented to clients, as described in Section 3. 2.1. Server Exports On a UNIX server, the name space describes all the files reachable by pathnames under the root directory or "/". On a Windows NT server the name space constitutes all the files on disks named by mapped disk letters. NFS server administrators rarely make the entire server's filesystem name space available to NFS clients. More often portions of the name space are made available via an "export" feature. In previous versions of the NFS protocol, the root filehandle for each export is obtained through the MOUNT protocol; the client sends a string that identifies the export of name space and the server returns the root filehandle for it. The MOUNT protocol supports an EXPORTS procedure that will enumerate the server's exports. 2.2. Browsing Exports The NFS version 4 protocol provides a root filehandle that clients can use to obtain filehandles for the exports of a particular server, via a series of LOOKUP operations within a COMPOUND, to traverse a path. A common user experience is to use a graphical user interface (perhaps a file "Open" dialog window) to find a file via progressive browsing through a directory tree. The client must be able to move from one export to another export via single-component, progressive LOOKUP operations. This style of browsing is not well supported by the NFS version 2 and 3 protocols. The client expects all LOOKUP operations to remain within a single server filesystem. For example, the device attribute will not change. This prevents a client from taking name space paths that span exports. An automounter on the client can obtain a snapshot of the server's name space using the EXPORTS procedure of the MOUNT protocol. If it understands the server's pathname syntax, it can create an image of the server's name space on the client. The parts of the name space that are not exported by the server are filled in with a "pseudo filesystem" that allows the user to browse from one mounted filesystem to another. There is a drawback to this representation of the server's name space on the client: it is static. If the server Noveck Expires December 21, 2006 [Page 5] Internet-Draft Location Chapters for 4.1 June 2006 administrator adds a new export the client will be unaware of it. 2.3. Server Pseudo Filesystem NFS version 4 servers avoid this name space inconsistency by presenting all the exports for a given server within the framework of a single namespace, for that server. An NFS version 4 client uses LOOKUP and READDIR operations to browse seamlessly from one export to another. Portions of the server name space that are not exported are bridged via a "pseudo filesystem" that provides a view of exported directories only. A pseudo filesystem has a unique fsid and behaves like a normal, read only filesystem. Based on the construction of the server's name space, it is possible that multiple pseudo filesystems may exist. For example, /a pseudo filesystem /a/b real filesystem /a/b/c pseudo filesystem /a/b/c/d real filesystem Each of the pseudo filesystems are considered separate entities and therefore will have its own unique fsid. 2.4. Multiple Roots The DOS and Windows operating environments are sometimes described as having "multiple roots". Filesystems are commonly represented as disk letters. MacOS represents filesystems as top level names. NFS version 4 servers for these platforms can construct a pseudo file system above these root names so that disk letters or volume names are simply directory names in the pseudo root. 2.5. Filehandle Volatility The nature of the server's pseudo filesystem is that it is a logical representation of filesystem(s) available from the server. Therefore, the pseudo filesystem is most likely constructed dynamically when the server is first instantiated. It is expected that the pseudo filesystem may not have an on disk counterpart from which persistent filehandles could be constructed. Even though it is preferable that the server provide persistent filehandles for the pseudo filesystem, the NFS client should expect that pseudo file system filehandles are volatile. This can be confirmed by checking the associated "fh_expire_type" attribute for those filehandles in question. If the filehandles are volatile, the NFS client must be prepared to recover a filehandle value (e.g. with a series of LOOKUP operations) when receiving an error of NFS4ERR_FHEXPIRED. Noveck Expires December 21, 2006 [Page 6] Internet-Draft Location Chapters for 4.1 June 2006 2.6. Exported Root If the server's root filesystem is exported, one might conclude that a pseudo-filesystem is unneeded. This not necessarily so. Assume the following filesystems on a server: / disk1 (exported) /a disk2 (not exported) /a/b disk3 (exported) Because disk2 is not exported, disk3 cannot be reached with simple LOOKUPs. The server must bridge the gap with a pseudo-filesystem. 2.7. Mount Point Crossing The server filesystem environment may be constructed in such a way that one filesystem contains a directory which is 'covered' or mounted upon by a second filesystem. For example: /a/b (filesystem 1) /a/b/c/d (filesystem 2) The pseudo filesystem for this server may be constructed to look like: / (place holder/not exported) /a/b (filesystem 1) /a/b/c/d (filesystem 2) It is the server's responsibility to present the pseudo filesystem that is complete to the client. If the client sends a lookup request for the path "/a/b/c/d", the server's response is the filehandle of the filesystem "/a/b/c/d". In previous versions of the NFS protocol, the server would respond with the filehandle of directory "/a/b/c/d" within the filesystem "/a/b". The NFS client will be able to determine if it crosses a server mount point by a change in the value of the "fsid" attribute. 2.8. Security Policy and Name Space Presentation The application of the server's security policy needs to be carefully considered by the implementor. One may choose to limit the viewability of portions of the pseudo filesystem based on the server's perception of the client's ability to authenticate itself properly. However, with the support of multiple security mechanisms and the ability to negotiate the appropriate use of these mechanisms, the server is unable to properly determine if a client will be able Noveck Expires December 21, 2006 [Page 7] Internet-Draft Location Chapters for 4.1 June 2006 to authenticate itself. If, based on its policies, the server chooses to limit the contents of the pseudo filesystem, the server may effectively hide filesystems from a client that may otherwise have legitimate access. As suggested practice, the server should apply the security policy of a shared resource in the server's namespace to the components of the resource's ancestors. For example: / /a/b /a/b/c The /a/b/c directory is a real filesystem and is the shared resource. The security policy for /a/b/c is Kerberos with integrity. The server should apply the same security policy to /, /a, and /a/b. This allows for the extension of the protection of the server's namespace to the ancestors of the real shared resource. For the case of the use of multiple, disjoint security mechanisms in the server's resources, the security for a particular object in the server's namespace should be the union of all security mechanisms of all direct descendants. Noveck Expires December 21, 2006 [Page 8] Internet-Draft Location Chapters for 4.1 June 2006 3. Multi-server Name Space NFSv4.1 supports attributes that allow a namespace to extend beyond the boundaries of a single server. Use of such multi-server namespaces is optional, and for many purposes, single-server namespace are perfectly acceptable. Use of multi-server namespaces can provide many advantages, however, by separating a file system's logical position in a name space from the (possibly changing) logistical and administrative considerations that result in particular file systems being located on particular servers. 3.1. Location attributes NFSv4 contains recommended attributes that allow file systems on one server to be associated with one or more instances of that file system on other servers. These attributes specify such file systems by specifying a server name (either a DNS name or an IP address) together with the path of that filesystem within that server's single-server name space. The fs_locations_info recommended attribute allows specification of one more file systems locations where the data corresponding to a given file system may be found. This attributes provides to the client, in addition to information about file system locations, extensive information about the various file system choices (e.g. priority for use, writability, currency, etc.) as well as information to help the client efficiently effect as seamless a transition as possible among multiple file system instances, when and if that should be necessary. The fs_locations recommended attribute is inherited from NFSv4.0 and only allows specification of the file system locations where the data corresponding to a given file system may be found. Servers should make this attribute available whenever fs_locations_info is supported, but client use of fs_locations_info is to be preferred. 3.2. File System Presence or Absence A given location in an NFSv4 namespace (typically but not necessarily a multi-server namespace) can have a number of file system locations associated with it (via the fs_locations or fs_locations_info attribute). There may also be an actual current file system at that location, accessible via normal namespace operations (e.g. LOOKUP). In this case there, the file system is said to be "present" at that position in the namespace and clients will typically use it, reserving use of additional locations specified via the location- related attributes to situations in which the principal location is no longer available. Noveck Expires December 21, 2006 [Page 9] Internet-Draft Location Chapters for 4.1 June 2006 When there is no actual filesystem at the namespace location in question, the file system is said to be "absent". An absent file system contains no files or directories other than the root and any reference to it, except to access a small set of attributes useful in determining alternate locations, will result in an error, NFS4ERR_MOVED. Note that if the server ever returns NFS4ERR_MOVED (i.e. file systems may be absent), it MUST support the fs_locations attribute and SHOULD support the fs_locations_info and fs_absent attributes. While the error name suggests that we have a case of a file system which once was present, and has only become absent later, this is only one possibility. A position in the namespace may be permanently absent with the file system(s) designated by the location attributes the only realization. The name NFS4ERR_MOVED reflects an earlier, more limited conception of its function, but this error will be returned whenever the referenced file system is absent, whether it has moved or not. Except in the case of GETATTR-type operations (to be discussed later), when the current filehandle at the start of an operation is within an absent file system, that operation is not performed and the error NFS4ERR_MOVED returned, to indicate that the filesystem is absent on the current server. Because a GETFH cannot succeed, if the current filehandle is within an absent file system, filehandles within an absent filesystem cannot be transferred to the client. When a client does have filehandles within an absent file system, it is the result of obtaining them when the file system was present, and having the file system become absent subsequently. It should be noted that because the check for the current filehandle being within an absent filesystem happens at the start of every operation, operations which change the current filehandle so that it is within an absent filesystem will not result in an error. This allows such combinations as PUTFH-GETATTR and LOOKUP-GETATTR to be used to get attribute information, particularly location attribute information, as discussed below. The recommended file system attribute fs_absent can used to interrogate the present/absent status of a given file system. 3.3. Getting Attributes for an Absent File System When a file system is absent, most attributes are not available, but it is necessary to allow the client access to the small set of attributes that are available, and most particularly those that give Noveck Expires December 21, 2006 [Page 10] Internet-Draft Location Chapters for 4.1 June 2006 information about the correct current locations for this file system, fs_locations and fs_locations_info. 3.3.1. GETATTR Within an Absent File System As mentioned above, an exception is made for GETATTR in that attributes may be obtained for a filehandle within an absent file system. This exception only applies if the attribute mask contains at least one attribute bit that indicates the client is interested in a result regarding an absent file system: fs_locations, fs_locations_info, or fs_absent. If none of these attributes is requested, GETATTR will result in an NFS4ERR_MOVED error. When a GETATTR is done on an absent file system, the set of supported attributes is very limited. Many attributes, including those that are normally mandatory will not be available on an absent file system. In addition to the attributes mentioned above (fs_locations, fs_locations_info, fs_absent), the following attributes SHOULD be available on absent file systems, in the case of recommended attributes at least to the same degree that they are available on present file systems. change: This attribute is useful for absent file systems and can be helpful in summarizing to the client when any of the location- related attributes changes. fsid: This attribute should be provided so that the client can determine file system boundaries, including, in particular, the boundary between present and absent file systems. mounted_on_fileid: For objects at the top of an absent file system this attribute needs to be available. Since the fileid is one which is within the present parent file system, there should be no need to reference the absent file system to provide this information. Other attributes SHOULD NOT be made available for absent file systems, even when it is possible to provide them. The server should not assume that more information is always better and should avoid gratuitously providing additional information. When a GETATTR operation includes a bit mask for one of the attributes fs_locations, fs_locations_info, or absent, but where the bit mask includes attributes which are not supported, GETATTR will not return an error, but will return the mask of the actual attributes supported with the results. Handling of VERIFY/NVERIFY is similar to GETATTR in that if the Noveck Expires December 21, 2006 [Page 11] Internet-Draft Location Chapters for 4.1 June 2006 attribute mask does not include fs_locations, fs_locations_info, or absent, the error NFS4ERR_MOVED will result. It differs in that any appearance in the attribute mask of an attribute not supported for an absent file system (and note that this will include some normally mandatory attributes), will also cause an NFS4ERR_MOVED result. 3.3.2. READDIR and Absent File Systems A READDIR performed when the current filehandle is within an absent file system will result in an NFS4ERR_MOVED error, since, unlike the case of GETATTR, no such exception is made for READDIR. Attributes for an absent file system may be fetched via a READDIR for a directory in a present file system, when that directory contains the root directories of one or more absent filesystems. In this case, the handling is as follows: o If the attribute set requested includes one of the attributes fs_locations, fs_locations_info, or absent, then fetching of attributes proceeds normally and no NFS4ERR_MOVED indication is returned, even when the rdattr_error attribute is requested. o If the attribute set requested does not include one of the attributes fs_locations, fs_locations_info, or fs_absent, then if the rdattr_error attribute is requested, each directory entry for the root of an absent file system, will report NFS4ERR_MOVED as the value of the rdattr_error attribute. o If the attribute set requested does not include any of the attributes fs_locations, fs_locations_info, fs_absent, or rdattr_error then the occurrence of the root of an absent file system within the directory will result in the READDIR failing with an NFSER_MOVED error. o The unavailability of an attribute because of a file system's absence, even one that is ordinarily mandatory, does not result in any error indication. The set of attributes returned for the root directory of the absent filesystem in that case is simply restricted to those actually available. 3.4. Uses of Location Information The location-bearing attributes (fs_locations and fs_locations_info), provide, together with the possibility of absent filesystems, a number of important facilities in providing reliable, manageable, and scalable data access. When a file system is present, these attribute can provide Noveck Expires December 21, 2006 [Page 12] Internet-Draft Location Chapters for 4.1 June 2006 alternative locations, to be used to access the same data, in the event that server failures, communications problems, or other difficulties, make continued access to the current file system impossible or otherwise impractical. Provision of such alternate locations is referred to as "replication" although there are cases in which replicated sets of data are not in fact present, and the replicas are instead different paths to the same data. When a file system is present and becomes absent, clients can be given the opportunity to have continued access to their data, at an alternate location. In this case, a continued attempt to use the data in the now-absent file system will result in an NFSERR_MOVED error and at that point the successor locations (typically only one but multiple choices are possible) can be fetched and used to continue access. Transfer of the file system contents to the new location is referred to as "migration", but it should be kept in mind that there are cases in which this term can be used, like "replication" when there is no actual data migration per se. Where a file system was not previously present, specification of file system location provides a means by which file systems located on one server can be associated with a name space defined by another server, thus allowing a general multi-server namespace facility. Designation of such a location, in place of an absent filesystem, is called "referral". 3.4.1. File System Replication The fs_locations and fs_locations_info attributes provide alternative locations, to be used to access data in place of the current file system. On first access to a filesystem, the client should obtain the value of the set alternate locations by interrogating the fs_locations or fs_locations_info attribute, with the latter being preferred. In the event that server failures, communications problems, or other difficulties, make continued access to the current file system impossible or otherwise impractical, the client can use the alternate locations as a way to get continued access to his data. The alternate locations may be physical replicas of the (typically read-only) file system data, or they may reflect alternate paths to the same server or provide for the use of various form of server clustering in which multiple servers provide alternate ways of accessing the same physical file system. How these different modes of file system transition are represented within the fs_locations and fs_locations_info attributes and how the client deals with file system transition issues will be discussed in detail below. Noveck Expires December 21, 2006 [Page 13] Internet-Draft Location Chapters for 4.1 June 2006 3.4.2. File System Migration When a file system is present and becomes absent, clients can be given the opportunity to have continued access to their data, at an alternate location, as specified by the fs_locations or fs_locations_info attribute. Typically, a client will be accessing the file system in question, get a an NFS4ERR_MOVED error, and then use the fs_locations or fs_locations_info attribute to determine the new location of the data. When fs_locations_info is used, additional information will be available which will define the nature of the client's handling of the transition to a new server. Such migration can be helpful in providing load balancing or general resource reallocation. The protocol does not specify how the filesystem will be moved between servers. It is anticipated that a number of different server-to-server transfer mechanisms might be used with the choice left to the server implementor. The NFSv4.1 protocol specifies the method used to communicate the migration event between client and server. The new location may be an alternate communication path to the same server, or, in the case of various forms of server clustering, another server providing access to the same physical file system. The client's responsibilities in dealing with this transition depend on the specific nature of the new access path and how and whether data was in fact migrated. These issues will be discussed in detail below. Although a single successor location is typical, multiple locations may be provided, together with information that allows priority among the choices to be indicated, via information in the fs_locations_info attribute. Where suitable clustering mechanisms make it possible to provide multiple identical file systems or paths to them, this allows the client the opportunity to deal with any resource or communications issues that might limit data availability. 3.4.3. Referrals Referrals provide a way of placing a file system in a location essentially without respect to its physical location on a given server. This allows a single server of a set of servers to present a multi-server namespace that encompasses filesystems located on multiple servers. Some likely uses of this include establishment of site-wide or organization-wide namespaces, or even knitting such together into a truly global namespace. Referrals occur when a client determines, upon first referencing a position in the current namespace, that it is part of a new file Noveck Expires December 21, 2006 [Page 14] Internet-Draft Location Chapters for 4.1 June 2006 system and that that file system is absent. When this occurs, typically by receiving the error NFS4ERR_MOVED, the actual location or locations of the file system can be determined by fetching the fs_locations or fs_locations_info attribute. Use of multi-server namespaces is enabled by NFSv4 but is not required. The use of multi-server namespaces and their scope will depend on the application used, and system administration preferences. Multi-server namespaces can be established by a single server providing a large set of referrals to all of the included filesystems. Alternatively, a single multi-server namespace may be administratively segmented with separate referral file systems (on separate servers) for each separately-administered section of the name space. Any segment or the top-level referral file system may use replicated referral file systems for higher availability. 3.5. Additional Client-side Considerations When clients make use of servers that implement referrals and migration, care should be taken so that a user who mounts a given filesystem that includes a referral or a relocated filesystem continue to see a coherent picture of that user-side filesystem despite the fact that it contains a number of server-side filesystems which may be on different servers. One important issue is upward navigation from the root of a server- side filesystem to its parent (specified as ".." in UNIX). The client needs to determine when it hits an fsid root going up the filetree. When at such a point, and needs to ascend to the parent, it must do so locally instead of sending a LOOKUPP call to the server. The LOOKUPP would normally return the ancestor of the target filesystem on the target server, which may not be part of the space that the client mounted. Another issue concerns refresh of referral locations. When referrals are used extensively, they may change as server configurations change. It is expected that clients will cache information related to traversing referrals so that future client side requests are resolved locally without server communication. This is usually rooted in client-side name lookup caching. Clients should periodically purge this data for referral points in order to detect changes in location information. When the change attribute changes for directories that hold referral entries or for the referral entries themselves, clients should consider any associated cached referral information to be out of date. Noveck Expires December 21, 2006 [Page 15] Internet-Draft Location Chapters for 4.1 June 2006 3.6. Effecting File System Transitions Transitions between file system instances, whether due to switching between replicas upon server unavailability, or in response to a server-initiated migration event are best dealt with together. Even though the prototypical use cases of replication and migration contain distinctive sets of features, when all possibilities for these operations are considered, the underlying unity of these operations, from the client's point of view is clear, even though for the server pragmatic considerations will normally force different implementation strategies for planned and unplanned transitions. A number of methods are possible for servers to replicate data and to track client state in order to allow clients to transition between file system instances with a minimum of disruption. Such methods vary between those that use inter-server clustering techniques to limit the changes seen by the client, to those that are less aggressive, use more standard methods of replicating data, and impose a greater burden on the client to adapt to the transition. The NFSv4.1 protocol does not impose choices on clients and servers with regard to that spectrum of transition methods. In fact, there are many valid choices, depending on client and application requirements and their interaction with server implementation choices. The NFSv4.1 protocol does define the specific choices that can be made, how these choices are communicated to the client and how the client is to deal with any discontinuities. In the sections below references will be made to various possible server implementation choices as a way of illustrating the transition scenarios that clients may deal with. The intent here is not to define or limit server implementations but rather to illustrate the range of issues that clients may face. In the discussion below, references will be made to a file system having a particular property or of two file systems (typically the source and destination) belonging to a common class of any of several types. Two file systems that belong to such a class share some important aspect of file system behavior that clients may depend upon when present, to easily effect a seamless transition between file system instances. Conversely, where the file systems do not belong to such a common class, the client has to deal with various sorts of implementation discontinuities which may cause performance or other issues in effecting a transition. Where the fs_locations_info attribute is available, such file system classification data will be made directly available to the client. See Section 3.10 for details. When only fs_locations is available, Noveck Expires December 21, 2006 [Page 16] Internet-Draft Location Chapters for 4.1 June 2006 default assumptions with regard to such classifications have to be inferred. See Section 3.9 for details. In cases in which one server is expected to accept opaque values from the client that originated from another server, it is a wise implementation practice for the servers to encode the "opaque" values in network byte order. If this is done, servers acting as replicas or immigrating filesystems will be able to parse values like stateids, directory cookies, filehandles, etc. even if their native byte order is different from that of other servers cooperating in the replication and migration of the filesystem. 3.6.1. Transparent File System Transitions Discussion of transition possibilities will start at the most transparent end of the spectrum of possibilities. When there are multiple paths to a single server, and there are network problems that force another path to be used, or when a path is to be put out of service, a replication or migration event may occur without any real replication or migration. Nevertheless, such events fit within the same general framework in that there is a transition between file system locations, communicated just as other, less transparent transitions are communicated. There are cases of transparent transitions that may happen independent of location information, in that a specific host name, may map to several IP addresses, allowing session trunking to provide alternate paths. In other cases, however multiple addresses may have separate location entries for specific file systems to preferentially direct traffic for those specific file systems to certain server addresses, subject to planned or unplanned, corresponding to a nominal replication or migrations event. The specific details of the transition depend on file system equivalence class information (as provided by the fs_locations_info and fs_locations attributes). o Where the old and new filesystems belong to the same _endpoint_ class, the transition consists of creating a new connection which is associated with the existing session to the old server endpoint. Where a connection cannot be associated with the existing session, the target server must be able to recognize the sessionid as invalid and force creation on a new session or a new client id. o Where the old and new filesystems do not belong to the same _endpoint_ classes, but to the same _server_ class, the transition consists of creating a new session, associated with the existing Noveck Expires December 21, 2006 [Page 17] Internet-Draft Location Chapters for 4.1 June 2006 clientid. Where the clientid is stale, the stale, the target server must be able to recognize the clientid as no longer valid and force creation of a new clientid. In either of the above cases, the file system may be shown as belonging to the same _sharing_ class, class allowing the alternate session or connection to be established in advance and used either to accelerate the file system transition when necessary (avoiding connection latency), or to provide higher performance by actively using multiple paths simultaneously. When two file systems belong to the same _endpoint_ class, or _sharing_ class, many transition issues are eliminated, and any information indicating otherwise is ignored as erroneous. In all such transparent transition cases, the following apply: o File handles stay the same if persistent and if volatile are only subject to expiration, if they would be in the absence of file system transition. o Fileid values do not change across the transition. o The file system will have the same fsid in both the old and new the old and new locations. o Change attribute values are consistent across the transition and do not have to be refetched. When change attributes indicate that a cached object is still valid, it can remain cached. o Session, client, and state identifier retain their validity across the transition, except where their staleness is recognized and reported by the new server. Except where such staleness requires it, no lock reclamation is needed. o Write verifiers are presumed to retain their validity and can be presented to COMMIT, with the expectation that if COMMIT on the new server accept them as valid, then that server has all of the data unstably written to the original server and has committed it to stable storage as requested. 3.6.2. Filehandles and File System Transitions There are a number of ways in which filehandles can be handled across a file system transition. These can be divided into two broad classes depending upon whether the two file systems across which the transition happens share sufficient state to effect some sort of continuity of filesystem handling. Noveck Expires December 21, 2006 [Page 18] Internet-Draft Location Chapters for 4.1 June 2006 When there is no such co-operation in filehandle assignment, the two file systems are reported as being in different _handle_ classes. In this case, all filehandles are assumed to expire as part of the file system transition. Note that this behavior does not depend on fh_expire_type attribute and supersedes the specification of FH4_VOL_MIGRATION bit, which only affects behavior when fs_locations_info is not available. When there is co-operation in filehandle assignment, the two file systems are reported as being in the same _handle_ classes. In this case, persistent filehandle remain valid after the file system transition, while volatile filehandles (excluding those while are only volatile due to the FH4_VOL_MIGRATION bit) are subject to expiration on the target server. 3.6.3. Fileid's and File System Transitions In NFSv4.0, the issue of continuity of fileid's in the event of a file system transition was not addressed. The general expectation had been that in situations in which the two filesystem instances are created by a single vendor using some sort of filesystem image copy, fileid's will be consistent across the transition while in the analogous multi-vendor transitions they will not. This poses difficulties, especially for the client without special knowledge of the of the transition mechanisms adopted by the server. It is important to note that while clients themselves may have no trouble with a fileid changing as a result of a file system transition event, applications do typically have access to the fileid (e.g. via stat), and the result of this is that an application may work perfectly well if there is no filesystem instance transition or if any such transition is among instances created by a single vendor, yet be unable to deal with the situation in which a multi-vendor transition occurs, at the wrong time. Providing the same fileid's in a multi-vendor (multiple server vendors) environment has generally been held to be quite difficult. While there is work to be done, it needs to be pointed out that this difficulty is partly self-imposed. Servers have typically identified fileid with inode number, i.e. with a quantity used to find the file in question. This identification poses special difficulties for migration of an fs between vendors where assigning the same index to a given file may not be possible. Note here that a fileid does not require that it be useful to find the file in question, only that it is unique within the given fs. Servers prepared to accept a fileid as a single piece of metadata and store it apart from the value used to index the file information can relatively easily maintain a fileid value across a migration event, allowing a truly transparent Noveck Expires December 21, 2006 [Page 19] Internet-Draft Location Chapters for 4.1 June 2006 migration event. In any case, where servers can provide continuity of fileids, they should and the client should be able to find out that such continuity is available, and take appropriate action. Information about the continuity (or lack thereof) of fileid's across a file system is represented by specifying whether the file systems in question are of the same _fileid_ class. 3.6.4. Fsid's and File System Transitions Since fsid's are only unique within a per-server basis, it is to be expected that they will change during a file system transition. Clients should not make the fsid's received from the server visible to application since they may not be globally unique, and because they may change during a file system transition event. Applications are best served if they are isolated from such transitions to the extent possible. 3.6.5. The Change Attribute and File System Transitions Since the change attribute is defined as a server-specific one, change attributes fetched from one server are normally presumed to be invalid on another server. Such a presumption is troublesome since it would invalidate all cached change attributes, requiring refetching. Even more disruptive, the absence of any assured continuity for the change attribute means that even if the same value is gotten on refetch no conclusions can drawn as to whether the object in question has changed. The identical change attribute could be merely an artifact, of a modified file with a different change attribute construction algorithm, with that new algorithm just happening to result in an identical change value. When the two file systems have consistent change attribute formats, and this fact is communicated to the client by reporting as in the same _change_ class, the client may assume a continuity of change attribute construction and handle this situation just as it would be handled without any filesystem transition. 3.6.6. Lock State and File System Transitions In a file system transition, the two file systems may have co- operated in state management. When this is the case, and the two file systems belong to the same _state_ class, the two file systems will have compatible state environments. In the case of migration, the servers involved in the migration of a filesystem SHOULD transfer all server state from the original to the new server. When this done, it must be done in a way that is transparent to the client. Noveck Expires December 21, 2006 [Page 20] Internet-Draft Location Chapters for 4.1 June 2006 With replication, such a degree of common state is typically not the case. Clients, however should use the information provided by the fs_locations_info attribute to determine whether such sharing is in effect when this is available, and only if that attribute is not available depend on these defaults. This state transfer will reduce disruption to the client when a file system transition If the servers are successful in transferring all state, the client will continue to use stateids assigned by the original server. Therefore the new server must recognize these stateids as valid. This holds true for the clientid as well. Since responsibility for an entire filesystem is transferred is with such an event, there is no possibility that conflicts will arise on the new server as a result of the transfer of locks. As part of the transfer of information between servers, leases would be transferred as well. The leases being transferred to the new server will typically have a different expiration time from those for the same client, previously on the old server. To maintain the property that all leases on a given server for a given client expire at the same time, the server should advance the expiration time to the later of the leases being transferred or the leases already present. This allows the client to maintain lease renewal of both classes without special effort. When the two servers belong to the same _state_ class, it does not necessarily mean that when dealing with the transition, the client will not have to reclaim state. However it does mean that the client may proceed using his current clientid and stateid's just as if there had been no file system transition event and only reclaim state when an NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID error is received. File systems co-operating in state management may actually share state or simply divide the id space so as to recognize (and reject as stale) each others state and clients id's. Servers which do share state may not do under all conditions or all times. The requirement for the server is that if it cannot be sure in accepting an id that it reflects the locks the client was given, it must treat all associated state as stale and report it as such to the client. When two file systems belong to different _state_ classes, the client must establish a new state on the destination, and reclaim if possible. In this case, old stateids and clientid's should not be presented to the new server since there is no assurance that they will not conflict with id's valid on that server. In either case, when actual locks are not known to be maintained, the destination server may establish a grace period specific to the given Noveck Expires December 21, 2006 [Page 21] Internet-Draft Location Chapters for 4.1 June 2006 file system, with non-reclaim locks being rejected for that file system, even though normal locks are being granted for other file systems. Clients should not infer the absence of a grace period for file systems being transitioned to a server from responses to requests for other file systems. In the case of lock reclamation for a given file system after a file system transition, edge conditions can arise similar to those for reclaim after server reboot (although in the case of the planned state transfer associated with migration, these can be avoided by securely recording lock state as part of state migration. Where the destination server cannot guarantee that locks will not be incorrectly granted, the destination server should not establish a file-system-specific grace period. In place of a file-system-specific version of RECLAIM_COMPLETE, servers may assume that an attempt to obtain a new lock, other than be reclaim, indicate the end of the client's attempt to reclaim locks for that file system. [NOTE: The alternative would be to adapt RECLAIM_COMPLETE to this task]. Information about client identity that may be propagated between servers in the form of nfs_client_id4 and associated verifiers, under the assumption that the client presents the same values to all the servers with which it deals. [NOTE: This contradicts what is currently said about SETCLIENTID, and interacts with the issue of what sessions should do about this.] Servers are encouraged to provide facilities to allow locks to be reclaimed on the new server after a file system transition. Often, however, in cases in which the two file systems are not of the same _state _ class, such facilities may not be available and client should be prepared to re-obtain locks, even though it is possible that the client may have his LOCK or OPEN request denied due to a conflicting lock. In some environments, such as the transition between read-only file systems, such denial of locks should not pose large difficulties in practice. When an attempt to re-establish a lock on a new server is denied, the client should treat the situation as if his original lock had been revoked. In all cases in which the lock is granted, the client cannot assume that no conflicting could have been granted in the interim. Where change attribute continuity is present, the client may check the change attribute to check for unwanted file modifications. Where even this is not available, and the file system is not read-only a client may reasonably treat all pending locks as having been revoked. Noveck Expires December 21, 2006 [Page 22] Internet-Draft Location Chapters for 4.1 June 2006 3.6.6.1. Leases and File System Transitions In the case of lease renewal, the client may not be submitting requests for a filesystem that has been transferred to another server. This can occur because of the lease renewal mechanism. The client renews leases for all filesystems when submitting a request to any one filesystem at the server. In order for the client to schedule renewal of leases that may have been relocated to the new server, the client must find out about lease relocation before those leases expire. To accomplish this, all operations which renew leases for a client (i.e. OPEN, CLOSE, READ, WRITE, RENEW, LOCK, LOCKT, LOCKU), will return the error NFS4ERR_LEASE_MOVED if responsibility for any of the leases to be renewed has been transferred to a new server. This condition will continue until the client receives an NFS4ERR_MOVED error and the server receives the subsequent GETATTR for the fs_locations or fs_locations_info attribute for an access to each filesystem for which a lease has been moved to a new server. [ISSUE: There is a conflict between this and the idea in the sessions text that we can have every op in the session implicitly renew the lease. This needs to be dealt with. D. Noveck will create an issue in the issue tracker.] When a client receives an NFS4ERR_LEASE_MOVED error, it should perform an operation on each filesystem associated with the server in question. When the client receives an NFS4ERR_MOVED error, the client can follow the normal process to obtain the new server information (through the fs_locations and fs_locations_info attributes) and perform renewal of those leases on the new server, unless information in fs_locations_info attribute shows that no state could have been transferred. If the server has not had state transferred to it transparently, the client will receive either NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID from the new server, as described above, and the client can then recover state information as it does in the event of server failure. 3.6.6.2. Transitions and the Lease_time Attribute In order that the client may appropriately manage its leases in the case of a file system transition, the destination server must establish proper values for the lease_time attribute. When state is transferred transparently, that state should include the correct value of the lease_time attribute. The lease_time attribute on the destination server must never be less than that on the source since this would result in premature expiration of leases Noveck Expires December 21, 2006 [Page 23] Internet-Draft Location Chapters for 4.1 June 2006 granted by the source server. Upon transitions in which state is transferred transparently, the client is under no obligation to re- fetch the lease_time attribute and may continue to use the value previously fetched (on the source server). If state has not been transferred transparently, either because the file systems are show as being in different state classes or because the client sees a real or simulated server reboot), the client should fetch the value of lease_time on the new (i.e. destination) server, and use it for subsequent locking requests. However the server must respect a grace period at least as long as the lease_time on the source server, in order to ensure that clients have ample time to reclaim their lock before potentially conflicting non-reclaimed locks are granted. 3.6.7. Write Verifiers and File System Transitions In a file system transition, the two file systems may be clustered in the handling of unstably written data. When this is the case, and the two file systems belong to the same _verifier_ class, valid verifiers from one system may be recognized by the other and superfluous writes avoided. There is no requirement that all valid verifiers be recognized, but it cannot be the case that a verifier is recognized as valid when it is not. [NOTE: We need to resolve the issue of proper verifier scope]. When two file systems belong to different _verifier_ classes, the client must assume that all unstable writes in existence at the time file system transition, have been lost since there is no way the old verifier can recognized as valid (or not) on the target server. 3.7. Effecting File System Referrals Referrals are effected when an absent file system is encountered, and one or more alternate locations are made available by the fs_locations or fs_locations_info attributes. The client will typically get an NFS4ERR_MOVED error, fetch the appropriate location information and proceed to access the file system on different server, even though it retains its logical position within the original namespace. The examples given in the sections below are somewhat artificial in that an actual client will not typically do a multi-component lookup, but will have cached information regarding the upper levels of the name hierarchy. However, these example are chosen to make the required behavior clear and easy to put within the scope of a small number of requests, without getting unduly into details of how specific clients might choose to cache things. Noveck Expires December 21, 2006 [Page 24] Internet-Draft Location Chapters for 4.1 June 2006 3.7.1. Referral Example (LOOKUP) Let us suppose that the following COMPOUND is issued in an environment in which /src/linux/2.7/latest is absent from the target server. This may be for a number of reasons. It may be the case that the file system has moved, or, it may be the case that the target server is functioning mainly, or solely, to refer clients to the servers on which various file systems are located. o PUTROOTFH o LOOKUP "src" o LOOKUP "linux" o LOOKUP "2.7" o LOOKUP "latest" o GETFH o GETATTR fsid,fileid,size,ctime Under the given circumstances, the following will be the result. o PUTROOTFH --> NFS_OK. The current fh is now the root of the pseudo-fs. o LOOKUP "src" --> NFS_OK. The current fh is for /src and is within the pseudo-fs. o LOOKUP "linux" --> NFS_OK. The current fh is for /src/linux and is within the pseudo-fs. o LOOKUP "2.7" --> NFS_OK. The current fh is for /src/linux/2.7 and is within the pseudo-fs. o LOOKUP "latest" --> NFS_OK. The current fh is for /src/linux/2.7/ latest and is within a new, absent fs, but ... the client will never see the value of that fh. o GETFH --> NFS4ERR_MOVED. Fails because current fh is in an absent fs at the start of the operation and the spec makes no exception for GETFH. o GETATTR fsid,fileid,size,ctime. Not executed because the failure of the GETFH stops processing of the COMPOUND. Noveck Expires December 21, 2006 [Page 25] Internet-Draft Location Chapters for 4.1 June 2006 Given the failure of the GETFH, the client has the job of determining the root of the absent file system and where to find that file system, i.e. the server and path relative to that server's root fh. Note here that in this example, the client did not obtain filehandles and attribute information (e.g. fsid) for the intermediate directories, so that he would not be sure where the absent file system starts. It could be the case, for example, that /src/linux/2.7 is the root of the moved filesystem and that the reason that the lookup of "latest" succeeded is that the filesystem was not absent on that op but was moved between the last LOOKUP and the GETFH (since COMPOUND is not atomic). Even if we had the fsid's for all of the intermediate directories, we could have no way of knowing that /src/linux/2.7/latest was the root of a new fs, since we don't yet have its fsid. In order to get the necessary information, let us re-issue the chain of lookup's with GETFH's and GETATTR's to at least get the fsid's so we can be sure where the appropriate fs boundaries are. The client could choose to get fs_locations_info at the same time but in most cases the client will have a good guess as to where fs boundaries are (because of where NFS4ERR_MOVED was gotten and where not) making fetching of fs_locations_info unnecessary. OP01: PUTROOTFH --> NFS_OK - Current fh is root of pseudo-fs. OP02: GETATTR(fsid) --> NFS_OK - Just for completeness. Normally, clients will know the fsid of the pseudo-fs as soon as they establish communication with a server. OP03: LOOKUP "src" --> NFS_OK OP04: GETATTR(fsid) --> NFS_OK - Get current fsid to see where fs boundaries are. The fsid will be that for the pseudo-fs in this example, so no boundary. OP05: GETFH --> NFS_OK - Current fh is for /src and is within pseudo-fs. OP06: LOOKUP "linux" --> NFS_OK Noveck Expires December 21, 2006 [Page 26] Internet-Draft Location Chapters for 4.1 June 2006 - Current fh is for /src/linux and is within pseudo-fs. OP07: GETATTR(fsid) --> NFS_OK - Get current fsid to see where fs boundaries are. The fsid will be that for the pseudo-fs in this example, so no boundary. OP08: GETFH --> NFS_OK - Current fh is for /src/linux and is within pseudo-fs. OP09: LOOKUP "2.7" --> NFS_OK - Current fh is for /src/linux/2.7 and is within pseudo-fs. OP10: GETATTR(fsid) --> NFS_OK - Get current fsid to see where fs boundaries are. The fsid will be that for the pseudo-fs in this example, so no boundary. OP11: GETFH --> NFS_OK - Current fh is for /src/linux/2.7 and is within pseudo-fs. OP12: LOOKUP "latest" --> NFS_OK - Current fh is for /src/linux/2.7/latest and is within a new, absent fs, but ... - The client will never see the value of that fh OP13: GETATTR(fsid, fs_locations_info) --> NFS_OK - We are getting the fsid to know where the fs boundaries are. Note that the fsid we are given will not necessarily be preserved at the new location. That fsid might be different and in fact the fsid we have for this fs might a valid fsid of a different fs on that new server. - In this particular case, we are pretty sure anyway that what has moved is /src/linux/2.7/latest rather than /src/linux/2.7 since we have the fsid of the latter and it is that of the pseudo-fs, which presumably cannot move. However, in other examples, we might not have this kind of information to rely on (e.g. /src/linux/2.7 might be a non-pseudo filesystem separate from /src/linux/2.7/ latest), so we need to have another reliable source information on the boundary of the fs which is moved. If, for example, the filesystem "/src/linux" had moved we would have a case of Noveck Expires December 21, 2006 [Page 27] Internet-Draft Location Chapters for 4.1 June 2006 migration rather than referral and once the boundaries of the migrated filesystem was clear we could fetch fs_locations_info. - We are fetching fs_locations_info because the fact that we got an NFS4ERR_MOVED at this point means that it most likely that this is a referral and we need the destination. Even if it is the case that "/src/linux/2.7" is a filesystem which has migrated, we will still need the location information for that file system. OP14: GETFH --> NFS4ERR_MOVED - Fails because current fh is in an absent fs at the start of the operation and the spec makes no exception for GETFH. Note that this has the happy consequence that we don't have to worry about the volatility or lack thereof of the fh. If the root of the fs on the new location is a persistent fh, then we can assume that this fh, which we never saw is a persistent fh, which, if we could see it, would exactly match the new fh. At least, there is no evidence to disprove that. On the other hand, if we find a volatile root at the new location, then the filehandle which we never saw must have been volatile or at least nobody can prove otherwise. Given the above, the client knows where the root of the absent file system is, by noting where the change of fsid occurred. The fs_locations_info attribute also gives the client the actual location of the absent file system, so that the referral can proceed. The server gives the client the bare minimum of information about the absent file system so that there will be very little scope for problems of conflict between information sent by the referring server and information of the file system's home. No filehandles and very few attributes are present on the referring server and the client can treat those it receives as basically transient information with the function of enabling the referral. 3.7.2. Referral Example (READDIR) Another context in which a client may encounter referrals is when it does a READDIR on directory in which some of the sub-directories are the roots of absent file systems. Suppose such a directory is read as follows: o PUTROOTFH o LOOKUP "src" Noveck Expires December 21, 2006 [Page 28] Internet-Draft Location Chapters for 4.1 June 2006 o LOOKUP "linux" o LOOKUP "2.7" o READDIR (fsid, size, ctime, mounted_on_fileid) In this case, because rdattr_error is not requested, fs_locations_info is not requested, and some of attributes cannot be provided the result will be an NFS4ERR_MOVED error on the READDIR, with the detailed results as follows: o PUTROOTFH --> NFS_OK. The current fh is at the root of the pseudo-fs. o LOOKUP "src" --> NFS_OK. The current fh is for /src and is within the pseudo-fs. o LOOKUP "linux" --> NFS_OK. The current fh is for /src/linux and is within the pseudo-fs. o LOOKUP "2.7" --> NFS_OK. The current fh is for /src/linux/2.7 and is within the pseudo-fs. o READDIR (fsid, size, ctime, mounted_on_fileid) --> NFS4ERR_MOVED. Note that the same error would have been returned if /src/linux/2.7 had migrated, when in fact it is because the directory contains the root of an absent fs. So now suppose that we reissue with rdattr_error: o PUTROOTFH o LOOKUP "src" o LOOKUP "linux" o LOOKUP "2.7" o READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid) The results will be: o PUTROOTFH --> NFS_OK. The current fh is at the root of the pseudo-fs. o LOOKUP "src" --> NFS_OK. The current fh is for /src and is within the pseudo-fs. Noveck Expires December 21, 2006 [Page 29] Internet-Draft Location Chapters for 4.1 June 2006 o LOOKUP "linux" --> NFS_OK. The current fh is for /src/linux and is within the pseudo-fs. o LOOKUP "2.7" --> NFS_OK. The current fh is for /src/linux/2.7 and is within the pseudo-fs. o READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid) --> NFS_OK. The attributes for "latest" will only contain rdattr_error with the value will be NFS4ERR_MOVED, together with an fsid value and an a value for mounted_on_fileid. So suppose we do another READDIR to get fs_locations_info, although we could have used a GETATTR directly, as in the previous section. o PUTROOTFH o LOOKUP "src" o LOOKUP "linux" o LOOKUP "2.7" o READDIR (rdattr_error, fs_locations_info, mounted_on_fileid, fsid, size, ctime) The results would be: o PUTROOTFH --> NFS_OK. The current fh is at the root of the pseudo-fs. o LOOKUP "src" --> NFS_OK. The current fh is for /src and is within the pseudo-fs. o LOOKUP "linux" --> NFS_OK. The current fh is for /src/linux and is within the pseudo-fs. o LOOKUP "2.7" --> NFS_OK. The current fh is for /src/linux/2.7 and is within the pseudo-fs. o READDIR (rdattr_error, fs_locations_info, mounted_on_fileid, fsid, size, ctime) --> NFS_OK. The attributes will be as shown below. The attributes for "latest" will only contain o rdattr_error (value: NFS4ERR_MOVED) o fs_locations_info ) Noveck Expires December 21, 2006 [Page 30] Internet-Draft Location Chapters for 4.1 June 2006 o mounted_on_fileid (value: unique fileid within referring fs) o fsid (value: unique value within referring server) The attribute entry for "latest" will not contain size or ctime. 3.8. The Attribute fs_absent In order to provide the client information about whether the current file system is present or absent, the fs_absent attribute may be interrogated. As noted above, this attribute, when supported, may be requested of absent filesystems without causing NFS4ERR_MOVED to be returned and it should always be available. Servers are strongly urged to support this attribute on all filesystems if they support it on any filesystem. 3.9. The Attribute fs_locations The fs_locations attribute is structured in the following way: struct fs_location { utf8str_cis server<>; pathname4 rootpath; }; struct fs_locations { pathname4 fs_root; fs_location locations<>; }; The fs_location struct is used to represent the location of a filesystem by providing a server name and the path to the root of the file system within that server's namespace. When a set of servers have corresponding file systems at the same path within their namespaces, an array of server names may be provided. An entry in the server array is an UTF8 string and represents one of a traditional DNS host name, IPv4 address, or IPv6 address. It is not a requirement that all servers that share the same rootpath be listed in one fs_location struct. The array of server names is provided for convenience. Servers that share the same rootpath may also be listed in separate fs_location entries in the fs_locations attribute. The fs_locations struct and attribute contains an array of such locations. Since the name space of each server may be constructed differently, the "fs_root" field is provided. The path represented by fs_root represents the location of the filesystem in the current Noveck Expires December 21, 2006 [Page 31] Internet-Draft Location Chapters for 4.1 June 2006 server's name space, i.e. that of the server from which the fs_locations attribute was obtained. The fs_root path is meant to aid the client by clearly referencing the root of the file system whose locations are being reported, no matter what object within the current file system, the current filehandle designates. As an example, suppose there is a replicated filesystem located at two servers (servA and servB). At servA, the filesystem is located at path "/a/b/c". At, servB the filesystem is located at path "/x/y/z". If the client were to obtain the fs_locations value for the directory at "/a/b/c/d", it might not necessarily know that the filesystem's root is located in servA's name space at "/a/b/c". When the client switches to servB, it will need to determine that the directory it first referenced at servA is now represented by the path "/x/y/z/d" on servB. To facilitate this, the fs_locations attribute provided by servA would have a fs_root value of "/a/b/c" and two entries in fs_locations. One entry in fs_locations will be for itself (servA) and the other will be for servB with a path of "/x/y/z". With this information, the client is able to substitute "/x/y/z" for the "/a/b/c" at the beginning of its access path and construct "/x/y/z/d" to use for the new server. Since fs_locations attribute lacks information defining various attributes of the various file system choices presented, it should only be interrogated and used when fs_locations_info is not available. When fs_locations is used, information about the specific locations should be assumed based on the following rules. The following rules are general and apply irrespective of the context. o When a DNS server name maps to multiple IP addresses, they should be considered identical, i.e. of the same _endpoint_ class. o Except in the case of servers sharing an _endpoint_ class, all listed servers should be considered as of the same _handle_ class, if and only if, the current fh_expire_type attribute does not include the FH4_VOL_MIGRATION bit. Note that in the case of referral, filehandle issues do not apply since there can be no filehandles known within the current file system nor is there any access to the fh_expire_type attribute on the referring (absent) file system. o Except in the case of servers sharing an _endpoint_ class, all listed servers should be considered as of the same _fileid_ class, if and only if, the fh_expire_type attribute indicates persistent filehandles and does not include the FH4_VOL_MIGRATION bit. Note that in the case of referral, fileid issues do not apply since Noveck Expires December 21, 2006 [Page 32] Internet-Draft Location Chapters for 4.1 June 2006 there can be no fileids known within the referring (absent) file system nor is there any access to the fh_expire_type attribute. o Except in the case of servers sharing an _endpoint_ class, all listed servers should be considered as of different _change_ classes. For other class assignments, handling depends of file system transitions depends on the reasons for the transition: o When the transition is due to migration, the target should be treated as being of the same _state_ and _verifier_ class as the source. o When the transition is due to failover to another replica, the target should be treated as being of a different _state_ and _verifier_ class from the source. The specific choices reflect typical implementation patterns for failover and controlled migration respectively. Since other choices are possible and useful, this information is better obtained by using fs_locations_info. See the section "Security Considerations" for a discussion on the recommendations for the security flavor to be used by any GETATTR operation that requests the "fs_locations" attribute. 3.10. The Attribute fs_locations_info The fs_locations_info attribute is intended as a more functional replacement for fs_locations which will continue to exist and be supported. Clients can use it get a more complete set of information about alternative file system locations. When the server does not support fs_locations_info, fs_locations can be used to get a subset of the information. A server which supports fs_locations_info MUST support fs_locations as well. There are several sorts of additional information present in fs_locations_info, that aren't available in fs_locations: o Attribute continuity information to allow a client to select a location which meets the transparency requirements of the applications accessing the data and to take advantage of optimizations that server guarantees as to attribute continuity may provide (e.g. change attribute). o Filesystem identity information which indicates when multiple replicas, from the clients point of view, correspond to the same Noveck Expires December 21, 2006 [Page 33] Internet-Draft Location Chapters for 4.1 June 2006 target filesystem, allowing them to be used interchangeably, without disruption, as multiple paths to the same thing. o Information which will bear on the suitability of various replicas, depending on the use that the client intends. For example, many applications need an absolutely up-to-date copy (e.g. those that write), while others may only need access to the most up-to-date copy reasonably available. o Server-derived preference information for replicas, which can be used to implement load-balancing while giving the client the entire fs list to be used in case the primary fails. The fs_locations_info attribute consists of a root pathname (just like fs_locations), together with an array of location4_item structures. Noveck Expires December 21, 2006 [Page 34] Internet-Draft Location Chapters for 4.1 June 2006 struct locations4_server { int32_t currency; uint32_t info<>; utf8str_cis server; }; const LIBX_GFLAGS = 0; const LIBX_TFLAGS = 1; const LIBX_CLSHARE = 2; const LIBX_CLSERVER = 3; const LIBX_CLENDPOINT = 4; const LIBX_CLHANDLE = 5; const LIBX_CLFILEID = 6; const LIBX_CLVERIFIER = 7; const LIBX_CLSTATE = 8; const LIBX_READRANK = 9; const LIBX_WRITERANK = 10; const LIBX_READORDER = 11; const LIBX_WRITEORDER = 12; const LIGF_WRITABLE = 0x01; const LIGF_CUR_REQ = 0x02; const LIGF_ABSENT = 0x04; const LIGF_GOING = 0x08; const LITF_RDMA = 0x01; struct locations4_item { locations4_server entries<>; pathname4 rootpath; }; struct locations4_info { pathname4 fs_root; locations4_item items<>; }; The fs_locations_info attribute is structured similarly to the fs_locations attribute. A top-level structure (fs_locations4 or locations4_info) contains the entire attribute including the root pathname of the fs and an array of lower-level structures that define replicas that share a common root path on their respective servers. Those lower-level structures in turn (fs_locations4 or location4_item) contain a specific pathname and information on one or more individual server replicas. For that last lowest-level Noveck Expires December 21, 2006 [Page 35] Internet-Draft Location Chapters for 4.1 June 2006 information, fs_locations has a server name in the form of utf8str_cis, while fs_locations_info has a location4_server structure that contains per-server-replica information in addition to the server name. The location4_server structure consists of the following items: o An indication of file system up-to-date-ness (currency) in terms of approximate seconds before the present. A negative value indicates that the server is unable to give any reasonably useful value here. A zero indicates that filesystem is the actual writable data or a reliably coherent and fully up-to-date copy. Positive values indicate how out- of-date this copy can normally be before it is considered for update. Such a value is not a guarantee that such updates will always be performed on the required schedule but instead serve as a hint about how far behind the most up-to-date copy of the data, this copy would normally be expected to be. o A counted array of 32-but words containing various sorts of data, about the particular file system instance. This data includes general flags, transport capability flags, file system equivalence class information, and selection priority information. The encoding will be discussed below. o The server string. For the case of the replica currently being accessed (via GETATTR), a null string may be used to indicate the current address being used for the RPC call. Data within the info array, is in the form of 8-bit data items even though that array is, from XDR's point of view an array of 32-bit integers. This definition was chosen because: o The kinds of data in the info array, representing, flags, file system classes and priorities among set of file systems representing the same data are such that eight bits provides a quite acceptable range of values. Even where there might be more than 256 such file system instances, having more than 256 distinct classes or priorities is unlikely. o XDR does not have any means to declare an 8-bit data type, other than an ASCII string, and using 32-bit data types would lead to significant space inefficiency. o Explicit definition of the various specific data items within XDR would limit expandability in that any extension within a subsequent minor version would require yet another attribute, leading to specification and implementation clumsiness. Noveck Expires December 21, 2006 [Page 36] Internet-Draft Location Chapters for 4.1 June 2006 o Such explicit definitions would also make it impossible to propose standards-track extensions apart from a full minor version. Each 8-bit successive field within this array is designated by a constant byte-index as defined above. More significant bit fields within a single word have successive indices with a transition to the next word following the most significant 8-bit field in each word. The set of info data is subject to expansion in a future minor version, or in a standard-track RFC, within the context of a single minor version. The server SHOULD NOT send and the client MUST not use indices within the info array that are not defined in standards- track RFC's. The following fragment of c++ code (with Doxygen-style comments) illustrates how data items within the info array can be found using a byte-index such as specified by the constants beginning with "LIBX_". The associated InfoArray object is assume to be initialized with "Length" containing the XDR-specified length in terms of 32-bit words and "Data" containing the array of words encoded by the "info<>" specification. Noveck Expires December 21, 2006 [Page 37] Internet-Draft Location Chapters for 4.1 June 2006 class InfoArray { private: uint32_t Length; uint32_t Data[]; public: uint8_t GetValue(int byteIndex); }; /// @brief Get the value of a locations4_server info value /// /// This method obtains the specific info value given a /// byte index defined in the NFSv4.1 spec or another /// later standards-track document. /// /// @param[in] byteIndex The byte index identifying the /// item requested. /// @returns The value of the requested item. uint8_t InfoArray::GetItem(int byteIndex) { int wordIndex = byteIndex/4; int byteWithinWord = byteIndex % 4; if (wordIndex >= Length) { return (0); } uint32_t ourWord = Data[wordIndex]; return ((ourWord >> (byteWithinWord*8)) & 0xff); } The info array contains within it: o Two 8-bit flag fields, one devoted to general file-system characteristics and a second reserved for transport-related capabilities. o Seven 8-bit class values which define various file system equivalence classes as explained below. o Four 8-bit priority values which govern file system selection as explained below. The general file system characteristics flag (at byte index LIBX_GFLAGS) has the following bits defined within it: Noveck Expires December 21, 2006 [Page 38] Internet-Draft Location Chapters for 4.1 June 2006 o LIGF_WRITABLE indicates that this fs target is writable, allowing it to be selected by clients which may need to write on this filesystem. When the current filesystem instance is writable, then any other filesystem to which the client might switch must incorporate within its data any committed write made on the current filesystem instance. See the section on verifier class, for issues related to uncommitted writes. While there is no harm in not setting this flag for a filesystem that turns out to be writable, turning the flag on for read-only filesystem can cause problems for clients who select a migration or replication target based on it and then find themselves unable to write. o LIGF_CUR_REQ indicates that this replica is the one on which the request is being made. Only a single server entry may have this flag set and in the case of a referral, no entry will have it. o LIGF_ABSENT indicates that this entry corresponds an absent filesystem replica. It can only be set if LIGF_CUR_REQ is set. When both such bits are set it indicates that a filesystem instance is not usable but that the information in the entry can be used to determine the sorts of continuity available when switching from this replica to other possible replicas. Since this bit can only be true if LIGF_CUR_REQ is true, the value could be determined using the fs_absent attribute but the information is also made available here for the convenience of the client. An entry with this bit, since it represents a true filesystem (albeit absent) does not appear in the event of a referral, but only where a filesystem has been accessed at this location and subsequently been migrated. o LIGF_GOING indicates that a replica, while still available, should not be used further. The client, if using it, should make an orderly transfer to another filesystem instance as expeditiously as possible. It is expected that file systems going out of service will be announced as LIGF_GOING some time before the actual loss of service and that the valid_for value will be sufficiently small to allow servers to detect and act on scheduled events while large enough that the cost of the requests to fetch the fs_locations_info values will not be excessive. Values on the order of ten minutes seem reasonable. The transport-flag field (at byte index LIBX_TFLAGS) contains the following bits related to the transport capabilities of the specific file system. o LITF_RDMA indicates that this file system provides NFSv4.1 file system access using an RDMA-capable transport. Noveck Expires December 21, 2006 [Page 39] Internet-Draft Location Chapters for 4.1 June 2006 Attribute continuity and filesystem identity information are expressed by defining equivalence relations on the sets of file systems presented to the client. Each such relation is expressed as a set of file system equivalence classes. For each relation, a file system has an 8-bit class number. Two file systems belong to the same class if both have identical non-zero class numbers. Zero is treated as non-matching. Most often, the relevant question for the client will be whether a given replica is identical-with/ continuous-to the current one in a given respect but the information should be available also as to whether two other replicas match in that respect as well. The following fields specify the file system's class numbers for the equivalence relations used in determining the nature of file system transitions. See Section 3.6 for details about how this information is to be used. o The field with byte-index LIBX_CLSHARE defines the sharing class for the file system. o The field with byte-index LIBX_CLSERVER defines the server class for the file system. o The field with byte-index LIBX_CLENDPOINT defines the endpoint class for the file system. o The field with byte-index LIBX_CLHANDLE defines the handle class for the file system. o The field with byte-index LIBX_CLFILEID defines the fileid class for the file system. o The field with byte-index LIBX_CLVERIFIER defines the verifier class for the file system. o The field with byte-index LIBX_CLSTATE defines the state class for the file system. Server-specified preference information is also provided via 8-bit values within the info array. The values provide a rank and an order (see below) to be used with separate values specifiable for the cases of read-only and writable file systems. These values are compared for different file systems to establish the server-specified preference, with lower values indicating "more preferred". Rank is used to express a strict server-imposed ordering on clients, with lower values indicating "more preferred." Clients should attempt to use all replicas with a given rank before they use one Noveck Expires December 21, 2006 [Page 40] Internet-Draft Location Chapters for 4.1 June 2006 with a higher rank. Only if all of those file systems are unavailable should the client proceed to those of a higher rank. Within a rank, the order value is used to specify the server's preference to guide the client's selection when the client's own preferences are not controlling, with lower values of order indicating "more preferred." If replicas are approximately equal in all respects, clients should defer to the order specified by the server. When clients look at server latency as part of their selection, they are free to use this criterion but it is suggested that when latency differences are not significant, the server- specified order should guide selection. o The field at byte index LIBX_READRANK gives the rank value to be used for read-only access. o The field at byte index LIBX_READOREDER gives the order value to be used for read-only access. o The field at byte index LIBX_WRITERANK gives the rank value to be used for writable access. o The field at byte index LIBX_WRITEOREDER gives the order value to be used for writable access. Depending on the potential need for write access by a given client, one of the pairs of rank and order values is used. The read rank and order should only be used if the client knows that only reading will ever be done or if it is prepared to switch to a different replica in the event that any write access capability is required in the future. The locations4_info structure, encoding the fs_locations_info attribute contains the following: o The fs_root field which contains the pathname of the root of the current filesystem on the current server, just as it does the fs_locations4 structure. o An array of locations4_item structures, which contain information about replicas of the current filesystem. Where the current filesystem is actually present, or has been present, i.e. this is not a referral situation, one of the locations4_item structure will contain a locations4_server for the current server. This structure will have LIGF_ABSENT set if the current filesystem is absent, i.e. normal access to it will return NFS4ERR_MOVED. o The valid_for field specifies a time for which it is reasonable for a client to use the fs_locations_info attribute without Noveck Expires December 21, 2006 [Page 41] Internet-Draft Location Chapters for 4.1 June 2006 refetch. The valid_for value does not provide a guarantee of validity since servers can unexpectedly go out of service or become inaccessible for any number of reasons. Clients are well- advised to refetch this information for actively accessed filesystem at every valid_for seconds. This is particularly important when filesystem replicas may go out of service in a controlled way using the LIGF_GOING flag to communicate an ongoing change. The server should set valid_for to a value which allows well-behaved clients to notice the LIF_GOING flag and make an orderly switch before the loss of service becomes effective. If this value is zero, then no refetch interval is appropriate and the client need not refetch this data on any particular schedule. In the event of a transition to a new filesystem instance, a new value of the fs_locations_info attribute will be fetched at the destination and it is to be expected that this may have a different valid_for value, which the client should then use, in the same fashion as the previous value. As noted above, the fs_locations_info attribute, when supported, may be requested of absent filesystems without causing NFS4ERR_MOVED to be returned and it is generally expected that will be available for both present and absent filesystems even if only a single location_server entry is present, designating the current (present) filesystem, or two location_server entries designating the current (and now previous) location of an absent filesystem and its successor location. Servers are strongly urged to support this attribute on all filesystems if they support it on any filesystem. 3.11. The Attribute fs_status In an environment in which multiple copies of the same basic set of data are available, information regarding the particular source of such data and the relationships among different copies, can be very helpful in providing consistent data to applications. Noveck Expires December 21, 2006 [Page 42] Internet-Draft Location Chapters for 4.1 June 2006 enum status4_type { STATUS4_FIXED = 1, STATUS4_UPDATED = 2, STATUS4_INTERLOCKED = 3, STATUS4_WRITABLE = 4, STATUS4_ABSENT = 5 }; struct fs4_status { status4_type type; utf8str_cs source; utf8str_cs current; int32_t age; nfstime4 version; }; The type value indicates the kind of filesystem image represented. This is of particular importance when using the version values to determine appropriate succession of filesystem images. Five types are distinguished: o STATUS4_FIXED which indicates a read-only image in the sense that it will never change. The possibility is allowed that as a result of migration or switch to a different image, changed data can be accessed but within the confines of this instance, no change is allowed. The client can use this fact to aggressively cache. o STATUS4_UPDATED which indicates an image that cannot be updated by the user writing to it but may be changed exogenously, typically because it is a periodically updated copy of another writable filesystem somewhere else. o STATUS4_VERSIONED which indicates that the image, like the STATUS4_UPDATED case, is updated exogenously, but it provides a guarantee that the server will carefully update the associated version value so that the client, may if it chooses, protect itself from a situation in which it reads data from one version of the filesystem, and then later reads data from an earlier version of the same filesystem. See below for a discussion of how this can be done. o STATUS4_WRITABLE which indicates that the filesystem is an actual writable one. The client need not of course actually write to the filesystem, but once it does, it should not accept a transition to anything other than a writable instance of that same filesystem. Noveck Expires December 21, 2006 [Page 43] Internet-Draft Location Chapters for 4.1 June 2006 o STATUS4_ABSENT which indicates that the information is the last valid for a filesystem which is no longer present. The opaque strings source and current provide a way of presenting information about the source of the filesystem image being present. It is not intended that client do anything with this information other than make it available to administrative tools. It is intended that this information be helpful when researching possible problems with a filesystem image that might arise when it is unclear if the correct image is being accessed and if not, how that image came to be made. This kind of debugging information will be helpful, if, as seems likely, copies of filesystems are made in many different ways (e.g. simple user-level copies, filesystem- level point-in-time copies, cloning of the underlying storage), under a variety of administrative arrangements. In such environments, determining how a given set of data was constructed can be very helpful in resolving problems. The opaque string 'source' is used to indicate the source of a given filesystem with the expectation that tools capable of creating a filesystem image propagate this information, when that is possible. It is understood that this may not always be possible since a user- level copy may be thought of as creating a new data set and the tools used may have no mechanism to propagate this data. When a filesystem is initially created associating with it data regarding how the filesystem was created, where it was created, by whom, etc. can be put in this attribute in a human- readable string form so that it will be available when propagated to subsequent copies of this data. The opaque string 'current' should provide whatever information is available about the source of the current copy. Such information as the tool creating it, any relevant parameters to that tool, the time at which the copy was done, the user making the change, the server on which the change was made etc. All information should be in a human- readable string form. The age provides an indication of how out-of-date the file system currently is with respect to its ultimate data source (in case of cascading data updates). This complements the currency field of locations4_server (See Section 3.10) in the following way: the information in locations4_server.currency gives a bound for how out of date the data in a file system might typically get, while the age gives a bound on how out of date that data actually is. Negative values imply no information is available. A zero means that this data is known to be current. A positive value means that this data is known to be no older than that number of seconds with respect to the ultimate data source. Noveck Expires December 21, 2006 [Page 44] Internet-Draft Location Chapters for 4.1 June 2006 The version field provides a version identification, in the form of a time value, such that successive versions always have later time values. When the filesystem type is anything other than STATUS4_VERSIONED, the server may provide such a value but there is no guarantee as to its validity and clients will not use it except to provide additional information to add to 'source' and 'current'. When the type is STATUS4_VERSIONED, servers should provide a value of version which progresses monotonically whenever any new version of the data is established. This allows the client, if reliable image progression is important to it, to fetch this attribute as part of each COMPOUND where data or metadata from the filesystem is used. When it is important to the client to make sure that only valid successor images are accepted, it must make sure that it does not read data or metadata from the filesystem without updating its sense of the current state of the image, to avoid the possibility that the fs_status which the client holds will be one for an earlier image, and so accept a new filesystem instance which is later than that but still earlier than updated data read by the client. In order to do this reliably, it must do a GETATTR of fs_status that follows any interrogation of data or metadata within the filesystem in question. Often this is most conveniently done by appending such a GETATTR after all other operations that reference a given filesystem. When errors occur between reading filesystem data and performing such a GETATTR, care must be exercised to make sure that the data in question is not used before obtaining the proper fs_status value. In this connection, when an OPEN is done within such a versioned filesystem and the associated GETATTR of fs_status is not successfully completed, the open file in question must not be accessed until that fs_status is fetched. The procedure above will ensure that before using any data from the filesystem the client has in hand a newly-fetched current version of the filesystem image. Multiple values for multiple requests in flight can be resolved by assembling them into the required partial order (and the elements should form a total order within it) and using the last. The client may then, when switching among filesystem instances, decline to use an instance which is not of type STATUS4_VERSIONED or whose version field is earlier than the last one obtained from the predecessor filesystem instance. Noveck Expires December 21, 2006 [Page 45] Internet-Draft Location Chapters for 4.1 June 2006 4. Other Changes This is a list of changes in other areas of the spec that need to be made to conform with what is written here. o Need to add fs_absent, fs_locations_info, and fs_status to the list of recommended attributes. o Need to add NFS4ERR_MOVED to all the ops that don't currently include it to match what the spec says. Alternatively, we may want to factor this out and create a list of errors that any op can receive. o Change the definition of NFS4ERR_MOVED in section 20 to indicate that it just means that the fs is not there and may never have really "moved". o Delete sections 6.14 and 6.14.* which have been incorporated in the new chapter. o In the spirit of the "minior" issue, fix instances of ampersand-lt which need to come out as less-than. Also in that spirit, fix errors marked "TDB" in the error list. o Add locations4_server, locations4_item, and locations4_info as the appropriate sections 1.2.*. o Add status4_type and fs4_status as the appropriate sections 1.2.*. o Delete the errors NFS4ERR_MOVED_DATA_AND_STATE and NFS4ERR_MOVED_DATA from section 20. o Replace the sixth paragraph of section 2.2.3 with the following text: FH4_VOL_MIGRATION The filehandle will expire as a result of a file system transition (migration or replication), in those case in which the continuity of filehandle use is not specified by _handle_ class information within the fs_locations_info attribute. When this bit is set, clients without access to fs_locations_info information should assume file handles will expire on file system transitions. o Note that the last sentence of the paragraph referred to above has been removed and was never true. It is one thing to say that a file handle may expire (i.e. that you have to be prepared for the server to tell you it is expired) and another to say that you must decide it is expired even if the server may not necessarily recognize as expired (because he has no idea what your handles look like). Noveck Expires December 21, 2006 [Page 46] Internet-Draft Location Chapters for 4.1 June 2006 o Replace the tenth paragraph of section 2.2.3 with the following text: Servers which provide volatile filehandles that may expire while open require special care as regards handling of RENAMESs and REMOVEs. This situation can arise if FH4_VOL_MIGRATION or FH4_VOL_RENAME is set, if FH4_VOLATILE_ANY is set and FH4_NOEXPIRE_WITH_OPEN not set, or if a non-readonly file system has a transition target in a different _handle _ class. In these cases, the server should deny a RENAME or REMOVE that would affect an OPEN file of any of the components leading to the OPEN file. In addition, the server should deny all RENAME or REMOVE requests during the grace period, in order to make sure that reclaims of files where filehandles may have expired do not do a reclaim for the wrong file. 5. References [1] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003. [2] Shepler, S., "NFSv4 Minor Version 1", draft-ietf-nfsv4-minorversion1-02 (work in progress), March 2006. Noveck Expires December 21, 2006 [Page 47] Internet-Draft Location Chapters for 4.1 June 2006 Author's Address David Noveck Network Appliance 1601 Trapelo Road, Suite 16 Waltham, MA 02454 US Phone: +1 781 961 9291 Email: dnoveck@netapp.com Noveck Expires December 21, 2006 [Page 48] Internet-Draft Location Chapters for 4.1 June 2006 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Noveck Expires December 21, 2006 [Page 49]