Network File System (NFS) Version 4 Minor Version 1 Protocol

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as specified in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

The above differs from the corresponding statement in earlier specification versions ([RFC5661] and [RFC8881]) which only referred to [RFC2119]. For further discussion of this change, see Appendix B.3¶

In some cases, such keywords will appear in all capitals within quotations, direct or indirect, from earlier documents. In such cases, these terms are to be interpreted just as above, except where it is explicitly noted that such an interpretation is not to be inferred. In some cases, it might be that this document's approach to the matter would not use those key words for reasons explained in the text. Such a shift might cause compatibility issues, if the previous keyword were actually relied upon but it also possible that it was not relied upon while the implications of that use were ignored for various reasons.¶

The reader should be aware that, as discussed in Appendix C.1, there are uses of the keywords listed above in RFCs 5661 and 8881 which might not have been appropriate, even though the interpretation specified above was intended when the text was written and submitted for publication. In some cases, the text in this document has been updated to correct the issue but it should be understood that not all such questionable uses have been addressed and that this state of affairs might continue to exist until a later draft of this document is submitted for publication.¶

1.2. The Changed Role of this Specification

Previous specifications for this minor version ([RFC5661], [RFC8881]) have purported to describe the protocol in its entirety, without reference to features common to all minor versions of NFS Version 4. In contrast, this update relies on a set of base documents describing common aspects of the NFSv4 protocol that applies to all minor versions.¶

• Rules for extensions and creation of new minor versions appear only in [RFC8178], unlike previously in which they appeared in the NFSv4.1 specification. This eliminates the unfortunate situation in which each minor version was allowed to create its own extension rules.¶

• Handling of internationalization-related matters (for all minor versions) is now discussed in its own document, which is expected to be an RFC derived from [I-D.ietf-nfsv4-internationalization].¶

  That document, based in large part on the handling of internationalization for NFSv4 minor version zero outlined in [RFC7530], has been extended to cover all minor versions and enhanced to fully support case-insensitive handling of internationalized file names.¶

  This corrects the unfortunate situation in which internationalization for minor version one and subsequent minor versions (in [RFC5661] and [RFC8881]) had never been implemented and could never have been implemented by NFS Version four clients and servers.¶

• Handling of core security-related matters for all NFSv4 minor versions will be consolidated in a set of documents that are expected to be RFCs derived from [I-D.dnoveck-nfsv4-security] and [I-D.dnoveck-nfsv4-acls].¶

  This shift is made necessary by the following issues, many of which are of long standing and make the continuation of previous approaches to these issues insupportable:¶

  • The lack of a substantial threat analysis in the Security Considerations section of any existing minor version specification.¶

  • The unfortunate designation of AUTH_SYS an "OPTIONAL means of authentication" which had the effect of obscuring the severe security problems with its common use. The use of "OPTIONAL" suggested that use of AUTH_SYS had no harmful consequences while the phrase "means of authentication" ignored the fact that no actual authentication took place.¶

  • The assumption that data confidentiality could be satisfactorily addressed as an occasionally-used optional facility.¶

  • The neglect of the need for dependable semantic description of the protocol's authorization semantics.¶

    Earlier versions of NFS had avoided the need for such descriptions by relying on POSIX semantics. The addition of non-POSIX semantic elements, including named attributes and ACLs, interfered with that approach although the necessity was not recognized as NFSv4 was being formulated.¶

  The availability of new transport-level security features such as those provided by RPC-with-TLS [RFC9289] provides a basis to correct many of the above issues.¶

  In providing that sort of correction, we need to be careful not to declare existing implementations non-compliant post facto, while still providing adequate warning of the security consequences of continuing to use the NFS Version 4 protocol insecurely, as described in previous specifications.¶

1.3. Addressing Protocol Defects

This section provides an overview of situations in which it was necessary to change the protocol to correct protocol issues that needed to be addressed going forward. Although all these issues can, from today's perspective, be viewed as mistakes, it is not clear whether that description is appropriate for decisions made so long ago, under very different circumstances. In any case, those questions will not be addressed here.¶

The correction of protocol defects often gives rise to compatibility issues and their possible presence will be discussed below. In addition, the question of when it is appropriate to address such issues using the protocol extension mechanism described in [RFC8178] needs to be considered. Section 9 of that document alludes to this possibility but we have to decide when defects are best addressed in that way.¶

These defects can be divided into two groups based on their origin.¶

• Defects that originated in minor version zero¶

  Many of these defects are addressed in the new NFSv4-wide documents ([I-D.dnoveck-nfsv4-security], [I-D.dnoveck-nfsv4-acls], and [I-D.ietf-nfsv4-internationalization]. For these defects , the greater change scope require more attention to compatibility issues. In addition, that greater scope limits the degree to which protocol extension can be used in providing a correction since that extension would need to be propagated to two non-extensible minor versions.¶

• Defects that originated in minor version one.¶

  A major source of defects was the result of the addition of a set of OPTIONAL features in v4.1, that have never been implemented, making it important to eliminate issues in the specification that have led to this situation.¶

  The presumptive non-implementation of these features will limit interoperability concerns. However, since we cannot be sure about the possible existence of implementations under development, we will try to provide for the possibility of interoperating with earlier implementations, even if that interoperation is hypothetical. Only in the case of features whose current specification makes implementation impossible can we ignore the possibility of interoperating with such implementations.¶

The following defects were addressed as part of this update effort:¶

1:

Internationalization is being thoroughly respecified in the NFSv4-wide document [I-D.ietf-nfsv4-internationalization].¶

There were no NFSv4.1 compatibility issues to deal with since the handling of internationalization approach mandated by [RFC8881] had never been implemented.¶

Potential NFSv4.0 compatibility issues were very limited since the approach followed in [RFC7530] was continued and that approach had been used by all implementations.¶

In one particular case, there is a potential compatibility issue arising from the transition of one potential troubling server behavior from being discouraged using SHOULD NOT to being prohibited using MUST NOT. However, in view of the unlikelihood of ongoing use of the discouraged behavior, this has not been considered problematic.¶

The respecification of the fs_charset_cap attribute raises the possibility of within-NFS4.1 compatibility issues. However the very limited use of this attribute by clients combine with the lack of clarity in the previous definition makes it unlikely that the use of protocol extension to support previous uses would be justified.¶

2:

Because the Persistent reply cache feature could not be implemented as described in [RFC8881], the entire area was respecified in Section 8.¶

The reasons for this respecification are discussed in Section 8.1.¶

Because the existing feature specification was unimplementable there were no compatibility issues to deal with.¶

No protocol extensions were needed since the small set of bits defined for the earlier feature could be coopted.¶

3:

Security for all minor versions is being thoroughly respecified in the NFSv4-wide document [I-D.dnoveck-nfsv4-security]. In this discussion, issues related to authorization semantics and to ACLs are being dealt with separately.¶

This respecification was made necessary by the lack of threat analyses for all minor versions, the absence of any discussion of the security problems associated with the use of AUTH_SYS, and the half-hearted approach to the security of over-the-wire transmission in which transmission in the clear was the default and the provision of secure transmission was an option requiring per-fs configuration.¶

4:

As part of the new handling of security, a more serious treatment of authorization semantics was necessary. As part of effecting this, the attributes mode, owner, and owner_group became REQUIRED, as it is impossible to effect security without them.¶

This change was not connected to the shift in terminology in which the attributes incorrectly described as "RECOMMENDED" became "OPTIONAL".¶

No significant compatibility issue are expected, since the existence of servers not supporting these attributes or of clients interacting with such servers, while possible theoretically, has to be considered extremely unlikely and none are known to the working group.¶

5:

A number of gaps in the description of authorization semantics needed to be addressed. These include the lack of a clear description of authorization for operations on named attribute directories and potential use of the "sticky" bit in controlling authorization of file deletion.¶

These matters are being addressed within [I-D.dnoveck-nfsv4-security] where they are being tracked as Consensus Items #66 and #6 respectively.¶

Working group consideration of the security document will involve resolving those two Consensus Items, as well as others.¶

6:

The handling of ACLs for all minor versions is being thoroughly respecified in the NFSv4-wide document [I-D.dnoveck-nfsv4-acls].¶

Such a respecification was made necessary by the profound underspecification of the ACL feature that arose from a misguided attempt to support two very different approaches to the provision of ACLs. The problems posed by the different semantics of these two were never clearly addressed since it was erroneously assumed that semantic description could be avoided. As a result, potential interoperability was compromised since there was no way for the client to determined what ACL-based facilities were supported by a particular server, given that the specification treated these differences as if they were quality-of-implementation issues.¶

The development of [I-D.dnoveck-nfsv4-acls] has included a respecification of the area in which support for a subset of draft POSIX ACLs, termed UNIX ACLs, was the core and the various additions to that core were considered additional OPTIONAL features. These included the features that motivated the extensions in the NFSv4 ACL model and further accommodations for the semantics of the draft POSIX ACL model.¶

The development of [I-D.dnoveck-nfsv4-acls] has involved use of protocol extension within NFSv4.1 in addition to necessary structural changes that did not involve XDR changes.¶

The development of [I-D.dnoveck-nfsv4-acls] to support the rfc5661bis effort will most likely be limited to providing interoperability for those using the facilities within the UNIX ACL core or within the draft POSIX acl model. Interoperability for features beyond that set is likely to be delayed to later ACL bis, while the deletion of unneeded proposed features will have to wait for a later minor version, e.g., NFSv4.3.¶

7:

It has been necessary to define a new read-only per-fs OPTIONAL attribute that will allow clients to determine which of the OPTIONAL extensions to the core UNIX ACL model are supported by the server.¶

While this was essential to make the NFSv4 extensions usable, it also has a critical role in making POSIX ACL support available within NFSv4, albeit with some client mapping/filtering,¶

8:

A new ACE flag was necessary to address the difference in the handling of partial ALLOW ACE satisfaction in the two ACL models to be supported.¶

No compatibility issues are expected to arise since Aclchoice will indicate to the client whether the server is aware of the new flag.¶

9:

In order to support the draft POSIX ACL approach to ACL inheritance, it was necessary to provide some protocol extensions despite the fact that, overall, the NFSv4 ACL inheritance model has a wider semantic range.¶

A new ACE flag was used to identify "default" ACEs defined as inherit-only ACEs applying to both file and subdirectories, that are, unlike most ACEs, not normally modified/obliterated by changes to ACL attribute. To control whether such ACEs, considered as part of the "default ACL" within draft POSIX, new flags have been defined within the na41_flag word used by the sacl and dacl attribute.¶

Because this approach is easier to deal with, it will be made available to all ACL users rather than being limited to those using the draft POSIX ACL model, for whom it is pretty much essential.¶

As in the previous case, no compatibility issues are expected to arise since Aclchoice will indicate to the client whether the server is aware of the new flags to support the draft POSIX ACL approach to inheritance.¶

10:

In order to enable the translation of reverse-slope modes to ACLs in environments which support for DENY ACEs was not available, it was necessary to enable ACLs to contain the special who values, GROUPNOTOWNER@ and OTHERS@.¶

While not, in itself a protocol extension, there were associated protocol extensions to enable Aclchoice to report about the support for these new values.¶

No compatibility issues are expected since the new values are unlikely to be seen by the client and can only be used by the client when Aclchoice indicates that support is present.¶

11:

The defects described in Appendix C.2.1 needed to be addressed together, in connection with making it clear that the term "Exactly-once Semantics" ignored the fact that there were valid reasons to give up on requests which could leave them unexecuted.¶

This change did not give rise to compatibility issues since the specification was changed to match existing implementations, and these are expected to remain as they are.¶

12:

The inadvertent prohibition of the use of RoCE in implementing NFSv4.1 using RPC-over-RDMA was removed.¶

This is another case in which compatibility issue are not expected because the spec has been changed to match existing implementations.¶

13:

The corrections discussed in Appendix C.2.3 had to be made since most of the worries expressed within it were the result of misunderstandings.¶

Although no compatibility issues are expected we will need to review the changes and reach consensus on them.¶

14:

There were a number of issues in the earlier specification of the directory delegation feature that need to be addressed to enable implementations of this needed feature to be produced. Given the lack of implementation during the long period since they were introduced in [RFC5661] many years ago.¶

While a significant part of the problems could be ascribed to clarity issues, there were also a set of defects, some of which required protocol extensions, as provided for in Section 9 of [RFC8178].¶

The defects which contributed substantially to this long-lasting lack of implementation includes the failure to fully address authorization issues for the use of cached directory data, implementability issues regarding the maintenance of cached attribute data, and the assumption that clients could maintain the cached directory contents only in the same format as used by the server. For more discussion of these defects, see Appendix C.2.2.¶

These issues were addressed by a major rewrite of Section 15.9 in which protocol extension was necessary, including the addition of new values to the enum notify_type4. In addition, there are complementary changes made to Section 23.39 and Section 25.4) and to operations that might result in notifications being sent.¶

Although no client and server implementations of this feature are known to exist, the possibility of them existing cannot be excluded. As a result, the revised specification takes care to deal appropriately with such hypothetical implementations, and to not prohibit their use unless that is necessary to avoid unacceptable system behavior.¶

15:

Change in the recommendations regarding handling of numeric strings to represent users and groups.¶

Formerly considered troublesome even in the AUTH_SYS case despite the fact that there is no explanation given as to how to effect mapping between numeric ids and strings. Instead, it is assumed that client and server will somehow agree to do this without the specification making it possible or giving a convincing reason that such mapping is needed.¶

Of the above, only the items 7, 8, 9, 10, and 14 required protocol extension to resolve. All will need to be incorporated in the eventual bis document superseding [RFC5662].¶

2.1. The NFS Version 4 Minor Version 1 Protocol

The NFS version 4 minor version 1 (NFSv4.1) protocol is the second minor version of the NFS version 4 (NFSv4) protocol. The first minor version, NFSv4.0, is now described in [RFC7530], as modified by [RFC7931] and [RFC8587]. Minor version 1 follows the guidelines for minor versioning presented in [RFC8178].¶

As a minor version, NFSv4.1 is consistent with the overall goals for NFSv4, but extends the protocol so as to better meet those goals, based on experiences with NFSv4.0. In addition, NFSv4.1 has adopted some additional goals, which motivate some of the major extensions in NFSv4.1, such as the use of the sessions model.¶

This minor version adds a considerable number of new operations including some that are not OPTIONAL and makes a number of NFSv4.0 operations MANDATORY to NOT implement. As a result, the vast majority of NFSv4.0 requests are not valid in NFSv4.1 and vice versa. While clients and server that support both minor versions are common, such implementations treat the two versions as distinct protocols sharing a substantial common heritage.¶

2.2. Scope of This Document

This document describes the NFSv4.1 protocol. With respect to NFSv4.0, this document does not:¶

• describe the NFSv4.0 protocol, except where needed to contrast it with NFSv4.1.¶
• modify the specification of the NFSv4.0 protocol.¶
• clarify the NFSv4.0 protocol.¶

2.3. NFSv4 Goals

The NFSv4 protocol is a further revision of the NFS protocol defined already by NFSv3 [RFC1813]. It retains the essential characteristics of previous versions: easy recovery; independence of transport protocols, operating systems, and file systems; simplicity; and good performance. NFSv4 had the following goals:¶

• Improved access and good performance on the Internet.¶

  The protocol is designed to transit firewalls easily, perform well where latency is high and bandwidth is low, and scale to very large numbers of clients per server.¶

• Strong security with facilities for negotiation of security handling built into the protocol.¶

  The protocol has built on the work of the ONCRPC working group in supporting the RPCSEC_GSS protocol. Additionally, the NFSv4.1 protocol provides a mechanism to allow clients and servers the ability to negotiate security and has provisions requiring or recommending client and server support for a minimal set of security schemes.¶

  The protocol now takes advantage of the ability of RPC to make confidentiality available by using TLS-based encryption on connections to be used for NFSv4.1, which may limit the need for negotiation regarding facilities such as privacy.¶

• Good cross-platform interoperability.¶

  The protocol embraces a file system model that provides a useful, common set of features that does not unduly favor one file system or operating system over another.¶

• Designed for protocol extensions via minor versioning.¶

  The protocol is designed to accept standard extensions within a framework that enables and encourages backward compatibility.¶

  When extensions are OPTIONAL, they can be added to an existing extensible minor version.¶

2.4. NFSv4.1 Goals

NFSv4.1 has the following goals, within the framework established by the overall NFSv4 goals.¶

• To correct significant structural weaknesses and oversights discovered in the base protocol.¶

• To add clarity and specificity to areas left unaddressed or not addressed in sufficient detail in the base protocol. However, as stated in Section 2.2, it is not a goal to clarify the NFSv4.0 protocol in the NFSv4.1 specification.¶

• To add specific features based on experience with the existing protocol and recent industry developments.¶

• To provide protocol support to take advantage of clustered server deployments including the ability to provide scalable parallel access to sets of files distributed among multiple servers, using a range of data access protocols.¶

  This parallel access may involve striping of single files among a set of servers within the cluster.¶

  Alternatively, parallel access may be provided by distributing unstriped files within the cluster allowing each client to contact the server holding each particular file directly.¶

2.5. General Definitions

The following definitions provide an appropriate context for the reader.¶

Byte:

In this document, a byte is an octet, i.e., a datum exactly 8 bits in length.¶

Client:

The client is the entity that accesses the NFS server's resources. The client may be an application that contains the logic to access the NFS server directly. The client may also be a traditional operating system client that provides remote file system services for a set of applications.¶

A client is uniquely identified by a client owner.¶

With reference to byte-range locking, the client is also the entity that maintains a set of locks on behalf of one or more applications. This client is responsible for providing crash or failure recovery for those locks it manages, in order to deal with the possibility of server reboot.¶

Note that multiple clients may share the same transport and connection and multiple clients may exist on the same network node.¶

Client ID:

The client ID is a 64-bit quantity used as a unique, short-hand reference to a client-supplied client owner, consisting of a verifier and a client owner id. The server is responsible for supplying the client ID.¶

Client Owner:

The client owner includes a unique string, the client owner id, opaque to the server, that identifies a client. It also includes a verifier to enable successive instances of the same client to be distinguished. Multiple network connections and source network addresses originating from those connections may share a client owner. The server is expected to treat requests from connections with the same client owner as coming from the same client. See Section 5.5 for more detail.¶

File System:

A file system is the collection of objects accessed using a particular server (as identified by the major identifier of a server owner, which is defined later in this section) that share the same value of the fsid attribute (see Section 11.12.1.9).¶

Lease:

A lease is an interval of time defined by the server for which the client is granted locks. At the end of a lease period, unrecallable locks may be revoked if the lease has not been extended. Such a lock must be revoked if a conflicting lock has been granted after the lease interval. Revocation of unrecallable locks within the lease interval is expected to be an unusual event and clients normally expect such revocations to be rare.¶

A server grants a client a single lease for all of its associated locking state. Recallable locks such as layouts and delegations can be revoked within the lease period and are generally not affected by the state of the lease.¶

Lock:

The term "lock" is used to refer to byte-range (in UNIX environments, also known as record) locks, share reservations, delegations, or layouts unless specifically stated otherwise.¶

Secret State Verifier (SSV):

The SSV is a unique secret key shared between a client and server. The SSV serves as the secret key for an internal (that is, internal to NFSv4.1) Generic Security Services (GSS) mechanism (the SSV GSS mechanism; see Section 7.9). The SSV GSS mechanism uses the SSV to compute message integrity code (MIC) and Wrap tokens. See Section 7.8.3 for more details on how NFSv4.1 uses the SSV and the SSV GSS mechanism.¶

Server:

The Server is the entity responsible for coordinating client access to a set of file systems and is identified by a server owner. A server can span multiple network addresses.¶

Server Owner:

The server owner identifies the server to the client. The server owner consists of a major identifier and a minor identifier. When the client has two connections each to a peer with the same major identifier, the client assumes that both peers are the same server (the server namespace is the same via each connection) and that lock state is sharable across both connections. When each peer has both the same major and minor identifiers, the client assumes that each connection might be associated with the same session.¶

Stable Storage:

Stable storage is storage from which data stored by an NFSv4.1 server can be recovered without data loss from multiple power failures (including cascading power failures, that is, several power failures in quick succession), operating system failures, and/or hardware failure of components other than the storage medium itself (such as disk, nonvolatile RAM, flash memory, etc.).¶

Some examples of stable storage that are allowable for an NFS server include:¶

1. Media commit of data; that is, the modified data has been successfully written to the disk media, for example, the disk platter.¶
2. An immediate reply disk drive with battery-backed, on-drive intermediate storage or uninterruptible power system (UPS).¶
3. Server commit of data with battery-backed intermediate storage and recovery software.¶
4. Cache commit with uninterruptible power system (UPS) and recovery software.¶

Stateid:

A stateid is a 128-bit quantity returned by a server that uniquely defines the open and locking states provided by the server for a specific open-owner or lock-owner/open-owner pair for a specific file and type of lock.¶

Verifier:

A verifier is a 64-bit quantity that is changed to indicate that a corresponding change on one of the peers has occurred requiring the other peer to adjust to possibility of change. There are a number of 64-bit quantities identified as verifiers:¶

• In many cases, a verifier is a 64-bit quantity generated by the server that the client can use to determine if the client has restarted and potentially lost writes that had not been reliably committed to stable storage.¶
• Another type of verifier is used to indicate whether the server mapping between directory cookies and directory entries has changed, requiring the client to restart directory interrogations that normally may be continued across multiple READDIR requests.¶
• As described above, client owners include a verifier, allowing the server to determine when a client reboot has occurred¶

The Secret State Verifier is not a verifier in the sense given in this definition.¶

2.6. Overview of NFSv4.1 Features

The major features of the NFSv4.1 protocol will be reviewed in brief. This will be done to provide an appropriate context for both the reader who is familiar with the previous versions of the NFS protocol and the reader who is new to the NFS protocols. For the reader new to the NFS protocols, there is still a set of fundamental knowledge that is expected. The reader should be familiar with the External Data Representation (XDR) and Remote Procedure Call (RPC) protocols as described in [RFC4506] and [RFC5531]. A basic knowledge of file systems and distributed file systems is expected as well.¶

In general, this specification of NFSv4.1 will not distinguish those features added in minor version 1 from those present in the base protocol but will treat NFSv4.1 as a unified whole. See Section 4 for a summary of the differences between NFSv4.0 and NFSv4.1.¶

2.7. RPC and Security

As with previous versions of NFS, the External Data Representation (XDR) and Remote Procedure Call (RPC) mechanisms used for the NFSv4.1 protocol are those defined in [RFC4506] and [RFC5531], as extended by [RFC9289]) to provide TLS-based encryption and client-host authentication. NFSv4.1 security. A description of the basics of NFv4.1 security will appear in an NFSv4-wide security document, to be derived from [I-D.dnoveck-nfsv4-security].¶

NFSv4.1 introduces parallel access (see Section 2.8.2), through the use of pNFS. The security framework described above is significantly modified by the introduction of pNFS (see Section 17.9), because of the addition of additional actors and because data access sometimes does not rely on RPC for principal identification/authentication. The appropriate handling depends on the data access protocol used (see Section 17.2.5) which depends in turn on the layout type (see Section 17.2.7.) The sections 17.9.1 through 17.9.3 discuss the security implications of using different sorts of data access protocols.¶

2.8. Protocol Structure

2.9. File System Model

The general file system model used for the NFSv4.1 protocol is the same as for previous minor versions of NFSv4. The server file system is hierarchical with the regular files contained within being treated as opaque byte streams. File names MAY be restricted to UTF-8-encoded strings of Unicode characters or treated as opaque. In addition, for some file systems, name handling MAY reflect the UTF-8 canonical equivalence relation, and in some cases, case-based equivalence relations as well.¶

The NFSv4.1 protocol does not rely on a separate protocol to provide for the initial mapping between path name and filehandle. All file systems exported by a server are presented as a tree so that all file systems are reachable from a special per-server global root filehandle. This allows LOOKUP operations to be used to perform functions previously provided by the MOUNT protocol. The server is responsible for providing any necessary pseudo file systems to bridge any gaps that arise due to unexported portions of the server-local name space that are between exported file systems.¶

5.1. Introduction

NFSv4.1 relies on core infrastructure common to nearly every operation. This core infrastructure is described in the remainder of this section.¶

5.2. RPC and XDR

The NFSv4.1 protocol is a Remote Procedure Call (RPC) application that uses RPC version 2 and the corresponding eXternal Data Representation (XDR) as defined in [RFC5531] and [RFC4506]. The transport-level encryption and client-host authentication facilities described in [RFC9289] can also be used.¶

5.3. RPC-Based Security

In addition to the above, as discussed in Section 5.3.1, some security flavors provide additional security services.¶

NFSv4.1 clients and servers MUST implement RPCSEC_GSS. (This requirement to implement is not a requirement to use.) Other flavors, such as AUTH_NONE and AUTH_SYS, can be implemented as well, although the security implications of doing so need to be carefully considered, particularly when the client host is not itself authenticated. In particular, it is RECOMMENDED by rpc-tls [RFC9289] that AUTH_SYS not be used when client host authentication is not in effect.¶

     service@hostname

     nfs

     nfs/hostname

5.4. COMPOUND and CB_COMPOUND

A significant departure from the versions of the NFS protocol before NFSv4 is the introduction of the COMPOUND procedure. For the NFSv4 protocol, in all minor versions, there are exactly two RPC procedures, NULL and COMPOUND. The COMPOUND procedure is defined as a series of individual operations and these operations perform the sorts of functions performed by traditional NFS procedures.¶

The operations combined within a COMPOUND request are evaluated in order by the server, without any atomicity guarantees. A limited set of facilities exist to pass results from one operation to another. Once an operation returns a failing result, the evaluation ends and the results of all evaluated operations are returned to the client.¶

With the use of the COMPOUND procedure, the client is able to build simple or complex requests. These COMPOUND requests allow for a reduction in the number of RPCs needed for logical file system operations. For example, multi-component look up requests can be constructed by combining multiple LOOKUP operations. Those can be further combined with operations such as GETATTR, READDIR, or OPEN plus READ to do more complicated sets of operation without incurring additional latency.¶

NFSv4.1 also contains a considerable set of callback operations in which the server makes an RPC directed at the client. Callback RPCs have a similar structure to that of the normal server requests. In all minor versions of the NFSv4 protocol, there are two callback RPC procedures: CB_NULL and CB_COMPOUND. The CB_COMPOUND procedure is defined in an analogous fashion to that of COMPOUND with its own set of callback operations.¶

The addition of new server and callback operations within the COMPOUND and CB_COMPOUND request framework provides a means of extending the protocol in subsequent minor versions.¶

Except for a small number of operations needed for session creation, server requests and callback requests are performed within the context of a session. Sessions provide a client context for every request and support robust replay protection for non-idempotent requests.¶

5.5. Client Identifiers and Client Owners

For each operation that obtains or depends on locking state, the specific client needs to be identifiable by the server.¶

Each distinct client instance is represented by a client ID. A client ID is a 64-bit identifier representing a specific client at a given time. The client ID is changed whenever the client re-initializes, and may change when the server re-initializes. Client IDs are used to support lock identification and crash recovery.¶

During steady state operation, the client ID associated with each operation is derived from the session (see Section 7) on which the operation is sent. A session is associated with a client ID when the session is created.¶

Unlike NFSv4.0, the only NFSv4.1 operations possible before a client ID is established are those needed to establish the client ID.¶

A sequence of an EXCHANGE_ID operation followed by a CREATE_SESSION operation using that client ID (eir_clientid as returned from EXCHANGE_ID) is required to establish and confirm the client ID on the server. Establishment of identification by a new incarnation of the client also has the effect of immediately releasing any locking state that a previous incarnation of that same client might have had on the server. Such released state would include all byte-range lock, share reservation, layout state. Also, where the server supports neither the CLAIM_DELEGATE_PREV nor the CLAIM_DELEG_PREV_FH claim types, all delegation state associated with the same client is released as well. For discussion of delegation state recovery, see Section 15.2.1. For discussion of layout state recovery, see Section 17.7.1.¶

Releasing such state requires that the server be able to determine that one client instance is the successor of another. Where this cannot be done, for any of a number of reasons, the locking state will remain for a time subject to lease expiration (see Section 13.3) and the new client will need to wait for such state to be removed, if it makes conflicting lock requests.¶

Client identification is encapsulated in the following client owner data type:¶

struct client_owner4 {
        verifier4       co_verifier;
        opaque          co_ownerid<NFS4_OPAQUE_LIMIT>;
};

The first field, co_verifier, is a client incarnation verifier, allowing the server to distinguish successive incarnations (e.g., reboots) of the same client. The server will start the process of canceling the client's leased state if co_verifier is different than what the server has previously recorded for the identified client (as specified in the co_ownerid field).¶

The second field, co_ownerid, contains the client owner id. This is a variable-length string that uniquely defines the client so that subsequent instances of the same client bear the same co_ownerid with a different verifier.¶

There are several considerations for how the client generates the co_ownerid string:¶

• The string should be unique so that multiple clients do not present the same string. The consequences of two clients presenting the same string range from one client getting an error to one client having its leased state abruptly and unexpectedly cancelled.¶
• The string should be selected so that subsequent incarnations (e.g., restarts) of the same client cause the client to present the same string. The implementer is cautioned from an approach that requires the string to be recorded in a local file because this precludes the use of the implementation in an environment where there is no local disk and all file access is from an NFSv4.1 server.¶
• The string should be the same for each server network address that the client accesses. This way, if a server has multiple interfaces, the client can trunk traffic over multiple network paths as described in Section 7.5. (Note: the precise opposite was advised in the NFSv4.0 specification [RFC3530].)¶
• The algorithm for generating the string should not assume that the client's network address will not change, unless the client implementation knows it is using statically assigned network addresses. This includes changes between client incarnations and even changes while the client is still running in its current incarnation. Thus, with dynamic address assignment, if the client includes just the client's network address in the co_ownerid string, there is a real risk that after the client gives up the network address, another client, using a similar algorithm for generating the co_ownerid string, would generate a conflicting co_ownerid string.¶

Given the above considerations, an example of a well-generated co_ownerid string is one that includes:¶

• If applicable, the client's statically assigned network address.¶

• Additional information that tends to be unique, such as one or more of:¶

  • The client machine's serial number (for privacy reasons, it is best to perform some one-way function on the serial number).¶
  • A Media Access Control (MAC) address (again, a one-way function should be performed).¶
  • The timestamp of when the NFSv4.1 software was first installed on the client (though this is subject to the previously mentioned caution about using information that is stored in a file, because the file might only be accessible over NFSv4.1).¶
  • A true random number. However, since this number ought to be the same between client incarnations, this shares the same problem as that of using the timestamp of the software installation.¶
• For a user-level NFSv4.1 client, it should contain additional information to distinguish the client from other user-level clients running on the same host, such as a process identifier or other unique sequence.¶

The client ID is assigned by the server (the eir_clientid result from EXCHANGE_ID) and should be chosen so that it will not conflict with a client ID previously assigned by the server. This applies across server restarts.¶

In the event of a server restart, a client may find out that its current client ID is no longer valid when it receives an NFS4ERR_STALE_CLIENTID error. The precise circumstances depend on the characteristics of the sessions involved, specifically whether the session is persistent (see Section 8), but in each case the client will receive this error when it attempts to establish a new session with the existing client ID and receives the error NFS4ERR_STALE_CLIENTID, indicating that a new client ID needs to be obtained via EXCHANGE_ID and the new session established with that client ID.¶

When a session is not persistent, the client will find out that it needs to create a new session as a result of getting an NFS4ERR_BADSESSION, since the session in question was lost as part of a server restart. When the existing client ID is presented to a server as part of creating a session and that client ID is not recognized, as would happen after a server restart, the server will reject the request with the error NFS4ERR_STALE_CLIENTID.¶

In the case of the session being persistent, the client will re-establish communication using the existing session after the restart. This session will be associated with the existing client ID but may only be used to retransmit operations that the client previously transmitted and did not see replies to. Replies to operations that the server previously performed will come from the reply cache; otherwise, NFS4ERR_DEADSESSION will be returned. Hence, such a session is referred to as "dead". In this situation, in order to perform new operations, the client needs to establish a new session. If an attempt is made to establish this new session with the existing client ID, the server will reject the request with NFS4ERR_STALE_CLIENTID.¶

When NFS4ERR_STALE_CLIENTID is received in either of these situations, the client needs to obtain a new client ID by use of the EXCHANGE_ID operation, then use that client ID as the basis of a new session, and then proceed to any other necessary recovery for the server restart case (see Section 13.4.2).¶

See the descriptions of EXCHANGE_ID (Section 23.35) and CREATE_SESSION (Section 23.36) for a complete specification of these operations.¶

5.6. Server Owners

The server owner is similar to a client owner (Section 5.5), but unlike the client owner, there is no shorthand server ID. The server owner is defined in the following data type:¶

struct server_owner4 {
 uint64_t       so_minor_id;
 opaque         so_major_id<NFS4_OPAQUE_LIMIT>;
};

The server owner is returned from EXCHANGE_ID. When the so_major_id fields are the same in two EXCHANGE_ID results, the connections that each EXCHANGE_ID were sent over can be assumed to address the same server (as defined in Section 2.5). If the so_minor_id fields are also the same, then not only do both connections connect to the same server, but the session can be shared across both connections. The reader is cautioned that multiple servers may deliberately or accidentally claim to have the same so_major_id or so_major_id/so_minor_id; the reader should examine Sections 7.5 and 23.35 in order to avoid acting on falsely matching server owner values.¶

The considerations for generating an so_major_id are similar to that for generating a co_ownerid string (see Section 5.5). The consequences of two servers generating conflicting so_major_id values are less dire than they are for co_ownerid conflicts because the client can use RPCSEC_GSS to compare the authenticity of each server (see Section 7.5).¶

5.7. Transport Layers

6.1. NFSv4.1-specific Recommendations and Requirements Regarding Security Services

[Author Aside]: Significant revisions have been made to address the hole created by the fact that the discussion of client support of data privacy uses the word "SHOULD".¶

Via the GSS-API, RPCSEC_GSS can be used to identify and authenticate users on clients to servers, and servers to users. Authentication of the client itself is not provided but can be provided by RPC independently of the use of RPCSEC_GSS.¶

GSS-API can also perform integrity checking on the entire RPC message, including the RPC header, and on the arguments or results. Finally, privacy/confidentiality, usually via encryption, is a service available with RPCSEC_GSS. Privacy is provided for the arguments and results. Note that if privacy is selected, integrity, authentication, and identification are enabled. If privacy is not selected, but integrity is selected, authentication and identification are enabled. If integrity and privacy are not selected, but authentication is enabled, identification is enabled. RPCSEC_GSS does not provide identification as a separate service.¶

Although GSS-API has an authentication service distinct from its privacy and integrity services, GSS-API's authentication service is not used for RPCSEC_GSS's authentication service. Instead, each RPC request and response header is integrity protected with the GSS-API integrity service, and this allows RPCSEC_GSS to offer per-RPC authentication and identity. See [RFC2203] for more information.¶

NFSv4.1 client and servers MUST support RPCSEC_GSS's integrity and authentication service. NFSv4.1 servers MUST support RPCSEC_GSS's privacy service.¶

NFSv4.1 clients SHOULD support RPCSEC_GSS's privacy service. Given that it is has never been made clear, as required by the definition of "SHOULD in [RFC2119], it has to be assumed that this statement, appearing in previous specifications has been treated as providing permission for clients not to support RPCSEC_GSS privacy. In light of this situation, it needs to be understood that, with regard to the use of "SHOULD" above, valid reasons to bypass this recommendation are limited to the reliance of implementors on those previous specifications and the difficulty of changing them now.¶

The following consequences need to be kept in mind by those not providing such support:¶

• Any data accessed on connections for which rpc-tls support is not provided will be available, in the clear, to those with the ability to monitor network traffic on a network segment used to effect the access.¶
• The existence of clients without privacy support would make it difficult or impossible to enforce privacy constraints that would otherwise be straightforward. The effect is to undercut the process of security negotiation, which is the only possible way to provide confidentiality when rpc-tls encryption is not in effect.¶

The reader is directed to Section 18.3.1 of [I-D.dnoveck-nfsv4-security] for a more complete discussion of security issues regarding data in flight.¶

6.2. NFSv4.1-specific Details of Security Negotiation

Unlike NFSv4.0, which only has the SECINFO operation, NFSv4.1 has the SECINFO_NO_NAME operation as well. As a result, many of the details of performing security negotiation will different from those in other minor versions and need to be discussed in this document, in the sections below.¶

7.1. Motivation and Overview

Previous versions and minor versions of NFS have suffered from the following:¶

• Lack of support for Exactly Once Semantics (EOS). This includes lack of support for EOS through server failure and recovery.¶
• Limited callback support, including no support for sending callbacks through firewalls, and races between replies to normal requests and callbacks.¶
• Limited trunking over multiple network paths.¶
• Requiring machine credentials for fully secure operation.¶

Through the introduction of a session, NFSv4.1 addresses the above shortfalls with practical solutions:¶

• EOS is enabled by a reply cache with a bounded size, making it feasible to keep the cache in persistent storage and enable EOS through server failure and recovery. One reason than previous revisions of NFS did not support EOS was because some EOS approaches often limited parallelism. As will be explained in Section 7.6, NFSv4.1 supports EOS without unduly limiting parallelism.¶
• The NFSv4.1 client (defined in Section 2.5) creates transport connections and provides them to the server to use for sending callback requests, thus solving the firewall issue (Section 23.34). Races between responses from client requests and callbacks caused by the requests are detected via the session's sequencing properties that are a consequence of EOS (Section 7.6.3).¶
• The NFSv4.1 client can associate an arbitrary number of connections with the session, and thus provide trunking (Section 7.5).¶
• The NFSv4.1 client and server produce a session key independent of client and server machine credentials which can be used to compute a digest for protecting critical session management operations (Section 7.8.3).¶
• The NFSv4.1 client can also create secure RPCSEC_GSS contexts for use by the session's backchannel that do not require the server to authenticate to a client machine principal (Section 7.8.2).¶

A session is a dynamically created, long-lived server object created by a client and used over time from one or more transport connections. Its function is to maintain the server's state relative to the connection(s) belonging to a client instance. This state is entirely independent of the connection itself, and indeed the state exists whether or not the connection exists. A client may have one or more sessions associated with it so that client-associated state may be accessed using any of the sessions associated with that client's client ID, when connections are associated with those sessions. When no connections are associated with any of a client ID's sessions for an extended time, such objects as locks, opens, delegations, layouts, etc. are subject to expiration. The session serves as an object representing a means of access by a client to the associated client state on the server, independent of the physical means of access to that state.¶

A single client may create multiple sessions. A single session MUST NOT serve multiple clients.¶

7.2. NFSv4 Integration

Sessions are part of NFSv4.1 and not NFSv4.0. Normally, a major infrastructure change such as sessions would require a new major version number to an Open Network Computing (ONC) RPC program like NFS. However, because NFSv4 encapsulates its functionality in a single procedure, COMPOUND, and because COMPOUND can support an arbitrary number of operations, sessions have been added to NFSv4.1 with little difficulty. COMPOUND includes a minor version number field, and for NFSv4.1 this minor version is set to 1. When the NFSv4 server processes a COMPOUND with the minor version set to 1, it expects a different set of operations than it does for NFSv4.0. NFSv4.1 defines the SEQUENCE operation, which is required for every COMPOUND that operates over an established session, with the exception of some session administration operations, such as DESTROY_SESSION (Section 23.37).¶

    +-----+--------------+-----------+------------+-----------+----
    | tag | minorversion | numops    |SEQUENCE op | op + args | ...
    |     |   (== 1)     | (limited) |  + args    |           |
    +-----+--------------+-----------+------------+-----------+----

    +------------+-----+--------+-------------------------------+--//
    |last status | tag | numres |status + SEQUENCE op + results |  //
    +------------+-----+--------+-------------------------------+--//
            //-----------------------+----
            // status + op + results | ...
            //-----------------------+----

7.3. Channels

A channel is not a connection. A channel represents a single direction in which ONC RPC requests are sent as part of a session..¶

Each session has one or two channels: the fore channel and the backchannel. Because there are at most two channels per session, and because each channel has a distinct purpose, channels are not assigned identifiers.¶

The fore channel is used for ordinary requests from the client to the server, and carries COMPOUND requests and responses. A session always has a fore channel.¶

The backchannel is used for callback requests from server to client, and carries CB_COMPOUND requests and responses. Whether or not there is a backchannel is decided by the client; however, many features of NFSv4.1 require a backchannel. NFSv4.1 servers MUST support backchannels.¶

Each session has resources for each channel, including separate reply caches (see Section 7.6.1). Note that even the backchannel requires a reply cache (or, at least, a slot table in order to detect retries) because some callback operations are non-idempotent.¶

7.4. Server Scope

Servers each specify a server scope value in the form of an opaque string eir_server_scope returned as part of the results of an EXCHANGE_ID operation. The purpose of the server scope is to allow a group of servers to indicate to clients that a set of servers sharing the same server scope value has arranged to use distinct values of opaque identifiers so that the two servers never assign the same value to two distinct objects. Thus, the identifiers generated by two servers within that set can be assumed compatible so that, in certain important cases, identifiers generated by one server in that set may be presented to another server of the same scope.¶

The use of such compatible values does not imply that a value generated by one server will always be accepted by another. In most cases, it will not. However, a server will not inadvertently accept a value generated by another server. When it does accept it, it will be because it is recognized as valid and carrying the same meaning as on another server of the same scope.¶

When servers are of the same server scope, this compatibility of values applies to the following identifiers:¶

• Filehandle values. A filehandle value accepted by two servers of the same server scope denotes the same object. A WRITE operation sent to one server is reflected immediately in a READ sent to the other.¶
• Server owner values. When the server scope values are the same, server owner value may be validly compared. In cases where the server scope values are different, server owner values are treated as different even if they contain identical strings of bytes.¶

The coordination among servers required to provide such compatibility can be quite minimal, and limited to a simple partition of the ID space. The recognition of common values requires additional implementation, but this can be tailored to the specific situations in which that recognition is desired.¶

Clients will have occasion to compare the server scope values of multiple servers under a number of circumstances, each of which will be discussed under the appropriate functional section:¶

• When server owner values received in response to EXCHANGE_ID operations sent to multiple network addresses are compared for the purpose of determining the validity of various forms of trunking, as described in Section 16.5.2.¶
• When network or server reconfiguration causes the same network address to possibly be directed to different servers, with the necessity for the client to determine when lock reclaim should be attempted, as described in Section 13.4.2.1.¶

When two replies from EXCHANGE_ID, each from two different server network addresses, have the same server scope, there are a number of ways a client can validate that the common server scope is due to two servers cooperating in a group.¶

• If both EXCHANGE_ID requests were sent with RPCSEC_GSS ([RFC2203], [RFC5403], [RFC7861]) authentication and the server principal is the same for both targets, the equality of server scope is validated. It is RECOMMENDED that two servers intending to share the same server scope and server_owner major_id also share the same principal name. In some cases, this simplifies the client's task of validating server scope.¶
• The client may accept the appearance of the second server in the fs_locations or fs_locations_info attribute for a relevant file system. For example, if there is a migration event for a particular file system or there are locks to be reclaimed on a particular file system, the attributes for that particular file system may be used. The client sends the GETATTR request to the first server for the fs_locations or fs_locations_info attribute with RPCSEC_GSS authentication. It may need to do this in advance of the need to verify the common server scope. If the client successfully authenticates the reply to GETATTR, and the GETATTR request and reply containing the fs_locations or fs_locations_info attribute refers to the second server, then the equality of server scope is supported. A client may choose to limit the use of this form of support to information relevant to the specific file system involved (e.g. a file system being migrated).¶

7.5. Trunking

Trunking is the use of multiple connections between a client and server in order to increase the speed of data transfer. NFSv4.1 supports two types of trunking: session trunking and client ID trunking.¶

In the context of a single server network address, it can be assumed that all connections are accessing the same server, and NFSv4.1 servers MUST support both forms of trunking. When multiple connections use a set of network addresses to access the same server, the server MUST support both forms of trunking. NFSv4.1 servers in a clustered configuration MAY allow network addresses for different servers to use client ID trunking.¶

Clients may use either form of trunking as long as they do not, when trunking between different server network addresses, violate the servers' mandates as to the kinds of trunking to be allowed (see below). With regard to callback channels, the client MUST allow the server to choose among all callback channels valid for a given client ID and MUST support trunking when the connections supporting the backchannel allow session or client ID trunking to be used for callbacks.¶

Session trunking is essentially the association of multiple connections, each with potentially different target and/or source network addresses, to the same session. When the target network addresses (server addresses) of the two connections are the same, the server MUST support such session trunking. When the target network addresses are different, the server MAY indicate such support using the data returned by the EXCHANGE_ID operation (see below).¶

Client ID trunking is the association of multiple sessions to the same client ID. Servers MUST support client ID trunking for two target network addresses whenever they allow session trunking for those same two network addresses. In addition, a server MAY, by presenting the same major server owner ID (Section 5.6) and server scope (Section 7.4), allow an additional case of client ID trunking. When two servers return the same major server owner and server scope, it means that the two servers are cooperating on locking state management, which is a prerequisite for client ID trunking.¶

Distinguishing when the client is allowed to use session and client ID trunking requires understanding how the results of the EXCHANGE_ID (Section 23.35) operation identify a server. Suppose a client sends EXCHANGE_IDs over two different connections, each with a possibly different target network address, but each EXCHANGE_ID operation has the same value in the eia_clientowner field. If the same NFSv4.1 server is listening over each connection, then each EXCHANGE_ID result MUST return the same values of eir_clientid, eir_server_owner.so_major_id, and eir_server_scope. The client can then treat each connection as referring to the same server (subject to verification; see Section 7.5.1 below), and it can use each connection to trunk requests and replies. The client's choice is whether session trunking or client ID trunking applies.¶

Session Trunking.

If the eia_clientowner argument is the same in two different EXCHANGE_ID requests, and the eir_clientid, eir_server_owner.so_major_id, eir_server_owner.so_minor_id, and eir_server_scope results match in both EXCHANGE_ID results, then the client is permitted to perform session trunking. If the client has no session mapping to the tuple of eir_clientid, eir_server_owner.so_major_id, eir_server_scope, and eir_server_owner.so_minor_id, then it creates the session via a CREATE_SESSION operation over one of the connections, which associates the connection to the session. If there is a session for the tuple, the client can send BIND_CONN_TO_SESSION to associate the connection to the session.¶

Of course, if the client does not desire to use session trunking, it is not required to do so. It can invoke CREATE_SESSION on the connection. This will result in client ID trunking as described below. It can also decide to drop the connection if it does not choose to use trunking.¶

Client ID Trunking.

If the eia_clientowner argument is the same in two different EXCHANGE_ID requests, and the eir_clientid, eir_server_owner.so_major_id, and eir_server_scope results match in both EXCHANGE_ID results, then the client is permitted to perform client ID trunking (regardless of whether the eir_server_owner.so_minor_id results match). The client can associate each connection with different sessions, where each session is associated with the same server.¶

The client completes the act of client ID trunking by invoking CREATE_SESSION on each connection, using the same client ID that was returned in eir_clientid. These invocations create two sessions and also associate each connection with its respective session. The client is free to decline to use client ID trunking by simply dropping the connection at this point.¶

When doing client ID trunking, locking state is shared across sessions associated with that same client ID. This requires the server to coordinate state across sessions and the client to be able to associate the same locking state with multiple sessions.¶

It is always possible that, as a result of various sorts of reconfiguration events, eir_server_scope and eir_server_owner values may be different on subsequent EXCHANGE_ID requests made to the same network address.¶

In most cases, such reconfiguration events will be disruptive and indicate that an IP address formerly connected to one server is now connected to an entirely different one.¶

Some guidelines on client handling of such situations follow:¶

• When eir_server_scope changes, the client has no assurance that any IDs that it obtained previously (e.g., filehandles) can be validly used on the new server, and, even if the new server accepts them, there is no assurance that this is not due to accident. Thus, it is best to treat all such state as lost or stale, although a client may assume that the probability of inadvertent acceptance is low and treat this situation as within the next case.¶
• When eir_server_scope remains the same and eir_server_owner.so_major_id changes, the client can use the filehandles it has, consider its locking state lost, and attempt to reclaim or otherwise re-obtain its locks. It might find that its filehandle is now stale. However, if NFS4ERR_STALE is not returned, it can proceed to reclaim or otherwise re-obtain its open locking state.¶
• When eir_server_scope and eir_server_owner.so_major_id remain the same, the client has to use the now-current values of eir_server_owner.so_minor_id in deciding on appropriate forms of trunking. This may result in connections being dropped or new sessions being created.¶

7.6. Exactly Once Semantics

[Author Aside]: This section, including some subsections, has been substantially modified from the corresponding section appearing in previous specifications [RFC5661] [RFC8881] and earlier drafts of this document. Change has been driven primarily by the incorrect use of RFC2119-defined keywords, most importantly in the case in which RPC requests need to be aborted, leading to some related changes to clarify the appropriate level of checking for the possibility of false retry. As part of this revised description, it is explained that, given the possibility of requests being aborted, the term "Exactly-once semantics" describes an aspiration and that what is really provided would better be called "at-most-once semantics. Also, the description of retry has been revised to properly use RFC2119 keywords. For more detailed information regarding changes which have been made, see Appendix C.2.1.¶

Via the session, NFSv4.1 offers what is termed "exactly once semantics" (EOS) for requests sent over a channel. EOS is supported on both the fore channel and backchannel.¶

Although this term is well-established and will not be changed, it should be noted that what is actually provided is at-most-once semantics to accommodate the possibility that the client will need to abort RPC requests, remaining unsure about whether the requested actions have been performed one time or not at all.¶

Each COMPOUND or CB_COMPOUND request that is sent with a leading SEQUENCE or CB_SEQUENCE operation needs to be executed by the receiver at most once. This requirement holds regardless of whether the request is sent with reply caching specified (see Section 7.6.1.3). The requirement also holds in the case in which NFSv4.1 is a pNFS data access protocol and the requester is sending the request over a session created between a pNFS data client and pNFS data server. To help understand the need for this requirement, we divide the requests sent to be executed into three categories:¶

• Non-idempotent requests.¶
• Idempotent modifying requests.¶
• Idempotent non-modifying requests.¶

An example of a non-idempotent request is RENAME. Obviously, if a replier executes the same RENAME request twice, and the first execution succeeds, the re-execution will fail. If the replier returns the result from the re-execution, this result is incorrect. Therefore, EOS is required for non-idempotent requests.¶

An example of an idempotent modifying request is a COMPOUND request containing a WRITE operation. Repeated execution of the same WRITE has the same effect as execution of that WRITE a single time. Nevertheless, enforcing EOS for WRITEs and other idempotent modifying requests is necessary to avoid data corruption, which could result from executing the same write request multiple times including some executions that occur after the completion of the first is noted by the requester.¶

Suppose a client sends WRITE A to a noncompliant server that does not enforce EOS, and receives no response, perhaps due to a network partition. The client reconnects to the server and re-sends WRITE A. Now, the server has outstanding two instances of A. The server can be in a situation in which it executes and replies to the retry of A, while the first A is still waiting in the server's internal I/O system for some resource. Upon receiving the reply to the second attempt of WRITE A, the client believes its WRITE is done so it is free to send WRITE B, which overlaps the byte-range of A. When the original A is dispatched from the server's I/O system and executed (thus the second time A will have been written), then what has been written by B can be overwritten and thus corrupted.¶

An example of an idempotent non-modifying request is a COMPOUND containing SEQUENCE, PUTFH, READLINK, and nothing else. The re-execution of such a request will not cause data corruption or produce an incorrect result. Nonetheless, to keep the implementation simple, the replier MUST enforce EOS for all requests, whether or not they are idempotent or modifying.¶

Note that fully complete EOS is not possible unless the server persists the reply cache in stable storage, and unless the server is somehow implemented to never require a restart (indeed, if such a server exists, the distinction between a reply cache kept in stable storage versus one that is not is one without meaning). See Section 8 for a discussion of persistence in the reply cache. Regardless, even if the server does not persist the reply cache, EOS improves robustness and correctness relative to previous versions of NFS because the earlier duplicate request/reply caches were based on the ONC RPC transaction identifier (XID). Section 7.6.1 explains the shortcomings of the XID as a basis for a reply cache and describes how NFSv4.1 sessions improve upon the XID.¶

7.7. RDMA Considerations

A complete discussion of the operation of RPC-based protocols over RDMA transports is in [RFC8166]. A discussion of the operation of NFSv4, including NFSv4.1, over RDMA is in [RFC8267]. Where RDMA is considered, this specification assumes the use of such a layering; it addresses only the upper-layer issues relevant to making best use of RPC/RDMA.¶

                 Sender gather:
     |RPC Request|Pad  bytes|Length| -> |User data...|
     \------+----------------------/      \
             \                             \
              \    Receiver scatter:        \-----------+- ...
         /-----+----------------\            \           \
         |RPC Request|Pad|Length|   ->  |FS buffer|->|FS buffer|->...

7.8. Session Security

7.9. The Secret State Verifier (SSV) GSS Mechanism

The SSV provides the secret key for a GSS mechanism internal to NFSv4.1 that NFSv4.1 uses for state protection. Contexts for this mechanism are not established via the RPCSEC_GSS protocol. Instead, the contexts are automatically created when EXCHANGE_ID specifies SP4_SSV protection. The only tokens defined are the PerMsgToken (emitted by GSS_GetMIC) and the SealedMessage token (emitted by GSS_Wrap).¶

The mechanism OID for the SSV mechanism is iso.org.dod.internet.private.enterprise.Michael Eisler.nfs.ssv_mech (1.3.6.1.4.1.28882.1.1). While the SSV mechanism does not define any initial context tokens, the OID can be used to let servers indicate that the SSV mechanism is acceptable whenever the client sends a SECINFO or SECINFO_NO_NAME operation (see Section 6.2).¶

The SSV mechanism defines four subkeys derived from the SSV value. Each time SET_SSV is invoked, the subkeys are recalculated by the client and server. The calculation of each of the four subkeys depends on each of the four respective ssv_subkey4 enumerated values. The calculation uses the HMAC [RFC2104] algorithm, using the current SSV as the key, the one-way hash algorithm as negotiated by EXCHANGE_ID, and the input text as represented by the XDR encoded enumeration value for that subkey of data type ssv_subkey4. If the length of the output of the HMAC algorithm exceeds the length of key of the encryption algorithm (which is also negotiated by EXCHANGE_ID), then the subkey MUST be truncated from the HMAC output, i.e., if the subkey is of N bytes long, then the first N bytes of the HMAC output MUST be used for the subkey. The specification of EXCHANGE_ID states that the length of the output of the HMAC algorithm MUST NOT be less than the length of subkey needed for the encryption algorithm (see Section 23.35).¶

/* Input for computing subkeys */
enum ssv_subkey4 {
        SSV4_SUBKEY_MIC_I2T     = 1,
        SSV4_SUBKEY_MIC_T2I     = 2,
        SSV4_SUBKEY_SEAL_I2T    = 3,
        SSV4_SUBKEY_SEAL_T2I    = 4
};

The subkey derived from SSV4_SUBKEY_MIC_I2T is used for calculating message integrity codes (MICs) that originate from the NFSv4.1 client, whether as part of a request over the fore channel or a response over the backchannel. The subkey derived from SSV4_SUBKEY_MIC_T2I is used for MICs originating from the NFSv4.1 server. The subkey derived from SSV4_SUBKEY_SEAL_I2T is used for encryption text originating from the NFSv4.1 client, and the subkey derived from SSV4_SUBKEY_SEAL_T2I is used for encryption text originating from the NFSv4.1 server.¶

The PerMsgToken description is based on an XDR definition:¶

/* Input for computing smt_hmac */
struct ssv_mic_plain_tkn4 {
  uint32_t        smpt_ssv_seq;
  opaque          smpt_orig_plain<>;
};

/* SSV GSS PerMsgToken token */
struct ssv_mic_tkn4 {
  uint32_t        smt_ssv_seq;
  opaque          smt_hmac<>;
};

The field smt_hmac is an HMAC calculated by using the subkey derived from SSV4_SUBKEY_MIC_I2T or SSV4_SUBKEY_MIC_T2I as the key, the one-way hash algorithm as negotiated by EXCHANGE_ID, and the input text as represented by data of type ssv_mic_plain_tkn4. The field smpt_ssv_seq is the same as smt_ssv_seq. The field smpt_orig_plain is the "message" input passed to GSS_GetMIC() (see Section 2.3.1 of [RFC2743]). The caller of GSS_GetMIC() provides a pointer to a buffer containing the plain text. The SSV mechanism's entry point for GSS_GetMIC() encodes this into an opaque array, and the encoding will include an initial four-byte length, plus any necessary padding. Prepended to this will be the XDR encoded value of smpt_ssv_seq, thus making up an XDR encoding of a value of data type ssv_mic_plain_tkn4, which in turn is the input into the HMAC.¶

The token emitted by GSS_GetMIC() is XDR encoded and of XDR data type ssv_mic_tkn4. The field smt_ssv_seq comes from the SSV sequence number, which is equal to one after SET_SSV (Section 23.47) is called the first time on a client ID. Thereafter, the SSV sequence number is incremented on each SET_SSV. Thus, smt_ssv_seq represents the version of the SSV at the time GSS_GetMIC() was called. As noted in Section 23.35, the client and server can maintain multiple concurrent versions of the SSV. This allows the SSV to be changed without serializing all RPC calls that use the SSV mechanism with SET_SSV operations. Once the HMAC is calculated, it is XDR encoded into smt_hmac, which will include an initial four-byte length, and any necessary padding. Prepended to this will be the XDR encoded value of smt_ssv_seq.¶

The SealedMessage description is based on an XDR definition:¶

/* Input for computing ssct_encr_data and ssct_hmac */
struct ssv_seal_plain_tkn4 {
  opaque          sspt_confounder<>;
  uint32_t        sspt_ssv_seq;
  opaque          sspt_orig_plain<>;
  opaque          sspt_pad<>;
};

/* SSV GSS SealedMessage token */
struct ssv_seal_cipher_tkn4 {
  uint32_t      ssct_ssv_seq;
  opaque        ssct_iv<>;
  opaque        ssct_encr_data<>;
  opaque        ssct_hmac<>;
};

The token emitted by GSS_Wrap() is XDR encoded and of XDR data type ssv_seal_cipher_tkn4.¶

The ssct_ssv_seq field has the same meaning as smt_ssv_seq.¶

The ssct_encr_data field is the result of encrypting a value of the XDR encoded data type ssv_seal_plain_tkn4. The encryption key is the subkey derived from SSV4_SUBKEY_SEAL_I2T or SSV4_SUBKEY_SEAL_T2I, and the encryption algorithm is that negotiated by EXCHANGE_ID.¶

The ssct_iv field is the initialization vector (IV) for the encryption algorithm (if applicable) and is sent in clear text. The content and size of the IV MUST comply with the specification of the encryption algorithm. For example, the id-aes256-CBC algorithm MUST use a 16-byte initialization vector (IV), which MUST be unpredictable for each instance of a value of data type ssv_seal_plain_tkn4 that is encrypted with a particular SSV key.¶

The ssct_hmac field is the result of computing an HMAC using the value of the XDR encoded data type ssv_seal_plain_tkn4 as the input text. The key is the subkey derived from SSV4_SUBKEY_MIC_I2T or SSV4_SUBKEY_MIC_T2I, and the one-way hash algorithm is that negotiated by EXCHANGE_ID.¶

The sspt_confounder field is a random value.¶

The sspt_ssv_seq field is the same as ssvt_ssv_seq.¶

The field sspt_orig_plain field is the original plaintext and is the "input_message" input passed to GSS_Wrap() (see Section 2.3.3 of [RFC2743]). As with the handling of the plaintext by the SSV mechanism's GSS_GetMIC() entry point, the entry point for GSS_Wrap() expects a pointer to the plaintext, and will XDR encode an opaque array into sspt_orig_plain representing the plain text, along with the other fields of an instance of data type ssv_seal_plain_tkn4.¶

The sspt_pad field is present to support encryption algorithms that require inputs to be in fixed-sized blocks. The content of sspt_pad is zero filled except for the length. Beware that the XDR encoding of ssv_seal_plain_tkn4 contains three variable-length arrays, and so each array consumes four bytes for an array length, and each array that follows the length is always padded to a multiple of four bytes per the XDR standard.¶

For example, suppose the encryption algorithm uses 16-byte blocks, and the sspt_confounder is three bytes long, and the sspt_orig_plain field is 15 bytes long. The XDR encoding of sspt_confounder uses eight bytes (4 + 3 + 1-byte pad), the XDR encoding of sspt_ssv_seq uses four bytes, the XDR encoding of sspt_orig_plain uses 20 bytes (4 + 15 + 1-byte pad), and the smallest XDR encoding of the sspt_pad field is four bytes. This totals 36 bytes. The next multiple of 16 is 48; thus, the length field of sspt_pad needs to be set to 12 bytes, or a total encoding of 16 bytes. The total number of XDR encoded bytes is thus 8 + 4 + 20 + 16 = 48.¶

GSS_Wrap() emits a token that is an XDR encoding of a value of data type ssv_seal_cipher_tkn4. Note that regardless of whether or not the caller of GSS_Wrap() requests confidentiality, the token always has confidentiality. This is because the SSV mechanism is for RPCSEC_GSS, and RPCSEC_GSS never produces GSS_wrap() tokens without confidentiality.¶

There is one SSV per client ID. There is a single GSS context for a client ID / SSV pair. All SSV mechanism RPCSEC_GSS handles of a client ID / SSV pair share the same GSS context. SSV GSS contexts do not expire except when the SSV is destroyed (causes would include the client ID being destroyed or a server restart). Since one purpose of context expiration is to replace keys that have been in use for "too long", hence vulnerable to compromise by brute force or accident, the client can replace the SSV key by sending periodic SET_SSV operations, which is done by cycling through different users' RPCSEC_GSS credentials. This way, the SSV is replaced without destroying the SSV's GSS contexts.¶

SSV RPCSEC_GSS handles can be expired or deleted by the server at any time, and the EXCHANGE_ID operation can be used to create more SSV RPCSEC_GSS handles. Expiration of SSV RPCSEC_GSS handles does not imply that the SSV or its GSS context has expired.¶

The client MUST establish an SSV via SET_SSV before the SSV GSS context can be used to emit tokens from GSS_Wrap() and GSS_GetMIC(). If SET_SSV has not been successfully called, attempts to emit tokens MUST fail.¶

The SSV mechanism does not support replay detection and sequencing in its tokens because RPCSEC_GSS does not use those features (see "Context Creation Requests", Section 5.2.2 of [RFC2203]). However, Section 7.10 discusses special considerations for the SSV mechanism when used with RPCSEC_GSS.¶

7.10. Security Considerations for RPCSEC_GSS When Using the SSV Mechanism

When a client ID is created with SP4_SSV state protection (see Section 23.35), the client is permitted to associate multiple RPCSEC_GSS handles with the single SSV GSS context (see Section 7.9). Because of the way RPCSEC_GSS (both version 1 and version 2, see [RFC2203] and [RFC5403]) calculate the verifier of the reply, special care must be taken by the implementation of the NFSv4.1 client to prevent attacks by a man-in-the-middle. The verifier of an RPCSEC_GSS reply is the output of GSS_GetMIC() applied to the input value of the seq_num field of the RPCSEC_GSS credential (data type rpc_gss_cred_ver_1_t) (see Section 5.3.3.2 of [RFC2203]). If multiple RPCSEC_GSS handles share the same GSS context, then if one handle is used to send a request with the same seq_num value as another handle, an attacker could block the reply, and replace it with the verifier used for the other handle.¶

There are multiple ways to prevent the attack on the SSV RPCSEC_GSS verifier in the reply. The simplest is believed to be as follows.¶

• Each time one or more new SSV RPCSEC_GSS handles are created via EXCHANGE_ID, the client SHOULD send a SET_SSV operation to modify the SSV. By changing the SSV, the new handles will not result in the re-use of an SSV RPCSEC_GSS verifier in a reply.¶
• When a requester decides to use N SSV RPCSEC_GSS handles, it SHOULD assign a unique and non-overlapping range of seq_nums to each SSV RPCSEC_GSS handle. The size of each range SHOULD be equal to MAXSEQ / N (see Section 5 of [RFC2203] for the definition of MAXSEQ). When an SSV RPCSEC_GSS handle reaches its maximum, it SHOULD force the replier to destroy the handle by sending a NULL RPC request with seq_num set to MAXSEQ + 1 (see Section 5.3.3.3 of [RFC2203]).¶
• When the requester wants to increase or decrease N, it SHOULD force the replier to destroy all N handles by sending a NULL RPC request on each handle with seq_num set to MAXSEQ + 1. If the requester is the client, it SHOULD send a SET_SSV operation before using new handles. If the requester is the server, then the client SHOULD send a SET_SSV operation when it detects that the server has forced it to destroy a backchannel's SSV RPCSEC_GSS handle. By sending a SET_SSV operation, the SSV will change, and so the attacker will be unavailable to successfully replay a previous verifier in a reply to the requester.¶

Note that if the replier carefully creates the SSV RPCSEC_GSS handles, the related risk of a man-in-the-middle splicing a forged SSV RPCSEC_GSS credential with a verifier for another handle does not exist. This is because the verifier in an RPCSEC_GSS request is computed from input that includes both the RPCSEC_GSS handle and seq_num (see Section 5.3.1 of [RFC2203]). Provided the replier takes care to avoid re-using the value of an RPCSEC_GSS handle that it creates, such as by including a generation number in the handle, the man-in-the-middle will not be able to successfully replay a previous verifier in the request to a replier.¶

7.11. Session Mechanics - Steady State

7.12. Session Inactivity Timer

The server MAY maintain a session inactivity timer for each session. If the session inactivity timer expires, then the server MAY destroy the session. To avoid losing a session due to inactivity, the client MUST renew the session inactivity timer. The length of session inactivity timer MUST NOT be less than the lease_time attribute (Section 11.12.1.11). As with lease renewal (Section 13.3), when the server receives a SEQUENCE operation, it resets the session inactivity timer, and MUST NOT allow the timer to expire while the rest of the operations in the COMPOUND procedure's request are still executing. Once the last operation has finished, the server MUST set the session inactivity timer to expire no sooner than the sum of the current time and the value of the lease_time attribute.¶

7.13. Session Mechanics - Recovery

7.14. Parallel NFS and Sessions

A client and server can potentially be a non-pNFS implementation, a metadata server implementation, a data server implementation, or two or three types of implementations. The EXCHGID4_FLAG_USE_NON_PNFS, EXCHGID4_FLAG_USE_PNFS_MDS, and EXCHGID4_FLAG_USE_PNFS_DS flags (not mutually exclusive) are passed in the EXCHANGE_ID arguments and results to allow the client to indicate how it wants to use sessions created under the client ID, and to allow the server to indicate how it will allow the sessions to be used. See Section 18.1 for pNFS sessions considerations.¶

8.1. Need for Feature Respecification

The original material has been modified substantially and extended in order address the three items listed below. As a result, the focus of the section has shifted to include all elements relevant to persistence across server failure, rather than dealing only with reply cache issues.¶

• Eliminate elements of the description that made the feature essentially unimplementable. These include overbroad requirements for atomicity and the assumption that all requests needed to be continued across server restart.¶

• Appropriately discuss lock persistence and its relation to reply cache persistence and session persistence.¶

• Provide new material describing the process by which the client finds out about the presence of persistence- related features in the event of server restart.¶

8.2. Persistence of Reply Cache

Since the reply cache is bounded, it is possible for the reply cache to be maintained in persistent storage so that it can be made available across server restarts. When the server undertakes to provide this support when the session is created (see Section 23.36 for details), it is uncertain whether what will provided is either:¶

• Persistence of the reply cache only.¶

• Persistent of the session including its membership within the clientid of which it is a part.¶

The replier needs to persist the following information if it agreed to provide persistence for the session (when the session was created;¶

• The session ID.¶

• The slot table¶

  This need to include the sequence ID and cached reply for each slot.¶

• Information about the connection(s) used by the server with is sufficient to determine whether a client attempting to connect after a server Restart.¶

This sort of information can be used to provide either of the two distinct sorts of session-based persistence. The server provides no specific commitment to provide either of these, although, as described in Section 8.4, the client will be able to determine which form, if any, has actually been provided, and respond appropriately¶

In describing persistence-related semantics it will be helpful to define the following two terms:¶

• An operation is said "reply-caching relevant" if it is either non-idempotent, modifying, or is the final operation (including the case of request termination because of an error) of a request that is specifically requested to be cached (i.e., has a SEQUENCE operation with sa_cachethis set to true).¶

• A request is said "reply-caching relevant" if it contains one or more operations which are non-idempotent or modifying or it is specifically requested to be cached (i.e., has a SEQUENCE operation with sa_cachethis set to true).¶

Whichever form of session-based persistence is provided by the server, any requests the client retries after the server restarts will return the results that are cached in the reply cache, However, these two forms differ with regard to the handling of new requests and the possible use of clientid-based persistence facilities:¶

• If only reply cache persistence is provided, any new requests will fail with NFS4ERR_DEADSESSION being returned as the result of the initial SEQUENCE operation.¶

  Because there is no need to use the sequence id to order future request the server does not need to update persistent storage, if two successive requests using the same slot are both not reply-caching relevant, although it does if one or both of the request is reply-cache relevant.¶

• If session persistence is provided, the existing session can be used after connection re-establishment to support the execution of new requests so that the client will be able to continue just as it would have if no session restart had occurred.¶

A persistent reply cache places certain demands on the server. Although it is not it is not necessary to execute successive operations within a COMPOUND atomically, the transfer of the results of a set of operations and their installation in the persistent cache must be immediate following the execution of any reply-cache relevant operation so that it is impossible for operations to be executed or have other visible effects while not appearing in persistent reply cache.¶

If a client were to retry a sequence of operations that was issued to the server, the only acceptable outcomes are:¶

• an indication that the request is still being processed.¶

• a cached reply reflecting the completion of the request,¶

• a cached reply reflecting the interruption of the request due to server failure.¶

• an indication that the client ID or session has been lost (indicating a catastrophic loss of the reply cache or a session that has been deleted because the client failed to use the session for an extended period of time).¶

The possibility exists of situations in which a server could fail and restart in the middle of a COMPOUND procedure that contains one or more non-idempotent or idempotent-but-modifying operations. If the server allows COMPOUND procedures to be continued after server failure, it creates significantly greater challenges for the execution of such requests and the atomic placement of results in the reply cache.¶

When a server providing a persistent reply cache does not continue a COMPOUND procedure that was interrupted by a server failure, the error NFS4ERR_DEADSESSION is returned on the last operation which was executed.¶

8.3. Persistence of Locking State

Servers may make locking state available across a server restart in a number of ways including the following:¶

• Data related to the existence of locks and their corresponding characteristics can be stored in persistent RAM and then used after restart if the address of that storage can be reliably obtained after restart.¶

• The storage of locking-related state can be integrated with the file system by treating locking state in the same fashion used for other metadata.¶

• Locking state information may be periodically logged to block-based low-latency persistent storage with logging of individual updates.¶

Although the details will vary with the means of providing persistence that is adopted, it is important that locking state made available across the server restart be consistent with locking state reflected in the results of requests made by clients.¶

The simplest part of this is to ensure that all locking state changes are effectively made available persistently before returning to the requester. In addition, when lock state additions or deletions are reflected in the processing of other operations, the state changes must be available persistently before allowing or denying some operation done by another client. For example, when opens denying write prevent file removal, granting such opens or doing corresponding closes need to be reflected persistently before denying or allowing corresponding file removal. Similar consideration apply to doing IO when mandatory byte-range locks are supported¶

The following items need to be kept in mind:¶

• There is no commitment by the server to provide this persistence and it may be dropped if for a particular client if unusual situations make it advisable.¶

  This decision is made separately for each client so that it is possible there will be server restarts where some, but not all, clients have persistent locking state available.¶

• While the fact that a reclaim on a reclaimable lock is part of the locking state which is to be persistent, the client's state of awareness of that need not be.¶

  There is thus no need for the reclaiming client to inform the server that it has completed specific individual reclaims after receiving the response.¶

8.4. Client Handling of Server Failure When Persistence Can be Used

When server failure occurs, the connection to the client will be disconnected and the client can then find out, as described below, whether server failure has occurred and what steps are necessary to continue use of the client with minimal disruption to those using the client.¶

This process includes the potential use of a persistent reply cache, as described in Section 8.5. The same process is followed depending on whether the server provided only a persistent reply cache or full session persistence.¶

If the server did not promise any session persistence, the client instead immediately does an EXCHANGE_ID followed by a CREATE_SESSION. On the other hand, if there was a possible use of a persistent reply cache, the use of EXCHANGE_ID/CREATE_SESSION is conditional and only happens if a new request has been completed with the error NFS4ERR_STALECLIENTID.¶

In either case, the next step depends on whether the clientid is the same as the one before the disconnection. If it is, then recovery is complete and new requests can be issued. This could happen if there were no server restart but also could if a combination of session-based and clientid-based persistence allowed the server failure to be dealt with essentially transparently.¶

In the case in which the clientid is different, the client need to reclaim its locks, as described in Section 8.6.¶

Even in the case in which lock persistence is available for a client, it is still possible that attempts to obtain new locks will fail with NFS4ERR_GRACE if other clients do not have their locks made available persistently.¶

8.5. Client Use of Session-based Persistence

After the connection to the server is re-established, the server will try to re-establish the connection, as the connection breakage occurred at a lower layer, without server restart. Although it is theoretically possible for an intermediary to hide such a disconnection, it would cause problems if it were to do so and the client had no knowledge of the server failure The discussion here assumes that no such disconnection-hiding implementation is in effect¶

After re-establishing the connection to the server, the client would initially attempt to continue use of the session, since it has no knowledge of whether the disconnection was the result of a server restart. If persistence not was requested when creating the session or the server indicated it was not present, then the client can legitimately conclude that EOS semantics was not available across server restart and needs to operate in that environment.¶

The continued use of the existing session could include both retries of requests issued before the disconnection and issuing new requests. As a result, the discussion below will deal with both type of requests. Given that context, one needs to note the following:¶

• Whether a given request is a retry or a new one may be judged differently by the client and the server.¶

  While it is virtually certain that a new request issued by the client will be perceived as such by the server, the reverse is not the case. Retries issued by the client might be perceived as new requests, if the original requests was lost before it was executed or its existence was noted in persistent storage.¶

• Although it might be desirable for a client to obtain information about existing requests before issuing new one, the discussion will not assume that clients take steps to prevent new requests from being issued.¶

  Since retries, as perceived by the client, may be considered as new requests by the server, the prevention of new requests by the client does not ensure that the server will not see and respond to such requests.¶

After re-establishing the connection, the client will be able to issue requests, including retries of requests already issued before the disconnection occurred. These retries need to be issued since there is no way the results of these requests could be communicated back to the client in the absence of a retry since the connection on which it was received no longer exists.¶

When responses to these requests are received, what is to be done depends primarily about the error, if any, associated with the response:¶

• In all the cases except the two special error codes noted in the bulleted items below, including receiving no error, the client can conclude that the request was executed to completion as reflected in the response. By design, the client is not aware of whether the execution occurred before or after the serve restart, or whether a server restart, in fact, occurred. However, if persistence was requested when the session was created and the server indicated it was present, the client can assume that the request was executed exactly once with the result reflected in the response.¶

  When this is the result that is returned for new requests, it can be because the server has provided full session persistence or because no server restart has occurred. In the former case, it must be true that the server has provided persistent storge of locking state for the d associated clientid since, if it had not, the error NFS4ERR_STALECLIENTID would have been returned.¶

• In the case that NFS4ERR_DEADSESSION is returned on the SEQUENCE operation, the most likely cause is that the request was, from the server's point of view, a new request and that session persistence was not provided by the server. In this case, the current request should be deferred until the results of all retried requests known to the client have been resolved. Others that are considered new by the server also need to be deferred until are reply cache information is obtained.¶

  In the case that NFS4ERR_DEADSESSION is returned on another operation, the request is one that was discontinued as a result of server restart. It is most likely that the request was one that contained more than one non-idempotent or modifying operations, with the server failing after one had been completed but before later operations were started. In this case the client has been informed of a partially complete request and needs to issue a new request to include the operations that were not performed as part of the initial request.¶

• In the case that the error NFS4ERR_STALECLIENTID is returned, the server has recognized a new request but was unable to continue its execution because the locking information it would use has been destroyed as part of the server restart. This can occur if no persistence was provided for the session, if the persistence was limited to the reply cache or if there was session persistence and client locking state was not maintained persistently.¶

  In this case lock recovery will be required but it will need to be delayed until all requests that were issued before the disconnection have been marked completed using the persisted reply cache.¶

Once the existing pending requests are disposed of, the client can proceed to doing new requests, although it might have to do lock recovery first. This can occur after a persistent reply cache is used to provide EOS or after it is found that there is no session persistence provided by the server.¶

8.6. Client Use of Clientid-based Persistence

At this point, lock recovery needs to begin if a new request is processed and completes returning the error NFS4ERR_STALECLIENTID. If no new requests have been issued at this point, the client can issue a request consisting only of a SEQUENCE operation to provide a test. If NFS4ERR_STALECLIENTID is not returned then the client will assume either that there has been no server restart or thar server restart as been accompanied with the recovery of locking state for the current clientid. Otherwise, lock recovery can be done as part of a server-provided grace period. The following three steps need to be taken:¶

When lock recovery is necessary, the client need to inform the new server of the existence of its locks before using stateids it obtained before the server restart. This process is referred to as reclaiming the client's locks, which is accomplished using the method listed below, depending on the type of lock to be reclaimed.¶

• Opens can generally be reclaimed by doing an OPEN with the claim type CLAIM_PREVIOUS.¶

  This includes the case of opens associated with delegation. For details, see Section 15.2.1,¶

  There is no specific way to reclaim delegations that have no associated open. In such cases, the client can open the file asking for an associated delegation, and return it immediately¶

• To reclaim byte-range locks, a LOCK operation with the reclaim parameter set to true is used.¶

  The associated open will need to be reclaimed first.¶

• There is no provision regarding reclaiming of layouts and thus no way to obtain them during a grace period.¶

  As a result, in case in which locking state is not made available by the server across a server failure, use of the data server is not immediately available and the client is best off doing IO through the MDS until obtaining needed layouts once the rest of lock reclamation is complete.¶

Once all reclaimable locks have been reclaimed, the client needs to do a global RECLAIM_COMPLETE to indicate that process is complete. The is necessary to allow new locks to be obtained. However, even after this done, such requests might still be rejected with NFS4ERR_GRACE if other clients have not completed their lock reclamations.¶

11.1. Categorization of File Attributes

In order to clarify the requirements for server support of particular attributes, and to provide guidance for clients dealing with non-support of particular attributes, all NFSv4.1 attributes are divided into the groups listed below:¶

All of these attributes are accommodated in the NFSv4.1 protocol by a specific, well-defined encoding and are identified by a number. They are interrogated by setting a bit in the bit vector sent in a GETATTR, request. The server response includes a bit vector to indicate which attributes were returned in the response.¶

The following attribute categories are defined:¶

• REQUIRED attributes, as discussed in Section 11.4.¶
• OPTIONAL attributes, as discussed in Section 11.5.¶
• Experimental attributes, as discussed in Section 11.6.¶

New attributes of any of these categories may be added to the NFSv4 protocol as part of a new minor version by publishing a Standards Track RFC that allocates a new attribute number value and defines the encoding for the attribute. In addition, new minor versions can move attributes between categories or make formerly OPTIONAL and Experimental attributes MANDATORY to NOT implement. Similarly, OPTIONAL attributes may be added to an existing extensible version by publishing a Standards Track RFC that allocates a new attribute number value and defines the encoding for the attribute. See [RFC8178] for further details¶

11.2. Changes in the Categorization of File Attributes

The categorization of file attributes appearing in this specification differs from that previously published for a number of reasons:¶

• The description of the attributes for which support is not REQUIRED no longer uses the RFC2119 keyword "RECOMMENDED" as this is not in accord with the definition of that term in [RFC2119].¶

  We now describe such attributes as OPTIONAL, leaving it to server to decide which are worthy of support and to clients to decide whether they wish to use servers on which they are not supported.¶

• The categorization of requirements/recommendation as to support for authorization-related attributes is now the responsibility of the NFSv4-wide security documents, to be derived from [I-D.dnoveck-nfsv4-security] and [I-D.dnoveck-nfsv4-acls].¶

  Currently, given the likely lack of agreement on the semantics of ACLs, it is likely that acl would best be described as an Experimental attribute. See Section 11.3 for further discussion.¶

As one illustration of the new approach to these matters, and its differences from older approaches, let us consider the following statement from Section 5.1 of [RFC8881]. Referring to the REQUIRED attributes, it states:¶

• The client is expected to be able to function with an attribute set limited to these attributes. With just the REQUIRED attributes some client functionality may be impaired or limited in some ways. In the case of servers not supporting the owner, mode, or acl-related attributes, there would be no ability to provide substantial security-related functionality.¶

This expectation was not a reasonable one when first formulated and as the NFSv4 protocols have been developed, there have never been any cases of it being realized. There is no reason to implement a server without the minimal authorization-related attributes derived from NFSv3 and no point in working to develop clients capable of interoperating with it. There is no motivation for the working group to devote any time to defining how such a combination is to operate or for implementers to experiment to try to implement remote file access without any meaningful authorization process.¶

Further, the above also seems to conflict with the following, appearing in Section 5.2 of [RFC8881]:¶

• It is expected that servers will support all attributes they comfortably can and only fail to support attributes that are difficult to support in their operating environments.¶

Together, these imply that there are operating environments in which it difficult to support all of mode, owner, group, and acl attributes. It is hard to believe that any such environments exist or that there would be any point in implementing an NFSv4.1 server using then, if they did exist.¶

11.3. Categorization of Authorization-related Attributes

This section provides an overview of the issues in involved in appropriately categorizing the authorization-related attributes, although the final categorization of these will appear in NFSv4-wide security documents, expected to be based on [I-D.dnoveck-nfsv4-security] and [I-D.dnoveck-nfsv4-acls].¶

Authorization-related attributes that are part of NFSv4.1 can be divided into those connected to he POSIX-based authorization model used in NFSv3 and those related to the use of ACLs to provide a more flexible authorization model. Within the context of NFSv4.1, the following should be noted:¶

• The attributes mode, owner, and owner_group need to be considered REQUIRED, as they are in [I-D.dnoveck-nfsv4-security].¶

  This is despite the fact that previous specifications have considered these attributes as OPTIONAL, although the word "RECOMMENDED" was sometime used. In any case, the new categorization in [I-D.dnoveck-nfsv4-acls] has to be considered dispositive both with regard to NFSv4.1 an other minor versions.¶

• The attributes acl, sacl, and dacl, although designated as OPTIONAL, have never been documented in a manner allowing effective client-server interoperability, suggesting that they would more appropriately be designated as "Experimental".¶

  While it is possible that tightening of the specifications being done in [I-D.dnoveck-nfsv4-security] and [I-D.dnoveck-nfsv4-acls] as part of the rfc5661bis effort might allow this to change, that is not yet assured¶

  In any case, efforts to provide a path to interoperability will continue and might affect this categorization in later minor versions, even if NFSv4.1 is not affected. See [I-D.dnoveck-nfsv4-acls] for details,¶

• The attribute aclsupport is appropriately designated as OPTIONAL, as it in [I-D.dnoveck-nfsv4-security].¶

11.4. REQUIRED Attributes

These MUST be supported by every NFSv4.1 client and server in order to ensure a minimum level of interoperability. The server MUST store and return these attributes when requested. A client may ask for the value of any of these attributes to be returned by setting a bit in the GETATTR request, and the server MUST return their value.¶

The client is expected to be able to function with an attribute set limited to these attributes. With just the REQUIRED attributes some client functionality may be unavailable or functionally limited .¶

11.5. OPTIONAL Attributes

These attributes are understood well enough to warrant support in the NFSv4.1 protocol. However, they might not be supported on all servers or used by all clients. A client may ask for any of these attributes to be returned by setting a bit in the GETATTR request but need to handle the case where the server does not return them. A client MAY ask for the set of attributes the server supports within a given file system and has no reason to request attributes the server does not support. A server is REQUIRED to be deal with requests for unsupported attributes by not returning values for them rather than by considering the request an error.¶

Previous versions of the NFSv4.1 specification [RFC5661] [RFC8881] have described these attributes as "RECOMMENDED" even though that description is not accord with [RFC2119]. The NFSv4.0 specification [RFC7530] still uses "RECOMMENDED" although explicitly disclaiming the assumption that the RFC2119 definition applies in this case. The description of these attribute as OPTIONAL connects them appropriately to provisions for protocol extension and minor versioning in which attributes are to be treated as OPTIONAL.¶

11.6. Experimental Attributes

While the vast majority of attributes are, as described in Section 11.5, "understood well enough to warrant support in the NFSv4.1 protocol", it appears to be the case that, for several attributes, that understanding was never properly recorded in existing NFSv4.1 specification documents. While it might be possibly to rectify that issue before eventual publication of this document, the likely existence of multiple incompatible implementations of such attributes make that unlikely¶

Although the existence of such attributes has never been acknowledged before as part of the categorization of NFSv4 attributes. Nevertheless, such attributes have existed in all NFSv4 minor versions and the necessary clarification, if it occurs, is not likely to be complete for some time.¶

While the intention has always been that attribute not be included in Proposed Standards unless they are described adequately to allow interoperable implementation to be developed. Despite that intention, such attributes have been included in multiple minor versions. Given the need to correct that situation, we need to be clear about the issues that have led to these unfortunate situations, so that we can, over time, address them.¶

11.7. Named Attributes

These attributes are not supported by direct encoding in the NFSv4 protocol but are accessed using string names rather than numbers and each corresponds to an uninterpreted stream of bytes that is stored in its own file system object. The namespace for these attributes may be accessed by using the OPENATTR operation as described below. The OPENATTR operation returns a filehandle for a "named attribute directory", and further perusal and modification of the namespace may be done using operations that work on more typical directories, subject to restrictions discussed below. In particular, READDIR may be used to get a list of such named attributes, and LOOKUP and OPEN may select a particular attribute. Creation of a new named attribute can be accomplished using an OPEN specifying file creation.¶

OPENATTR takes a filehandle for the object and returns the filehandle for the attribute directory. The filehandle for the named attributes designates a directory object accessible by LOOKUP or READDIR and contains files whose names identify the named attributes and whose data bytes are the value of those attributes. For example:¶

Named attributes are intended for data needed by applications rather than by NFS client implementations. NFS implementers who wish to define new attributes need to specify them as OPTIONAL attributes using the protocol extension facilities specified in [RFC8178].¶

Once an OPEN is done, named attributes may be examined and changed using READ and WRITE operations referencing the filehandles and stateids returned by OPEN.¶

Named attributes may have their own (non-named) attributes. Each of these objects MUST have all of the REQUIRED attributes and may have additional attributes which are not REQUIRED. However, the sets of supported attributes for named attributes need not be, and typically will not be, as large as that for other objects in that file system. Nevertheless, the value of the supported_attrs attribute should reflect the supported attributes for the file system and will not reflect the restricted attribute sets for these special objects.¶

Named attributes and the named attribute directories can be the target of delegations (in the case of the named attribute directory, these will be directory delegations). However, since granting of delegations is at the server's discretion, a server need not support delegations on named attributes or on named attribute directories.¶

Support for named attributes is OPTIONAL and clients need to be prepared to deal with servers that do not support them. However, clients are entitled to assume that if OPENATTR is supported, there will be support for arbitrarily named attributes, rather than support for a few specific names known to the server. If a server does support named attributes, a client that is also able to handle them should be able to copy a file's data and metadata with complete transparency from one location to another since names allowed for regular directory entries are expected to be valid for named attribute names as well.¶

In NFSv4.1, the structure of named attribute directories is restricted in a number of ways, in order to prevent the development of non-interoperable implementations in which some servers support a fully general hierarchical directory structure for named attributes while others support a limited, non-hierarchal structure for named attributes. In such a mixed environment, clients or applications might come to depend on non-portable extensions. The restrictions are:¶

• CREATE is not allowed in a named attribute directory. Thus, such objects as symbolic links and special files are not allowed to be named attributes. Further, directories may not be created in a named attribute directory, so a hierarchical structure of named attributes for a single object is not allowed.¶
• If OPENATTR is done on a named attribute directory or on a named attribute, the server MUST return NFS4ERR_WRONG_TYPE.¶
• Doing a RENAME of a named attribute to a different named attribute directory or to an ordinary (i.e., non-named-attribute) directory is not allowed.¶
• Creating hard links between named attribute directories or between named attribute directories and ordinary directories is not allowed.¶

Names of attributes will not be controlled by this document or other IETF Standards Track documents, beyond what is necessary to regulate the names of files within directories to handle internationalization and case-insensitivity. See Section 27.2 for further discussion.¶

11.9. Set-Only and Get-Only Attributes

Some of the protocol-defined attributes are set-only; i.e., they can be set via SETATTR but not retrieved via GETATTR. Similarly, some protocol-defined attributes are get-only; i.e., they can be retrieved via GETATTR but not set via SETATTR. If a client attempts to set a get-only attribute or get a set-only attributes, the server MUST return NFS4ERR_INVAL.¶

11.10. REQUIRED Attributes - List and Definition References

The list of REQUIRED attributes appears in Table 4. The meaning of the columns of the table are:¶

Name:

The name of the attribute.¶

Id:

The number assigned to the attribute. In the event of conflicts between the assigned number and [RFC5662], the latter is likely authoritative, but should be resolved with Errata to this document and/or [RFC5662]. See [errata] for the Errata process.¶

Data Type:

The XDR data type of the attribute.¶

Acc:

Access allowed to the attribute. R means read-only (GETATTR may retrieve, SETATTR may not set). W means write-only (SETATTR may set, GETATTR may not retrieve). R W means read/write (GETATTR may retrieve, SETATTR may set).¶

Defined in:

The section of this specification that describes the attribute.¶

11.11. OPTIONAL Attributes - List and Definition References

The OPTIONAL attributes are defined in Table 5. The meanings of the column headers are the same as Table 4; see Section 11.10 for the meanings.¶

11.12. Attribute Definitions

11.13. Interpreting owner and owner_group

The attributes "owner" and "owner_group" (and also users and groups within the "acl" attribute) are transferred in the form of a UTF-8 string. This string can be used to identify users and groups in several ways:¶

• A string of the form "name@domain" can be used to give a user or group name together with a domain in which those are defined.¶

  This form provides greater degree of extensibility than was possible in NFSv3 which limited these identifiers to 32-bit unsigned integers whose values are all centrally administered as members within a common domain.¶

• Numeric ids converted to string form.¶

  Using this format maintains the strengths and weaknesses of the NFSv3 approach.¶

The following issues are relevant in selected the form to use.¶

• The use of the form "name@domain" provides greater flexibility, both with regard to the number of users that can be accommodated and to the management of multiple sets of users in separate domains.¶

  Taking advantage of this flexibility often requires extensive work because of limitations of the API's used to reference users and groups.¶

• The use of the form "name@domain" allows clients and servers to work together even if they have different internal formats for user and groups.¶

  In many cases, there is no need for such mapping.¶

  Providing this mapping requires extra implementation and raises potential security issues.¶

For detailed discussions regarding which of the forms clients and server are to use for these values, see Section 5.1 of [I-D.dnoveck-nfsv4-security].¶

11.14. Character Case Attributes

With respect to the case_insensitive and case_preserving attributes, each UCS-4 character (which UTF-8 encodes) can be mapped to an equivalent character of different case or compared in a case-insensitive manner. The details vary based on the Unicode version implemented by the server for the current file system. Details of the process and how the client can best deal with uncertainty about the process will be discussed in the NFSv4-wide internationalization document (See [I-D.ietf-nfsv4-internationalization] for the latest version)¶

11.15. Directory Notification Attributes

As described in Section 23.39, the client can request a minimum delay for notifications of changes to attributes, but the server is free to ignore what the client requests. The client can determine in advance what notification delays the server will accept by sending a GETATTR operation for either or both of two directory notification attributes. When the client calls the GET_DIR_DELEGATION operation and asks for attribute change notifications, it should request notification delays that are no less than the values in the server-provided attributes.¶

11.16. pNFS Attribute Definitions

11.17. Retention Attributes

Retention is a concept whereby a file object can be placed in an immutable, undeletable, unrenamable state for a fixed or infinite duration of time. Once in this "retained" state, the file cannot be moved out of the state until the duration of retention has been reached.¶

When retention is enabled, retention MUST extend to the data of the file, and the name of file. The server MAY extend retention to any other property of the file, including any subset of REQUIRED, OPTIONAL, and named attributes, with the exceptions noted in this section.¶

Servers MAY support or not support retention on any file object type.¶

The five retention attributes are explained in the next subsections.¶

const RET4_DURATION_INFINITE    = 0xffffffffffffffff;
struct retention_get4 {
        uint64_t        rg_duration;
        nfstime4        rg_begin_time<1>;
};

struct retention_set4 {
        bool            rs_enable;
        uint64_t        rs_duration<1>;
};

11.18. Access Control Attributes

The use of the access control attributes are fully described in various sections of the NFSv4-wide security documents [I-D.dnoveck-nfsv4-security] [I-D.dnoveck-nfsv4-acls].¶

• The mode, mode_set_masked, owner, and owner_group attributes are described in Sections 5.3.1 though 5.3.4 of [I-D.dnoveck-nfsv4-security].¶

• The acl, aclsupport, sacl, and dacl attributes are described in Sections 3.4, 3.5, 3.6, and 3.8 of [I-D.dnoveck-nfsv4-acls].¶

12.1. Server Exports

On a UNIX server, the namespace describes all the files reachable by pathnames under the root directory or "/". On a Windows server, the namespace constitutes all the files on disks named by mapped disk letters. NFS server administrators rarely make the entire server's file system namespace available to NFS clients. More often, portions of the namespace are made available via an "export" feature. In previous versions of the NFS protocol, the root filehandle for each export is obtained through the MOUNT protocol; the client sent a string that identified the export name within the namespace and the server returned the root filehandle for that export. The MOUNT protocol also provided an EXPORTS procedure that enumerated the server's exports.¶

12.2. Browsing Exports

The NFSv4.1 protocol provides a root filehandle that clients can use to obtain filehandles for the exports of a particular server, via a series of LOOKUP operations within a COMPOUND, to traverse a path. A common user experience is to use a graphical user interface (perhaps a file "Open" dialog window) to find a file via progressive browsing through a directory tree. The client must be able to move from one export to another export via single-component, progressive LOOKUP operations.¶

This style of browsing is not well supported by the NFSv3 protocol. In NFSv3, the client expects all LOOKUP operations to remain within a single server file system. For example, the device attribute will not change. This prevents a client from taking namespace paths that span exports.¶

In the case of NFSv3, an automounter on the client can obtain a snapshot of the server's namespace using the EXPORTS procedure of the MOUNT protocol. If it understands the server's pathname syntax, it can create an image of the server's namespace on the client. The parts of the namespace that are not exported by the server are filled in with directories that might be constructed similarly to an NFSv4.1 "pseudo file system" (see Section 12.3) that allows the user to browse from one mounted file system to another. There is a drawback to this representation of the server's namespace on the client: it is static. If the server administrator adds a new export, the client will be unaware of it.¶

12.3. Server Pseudo File System

NFSv4.1 servers avoid this namespace inconsistency by presenting all the exports for a given server within the framework of a single namespace for that server. An NFSv4.1 client uses LOOKUP and READDIR operations to browse seamlessly from one export to another.¶

Where there are portions of the server namespace that are not exported, clients require some way of traversing those portions to reach actual exported file systems. A technique that servers may use to provide for this is to bridge the unexported portion of the namespace via a "pseudo file system" that provides a view of exported directories only. A pseudo file system has a unique fsid and behaves like a normal, read-only file system.¶

Based on the construction of the server's namespace, it is possible that multiple pseudo file systems may exist. For example,¶

        /a              pseudo file system
        /a/b            real file system
        /a/b/c          pseudo file system
        /a/b/c/d        real file system

Each of the pseudo file systems is considered a separate entity and therefore MUST have its own fsid, unique among all the fsids for that server.¶

12.5. Filehandle Volatility

The nature of the server's pseudo file system is that it is a logical representation of file system(s) available from the server. Therefore, the pseudo file system is most likely constructed dynamically when the server is first instantiated. It is expected that the pseudo file system may not have an on-disk counterpart from which persistent filehandles could be constructed. Even though it is preferable that the server provide persistent filehandles for the pseudo file system, the NFS client should expect that pseudo file system filehandles are volatile. This can be confirmed by checking the associated "fh_expire_type" attribute for those filehandles in question. If the filehandles are volatile, the NFS client must be prepared to recover a filehandle value (e.g., with a series of LOOKUP operations) when receiving an error of NFS4ERR_FHEXPIRED.¶

Because it is quite likely that servers will implement pseudo file systems using volatile filehandles, clients need to be prepared for them, rather than assuming that all filehandles will be persistent.¶

14.4. Stateid Seqid Values and Byte-Range Locks

When a LOCK or LOCKU operation is performed, the stateid returned has the same "other" value as the argument's stateid, and a "seqid" value that is incremented (relative to the argument's stateid) to reflect the occurrence of the LOCK or LOCKU operation. The server MUST increment the value of the "seqid" field whenever there is any change to the locking status of any byte offset as described by any of the locks covered by the stateid. A change in locking status includes a change from locked to unlocked or the reverse or a change from being locked for READ_LT to being locked for WRITE_LT or the reverse.¶

When there is no such change, as, for example, when a range already locked for WRITE_LT is locked again for WRITE_LT, the server MAY increment the "seqid" value.¶

14.5. Issues with Multiple Open-Owners

When the same file is opened by multiple open-owners, a client will have multiple OPEN stateids for that file, each associated with a different open-owner. In that case, there can be multiple LOCK and LOCKU requests for the same lock-owner sent using the different OPEN stateids, and so a situation may arise in which there are multiple stateids, each representing byte-range locks on the same file and held by the same lock-owner but each associated with a different open-owner.¶

In such a situation, the locking status of each byte (i.e., whether it is locked, the READ_LT or WRITE_LT type of the lock, and the lock-owner holding the lock) MUST reflect the last LOCK or LOCKU operation done for the lock-owner in question, independent of the stateid through which the request was sent.¶

When a byte is locked by the lock-owner in question, the open-owner to which that byte-range lock is assigned SHOULD be that of the open-owner associated with the stateid through which the last LOCK of that byte was done. When there is a change in the open-owner associated with locks for the stateid through which a LOCK or LOCKU was done, the "seqid" field of the stateid MUST be incremented, even if the locking, in terms of lock-owners has not changed. When there is a change to the set of locked bytes associated with a different stateid for the same lock-owner, i.e., associated with a different open-owner, the "seqid" value for that stateid MUST NOT be incremented.¶

14.6. Blocking Locks

Some clients require the support of blocking locks. While NFSv4.1 provides a callback when a previously unavailable lock becomes available, this is an OPTIONAL feature and clients cannot depend on its presence. Clients need to be prepared to continually poll for the lock. This presents a fairness problem. Two of the lock types, READW_LT and WRITEW_LT, are used to indicate to the server that the client is requesting a blocking lock. When the callback is not used, the server should maintain an ordered list of pending blocking locks. When the conflicting lock is released, the server may wait for the period of time equal to lease_time for the first waiting client to re-request the lock. After the lease period expires, the next waiting client request is allowed the lock. Clients are required to poll at an interval sufficiently small that it is likely to acquire the lock in a timely manner. The server is not required to maintain a list of pending blocked locks as it is used to increase fairness and not correct operation. Because of the unordered nature of crash recovery, storing of lock state to stable storage would be required to guarantee ordered granting of blocking locks.¶

Servers may also note the lock types and delay returning denial of the request to allow extra time for a conflicting lock to be released, allowing a successful return. In this way, clients can avoid the burden of needless frequent polling for blocking locks. The server should take care in the length of delay in the event the client retransmits the request.¶

If a server receives a blocking LOCK operation, denies it, and then later receives a nonblocking request for the same lock, which is also denied, then it should remove the lock in question from its list of pending blocking locks. Clients should use such a nonblocking request to indicate to the server that this is the last time they intend to poll for the lock, as may happen when the process requesting the lock is interrupted. This is a courtesy to the server, to prevent it from unnecessarily waiting a lease period before granting other LOCK operations. However, clients are not required to perform this courtesy, and servers must not depend on them doing so. Also, clients must be prepared for the possibility that this final locking request will be accepted.¶

When a server indicates, via the flag OPEN4_RESULT_MAY_NOTIFY_LOCK, that CB_NOTIFY_LOCK callbacks might be done for the current open file, the client should take notice of this, but, since this is a hint, cannot rely on a CB_NOTIFY_LOCK always being done. A client may reasonably reduce the frequency with which it polls for a denied lock, since the greater latency that might occur is likely to be eliminated given a prompt callback, but it still needs to poll. When it receives a CB_NOTIFY_LOCK, it should promptly try to obtain the lock, but it should be aware that other clients may be polling and that the server is under no obligation to reserve the lock for that particular client.¶

14.7. Share Reservations

A share reservation is a mechanism to control access to a file. It is a separate and independent mechanism from byte-range locking. When a client opens a file, it sends an OPEN operation to the server specifying the type of access required (READ, WRITE, or BOTH) and the type of access to deny others (OPEN4_SHARE_DENY_NONE, OPEN4_SHARE_DENY_READ, OPEN4_SHARE_DENY_WRITE, or OPEN4_SHARE_DENY_BOTH). If the OPEN fails, the client will fail the application's open request.¶

Pseudo-code definition of the semantics:¶

        if (request.access == 0) {
          return (NFS4ERR_INVAL)
        } else {
          if ((request.access & file_state.deny)) ||
             (request.deny & file_state.access)) {
            return (NFS4ERR_SHARE_DENIED)
        }
        return (NFS4ERR_OK);

When doing this checking of share reservations on OPEN, the current file_state used in the algorithm includes bits that reflect all current opens, including those for the open-owner making the new OPEN request.¶

The constants used for the OPEN and OPEN_DOWNGRADE operations for the access and deny fields are as follows:¶

const OPEN4_SHARE_ACCESS_READ   = 0x00000001;
const OPEN4_SHARE_ACCESS_WRITE  = 0x00000002;
const OPEN4_SHARE_ACCESS_BOTH   = 0x00000003;

const OPEN4_SHARE_DENY_NONE     = 0x00000000;
const OPEN4_SHARE_DENY_READ     = 0x00000001;
const OPEN4_SHARE_DENY_WRITE    = 0x00000002;
const OPEN4_SHARE_DENY_BOTH     = 0x00000003;

14.9. Open Upgrade and Downgrade

When an OPEN is done for a file and the open-owner for which the OPEN is being done already has the file open, the result is to upgrade the open file status maintained on the server to include the access and deny bits specified by the new OPEN as well as those for the existing OPEN. The result is that there is one open file, as far as the protocol is concerned, and it includes the union of the access and deny bits for all of the OPEN requests completed. The OPEN is represented by a single stateid whose "other" value matches that of the original open, and whose "seqid" value is incremented to reflect the occurrence of the upgrade. The increment is required in cases in which the "upgrade" results in no change to the open mode (e.g., an OPEN is done for read when the existing open file is opened for OPEN4_SHARE_ACCESS_BOTH). Only a single CLOSE will be done to reset the effects of both OPENs. The client may use the stateid returned by the OPEN effecting the upgrade or with a stateid sharing the same "other" field and a seqid of zero, although care needs to be taken as far as upgrades that happen while the CLOSE is pending. Note that the client, when sending the OPEN, may not know that the same file is in fact being opened. The above only applies if both OPENs result in the OPENed object being designated by the same filehandle.¶

When the server chooses to export multiple filehandles corresponding to the same file object and returns different filehandles on two different OPENs of the same file object, the server MUST NOT "OR" together the access and deny bits and coalesce the two open files. Instead, the server must maintain separate OPENs with separate stateids and will require separate CLOSEs to free them.¶

When multiple open files on the client are merged into a single OPEN file object on the server, the close of one of the open files (on the client) may necessitate change of the access and deny status of the open file on the server. This is because the union of the access and deny bits for the remaining opens may be smaller (i.e., a proper subset) than previously. The OPEN_DOWNGRADE operation is used to make the necessary change and the client should use it to update the server so that share reservation requests by other clients are handled properly. The stateid returned has the same "other" field as that passed to the server. The "seqid" value in the returned stateid MUST be incremented, even in situations in which there is no change to the access and deny bits for the file.¶

14.10. Parallel OPENs

Unlike the case of NFSv4.0, in which OPEN operations for the same open-owner are inherently serialized because of the owner-based seqid, multiple OPENs for the same open-owner may be done in parallel. When clients do this, they may encounter situations in which, because of the existence of hard links, two OPEN operations may turn out to open the same file, with a later OPEN performed being an upgrade of the first, with this fact only visible to the client once the operations complete.¶

In this situation, clients may determine the order in which the OPENs were performed by examining the stateids returned by the OPENs. Stateids that share a common value of the "other" field can be recognized as having opened the same file, with the order of the operations determinable from the order of the "seqid" fields, mod any possible wraparound of the 32-bit field.¶

When the possibility exists that the client will send multiple OPENs for the same open-owner in parallel, it may be the case that an open upgrade may happen without the client knowing beforehand that this could happen. Because of this possibility, CLOSEs and OPEN_DOWNGRADEs should generally be sent with a non-zero seqid in the stateid, to avoid the possibility that the status change associated with an open upgrade is not inadvertently lost.¶

14.11. Reclaim of Open and Byte-Range Locks

Special forms of the LOCK and OPEN operations are provided when it is necessary to re-establish byte-range locks or opens after a server failure.¶

• To reclaim existing opens, an OPEN operation is performed using a CLAIM_PREVIOUS. Because the client, in this type of situation, will have already opened the file and have the filehandle of the target file, this operation requires that the current filehandle be the target file, rather than a directory, and no file name is specified.¶
• To reclaim byte-range locks, a LOCK operation with the reclaim parameter set to true is used.¶

Reclaims of opens associated with delegations are discussed in Section 15.2.1.¶

15.2. Delegation and Callbacks

Recallable delegation of server responsibilities for a file to a client improves performance by avoiding repeated requests to the server in the absence of inter-client conflict. With the use of a "callback" RPC from server to client, a server recalls delegated responsibilities when another client engages in sharing of a delegated file.¶

A delegation is passed from the server to the client, specifying the object of the delegation and the type of delegation. There are different types of delegations, but each type contains a stateid to be used to represent the delegation when performing operations that depend on the delegation. This stateid is similar to those associated with locks and share reservations but differs in that the stateid for a delegation is associated with a client ID and may be used on behalf of all the open-owners for the given client. A delegation is made to the client as a whole and not to any specific process or thread of control within it.¶

The backchannel is established by CREATE_SESSION and BIND_CONN_TO_SESSION, and the client is required to maintain it. Because the backchannel may be down, even temporarily, correct protocol operation does not depend on them. Preliminary testing of backchannel functionality by means of a CB_COMPOUND procedure with a single operation, CB_SEQUENCE, can be used to check the continuity of the backchannel. A server avoids delegating responsibilities until it has determined that the backchannel exists. Because the granting of a delegation is always conditional upon the absence of conflicting access, clients MUST NOT assume that a delegation will be granted and they MUST always be prepared for OPENs, WANT_DELEGATIONs, and GET_DIR_DELEGATIONs to be processed without any delegations being granted.¶

Unlike locks, an operation by a second client to a delegated file will cause the server to recall a delegation through a callback. For individual operations, we will describe, under IMPLEMENTATION, when such operations are required to effect a recall. A number of points should be noted, however.¶

• The server is free to recall a delegation whenever it feels it is desirable and may do so even if no operations requiring recall are being done.¶
• Operations done outside the NFSv4.1 protocol, due to, for example, access by other protocols including other minor version of NFSv4, or by local access, also need to result in delegation recall when they make analogous changes to file system data, including the delegated file's contents, its attributes and the set of names linked to that file. What is crucial is if the change would invalidate the guarantees provided by the delegation. When this is possible, the delegation needs to be recalled and MUST be returned or revoked before allowing the operation to proceed.¶
• The semantics of the file system are crucial in defining when delegation recall is required. If a particular change within a specific implementation causes change to a file attribute, then delegation recall is required, whether that operation has been specifically listed as requiring delegation recall. Again, what is critical is whether the guarantees provided by the delegation are being invalidated.¶

Despite those caveats, the implementation sections for a number of operations describe situations in which delegation recall would be required under some common circumstances:¶

• For GETATTR, see Section 23.7.4.¶
• For LINK, see Section 23.9.4.¶
• For OPEN, see Section 23.16.4.¶
• For READ, see Section 23.22.4.¶
• For REMOVE, see Section 23.25.4.¶
• For RENAME, see Section 23.26.4.¶
• For SETATTR, see Section 23.30.4.¶
• For WRITE, see Section 23.32.4.¶

On recall, the client holding the delegation needs to flush modified state (such as modified data) to the server and return the delegation. The conflicting request will not be acted on until the recall is complete. The recall is considered complete when the client returns the delegation or the server times its wait for the delegation to be returned and revokes the delegation as a result of the timeout. In the interim, the server will either delay responding to conflicting requests or respond to them with NFS4ERR_DELAY. Following the resolution of the recall, the server has the information necessary to grant or deny the second client's request.¶

At the time the client receives a delegation recall, it may have substantial state that needs to be flushed to the server. Therefore, the server should allow sufficient time for the delegation to be returned since it may involve numerous RPCs to the server. If the server is able to determine that the client is diligently flushing state to the server as a result of the recall, the server may extend the usual time allowed for a recall. However, the time allowed for recall completion should not be unbounded.¶

An example of this is when responsibility to mediate opens on a given file is delegated to a client (see Section 15.4). The server will not know what opens are in effect on the client. Without this knowledge, the server will be unable to determine if the access and deny states for the file allow any particular open until the delegation for the file has been returned.¶

A client failure or a network partition can result in failure to respond to a recall callback. In this case, the server will revoke the delegation, which in turn will render useless any modified state still on the client.¶

15.4. Open Delegation

When a file is being OPENed, the server may delegate further handling of opens and closes for that file to the opening client. Any such delegation is recallable since the circumstances that allowed for the delegation are subject to change. In particular, if the server receives a conflicting OPEN from another client, the server must recall the delegation before deciding whether the OPEN from the other client may be granted. Making a delegation is up to the server, and clients should not assume that any particular OPEN either will or will not result in an OPEN delegation. The following is a typical set of conditions that servers might use in deciding whether an OPEN should be delegated:¶

• The client must be able to respond to the server's callback requests. If a backchannel has been established, the server will send a CB_COMPOUND request, containing a single operation, CB_SEQUENCE, for a test of backchannel availability.¶
• The client must have responded properly to previous recalls.¶
• There must be no current OPEN conflicting with the requested delegation.¶
• There should be no current delegation that conflicts with the delegation being requested.¶
• The probability of future conflicting open requests should be low based on the recent history of the file.¶
• The existence of any server-specific semantics of OPEN/CLOSE that would make the required handling incompatible with the prescribed handling that the delegated client would apply (see below).¶

There are two types of OPEN delegations: OPEN_DELEGATE_READ and OPEN_DELEGATE_WRITE. An OPEN_DELEGATE_READ delegation allows a client to handle, on its own, requests to open a file for reading that do not deny OPEN4_SHARE_ACCESS_READ access to others. Multiple OPEN_DELEGATE_READ delegations may be outstanding simultaneously and do not conflict. An OPEN_DELEGATE_WRITE delegation allows the client to handle, on its own, all opens. Only one OPEN_DELEGATE_WRITE delegation may exist for a given file at a given time, and it is inconsistent with any OPEN_DELEGATE_READ delegations.¶

When a client has either type of open delegation, it is assured that neither the contents, the attributes (with the exception of time_access), nor the names of any links to the file will change without its knowledge, so long as the delegation is held. When a client has an OPEN_DELEGATE_WRITE delegation, it may modify the file data locally since no other client will be accessing the file's data. The client holding an OPEN_DELEGATE_WRITE delegation may only locally affect file attributes that are intimately connected with the file data: size, change, time_access, time_metadata, and time_modify. All other attributes must be reflected on the server.¶

When a client has an OPEN delegation, it does not need to send OPENs or CLOSEs to the server. Instead, the client may update the appropriate status internally. For an OPEN_DELEGATE_READ delegation, opens that cannot be handled locally (opens that are for OPEN4_SHARE_ACCESS_WRITE/OPEN4_SHARE_ACCESS_BOTH or that deny OPEN4_SHARE_ACCESS_READ access) must be sent to the server.¶

When an OPEN delegation is made, the reply to the OPEN contains an OPEN delegation structure that specifies the following:¶

• the type of delegation (OPEN_DELEGATE_READ or OPEN_DELEGATE_WRITE).¶
• space limitation information to control flushing of data on close (OPEN_DELEGATE_WRITE delegation only; see Section 15.4.1)¶
• an nfsace4 specifying read and write permissions¶
• a stateid to represent the delegation¶

The delegation stateid is separate and distinct from the stateid for the OPEN proper. The standard stateid, unlike the delegation stateid, is associated with a particular lock-owner and will continue to be valid after the delegation is recalled and the file remains open.¶

When a request internal to the client is made to open a file and an OPEN delegation is in effect, it will be accepted or rejected solely on the basis of the following conditions. Any requirement for other checks to be made by the delegate should result in the OPEN delegation being denied so that the checks can be made by the server itself.¶

• The access and deny bits for the request and the file as described in Section 14.7.¶
• The read and write permissions as determined below.¶

The nfsace4 passed with delegation can be used to avoid frequent ACCESS calls. The permission check should be as follows:¶

• If the nfsace4 indicates that the open may be done, then it should be granted without reference to the server.¶
• If the nfsace4 indicates that the open may not be done, then an ACCESS request must be sent to the server to obtain the definitive answer.¶

The server may return an nfsace4 that is more restrictive than the actual ACL of the file. This includes an nfsace4 that specifies denial of all access. Note that some common practices such as mapping the traditional user "root" to the user "nobody" (see Section 11.13) may make it incorrect to return the actual ACL of the file in the delegation response.¶

The use of a delegation together with various other forms of caching creates the possibility that no server authentication and authorization will ever be performed for a given user since all of the user's requests might be satisfied locally. Where the client is depending on the server for authentication and authorization, the client should be sure authentication and authorization occurs for each user by use of the ACCESS operation. This should be the case even if an ACCESS operation would not be required otherwise. As mentioned before, the server may enforce frequent authentication by returning an nfsace4 denying all access with every OPEN delegation.¶

    if (!modified) {
        do CB_GETATTR for change and size;

        if (cc != sc)
            modified = TRUE;
    } else {
        do CB_GETATTR for size;
    }

    if (modified) {
        sc = sc + 1;
        time_modify = time_metadata = current_time;
        update sc, time_modify, time_metadata into file's metadata;
    }

15.5. Data Caching and Revocation

When locks and delegations are revoked, the assumptions upon which successful caching depends are no longer guaranteed. For any locks or share reservations that have been revoked, the corresponding state-owner needs to be notified. This notification includes applications with a file open that has a corresponding delegation that has been revoked. Cached data associated with the revocation must be removed from the client. In the case of modified data existing in the client's cache, that data must be removed from the client without being written to the server. As mentioned, the assumptions made by the client are no longer valid at the point when a lock or delegation has been revoked. For example, another client may have been granted a conflicting byte-range lock after the revocation of the byte-range lock at the first client. Therefore, the data within the lock range may have been modified by the other client. Obviously, the first client is unable to guarantee to the application what has occurred to the file in the case of revocation.¶

Notification to a state-owner will in many cases consist of simply returning an error on the next and all subsequent READs/WRITEs to the open file or on the close. Where the methods available to a client make such notification impossible because errors for certain operations may not be returned, more drastic action such as signals or process termination may be appropriate. The justification here is that an invariant on which an application depends may be violated. Depending on how errors are typically treated for the client-operating environment, further levels of notification including logging, console messages, and GUI pop-ups may be appropriate.¶

15.7. Data and Metadata Caching and Memory Mapped Files

Some operating environments include the capability for an application to map a file's content into the application's address space. Each time the application accesses a memory location that corresponds to a block that has not been loaded into the address space, a page fault occurs and the file is read (or if the block does not exist in the file, the block is allocated and then instantiated in the application's address space).¶

As long as each memory-mapped access to the file requires a page fault, the relevant attributes of the file that are used to detect access and modification (time_access, time_metadata, time_modify, and change) will be updated. However, in many operating environments, when page faults are not required, these attributes will not be updated on reads or updates to the file via memory access (regardless of whether the file is local or is accessed remotely). A client or server MAY fail to update attributes of a file that is being accessed via memory-mapped I/O. This has several implications:¶

• If there is an application on the server that has memory mapped a file that a client is also accessing, the client may not be able to get a consistent value of the change attribute to determine whether or not its cache is stale. A server that knows that the file is memory-mapped could always pessimistically return updated values for change so as to force the application to always get the most up-to-date data and metadata for the file. However, due to the negative performance implications of this, such behavior is OPTIONAL.¶
• If the memory-mapped file is not being modified on the server, and instead is just being read by an application via the memory-mapped interface, the client will not see an updated time_access attribute. However, in many operating environments, neither will any process running on the server. Thus, NFS clients are at no disadvantage with respect to local processes.¶

• If there is another client that is memory mapping the file, and if that client is holding an OPEN_DELEGATE_WRITE delegation, the same set of issues as discussed in the previous two bullet points apply. However, it should be noted that it is very unlikely that such a delegation will be held since it is normally required that the file be open for read to be mapped into memory. Only if the file were not open and accessed using a special stateid could the delegation be retained while the file in question is mapped into another client's memory. For this reason, such use is highly undesirable.¶

  In this situation, when a server does a CB_GETATTR to a file that the client has modified in its cache, the reply from CB_GETATTR would not necessarily be accurate, assuming the delegation is not recalled at this point. As discussed earlier, the client's obligation is to report that the file has been modified since the delegation was granted, not whether it has been modified again between successive CB_GETATTR calls, and the server MUST assume that any file the client has modified in cache has been modified again between successive CB_GETATTR calls. Depending on the nature of the client's memory management system, it might not be possible to live up to this weak obligation. A client MAY return stale information in CB_GETATTR whenever the file is memory-mapped, if another client is accessing the file without opening it.¶

15.8. Name and Directory Caching without Directory Delegations

The NFSv4.1 directory delegation facility (described in Section 15.9 below) is OPTIONAL for servers to implement. Even where it is implemented, it may not always be functional because of resource availability issues or other constraints. Thus, it is important to understand how name and directory caching are done in the absence of directory delegations. These topics are discussed in the next two subsections.¶

15.9. Directory Delegations and Notifications

/*
 * Directory notification types and associated flags
 */
enum notify_type4 {
        /*
         * Present in RFCs 5661, 8881
         */
        NOTIFY4_CHANGE_CHILD_ATTRS = 0,
        NOTIFY4_CHANGE_DIR_ATTRS = 1,
        NOTIFY4_REMOVE_ENTRY = 2,
        NOTIFY4_ADD_ENTRY = 3,
        NOTIFY4_RENAME_ENTRY = 4,
        NOTIFY4_CHANGE_COOKIE_VERIFIER = 5,
        /*
         * Added in NFSv4.1 bis document
         */
        NOTIFY4_GFLAG_EXTEND = 6,
        NOTIFY4_AUFLAG_VALID = 7,
        NOTIFY4_AUFLAG_USER = 8,
        NOTIFY4_AUFLAG_GROUP = 9,
        NOTIFY4_AUFLAG_OTHER = 10,
        NOTIFY4_CHANGE_AUTH = 11,
        NOTIFY4_CFLAG_ORDER = 12,
        NOTIFY4_AUFLAG_GANOW = 13,
        NOTIFY4_AUFLAG_GALATER = 14,
        NOTIFY4_CHANGE_GA = 15,
        NOTIFY4_CHANGE_AMASK = 16

};