Internet-Draft Richard V. Huber LDAP Duplication/Replication/Update Gerald F. Maziarski Protocols WG AT&T Laboratories Intended Category: TBD Ryan D. Moats Expires: August 2001 Coreon, Inc. February 2001 General Usage Profile for LDAPv3 Replication draft-ietf-ldup-usage-profile-00.txt Status of This Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/lid-abstracts.txt. The list of Internet-Drafts Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (2001). All Rights Reserved. Abstract Support for replication in LDAP directory systems is often one of the key factors in the decision to deploy them. But replication brings design constraints along with its benefits. We discuss some of the factors that should be taken into consideration when designing a replicated directory system. Both programming and architectural/operational concerns are addressed. Huber, et al Expires August 2001 [Page 1] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Huber, et al Expires August 2001 [Page 2] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 Table of Contents 1 Introduction.......................................................4 2 Meta-data Considerations...........................................4 2.1 Schema Considerations...........................................4 2.2 Replication Agreements..........................................5 2.3 Access Control..................................................6 2.4 Change Logs.....................................................7 3 Naming Considerations..............................................7 4 Conflict Resolution Considerations.................................7 4.1 Consistent Access after Changes.................................7 4.2 Conflict Resolution in Single-Master Systems....................8 4.3 Problem Cases...................................................9 4.4 General Principles.............................................10 5 Failover Considerations...........................................11 5.1 Common Issues..................................................11 5.2 Single Master Issues...........................................11 5.3 Multi-Master Issues............................................12 6 Impact of Non-LDAP Changes/Constraints............................12 6.1 Changes Outside of LDAP........................................12 6.2 Application Triggers...........................................13 7 Security Considerations...........................................13 8 Acknowledgements..................................................13 9 References........................................................14 Authors' Addresses...................................................14 Full Copyright Statement.............................................15 Huber, et al Expires August 2001 [Page 3] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 1 Introduction As LDAP directories become part of critical infrastructure in many applications, the need for extremely high reliability and availability becomes significant. And as directories are more widely used, the load on individual directory servers gets higher. Distributed, replicated directories can reduce the problems of reliability and capacity. But applications that work well with a single, standalone directory will develop problems in a distributed environment unless both the application and the environment are carefully designed. The particular areas of concern depend partly on whether the distributed directory is a single- or multi-master system, but there are many concerns that are common to both. In the remainder of the document, we have flagged some sections as being specific to single- or multi-master directories. Unflagged sections pertain to both. This document addresses general issues. There may be additional drafts in the future that address specific applications. 2 Meta-data Considerations Any LDAP directory contains meta-data as well as the user data in the directory. Examples of this meta-data include descriptions of the data in the directory (e.g. schema), policies for use of the data (e.g. access controls), and configuration/status information (e.g. replication agreements); this is not an exhaustive list. Many types of meta-data are stored in the directory itself; accessible as regular data or as operational attributes. Since meta-data can appear in the directory, issues arise when it is replicated. 2.1 Schema Considerations If the schema of one or more of the copies of a replica differs from the schema of the other replicas, then there exists the possibility of schema mismatch when data is exchanged between them. The extensibility feature of LDAP schema nearly guarantees that replica rings comprised of a heterogeneous mix of systems will not contain homogeneous schema because of directory vendors' built-in extensions. Because a given directory may not utilize its complete schema, not all of the potential Huber, et al Expires August 2001 [Page 4] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 schema differences in a ring will result in a schema mismatch under replication. Schema mismatch issues are further complicated by the possibility of replicating the "subschemaSubentry" itself. Some directories distribute schema changes through that mechanism. Currently there is no standard for LDAP schema representation. In the absence of such a standard, schema interoperability is not possible in the IETF sense. Designers should establish common schema on all servers holding a replica. It is useful to separate out two classes of schema mismatch: "latent", when the differences do not intersect the collection of data in the ring; and "salient", when they do. Schema mismatches that can cause data corruption in one or more of the replicas must result in meta-data (e.g. log entries) to comply with Requirement MM5 of [ReplicaReq]. However, not all schema differences produce corruption in all circumstances. Some schema differences may have little or nor impact on the proper storage of replicated data. However, any time data is added to the directory it could change a latent mismatch into a salient mismatch. It is therefore recommended that all schema mismatches be removed from, or otherwise rationalized among, the replica ring schemas, if possible. The tool described by requirement AM8 of [ReplicaReq] would help designers detect schema conflicts as early as possible. The other option that can be considered in this situation is the use of fractional replication to replicate only those attributes that do not have differences. Examples of mismatches: 1. Base syntax of an attribute type 2. Structure Rule of an object class 3. Optional vs. Mandatory attribute in an object class 4. Structural vs. Auxiliary in an object class 5. Object class not defined 6. Attribute type not defined 7. Object identifiers differs on an attribute type or on an object class 8. Type and number of attributes defined in a class 9. Multi-valued vs. Single valued attribute types 10. ACL format (and consequently, ACL calculus) 11. Matching rule of an attribute type 12. Naming collisions of attribute type names 13. Attribute name aliasing ("street" vs. "streetAddress") 2.2 Replication Agreements Replication Agreements are central to replication, they allow configuration of most of the aspects of the replication process, Huber, et al Expires August 2001 [Page 5] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 including the triggers for replica cycles (from Requirement M1 in [ReplicaReq]), critical OID information (from Requirement M6 in [ReplicaReq]), and replication process parameters (Requirement M7 in [ReplicaReq]). Through the use of a standard schema (Requirement S2) it is possible to replicate the replication agreement. If a replication agreement includes replication credentials, the agreement SHOULD be read protected in the directory and transport of the replication agreement SHOULD be encrypted. Further, the directory administrator should consider and understand the possibility of the replication of the agreement impacting replication processes. 2.3 Access Control - Access Control Information (ACI) is treated as though it were stored as attributes in the directory [ACModel] - LDAP declares [RFC2251] that changes resulting from a single LDAP MODIFY are atomic (but see caveats in multi-master case) - ACI affecting a given entry may not be part of that entry (it could be part of a group entry or part of an ancestor of the entry in question) - ACI cannot always be changed atomically with associated data changes - If you aren't careful, you can leave windows where data is unprotected - Access control changes SHOULD be made before adds and after deletes - Access control changes and the associated data changes SHOULD be made on same system. - Life gets interesting when access control policy is inherited from a superior partition, but that is mostly a partitioning issue, not a replication issue (once we figure out how partitioning works we may find out it is also a replication issue). Even when ACL information is faithfully replicated among heterogeneous members of a replica ring that agree on transfer format, there is no guarantee that an ACI change is expressed similarly everywhere in the ring. This caveat is partly due to partitioning effects mentioned above, and partly due to vendor differences with regard to the expression of security policy. The access control mechanisms specified in [ACModel] are neutral with respect to policy inheritance mechanisms, explicit vs. implicit denial, and group nesting; all of which may differ between vendors, and will affect or modify the expression of access control information. Huber, et al Expires August 2001 [Page 6] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 2.4 Change Logs Requirement G4 of [ReplicaReq] states that meta-data must not grow without bound. Since it is unrealistic to assume that meta-data won't be needed during replication, how and when meta-data can be purged must be considered. Replicas that use connections with intermittent quality SHOULD use explicit replica cycle scheduling. This allows notification of a delayed replica and manual intervention before the meta-data grows without bound. In extreme cases, it may be necessary to remove these replicas from the replication ring and re-add them add once better connectivity is available. Second, it is also possible for a consumer to receive changes that cannot be applied. The replication agreement SHOULD include information for when the consumer should stop deferring the change and declare the changes as inapplicable, thus invoking whatever procedure is in place to comply with requirement MM5 of [ReplicaReq] during multi-master replication. 3 Naming Considerations A number of naming models have been proposed for directories ([RFC1255], [RFC2377], [CIMNames]), and many others have been implemented on an ad hoc basis. Each of these models specifies the naming attributes to be used and provides rules for using them. The naming scheme may also specify containment rules. The naming plan applies to the directory as a whole, not the individual servers holding replicas. In a heterogeneous replicated environment, all of the replicating servers must be capable of supporting all of the rules for the naming plan in use for that directory. Some directory implementations have naming constraints. If one of them is part of a replicated directory, those constraints will have to be observed by all participating directories. If the environment contains implementations with incompatible constraints there is a major problem. This SHOULD be checked as early in the design phase as possible to avoid costly problems. 4 Conflict Resolution Considerations 4.1 Consistent Access after Changes Many operations on a directory are done as a set of steps. For example, a new object may be created in one operation, and its values may be filled in as part of a separate LDAP operation. An Huber, et al Expires August 2001 [Page 7] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 administrator may add a user to a directory, and that user may then try to log in using the new entry. Replicated LDAP directories provide loose consistency [ReplicaReq]. A new entry or a change to an existing entry will not reach all replicas immediately; there will be some delay before changes are available on all replicas. Changes made (e.g. adding a new user) on one physical system may appear to be "lost" if checked on another physical system before replication is complete. In general, LDAP applications SHOULD be prepared to operate correctly in the face of replication delays. In some cases, this means designing to allow for delay. In the case of the newly created user, it SHOULD be standard practice to ask the user to wait a while before trying to use the entry. In the case where the new object must be filled in, the application SHOULD make appropriate use of LDAP sessions to make sure that the same server is reached for both operations. As a general rule, an LDAP application SHOULD bind once and not unbind until a complete set of related operations have been performed. In the single-master case, all write requests go to one server. If a set of related reads and writes are done, they SHOULD all be done on the master if possible. Ideally, only sets of related operations that can not include a write SHOULD go to one of the slave servers. But load balancing concerns may make this impractical. In some cases, related requests will deal with data in different partitions which are not all available on a single server. In this case, it is safer to keep sessions open to all servers rather than closing the session with one server and opening one with another server. It may not always be obvious to clients that they are using different servers. If a load distribution system is used between the client and the server, the client may find that a change request and a subsequent lookup are directed to different physical servers even though the original requests were sent to the same server name and/or address. Since LDAP is session oriented, any load distribution system used should take sessions into account. Thus, keeping all related read and write requests within a single bind/unbind session SHOULD be the goal in this situation as well. 4.2 Conflict Resolution in Single-Master Systems It is possible that resolution conflicts could occur in a single master replication system. Because requirement SM2 of [ReplicaReq] is a Huber, et al Expires August 2001 [Page 8] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 SHOULD and not a MUST, it is possible for implementers to reorder changes. If changes are reordered, it is quite possible for a conflict to occur. If schema changes are critical and must be moved to the front of the replication queue, then a schema change that deletes an attribute for some object class that follows some changes that delete values of the soon-to-be-deleted attribute can cause conflict. If the schema change moves to the head of the queue, the consumer servers might have to delete an attribute that still has values, and then receive requests to delete the values of an attribute that is no longer defined. However, directory administrators may have scenarios where re-ordering of replication information is desirable. On a case-by-case basis, the directory administrator should make such decisions. 4.3 Problem Cases 4.3.1 Atomicity The fact that replication does not guarantee the time order arrival of changes at a consumer allows situations where changes that were applied successfully at the supplier may fail in part when an attempt is made to apply the same change at the consumer. Examples appear below. 4.3.1.1 Locking There is an entry with distinguished name DN that contains attributes X, Y, and Z. The value of X is 1. On replica A, a ModifyRequest is processed which includes modifications to change that value of X from 1 to 0 and to set the value of Y to "USER1". At the same time, replica B process a ModifyRequest which includes modifications to change the value of X from 1 to 0 and to set the value of Y to "USER2" and the value of Z to 42. The application in this case is using X as a lock and is depending on the atomic nature of ModifyRequests to provide mutual exclusion for lock access. In the single-server case, the two operations would have occurred sequentially. Since a ModifyRequest is atomic, the entire first operation would succeed. The second ModifyRequest would fail, since the value of X would be 0 when it was attempted, and the modification changing X from 1 to 0 would thus fail. The atomicity rule would cause all other modifications in the ModifyRequest to fail as well. In the multi-master case, it is inevitable that at least some of the changes will be reversed despite the use of the lock. Assuming the changes from A have priority per the conflict resolution algorithm, the value of X should be 0 and the value of Y should be "USER1" The interesting question is the value of Z at the end of the replication Huber, et al Expires August 2001 [Page 9] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 cycle. If it is 42, the atomicity constraint on the change from B has been violated. But for it to revert to its previous value, grouping information must be retained and it is not clear when that information can be safely discarded. Thus, requirement G6 in [ReplicaReq] may be violated. The utility of locking mechanisms cannot be guaranteed, and therefore are likely to be misleading. Its use in multi-master environments is therefore deprecated. 4.3.1.2 Partitioning Suppose two servers, A and B, are members of Replica-ring X; and servers B and C are members of replica-ring Y. A ModifyRDN operation on server B moves an entry from ring X to ring Y. If the appropriate delete is done on A, the operation may be considered successful even though hidden circumstances on ring Y prevent the move. If on C another change made the modifyRDN that worked on B impossible, because an RDN of a different GUID exists there already, then the change on A is inconsistent and will need to be reversed. Others examples of cases of this class include group membership modification and access control scoping. 4.4 General Principles The examples above discuss some of the most difficult problems that can arise in multi-master replication. While they can be dealt with, dealing with them is difficult and can lead to situations that are quite confusing to the application and to users. The common characteristics of the examples are: . Several directory users/applications are changing the same data . They are changing the data at the same time . They are using different directory servers to make these changes . They are changing data that are parts of a distinguished name or they are using ModifyRequest to both read and write a given attribute value in a single atomic request If any one of these conditions is reversed, the types of problems described above will not occur. There are many useful applications of multi-master directories where at least one of the above conditions Huber, et al Expires August 2001 [Page 10] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 does not occur. For cases where all four do occur, application designers should be aware of the possible consequences. 5 Failover Considerations One of the major reasons to use directory replication is to improve reliability of the directory system as a whole. Replication permits hot- and warm-standby configurations to be built easily. But there are some issues that must be considered during design. In this situation, single-master systems actually raise more concerns than multi-master. Both are addressed below. 5.1 Common Issues In both the single- and multi-master cases, clients must be able to find an alternate quickly when a server fails. Some possible ways to do this are detailed in [FindingLDAP] and [LDAPinDNS]. If all else fails, a list of possible servers can be built into client applications. Designers should consider how clients are notified that the server is again available. When the failed server comes back up, it is brought back into synchronization with the other servers and is ready to take requests. Note that the process used to bring a failed server back into replication can also be used to add a server to a set of replicating servers. In this case, the new server might be initialized from a backed-up copy of the directory or it may acquire the entire DIB via replication. The former method is usually preferable when the directory is large. 5.2 Single Master Issues In a single-master system, the master is a single point of failure, as all modification has to originate at the master server. When high availability is a requirement, a quick, automated failover process for converting a slave replica to a new master is desirable, as the failover process time becomes a major factor in determining system availability. The considerations in section 5.1 apply here; clients must know how to find the new master or a new slave in case of failure. To aid in promotion of a slave replica, the master could replicate control information and meta-data (included replication credentials) so that this information is available during failover promotion. This data may either be replicated on a single "failover designate" slave (which would become the master in during failover) or it could be Huber, et al Expires August 2001 [Page 11] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 replicated to all slaves. The first possibility has the advantage of minimizing the amount of extra replication while the second more robustly handles multiple failovers (i.e. failover of the newly promoted master to another slave before the original master has been restored). If this method is followed, data privacy mechanisms SHOULD be used to protect the replication session. In addition, if data privacy mechanisms (e.g. encryption) were used to protect the replication session, the new master MUST have the necessary key information. Further this key information should be independent of the master that is using it (i.e. not tied to the IP address of the master server). Restoration of the failed or broken master can be handled in one of two ways: - It could join the ring and function as a slave. - It could join the ring and negotiate with the new master to synchronize and then take over as master. In either case, clients need a way to know that a new server is available. The slave replicas may also use the replication agreement to filter which master is allowed to submit changes. Such a model allows the slave servers to function correctly when the master server is "broken" and sending out incorrect updates. However, then it is necessary to update the replication agreement during the fail over process so that the slaves will accept updates from the new master. This is the case for both the original failure and the restoration of the restored master if that is how the restored master rejoins the replica ring. 5.3 Multi-Master Issues Typically, a multi-master configuration was used because high availability is required for writes as well as reads in the directory. Multi-master implies that there are multiple active servers prepared to take write requests, so there is no "switchover" time in this case. But clients still need to be able to find an alternate server, so the considerations of Section 5.1 apply here. 6 Impact of Non-LDAP Changes/Constraints 6.1 Changes Outside of LDAP Huber, et al Expires August 2001 [Page 12] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 LDAP directories are typically built on top of some database or file system. Thus there are ways to change the data that do not go through the normal LDAP change mechanisms (e.g. ModifyRequest). If the data is modified outside of LDAP, the changes will not be checked for schema conformance nor will access controls be checked as the changes are made. Since both integrity and security checks are omitted, security can be adversely affected. Also, many systems use the normal LDAP modification mechanisms to trigger replication. Changes made using non-LDAP mechanisms may not be replicated at all, leading to inconsistencies between replica copies. 6.2 Application Triggers Directory servers commonly integrate one or more specific applications. To achieve this integration the directory server may intercept updates and run application-specific "trigger" code. Such triggers enforce directory invariants that cannot be expressed by the LDAP schema. A simple trigger example is password policy enforcement. A directory server might interpret a request to replace the current value of the userPassword attribute with some new value as a request to first check that the new value conforms to the server's password policy (e.g. the value is sufficiently long and complex) before storing the new value. Using this trigger the directory server voids the security risk associated with passwords that are easy to attack. A more complex trigger example is password hashing. A directory server might interpret a request to replace the current value of the userPassword attribute with some new value as a request to compute one or more secure hashes of the new value and store these hashes in one or more attributes, storing no value in the userPassword attribute. Using this trigger the directory server avoids the security exposure of storing the plaintext password. Replication between directory servers with different application triggers will compromise directory integrity. 7 Security Considerations The document discusses issues that arise in replication. Some of these issues are security related (e.g. replication of access control information) and the security implications are discussed in the relevant sections. 8 Acknowledgements Huber, et al Expires August 2001 [Page 13] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 The authors would like to thank Mark Brown for identifying the issues discussed in section 6. 9 References [ACModel] E. Stokes, R. Blakeley, D. Rinkevich, R. Byrne, "Access Control Model for LDAPv3", Internet Draft, draft-ietf-ldapext-acl- model-06.txt, July 2000. [CIMNames] Desktop Management Task Force, "Guidelines for CIM-to-LDAP Directory Mappings", DMTF Specification DSP0100, May 2000 (available online at http://www.dmtf.org/spec/DEN/DSP0100.htm). [FindingLDAP] R. Moats, R. Hedberg, "A Taxonomy of Methods for LDAP Clients Finding Servers", Internet Draft, draft-ietf-ldapext-ldap- taxonomy-04.txt, October 2000. [LDAPinDNS] M. Armijo, L. Esibov, P. Leach, R. L. Morgan, "Discovering LDAP Services with DNS", Internet Draft, draft-ietf-ldapext-locate- 04.txt, August 2000. [ReplicaReq] E. Stokes, R. Weiser, R. Moats, R. Huber, "LDAPv3 Replication Requirements", Internet Draft, draft-ietf-ldup-replica-req- 06.txt, January 2001. [RFC1255] The North American Directory Forum, "A Naming Scheme for c=US", RFC 1255, September 1991. [RFC2119] S. Bradner, "Key Words for Use in RFCs to Indicate Requirement Levels", RFC 2119, March 1997. [RFC2251] M. Wahl, T. Howes, S. Kille, "Lightweight Directory Access Protocol", RFC 2251, December 1997. [RFC2377] A. Grimstad, R. Huber, S. Sataluri, M. Wahl, "Naming Plan for Internet Directory-Enabled Applications", RFC 2377, September 1998. Authors' Addresses Richard V. Huber Room C3-3B30 AT&T Laboratories 200 Laurel Avenue South Middletown, NJ 07748 USA E-Mail: rvh@att.com Telephone: +1 732 420 2632 Fax: +1 732 368 1690 Huber, et al Expires August 2001 [Page 14] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 Gerald F. Maziarski Room C3-3Z01 AT&T Laboratories 200 Laurel Avenue South Middletown, NJ 07748 USA E-Mail: gfm@att.com Telephone: +1 732 420 2162 Fax: +1 732 368 1690 Ryan D. Moats Coreon, Inc. 15621 Drexel Circle Omaha, NE 68135 USA E-Mail: rmoats@coreon.net Telephone: +1 402 894 9456 Full Copyright Statement Copyright (C) The Internet Society (2000). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgement Huber, et al Expires August 2001 [Page 15] Internet-Draft Usage Profiles for LDAPv3 Replication February 2001 Funding for the RFC Editor function is currently provided by the Internet Society. Huber, et al Expires August 2001 [Page 16]