2.1.8 LDAP Duplication/Replication/Update Protocols (ldup)

NOTE: This charter is a snapshot of the 44th IETF Meeting in Minneapolis, Minnesota. It may now be out-of-date. Last Modified: 16-Feb-99


Chris Apple <capple@att.com>
John C. Strassner <johns@cisco.com>

Applications Area Director(s):

Keith Moore <moore@cs.utk.edu>
Patrik Faltstrom <paf@swip.net>

Applications Area Advisor:

Patrik Faltstrom <paf@swip.net>

Mailing Lists:

General Discussion:ietf-ldup@imc.org
To Subscribe: ietf-ldup-request@imc.org
In Body: subscribe
Archive: http://www.imc.org/ietf-ldup/

Description of Working Group:

As LDAPv3 becomes more widely deployed, replication of data across servers running different implementations becomes an important part of providing a distributed directory service. However, the LDAPv3 community, to date, has focused on standardizing the client-server access protocol. Therefore, this group will standardize master-slave and multi-master LDAPv3 replication as defined below:

o Multi-Master Replication - A replication model where entries can be written and updated on any of several replica copies, without requiring communication with other masters before the write or update is performed.

o Master-Slave, or Single-Master Replication - A replication model that assumes only one server, the master, allows write access to the replicated data. Note that Master-Slave replication can be considered a proper subset of multi-master replication.

The WG's approach is to first develop a set of requirements for LDAPv3 directory replication and write an applicability statement defining scenarios on which replication requirements are based. An engineering team was formed consisting of different vendors and the co-chairs in order to harmonize the existing approaches into a single standard approach. All of these have been accomplished during the pre-working group stage. It should be noted, however, that replication using heterogeneous servers is dependent on resolving access control issues, which are the domain of other working groups.

The new replication architecture support all forms of replication mentioned above. Six areas of working group focus have been identified through LDUP Engineering Team discussions, each leading to one or more documents to be published:

o Abstract Model of LDAPv3 Replication

This documents a general-purpose LDAPv3 replication architecture, defines key components of this architecture, describes how these key components functionally behave, and describes how these components interact with each other when in various modes of operation

o LDAPv3 Replication Information Model

Defines the schema and semantics of information used to operate, administer, maintain, and provision replication between LDAPv3 servers. Specifically, this document will contain common schema specifications intended to facilitate interoperable implementations with respect to:

+ replication agreements + consistency models + replication topologies + managing deleted objects and their states + administration and management

o LDAPv3 Replication Information Transport Protocol

LDAPv3 extended operation and control specifications required to allow LDAPv3 to be used as the transport protocol for information being replicated

o LDAPv3 Mandatory Replica Management

Specifications required to allow administration, maintenance, and provisioning of replicas and replication agreements. These specifications may take the form of definitions for LDAPv3 extended operations, controls, and/or new schema elements.

o LDAPv3 Conflict Detection and Resolution Procedures

Procedures for detection and resolution of conflicts between the state of multiple replicas that contain information from the same unit of replication.

o LDAPv3 Profiles

Including the Abstract Replication Model, Information Model, LDAPv3 Protocol Extensions, and Conflict Detection and Resolution Procedures for:

+ Master-Slave LDAPv3 Directory Replication + Multi-Master LDAPv3 Directory Replication

The LDUP WG Chairs will assign to one or two persons to be official LDUP WG liasons to ITU, to monitor X.500 replication work in ITU, and to coordinate the work of the LDUP WG with similar work in ITU.

Goals and Milestones:

Nov 98


LDAPv3 Directory Replication Requirements and Applicability Statement I-D goes to WG Last Call as Informational.

Nov 98


Submit Internet-Draft on LDAPv3 Replication Information Model

Nov 98


Submit I-D on LDAPv3 Mandatory Replica Management

Dec 98


Submit I-D on LDAPv3 Conflict Detection and Resolution Procedures

Dec 98


LDAPv3 Abstract Model of Replication goes to WG Last Call as Informational.

Dec 98


Submit I-D on LDAPv3 Multi-Master Replication Profile

Dec 98


Submit I-D on LDAPv3 Master-Slave Replication Profile

Dec 98


Submit I-D on LDAPv3 Replication Information Transport Protocol

Jan 99


LDAPv3 Replication Information Model goes to WG Last Call as Proposed Standard.

Mar 99


LDAPv3 Mandatory Replica Management goes to WG Last Call as Proposed Standard.

Mar 99


LDAPv3 Replication Information Transport Protocol goes to WG Last Call as Proposed Standard.

Apr 99


LDAPv3 Conflict Detection and Resolution Procedures goes to WG Last Call as Proposed Standard.

Apr 99


LDAPv3 Master-Slave LDAPv3 Replication Profile goes to WG Last Call as Proposed Standard.

Apr 99


LDAPv3 Multi-Master LDAPv3 Replication Profile goes to WG Last Call as Proposed Standard.


No Request For Comments

Current Meeting Report

Minutes of the LDUP WG Meeting, 44th IETF (Minneapolis, MN)
Thursday, March 18, 1999

Minutes reported by John Strassner


1. Introduction and agenda bashing


2. Requirements draft update

Russ Weiser

3. Architecture and Model update

John Merrells

4. Replication Transfer Protocol

Ellen Stokes

5. Conflict Detection and Resolution

Steven Legg

Comments from Keith Moore:
(this one item is out of place as to when it happened, but contains general information, and so is put here for convenience)

Keith: the IESG frowns on publishing an informational document that is called a â__Requirementsâ__ document. Perhaps the title should be â__Design Considerationsâ__. (Chairs and author agree to change the name to Design Considerations). Keith also recommends that we socialize LDUP with the other wgs in APPS that are using LDAP (at least 12 now). Chairs agree, and we will designate one or more people to do so.

Russ Weiser - Requirements draft update

Changes made: clarification of terminology, addition of an applicability statement, added requirements references to scenarios, and clarifications to the document that address previous comments. The following are comments to the draft:

Are attribute and object resurrection requirements? (reference 5.1.6). Russ posits that it still is a requirement. In archive, Mike from Xerox brought up a synchronization scenario involving two palm pilots and synchronizing their bank accounts. Russ thinks that this is out of scope. Mike responds that the applicability statement should be broadened â_" the existing examples are really more about administration, and donâ_TMt characterize the nature of the application. General agreement on this, Russ to revise.

Need a definition of what a conflict is. This may belong in this draft, and also possibly in the model document. (Current draft talks about conflicts, but doesnâ_TMt define it). Russ to include.

Need more words about partial replication. This will be followed up in email discussions on the list.

Need more clarification on naming context. Naming contexts donâ_TMt really overlap, unless the root of one resides in the other. What you would like to do is to cache the mount point of the subordinate in the superiorâ_TMs naming context. In terms of DNs, it is in the tree, but in terms of where it resides on servers, it could be a reference to another server.

What is the distinction between transient and looser consistency? Transient will settle out over time more quickly than â__looserâ__ will. In addition, transient will reach a converged state faster than looser will. In transient, you have global knowledge whereas in looser, you donâ_TMt. This helps convergence. This means that you can identify when an update has finished so you can purge it, whereas in looser you may purge it before the update has reached everywhere it wants to go. (This applies to purging death warrants).

Mike has additional comments, but will send them to the list for further discussion.

Goal: Russ to have an update out in 1-2 weeks.

Note from the chairs: Please everyone, focus your attention on this draft. We need to move it along to help progress the others. This should go into Last Call before Oslo.

John Merrells - Architecture and Model

Status is that it is being moved into the wg namespace from the individual namespace. This draft is, of course, affected by different drafts as people start to implement these ideas. The list of topics that affect the model include replay ordering, CSN-TSR/TRS/TSRs, Partial Replicas, Lost+Found entry, replication management, and the high-level URP specification.

The draft supports out of order replay. If you put more constraints on the ordering of updates, then you donâ_TMt need any update resolution. But this is impractical, hence the need to support replay.

The sequence number is really serving two purposes â_" making time more granular (to help with ensuring that order is correct) as well as to ensure the atomicity of the primitives in LDAP operations (e.g., a primitive should not be able to be inserted). The problem is that TSR is natural for replication, and TRS is natural for state systems. Thus, we are considering TSRS, which has the time split up as described above. We have in fact converged on TSRS. We need to hear from the list if this is OK.

Partial replicas are proving to be troublesome. We need to address them in detail (we have been trying to ignore them so far). By taking this on, we now are in effect considering sparse replication. But the problem is that entries move in and out of the replication scope just as a matter of fact of LDAP operations. So we need to track entry movement, which greatly complicates things. Moved to the list for more discussion.

Weâ_TMve decided that there is a lost+found entry (this is for entries that lose their superior reference), but we donâ_TMt know where it is. Possibilities include keeping it in the subtree (perhaps, make it a subordinate of the root of the naming context). Mark points out that we must ensure that if we do that, we must ensure that it can not be seen by processes other than LDUP. Moved to the list for more discussion.

Replication management is being dealt with in a separate I-D from Mark Wahl. First draft due at the end of April.

Suggestion is to bridge between the model document (too high-level) and the URP document (too low-level). This will be done done by John Merrells, Russ Weiser, and Mike Spreitzer. Probably will go as a new draft. Due date to be supplied to the chairs within a couple of weeks.

Schema replication. In log-based systems, you send the schema modifications exactly at the point where the client sent them. But in a state-based system, you really want to send the schema first, and then the updates. This may mean that we must impose a constraint on the ordering of updates. Needs discussion on mailing list.

Replication Transfer Protocol Discussion â_" Ellen Stokes

Will be a draft out shortly. Basic flow is:

S: bind to consumer
C: bind response (identity/credential is implementation defined)
S: sends StartReplicationExtended Request repl root, sender's replica ID, protocol OID=full|incremental, SIR|CIR[|both]
C: send StartReplicationExtendedResponse (does access control decision for identity allowed to do this operation) result, updateVector
S: â_¦ (transmits all required updates)
S: sends EndReplicationExtendedRequest updateVector only on full update: returnUpdateVector
C: sends EndReplicationExtendedResponse update vector needed for read-only replicas
S: disconnects


Supplier-initiated replication (SIR)
Consumer-initiated replication (CIR)
Full replica
Incremental Update

(see Ellenâ_TMs charts on protocol flow details for combinations)

Note that there could be problems if there is a firewall between the consumer and the server.

Questions: how do we handle unsolicited notifications? (there is an existing ldap draft, but it requires a modification to that. What about data compression? We know that there are several things that could benefit from this, and perhaps we should consider an additional compression phase in the protocol. This might make us revisit the startTLS draft. Keith recommends using a separable compression layer, as opposed to tweaking the data structures â_" please look at RFC 2393-2395.

How many extensions are you looking at? Four minimum, six maximum (we think). Note that we have an advantage in keeping the extensions separate, because as we evolve the protocol, this provides flexibility in determining what product supports what extension.

CIR should use SIR instead of turning around the connection.

Full and incremental update differ only in the protocol OID. Note that both use the same URP primitives.

General note to authors of this document: please be specific as to error codes and what they mean.

Further note: access control section is important, but there is an LDAPEXT sub-committee to do this, and all of the members of that subcommitte are also members of LDUP.

When you have two masters that are peers, itâ_TMs more complicated than just throwing away the updates of one â_" you need to bring them both into consistency.

Steven Legg - Conflict Detection and Resolution

Consistency: the contents of two DSAs that are replicating data should be the same.

There are a number of integrity constraints on the directory data, called DIT integrity (e.g., the constraints imposed by the underlying information model) and business rule integrity (e.g., every employee ID number must be unique). Note that an explicit assumption is that transient inconsistency can be tolerated.

In an enterprise database environment, there is a single administrative authority for the database. There is a unified and compatible set of business rule integrity constraints, imposed within a particular administrative or application domain. Every database application understands and obeys the same set of business rules.

In an Internet directory environment, things are harder . There are many administrative authorities, each with its own business rules. The only thing that we can get agreement on is DIT constraints. And since there is still major disagreement on attributes (e.g., in LDAPEXT, there was a debate on the use of cn â_" some people used it as a name, others put in simple identification information (e.g., employee ID), we can only rely on DIT constraints, meaning that business integrity is a real problem).

So therefore, we should be able to make updates to occur at a master DSA without communicating with any of the co-masters, and conflicts are detected after the fact. So what do we do? Disregard one of the updates in a conflict, or provide rules to harmonize the updates in the conflict by making corrections. This would be done by the administrator.

Can a conflict be ignored? And if so, can it be ignored silently? Second is definitely not good â_" we should at least tell the client what we are doing. And if a conflict resolution requires a human to resolve it, then we should tell that to the client too. The worry is that ignoring one update can have cascading effects on earlier and/or later updates. This has additional problems, as updates are usually interdependent, and puts an extra load on DSAs as updates are undone and redone. It also has some scaling problems. Finally it can be potentially confusing for users, as previously accepted updates are undone sometime later.

Concern over (mis)use of LDAP, and that LDUP may exacerbate the problem. Perhaps, as part of the LDUP activity, we could have the LDUP server provide operational, etc. information to the client to aid in its deployment.

Important thing at this stage in your work is to make certain that you can identify all cases and talk sensibly about what happens. Given that there are some set of failure cases, we better identify in advance what the failure mode is and how we will deal with it. In particular, what happens with an old client or server using a (new) LDUP server? The point is for us to understand the failure mode and document it.

There are things that we didn't do. It's very difficult to implement without rewinding the directory database to an earlier state. In addition, a partial replica may not see the same conflicts as the full replica. Finally, what do you do about lost updates? There's no guarantee that a user will catch these, so the burden is probably on the administrator.

What we are using is a locally-defined conflict resultion procedure. This is because the same conflict resolution algorithm MUST be used at all DSAs where an entry is mastered. The interaction of different conflict resolution algorithms will be too complex and subtle for directory administrators to get right, so it is MUCH easier if the SAME algorithm is applied everywhere.

The Update Reconciliation Procedures draft includes procedures for breaking down LDAP, DAP and DSP operations into more primitive actions. It appears to be compatiable with partial replica filtering, though arguably more work needs to be done here. It centers around defining procedures for applying replication primitives against the local directory database. A key point is that this draft is specifically written to allow extensions to these protocols, as well as adding whole new access protocols, to the list.

The main features include:
- updates may be seen out-of-order by mastering DSAs (URP accommodates arbitrary reordering)
- only the latest state of a directory item is of interest (changes to achieve that state are unimportant)
- URP ignores business rule integrity constraints (impossible to standardize)

(unfortunately, we ran out of time at this juncture. the chairs encourage you to go through Steven's full presentation and draft, and participate in ensuing discussions on the list)


Replication Transfer Protocol