15.6. Configuring Multi-Master Replication

In a multi-master configuration, many suppliers can accept updates, synchronize with each other, and update all consumers. The consumers can send referrals for updates to all masters.
Directory Server supports 20-way multi-master replication, meaning that there can be up to 20 masters (and an unlimited number of hub suppliers) in a single replication scenario. Directory Server allows an unlimited number of consumers.
To set up multi-master replication, set up all of the consumers first, then set up the suppliers, and last, initialize all of the databases.

Note

More than 10 databases running with replication or more than on a supplier can cause performance degradation. To support that many consumers, introduce hub replicas between the suppliers and consumers. See Section 15.7, “Configuring Cascading Replication”.

15.6.1. Configuring the Read-Write Replicas on the Supplier Servers

Set up each supplier server. The first supplier configured should be used to initialize the other suppliers in the multi-master replication environment.
  1. Specify the supplier settings for the server.
    1. In the Directory Server Console, select the Configuration tab.
    2. In the navigation tree, select the Replication folder.
    3. In the right-hand side of the window, select the Supplier Settings tab.
    4. Check the Enable Changelog check box.
      This activates all of the fields in the pane below that were previously grayed out.
    5. Specify a changelog by clicking the Use default button, or click the Browse button to display a file selector.
    6. Set the changelog parameters for the number and age of the log files.
      Clear the unlimited check boxes to specify different values.
    7. Click Save.
  2. Create the entry for the supplier bind DN on the consumer server if it does not exist. This is the special entry that the other suppliers will use to bind to this supplier, as in other supplier-consumer relationships. This is described in Section 15.4, “Creating the Supplier Bind DN Entry”.

    Note

    For multi-master replication, it is necessary to create this supplier bind DN on the supplier servers as well as the consumers because the suppliers act as both consumer and supplier to the other supplier servers.
  3. Specify the replication settings for the multi-mastered read-write replica.
    1. In the Directory Server Console, select the Configuration tab.
    2. In the navigation tree, expand the Replication folder, and highlight the replica database.
      The Replica Settings tab for that database opens in the right-hand side of the window.
    3. Check the Enable Replica check box.
    4. In the Replica Role section, select the Multiple Master radio button.
    5. In the Common Settings section, specify a Replica ID, which is an integer between 1 and 65534, inclusive.
      The replica ID must be unique for a given suffix, different from any other ID used for read-write replicas on this server and on other servers.
    6. In the Common Settings section, specify a purge delay in the Purge delay field.
      The purge delay is how often the state information stored for the replicated entries is deleted.
    7. In the Update Settings section, specify the bind DN that the supplier will use to bind to the replica. Enter the supplier bind DN in the Enter a new Supplier DN field, and click Add. The supplier bind DN appears in the Current Supplier DNs list.
      The supplier bind DN should be the entry created in step 2. The supplier bind DN is a privileged user because it is not subject to access control in the replicated database.

      Note

      There can be multiple supplier bind DNs per consumer but only one supplier DN per replication agreement.
    8. Specify the LDAP URL (ldap://hostname:port or ldap://IP_address:port, with IPv4 or IPv6 addresses) for any supplier servers to which to refer updates, such as the other suppliers in the multi-master replication set. Only specify the URL for the supplier server.
      For clients to bind using TLS, specify a URL beginning with ldaps://.
    9. Click Save.

15.6.2. Configuring the Read-Only Replicas on the Consumer Servers

First, configure every consumer before creating any replication agreements.
  1. Create the database for the read-only replica if it does not exist. See Section 2.1.1, “Creating Suffixes” for instructions on creating suffixes.
  2. Create the entry for the supplier bind DN on the consumer server if it does not exist. The supplier bind DN is the special entry that the supplier will use to bind to the consumer. This is described in Section 15.4, “Creating the Supplier Bind DN Entry”.
  3. Specify the replication settings required for a read-only replica.
    1. In the Directory Server Console, select the Configuration tab.
    2. In the navigation tree, expand the Replication folder, and highlight the replica database.
      The Replica Settings tab for that database opens in the right-hand side of the window.
    3. Check the Enable Replica check box.
    4. In the Replica Role section, select the Dedicated Consumer radio button.
    5. In the Common Settings section, specify a purge delay in the Purge delay field.
      This option indicates how often the state information stored for the replicated entries is purged.
    6. In the Update Settings section, specify the bind DN that the supplier will use to bind to the replica. Enter the supplier bind DN in the Enter a new Supplier DN field, and click Add. The supplier bind DN appears in the Current Supplier DNs list.
      The supplier bind DN should be the entry created in step 2. The supplier bind DN is a privileged user because it is not subject to access control in the replicated database.

      Note

      There can be multiple supplier bind DNs per consumer but only one supplier DN per replication agreement.
    7. Specify the URL for any supplier servers to which to refer updates.
      By default, all updates are first referred to the supplier servers that are specified here. If no suppliers are set here, updates are referred to the supplier servers that have a replication agreement that includes the current replica.
      Automatic referrals assume that clients bind over a regular connection; this has a URL in the form ldap://hostname:port. For clients to bind to the supplier using TLS, use this field to specify a referral of the form ldaps://hostname:port, where the s in ldaps indicates a secure connection.

      Note

      It is also possible to use IPv4 or IPv6 addresses instead of the host name.
  4. Click Save.
Repeat these steps for every consumer server in the replication configuration.

15.6.3. Setting up the Replication Agreements

Note

  1. First set up replication agreements on a single supplier, the data master, between the other multi-master suppliers, and initialize all of the other suppliers.
  2. Then create replication agreements for all other suppliers in the multi-master replication set, but do not reinitialize any of the suppliers.
  3. Then create replication agreements for all of the consumers from the single data master, and initialize the consumers.
  4. Then create replication agreements for all of the consumers from for all of the other suppliers, but do not reinitialize any of the consumers.
  1. In the navigation tree of the Configuration tab, right-click the database to replicate, and select New Replication Agreement.
    Alternatively, highlight the database, and select New Replication Agreement from the Object menu to start the Replication Agreement Wizard.
  2. In the first screen, fill in a name and description for the replication agreement, and hit Next.
  3. In the Source and Destination screen, fill in the URL (hostname:port or IP_address:port, with IPv4 or IPv6 addresses) for the consumer and the supplier bind DN and password on that consumer. If the target server is not available, hit in other to fill in the information manually.
    • Unless there is more than one instance of Directory Server configured, by default, there are no consumers available in the drop-down menu. The server URL can be entered manually, in the format hostname:port or IP_address:port, with IPv4 or IPv6 addresses.
    • The port listed is the non-TLS port, even if the Directory Server instance is configured to run over TLS. This port number is used only for identification of the Directory Server instance in the Console; it does not specify the actual port number or protocol that is used for replication.
    • If TLS is enabled on the servers, it is possible to select the Using encrypted SSL connection radio button for TLS client authentication. Otherwise, fill in the supplier bind DN and password.

      Note

      If attribute encryption is enabled, a secure connection is required for the encrypted attributes to be replicated.
  4. Select the connection type. There are three options:
    • Use LDAP. This sets a standard, unencrypted connection.
    • Use TLS/SSL. This uses a secure connection over the server's secure LDAPS port, such as 636. This setting is required to use TLS.
    • Use Start TLS. This uses Start TLS to establish a secure connection over the server's standard port.

    Note

    If secure binds are required for simple password authentication (Section 19.11.1, “Requiring Secure Binds”), then any replication operations will fail unless they occur over a secure connection. Using a secure connection (TLS and Start TLS connections or SASL authentication) is recommended, anyway.
  5. Select the appropriate authentication method and supply the required information. This gives the information that the supplier uses to authenticate and bind to the consumer server to send updates.
    • Simple means that the server connects over the standard port with no encryption. The only required information is the bind DN and password for the Replication Manager (which must exist on the consumer server).
    • Server TLS/SSL Certificate uses the supplier's TLS certificate to authenticate to the consumer server. A certificate must be installed on the supplier for certificate-based authentication, and the consumer server must have certificate mapping configured so that it can map the subject DN in the supplier's certificate to its Replication Manager entry.
      Configuring TLS and certificate mapping is described in Section 9.4, “Enabling TLS”.
    • SASL/DIGEST-MD5, like simple authentication, requires only the bind DN and password to authenticate. This can run over a standard or TLS connection.
    • SASL/GSSAPI requires the supplier server to have a Kerberos keytab (as in Section 9.10.2.2, “About the KDC Server and Keytabs”), and the consumer server to have a SASL mapping to map the supplier's principal to the real replication manager entry (as in Section 9.9.3.1, “Configuring SASL Identity Mapping from the Console”).
  6. Hit Next.
  7. Fractional replication controls which entry attributes are replicated between servers. By default, all attributes are replicated. To select attributes that will not be replicated to the consumer, check the Enable Fractional Replication check box. Then, highlight the attribute (or attributes) in the Included column on the right, and click Remove. All attributes that will not be replicated are listed in the Excluded column on the left, as well as in the summary the replication agreement is complete.
  8. Set the schedule for when replication runs. By default, replication runs continually.

    Note

    The replication schedule cannot cross midnight (0000). So, it is possible to set a schedule that begins at 0001 and ends at 2359 on the same day, but it is not possible to set one that begins at 2359 on one day and ends at 0001 on the next.
    Hit Next.
  9. Set when the consumer is initialized. Initializing a consumer manually copies all data over from the supplier to the consumer. The default is to create an initialization file (an LDIF of all supplier data) so that the consumer can be initialized later. It is also possible to initialize the consumer as soon as the replication agreement is completed or not at all. For information on initializing consumers, see Section 15.18, “Initializing Consumers”. For multi-master replication, consider the following:
    • Ensure one supplier has the complete set of data to replicate to the other suppliers. Use this one supplier to initialize the replica on all other suppliers in the multi-master replication set.
    • Initialize the replicas on the consumer servers from any of the multi-master suppliers.
    • Do not try to reinitialize the servers when the replication agreements are set up. For example, do not initialize server1 from server2 if server2 has already been initialized from server1. In this case, select Do not initialize consumer.

    Note

    Replication will not begin until the consumer is initialized.

    Important

    For multi-master replication, be sure that consumers are only initialized once, by one supplier. When checking the replication status, be sure to check the replication agreement entry, on the appropriate supplier, which was used to initialize the consumer.
    Hit Next.
  10. The final screen shows the settings for the replication agreement, as it will be included in the dse.ldif file. Hit Done to save the agreement.
The replication agreement is set up.

Note

At the end of this procedure, all supplier servers will have mutual replication agreements, which means that they can accept updates from each other.

Note

After creating a replication agreement, the connection type (TLS or non-TLS) cannot be changed because LDAP and LDAPS connections use different ports. To change the connection type, re-create the replication agreement.

15.6.4. Preventing Monopolization of a Consumer in Multi-Master Replication

One of the features of multi-master replication is that a supplier acquires exclusive access to the consumer for the replicated area. During this time, other suppliers are locked out of direct contact with the consumer. If a supplier attempts to acquire access while locked out, the consumer sends back a busy response, and the supplier sleeps for several seconds before making another attempt. During a low update load, the supplier sends its update to another consumer while the first consumer is locked and then send updates when the first consumer is free again.
A problem can arise if the locking supplier is under a heavy update load or has a lot of pending updates in the changelog. If the locking supplier finishes sending updates and then has more pending changes to send, it will immediately attempt to reacquire the consumer and will most likely succeed, since the other suppliers usually will be sleeping. This can cause a single supplier to monopolize a consumer for several hours or longer.
The following attributes address this issue:
nsds5ReplicaBusyWaitTime
Sets the time in seconds a supplier waits after a consumer sends back a busy response before making another attempt to acquire access.
For example, to configure that a supplier waits 5 seconds before making another acquire attempt:
# ldapmodify -D "cn=Directory Manager" -W -p 389 -h server.example.com -x

dn: cn=Replication_Agreement_Name,cn=replica,cn=suffix_Name,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaBusyWaitTime
nsds5ReplicaBusyWaitTime: 5
nsds5ReplicaSessionPauseTime
Sets the time in seconds a supplier waits between two update sessions. If you set a value lower or equal than the value specified in nsds5ReplicaBusyWaitTime, Directory Server automatically uses a value for the nsds5ReplicaSessionPauseTime parameter, that is one second higher than the value set in nsds5ReplicaBusyWaitTime.
For example, to configure that the supplier waits 10 seconds between two update sessions:
# ldapmodify -D "cn=Directory Manager" -W -p 389 -h server.example.com -x

dn: cn=Replication_Agreement_Name,cn=replica,cn=suffix_Name,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaSessionPauseTime
nsds5ReplicaSessionPauseTime: 10
nsds5ReplicaReleaseTimeout
Sets the timeout after which a master will release the replica, whether or not it has finished sending its updates. This prevents a single master from monopolizing a replica.
For example, to configure in a heavy replication environment that a master will release a replica after 90 seconds:
# ldapmodify -D "cn=Directory Manager" -W -p 389 -h server.example.com -x

dn: cn=replica,cn=suffix_Name,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaReleaseTimeout
nsds5ReplicaReleaseTimeout: 90
For further details, see the parameter descriptions in the Red Hat Directory Server Configuration, Command, and File Reference.
To log replica busy errors, enable Replication error logging (log level 8192). See Section 20.3.7, “Configuring Log Levels”.