15.3. Multi-Master Replication

In a multi-master replication scenario, the master copies of the directory data are stored on multiple read-write replicas. Each of these servers maintains a changelog for the read-write replica. Directory Server supports up to 20 masters in a replication topology.

Note

Each supplier in a multi-master replication environment is also a consumer automatically.
The following diagram shows a multi-master replication environment with two masters:
Multi-master Replication with Two Masters

Figure 15.2. Multi-master Replication with Two Masters

In complex environments, replication topologies often contain multiple read-write masters as well as read-only consumers. The following diagram shows a topology where each master is configured with ten replication agreements to replicate data to two other masters and eight consumers:
Complex Replication Scenario with Four Masters and Eight Consumers

Figure 15.3. Complex Replication Scenario with Four Masters and Eight Consumers

Note

The replication speed depends on:
  • The speed of the network.
  • The number of outgoing and incoming replication agreements.
Use the command line or web console to set up a multi-master replication topology. See:

15.3.1. Setting up Multi-master Replication Using the Command Line

The following example assumes that you have an existing Directory Server instance running on a host named master1.example.com. The following procedures describe how to add another write-write replica named master2.example.com to the topology, and how to configure multi-master replication for the dc=example,dc=com suffix.

Preparing the New Server to Join

On the master2.example.com host:
  1. Install Directory Server, and create an instance. For details, see the Red Hat Directory Server Installation Guide.
  2. In case you created the instance without a database, create the database for the suffix. For example, to create a database named userRoot for the dc=example,dc=com suffix:
    # dsconf -D "cn=Directory Manager" ldap://master2.example.com backend \
        create --suffix="dc=example,dc=com" --be-name="userRoot"
    For details on creating a database for a suffix, see Section 2.1.1, “Creating Suffixes”.
  3. Enable replication for the suffix, and create the replication manager account:
    # dsconf -D "cn=Directory Manager" ldap://master2.example.com replication \
        enable --suffix="dc=example,dc=com" --role="master" --replica-id=1 \
        --bind-dn="cn=replication manager,cn=config" --bind-passwd="password"
    This command configures the master2.example.com host as a master for the dc=example,dc=com suffix, and sets the replica ID for this entry to 1. Additionally, the server creates the cn=replication manager,cn=config user with the specified password, and allows this account to replicate changes for the suffix to this host.

    Important

    The replica ID must be a unique integer between 1 and 65534 for a suffix across all masters in the topology.

Configuring the Existing Server as a Supplier

On the master1.example.com host:
  1. Similarly to the command you ran on the new server to join, enable replication for the dc=example,dc=com suffix, and create the replication manager account:
    # dsconf -D "cn=Directory Manager" ldap://master1.example.com replication \
        enable --suffix="dc=example,dc=com" --role="master" --replica-id=2 \
        --bind-dn="cn=replication manager,cn=config" --bind-passwd="password"
    The replica ID must be different than the one created in the section called “Preparing the New Server to Join”, but the replication manager account can use the same DN.
  2. Add the replication agreement, and initialize a new server. For example:
    # dsconf -D "cn=Directory Manager" ldap://master1.example.com repl-agmt \
         create --suffix="dc=example,dc=com" --host="master2.example.com" --port=636 \
         --conn-protocol=LDAPS --bind-dn="cn=replication manager,cn=config" \
         --bind-passwd="password" --bind-method=SIMPLE --init \
         example-agreement-master1-to-master2
    This command creates a replication agreement named example-agreement-master1-to-master2. The replication agreement defines settings, such as the consumer's host name, protocol, and authentication information that the supplier uses when connecting and replicating data to the consumer.
    After the agreement was created, Directory Server initializes the consumer. To initialize the consumer later, omit the --init option. Note that replication does not start before you initialize the consumer. For details on initializing a consumer, see Section 15.7.3, “Initializing a Consumer”.
    For further details about the options used in the command, enter:
    # dsconf -D "cn=Directory Manager" ldap://master1.example.com repl-agmt --help
  3. Verify whether the initialization was successful:
    # dsconf -D "cn=Directory Manager" ldap://master1.example.com repl-agmt \
         init-status --suffix="dc=example,dc=com" example-agreement-master1-to-master2
    Agreement successfully initialized.
    Depending on the amount of data to replicate, the initialization can take be time-consuming.

Configuring the New Server as a Supplier

On the master2.example.com host:

Warning

Do not continue if you have not initialized the replication agreement on the existing server as described in the section called “Configuring the Existing Server as a Supplier”. Otherwise, the empty database from the new server overrides the database on the existing master.
  1. Add the replication agreement, and initialize the server. For example:
    # dsconf -D "cn=Directory Manager" ldap://master2.example.com repl-agmt \
         create --suffix="dc=example,dc=com" --host="master2.example.com" --port=636 \
         --conn-protocol=LDAPS --bind-dn="cn=replication manager,cn=config" \
         --bind-passwd="password" --bind-method=SIMPLE --init \
         example-agreement-master2-to-master1
    After the agreement was created, Directory Server initializes the consumer. To initialize the consumer later, omit the --init option.
  2. Verify whether the initialization was successful:
    # dsconf -D "cn=Directory Manager" ldap://master1.example.com repl-agmt \
         init-status --suffix="dc=example,dc=com" example-agreement-master2-to-master1
    Agreement successfully initialized.

15.3.2. Setting up Multi-master Replication Using the Web Console

The following example assumes that you have an existing Directory Server instance running on a host named master1.example.com. The following procedures describe how to add another write-write replica named master2.example.com to the topology, and how to configure multi-master replication for the dc=example,dc=com suffix.

Preparing the New Server to Join

On the master2.example.com host:
  1. Install Directory Server, and create an instance. For details, see the Red Hat Directory Server Installation Guide.
  2. Open the Directory Server user interface in the web console. See Section 1.4, “Logging Into Directory Server Using the Web Console”.
  3. Select the instance.
  4. In case you created the instance without a database, create the database from the suffix. For details about creating a database for a suffix, see Section 2.1.1, “Creating Suffixes”.
  5. Enable replication for the suffix:
    1. Open the Replication menu.
    2. Select the dc=example,dc=com suffix, and click Enable Replication.
    3. Select Master in the Replication Role field, enter a replica ID, as well as the DN and password of the replication manager account to create. For example:
      These settings configure the master2.example.com host as a master for the dc=example,dc=com suffix, and set the replica ID for this entry to 1. Additionally, the server creates the cn=replication manager,cn=config user with the specified password, and allows this account to replicate changes for the suffix to this host.

      Important

      The replica ID must be a unique integer between 1 and 65534 for a suffix across all masters in the topology.
    4. Click Enable Replication.

Configuring the Existing Server as a Supplier

On the master1.example.com host:
  1. Open the Directory Server user interface in the web console. See Section 1.4, “Logging Into Directory Server Using the Web Console”.
  2. Select the instance.
  3. Similarly to the settings on the new server to join, enable replication for the dc=example,dc=com suffix, and create a replication manager account:
    1. Open the Replication menu.
    2. Select the dc=example,dc=com suffix, and click Enable Replication.
    3. Select Master in the Replication Role field, enter a replica ID, as well as the DN and password of the replication manager account to create. For example:
      The replica ID must be different than the one created in the section called “Preparing the New Server to Join”, but the replication manager account can use the same DN.
    4. Click Enable Replication.
  4. Add the replication agreement and initialize the consumer:
    1. Open the Replication menu, and select the Agreements entry.
    2. Click Create Replication Agreement, and fill the fields. For example:
      These settings create a replication agreement named example-agreement-master1-to-master2. The replication agreement defines settings, such as the consumer's host name, protocol, and authentication information that the supplier uses when connecting and replicating data to the consumer.
    3. Select Do Online Initialization in the Consumer Initialization field to automatically initialize the consumer after saving the agreement.
      To initialize the consumer later, select Do Not Initialize. Note that replication does not start before you initialize the consumer. For details on initializing a consumer, see Section 15.7.3, “Initializing a Consumer”.
    4. Click Save Agreement.
  5. Verify whether the initialization was successful:
    1. Open the Replication menu.
    2. Select the Agreements entry.
      For a successfully-completed initialization, the web console displays the Error (0) Replica acquired successfully: Incremental update succeeded message in the Last Update Status column.
      Depending on the amount of data to replicate, the initialization can be time-consuming.

Configuring the New Server as a Supplier

On the master2.example.com host:

Warning

Do not continue if you have not initialized the replication agreement on the existing server as described in the section called “Configuring the Existing Server as a Supplier”. Otherwise, the empty database from the new server overrides the database on the existing master.
  1. Add the replication agreement, and initialize the consumer:
    1. Open the Replication menu, and select the Agreements entry.
    2. Click Create Replication Agreement, and fill the fields. For example:
      These settings create a replication agreement named example-agreement-master2-to-master1.
    3. Select Do Online Initialization in the Consumer Initialization field to automatically initialize the consumer after saving the agreement.
      To initialize the consumer later, select Do Not Initialize. Note that replication does not start before you initialize the consumer. For details on initializing a consumer, see Section 15.7.3, “Initializing a Consumer”.
    4. Click Save Agreement.
  2. Verify whether the initialization was successful:
    1. Open the Replication menu.
    2. Select the Agreements entry.
      If the initialization completed successfully, the web console displays the Error (0) Replica acquired successfully: Incremental update succeeded message in the Last Update Status column.
      Depending on the amount of data to replicate, the initialization be time-consuming.

15.3.3. Preventing Monopolization of a Consumer in Multi-Master Replication

One of the features of multi-master replication is that a supplier acquires exclusive access to the consumer for the replicated area. During this time, other suppliers are locked out of direct contact with the consumer. If a supplier attempts to acquire access while locked out, the consumer sends back a busy response, and the supplier sleeps for several seconds before making another attempt. During a low update load, the supplier sends its update to another consumer while the first consumer is locked, and then sends updates when the first consumer is free again.
A problem can arise if the locking supplier is under a heavy update load or has a lot of pending updates in the changelog. If the locking supplier finishes sending updates and has multiple pending changes to send, it immediately attempts to reacquire the consumer. Such attempt in most cases succeeds, because other suppliers are usually sleeping. This can cause a single supplier to monopolize a consumer for several hours or longer.
The following attributes address this issue:
nsds5ReplicaBusyWaitTime
Sets the time in seconds for a supplier to wait after a consumer sends back a busy response before making another attempt to acquire access.
For example, to configure that a supplier waits 5 seconds before making another acquire attempt:
# dsconf -D "cn=Directory Manager" ldap://supplier.example.com repl-agmt \
     set --suffix="suffix" --busy-wait-time=5 agreement_name
nsds5ReplicaSessionPauseTime
Sets the time in seconds for a supplier to wait between two update sessions. If you set a value lower or equal than the value specified in nsds5ReplicaBusyWaitTime, Directory Server automatically uses the value for the nsds5ReplicaSessionPauseTime parameter that is one second higher than the value set in nsds5ReplicaBusyWaitTime.
For example, to configure that the supplier waits 10 seconds between two update sessions:
# dsconf -D "cn=Directory Manager" ldap://supplier.example.com repl-agmt \
     set --suffix="suffix" --session-pause-time=10 agreement_name
nsds5ReplicaReleaseTimeout
Sets the timeout after which a master releases the replica, whether or not it has finished sending its updates. This prevents a single master from monopolizing a replica.
For example, to configure a master to release a replica after 90 seconds in a heavy replication environment:
# dsconf -D "cn=Directory Manager" ldap://supplier.example.com replication \
     set --suffix="suffix" --repl-release-timeout=90
For further details, see the parameter descriptions in the Red Hat Directory Server Configuration, Command, and File Reference.
To log replica busy errors, enable Replication error logging (log level 8192). See Section 21.3.7, “Configuring the Log Levels”.