Red Hat Directory Server / RHDS replication and status "Can't acquire busy replica. Code 1"

Solution Verified - Updated -

Environment

RHEL 7, 8,9
RHDS 10, 11, 12

Issue

There is a high traffic Red Hat Directory Server / RHDS replication environment, and the Directory Server console show a replication status such as:

Can't acquire busy replica. Code 1

Why is that? Is replication broken?

Resolution

This replication status message is "normal", and may happen in busy replica environment, it does not mean replication is broken, it can be ignored as long as updates are sent through other replica and there is no divergence.

Root Cause

Getting a "replica busy" message is "normal".
This can happen when two suppliers attempt to update the same consumer at the same time.
One of them gets through, the other has to wait.
The way replication works is that the supplier acquires an exclusive lock on the consumer for the duration of replication session.
Any other supplier attempting to update that particular consumer will get a "replica busy" reply.
If the same supplier keeps getting the same consumer and always times it just right to lock out the other suppliers, this is not a problem - updates from the "locked out" suppliers will still go to the "locking" supplier and then to the consumer.

Diagnostic Steps

Example:

ldapsearch -xLLL -D "cn=directory manager" -W -b cn=mapping\ tree,cn=config cn=m1toh1 nsds5replicaLastUpdateStatus nsds5replicaUpdateInProgress 
dn: cn=m1toh1,cn=replica,cn=dc\3Dexample,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: 1 Can't acquire busy replica
nsds5replicaUpdateInProgress: FALSE

Source code:

ldapserver/ldap/servers/plugins/replication/repl5.h:#define NSDS50_REPL_REPLICA_BUSY 0x01 /* Replica busy, try later */

ldapserver/ldap/servers/plugins/replication/repl5_agmt.c:                                   "%d Can't acquire busy replica", replrc );

void
agmt_set_last_update_status (Repl_Agmt *ra, int ldaprc, int replrc, const char *message)
{
        PR_ASSERT(NULL != ra);
        if (NULL != ra)
        {
                if (replrc == NSDS50_REPL_UPTODATE)
                {
                        /* no session started, no status update */
                }
                else if (ldaprc != LDAP_SUCCESS)
                {
...
                else if (replrc != 0)
                {
                        if (replrc == NSDS50_REPL_REPLICA_BUSY)
                        {
                                PR_snprintf(ra->last_update_status, STATUS_LEN,
                                        "%d Can't acquire busy replica", replrc ); 
                        }
...

and set in 4 places in 
./ldap/servers/plugins/replication/repl_extop.c
can be a failed SLAPI connection, replica in config, total update against other replica, or try to lock a session

Online documentation links:
Used to be in:

Red Hat Directory Server 9.0 Administration Guide
Publication date: December 7, 2011, updated April 2, 2012
11.19.2. Monitoring Replication Status from Administration Express

in RHDS 8.2:

https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Administration_Guide/Managing_Replication-Monitoring_Replication_Status.html
"
Update Status   The number is the status code, and the string is the implication of the status code. Watch this column for possible deadlock if all the suppliers complain that they cannot acquire the busy replica. It is normal if one of the suppliers is doing an update while the others can't acquire the busy replica.
"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments