Remove and re-add IPA replica on IdM cluster

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • IPA Server

Issue

If an IPA replica becomes un-fixable or the solution will take too much time, the option exists to remove and re-add the replica. This will usually fix the issue, although this procedure has the potential to detract from a root cause analysis, so this consideration must be considered.

Resolution

1. Remove the replica from the topology. These must be executed on a system other than the one that needs to be uninstalled.

# ipa-replica-manage del  <fqdn of replica to be removed> --force --cleanup
# ipa server-del <fqdn of the replica to be removed>
# ipa host-del <fqdn of the replica to be removed>

2. Check to make sure the IPA server can't find the replica.

# ipa host-find replica1.idm.example.com
# ipa topologysegment-find domain | grep -i replica1.idm.example.com

3. Use the ipa-replica-manage clean-ruv command to remove the remaining replica update vectors from the LDAP entries. Provide the replica update vector's ID as an argument.

[user@host ~]# ipa-replica-manage list-ruv
Directory Manager password: password

Replica Update Vectors:
        host.example.com:389: 4
        replica1.idm.example.com:389: 9
        replica2.idm.example.com:389: 10
Certificate Server Replica Update Vectors:
        host.example.com:389: 6
[user@host ~]# ipa-replica-manage clean-ruv 9

4. From the server that has the problem, run the uninstall commands:

# ipa-server-install --uninstall

5. Remove all pki-tomcat files and the PKI instance that may exist.

# pkidestroy -s CA -i pki-tomcat
# rm -rf /var/log/pki/pki-tomcat
# rm -rf /etc/sysconfig/pki-tomcat 
# rm -rf /etc/sysconfig/pki/tomcat/pki-tomcat 
# rm -rf /var/lib/pki/pki-tomcat 
# rm -rf /etc/pki/pki-tomcat 
# rm -rf /root/ipa.csr

6. Install the IdM server again. This command will be run on the IPA server you are trying to reinstall. (Please understand that this a generic template for the command, and reference should be made to the appropriate installation documentation for your RHEL version, linked at the bottom of this article):

# ipa-replica-install --principal admin --admin-password <password in single quotes> --setup-ca --server <primary FQDN> --domain <domain>

NOTE: If the reinstallation fails with the warning message below, it may be due to an empty or incorrect file that needs to be manually removed from the client side.

"WARNING Using existing certificate '/etc/ipa/ca.crt'

a). Remove the old /etc/ipa/ca.crt file, and try to reinstall.

# rm /etc/ipa/ca.crt

References:

Uninstalling an IdM server - RHEL 7

Uninstalling an IdM server - RHEL 8

Uninstalling an IdM server - RHEL 9

Root Cause

  • After updating the system, some services no longer work correctly.
  • After running the ipa-retore command the database ID is different.
  • Any IPA services are not working
  • The directory services are currently not functioning properly.
  • Unable to fix certificates

Diagnostic Steps

1. Check the logs in /var/log/dirsrv/

Jun 28 08:34:43 replica1 ns-slapd[1763]: [28/Jun/2023:08:34:43.000015880 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meTocmspidm0
2.idm.example.com" (idmmaster:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()

2. Run a connect check command to check if the replica can't communicate with the master server.

# /usr/sbin/ipa-replica-conncheck --replica master1.idm.example.com
Check connection from master to remote replica 'idm01.example.com':
Failed to connect to port 389 tcp on 192.168.1.2
   Directory Service: Unsecure port (389): FAILED
Failed to connect to port 636 tcp on 192.168.1.2
   Directory Service: Secure port (636): FAILED
Failed to connect to port 88 tcp on 192.168.1.2
   Kerberos KDC: TCP (88): FAILED
Failed to connect to port 88 udp on 192.168.1.2
   Kerberos KDC: UDP (88): WARNING
Failed to connect to port 464 tcp on 192.168.1.2
   Kerberos Kpasswd: TCP (464): FAILED
Failed to connect to port 464 udp on 192.168.1.2
   Kerberos Kpasswd: UDP (464): WARNING
Failed to connect to port 80 tcp on 192.168.1.2
   HTTP Server: Unsecure port (80): FAILED
Failed to connect to port 443 tcp on 192.168.1.2
   HTTP Server: Secure port (443): FAILED
The following UDP ports could not be verified as open: 88, 464
This can happen if they are already bound to an application
and ipa-replica-conncheck cannot attach own UDP responder.
ERROR: Port check failed! Inaccessible port(s): 389 (TCP), 636 (TCP), 88

3. Check A dirsrv logs:

# less /var/log/dirsrv/slapd-IDMEXAMPLE-LOCAL/errors 
[26/Aug/2023:19:06:36.483515588 +0000] - ERR - init_schema_dse_ext - Could not add attribute type "objectClass" to the schema: attribute type objectClass: Unknown attribute syntax OID "1.3.6.1.4.1.1466.115.121.1.15"
[26/Aug/2023:19:06:36.492435570 +0000] - ERR - dse_read_one_file - The entry cn=schema in file /usr/share/dirsrv/schema/00core.ldif (lineno: 1) is invalid,  error code 21 (Invalid syntax) - attribute type aci: Unknown attribute syntax OID "1.3.6.1.4.1.1466.115.121.1.15"
[26/Aug/2023:19:06:36.494793701 +0000] - ERR - setup_internal_backends - Please edit the file to correct the reported problems and then restart the server.

4. Command ipa host-add replica.server.fqdn --random fails with an error message:

"ipa: ERROR: 'replica1.idm.example.com' does not have a corresponding DNS A/AAAA record"

5. After add the DNS record, it says:

" ipa: ERROR: Host with name 'replica1.idm.example.com' already exists "

6. When trying to re-initialize the replica and get the below error:

# ipa-replica-manage re-initialize --from idmmaster.idm.example.com

Update in progress, 1441 seconds elapsed
[ldaps://replica1.idm.example.com:636] reports: Update failed! Status: [Error (-1) - LDAP error: Can't contact LDAP server - no response received]

7. After patching IPA, can't start Directory Manager:

# less /var/log/ipaupgrade.log
CalledProcessError: CalledProcessError(Command ['/bin/systemctl', 'start', 'dirsrv@IDMEXAMPLE-LOCAL.service'] returned non-zero exit status 1: 'Job for dirsrv@IDMEXAMPLE-LOCAL.service failed because the control process exited with error code.\nSee "systemctl status dirsrv@IDMEXAMPLE-LOCAL.service" and "journalctl -xeu dirsrv@IDMEXAMPLE-LOCAL.service" for details.\n')
2023-08-28T15:01:56Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
  • Component
  • ipa

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments