Password changes fail to sync with Active Directory in IPA after IPA upgrade

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 7 and above

  • IPA server. The issue is observed in packages as early (but not limited to) the following:

    • ipa-server-4.5.0-22.el7_4.x86_64 and above
    • 389-ds-base-1.3.1.2-1.el7.x86_64 and above

  • IPA servers replicating with each other with one or more using WinSync to replicate with AD

  • IPA servers recently were upgraded via ipa-server-upgrade and passwords were changed on those servers

Issue

  • I am unable to resynchronize IPA server with Active Directory (AD)
  • Password changes on IPA with WinSync synchronize with AD, but replicated IPA servers do not synchronize password changes

  • IPA replication on our WinSync server is broken with the following error:

    # ipa-replica-manage list -v `hostname -f`
hostname0.thecompany.com: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (0) Replica acquired successfully: Incremental update succeeded
  last update ended: 2021-07-06 13:04:45+00:00
hostname1.thecompany.com: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (0) Replica acquired successfully: Incremental update succeeded
  last update ended: 2021-07-06 13:04:45+00:00
hostname2.thecompany.com: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (-1) Incremental update has failed and requires administrator action - LDAP error: Can't contact LDAP server
  last update ended: 2021-07-05 09:26:50+00:00
[...]

# grep hostname2.thecompany.com /var/log/dirsrv/slapd-THECOMPANY-COM/errors | tail -3
[21/Jun/2021:10:26:00.572514553 -0500] - ERR - NSMMReplicationPlugin - windows sync - windows_replay_update - agmt="cn=meTohostname2.thecompany.com" (hostname2:389) - Failed map dn for modify operation dn="uid=1000000,cn=users,cn=accounts,dc=thecompany,dc=com" rc=-1 remote_dn = [(null)]
[23/Jun/2021:14:09:22.345466330 -0500] - ERR - NSMMReplicationPlugin - windows sync - windows_replay_update - agmt="cn=meTohostname2.thecompany.com" (hostname2:389) - Failed map dn for modify operation dn="uid=1000000,cn=users,cn=accounts,dc=thecompany,dc=com" rc=-1 remote_dn = [(null)]
[23/Jun/2021:14:11:09.477790928 -0500] - ERR - NSMMReplicationPlugin - windows sync - windows_replay_update - agmt="cn=meTohostname2.thecompany.com" (hostname2:389) - Failed map dn for modify operation dn="uid=1000000,cn=users,cn=accounts,dc=thecompany,dc=com" rc=-1 remote_dn = [(null)]

Resolution

Red Hat Enterprise Linux 7

  • Red Hat advises to either upgrade to Red Hat Enterprise Linux 7.9 or migrate the environment to Red Hat Enterprise Linux 8.4 or above in order to proceed.
  • Once the system is fully upgraded to Red Hat Enterprise Linux 7.9 (or in case the environment migrates to Red Hat Enterprise Linux 8.4 or above) Red Hat recommends to either re-create the WinSync agreement or setup a cross-forest trust.

Red Hat Enterprise Linux 8

  • Red Hat advises to upgrade to Red Hat Enterprise Linux 8.4 or above in order to proceed.
  • Once the system is fully upgraded to Red Hat Enterprise Linux 8.4 or above, Red Hat recommends to either re-create the WinSync agreement or setup a cross-forest trust.

Tips on system upgrade:

  • Take a snapshot/make a backup of the systems before starting
  • Before starting the upgrade, make sure the systems with CA installed (check with 'ipa config-show') do not have expired certs using 'getcert list'.
  • Start the update process in the CA renewal master (identify the CA renewal master using 'ipa config-show' ).
  • Keep all servers in the same Red Hat Enterprise Linux version.
  • Give about 10 minutes between updating each system to let everything catch up and resynchronize.

Note WinSync is no longer actively developed in Red Hat Enterprise Linux 8. The preferred solution for indirect integration is cross-forest trust, which is also an option available in Red Hat Enterprise Linux 7.

Workaround

  • If neither option above is feasible, the following command will need to be ran after every IPA upgrade on each replica:

    # ldapmodify -D 'cn=directory manager' -W << EOF

dn: cn=config
changetype: modify
replace: nsslapd-unhashed-pw-switch
nsslapd-unhashed-pw-switch: on
EOF

Root Cause

  • The IPA server upgrade script was modified in order to disable keeping track of unhashed user passwords in the main replication changelog. Unfortunately, the change did not properly set the defaults to on for IPA servers with WinSync and instead disables management of unhashed passwords.
  • Traditionally, 389 DS stored unhashed passwords in changelogs to allow for the passwords to be handed to plugins. For times where this is not needed, disabling storing of unhashed passwords became preferable. As such, 389 DS was modified to optionally disable managing unhashed passwords in changelogs.
  • This change to 389 DS was revisited; the default configuration in the above RFEs was set to enable storing unhashed passwords, but the default needed to change depending on the use of IPA and instances of WinSync in IPA. Additional RFEs were filed in order to modify the default options to not log these unhashed passwords in changelogs in default IPA installations and enable it for IPA installations using WinSync replications with AD, as AD needs those passwords unhashed.
  • This change to default values were not appropriately set for WinSync replications within IPA (which need to be on) and instead set the default value (nolog) for IPA installations.

Diagnostic Steps

To reproduce the issue

  1. Ensure an IPA environment has multiple IPA servers replicating with each other.
  2. Ensure at least one of those IPA servers has a WinSync agreement with AD. This WinSync agreement sets nsslapd-unhashed-pw-switch: on for that single IPA server

  3. Ensure nsslapd-unhashed-pw-switch: on is set for all of the other non-WinSync IPA servers so password syncing can function, e.g.

    ldapmodify -D 'cn=directory manager' -W
dn: cn=config
changetype: modify
replace: nsslapd-unhashed-pw-switch
nsslapd-unhashed-pw-switch: on

  4. Change user passwords on each of the IPA servers

  5. Run ipa-server-upgrade on each of the IPA servers
  6. Change user passwords on each of the IPA servers

Checking results

  • Password changes on the IPA server with WinSync agreement are synced to AD
  • Password changes on the other IPA servers are still synced but not to AD

  • Commands and logs may contain errors similar to the following:

    # ipa-replica-manage list -v `hostname -f`
hostname0.thecompany.com: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (0) Replica acquired successfully: Incremental update succeeded
  last update ended: 2021-07-06 13:04:45+00:00
hostname1.thecompany.com: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (0) Replica acquired successfully: Incremental update succeeded
  last update ended: 2021-07-06 13:04:45+00:00
hostname2.thecompany.com: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (-1) Incremental update has failed and requires administrator action - LDAP error: Can't contact LDAP server
  last update ended: 2021-07-05 09:26:50+00:00
[...]

# grep hostname2.thecompany.com /var/log/dirsrv/slapd-THECOMPANY-COM/errors | tail -3
[21/Jun/2021:10:26:00.572514553 -0500] - ERR - NSMMReplicationPlugin - windows sync - windows_replay_update - agmt="cn=meTohostname2.thecompany.com" (hostname2:389) - Failed map dn for modify operation dn="uid=1000000,cn=users,cn=accounts,dc=thecompany,dc=com" rc=-1 remote_dn = [(null)]
[23/Jun/2021:14:09:22.345466330 -0500] - ERR - NSMMReplicationPlugin - windows sync - windows_replay_update - agmt="cn=meTohostname2.thecompany.com" (hostname2:389) - Failed map dn for modify operation dn="uid=1000000,cn=users,cn=accounts,dc=thecompany,dc=com" rc=-1 remote_dn = [(null)]
[23/Jun/2021:14:11:09.477790928 -0500] - ERR - NSMMReplicationPlugin - windows sync - windows_replay_update - agmt="cn=meTohostname2.thecompany.com" (hostname2:389) - Failed map dn for modify operation dn="uid=1000000,cn=users,cn=accounts,dc=thecompany,dc=com" rc=-1 remote_dn = [(null)]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments