IdM/IPA replica install error - Replica has a different generation ID than the local data. - scenario with multiple replica install and un-install
Environment
- Red Hat Enterprise Linux
- IPA IdM
Issue
An IdM / IPA replica configuration may fail with error:
Replica has a different generation ID than the local data.
There may be different cause for this error, and this note will only focus on the possible scenario of multiple replica install and un-install attempts.
Another article explain the case of network errors between an IPA master and an IPA replica.
Resolution
1) Remove pre-existing replication agreements left on an IPA master for the targeted replica, after the replica un-installation:
[root@ipaserver1 ~]# ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=meToipaserver2.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: delete
EOF
2) Remove any tombstone(s) left on the IPA master for the targeted replica by running a CLEANRUV.
In order to run the CLEANRUV task, you will need to know:
- the replica config entry DN - you can find this using ldapsearch
# ldapsearch -xLLL -D "cn=directory manager" -W -s sub -b cn=config objectclass=nsds5replica
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
cn: replica
... more output ...
In this case, the DN to use is cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config - it looks funny because of the DN escapes, but they are necessary.
- the replica ID - you can get this from the RUV tombstone entry like this:
# ldapsearch -xLLL -D "cn=directory manager" -W -b dc=example,dc=com \
'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
Look at the nsds50ruv attribute - the replica ID is the number after the "{replica " in the RUV element - for example, in
{replica 55 ldap://localhost.localdomain:9389} 4e6a27ca000000370000 4e6a27e8000000370000
The replica ID is 55
To execute the CLEANRUV task, use ldapmodify:
# ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV55
EOF
Where the dn: .... is the DN of the replica config entry, and the number after CLEANRUV in
nsds5task: CLEANRUV55
is the replica ID of the entry you want to remove.
After running ldapmodify, you can use
# ldapsearch -xLLL -D "cn=directory manager" -W -b dc=example,dc=com \
'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
to see that the entry for the obsolete replica has been removed.
If there are multiple entries you need to get rid of, just execute the ldapmodify again with each replica ID number.
NOTE: The CLEANRUV operation does not replicate. You will have to perform this operation on all of your servers (masters, hubs, consumers).
Root Cause
Uninstalling an IPA replica on a replica with version 2.1.3-9 does not fully remove the replication configuration and tombstone on the IPA masters.
See the "Diagnostic Steps" section for long steps and details of examples.
Also see Red Hat Bugzilla number 784378 - Run CLEANRUV task when completely deleting a replica
and FreeIPA upstream case: https://fedorahosted.org/freeipa/ticket/2303
and upstream port389.org: https://fedorahosted.org/389/ticket/337
When a master is removed from a replicated environment the meta-data for that master is still contained in the other servers.
There is a special task you can use to remove this meta-data - the CLEANRUV task.
http://directory.fedoraproject.org/wiki/Howto:CLEANRUV
We are going to attempt to make this invisible for IPA users, running this automatically when a master is removed, so no explicit action from an admin will be necessary.
Latest ipa-replica-manage also includes list-ruv & clean-ruv options which can be used to clean ruv for an old replication agreement.
Diagnostic Steps
The full error on the IPA replica may be like this:
[root@ipaserver2 ~]# ipa-replica-install -d --skip-conncheck /root/replica-info-ipaserver2.example.com.gpg
...snip...
Starting replication, please wait until this has completed.
[06/Jun/2012:05:13:32 -0700] NSMMReplicationPlugin - agmt="cn=meToipaserver1.example.com" (ipaserver1:389): Replica has a different generation ID than the local data.
[ipaserver1.example.com] reports: Update failed! Status: [-2 - System error]
creation of replica failed: Failed to start replication
root : DEBUG Failed to start replication
File "/usr/sbin/ipa-replica-install", line 482, in <module>
main()
File "/usr/sbin/ipa-replica-install", line 433, in main
ds = install_replica_ds(config)
File "/usr/sbin/ipa-replica-install", line 135, in install_replica_ds
pkcs12_info)
File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py", line 284, in create_replica
self.start_creation("Configuring directory server", 60)
File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 248, in start_creation
method()
File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py", line 297, in __setup_replica
r_bindpw=self.dm_password)
File "/usr/lib/python2.6/site-packages/ipaserver/install/replication.py", line 694, in setup_replication
raise RuntimeError("Failed to start replication")
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
[root@ipaserver2 ~]#
[root@ipaserver2 ~]# ipa-server-install --uninstall
On the master, list the existing agreements, in that case, the targeted IPA replica is ipaserver2.example.com, it was installed and uninstalled several times, and the agreement was not removed from the IPA master ipaserver1.example.com:
ldapsearch -xLLL -D "cn=directory manager" -W -s sub -b cn=config objectclass=nsds5replicationagreement
Enter LDAP Password:
dn: cn=meToipaserver2.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=map
ping tree,cn=config
cn: meToipaserver2.example.com
objectClass: nsds5replicationagreement
objectClass: top
nsDS5ReplicaTransportInfo: LDAP
description: me to ipaserver2.example.com
nsDS5ReplicaRoot: dc=example,dc=com
nsDS5ReplicaHost: ipaserver2.example.com
nsds5replicaTimeout: 120
nsDS5ReplicaPort: 389
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicaUpdateSchedule: 0000-2359 0123456
nsds50ruv: {replicageneration} 4f6e5d8e000000040000
nsds50ruv: {replica 6 ldap://ipaserver2.example.com:389} 4fcf42e5001d00060000
4fcf44d1000000060000
nsds50ruv: {replica 4 ldap://ipaserver1.example.com:389} 4f9ee89f000000040000
4fcf438a000000040000
nsds50ruv: {replica 5 ldap://ipaclient1.example.com:389} 4f6e6ba1001d00050000
4f6e8256000000050000
nsds50ruv: {replica 3 ldap://ipaserver2.example.com:389} 4f6e5d95001c00030000
4f8f565c000000030000
nsruvReplicaLastModified: {replica 6 ldap://ipaserver2.example.com:389} 000000
00
nsruvReplicaLastModified: {replica 4 ldap://ipaserver1.example.com:389} 000000
00
nsruvReplicaLastModified: {replica 5 ldap://ipaclient1.example.com:389} 000000
00
nsruvReplicaLastModified: {replica 3 ldap://ipaserver2.example.com:389} 000000
00
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 0
nsds5replicaLastUpdateEnd: 0
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: -1 - System error
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 20120606130751Z
nsds5replicaLastInitEnd: 0
nsds5replicaLastInitStatus: -2 - System error
[root@ipaserver1 ~]#
Delete on the IPA master the replication agreement for the target replica, ipaserver2.example.com:
[root@ipaserver1 ~]# ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=meToipaserver2.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: delete
EOF
Verify:
ldapsearch -xLLL -D "cn=directory manager" -W -s sub -b cn=config objectclass=nsds5replicationagreement
Enter LDAP Password:
[root@ipaserver1 ~]#
Now, the second step is to clean up the RUV on the IPA master.
Perform the following search on one of the IPA servers to verify the replica id's in the tombstone:
[root@ipaserver1 ~]# ldapsearch -xLLL -D "cn=directory manager" -W -b dc=example,dc=com '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
Enter LDAP Password:
dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 4f6e5d8e000000040000
nsds50ruv: {replica 4 ldap://ipaserver1.example.com:389} 4f9ee89f000000040000
4fcf496f000000040000
nsds50ruv: {replica 8 ldap://ipaserver2.example.com:389}
nsds50ruv: {replica 7 ldap://ipaserver2.example.com:389}
nsds50ruv: {replica 6 ldap://ipaserver2.example.com:389} 4fcf42e5001d00060000
4fcf44d1000000060000
nsds50ruv: {replica 5 ldap://ipaclient1.example.com:389} 4f6e6ba1001d00050000
4f6e8256000000050000
nsds50ruv: {replica 3 ldap://ipaserver2.example.com:389} 4f6e5d95001c00030000
4f8f565c000000030000
dc: example
nsruvReplicaLastModified: {replica 4 ldap://ipaserver1.example.com:389} 4fcf49
6c
nsruvReplicaLastModified: {replica 8 ldap://ipaserver2.example.com:389} 000000
00
nsruvReplicaLastModified: {replica 7 ldap://ipaserver2.example.com:389} 000000
00
nsruvReplicaLastModified: {replica 6 ldap://ipaserver2.example.com:389} 000000
00
nsruvReplicaLastModified: {replica 5 ldap://ipaclient1.example.com:389} 000000
00
nsruvReplicaLastModified: {replica 3 ldap://ipaserver2.example.com:389} 000000
00
[root@ipaserver1 ~]#
In that test, the IPA replica ipaserver2.example.com was configured and unconfigured several times, and each attempt to reconfigure/re-install the IPA replica did fail with the same error. See the multiple replica id's.
Delete them:
ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV3
-
replace: nsds5task
nsds5task: CLEANRUV6
-
replace: nsds5task
nsds5task: CLEANRUV7
-
replace: nsds5task
nsds5task: CLEANRUV8
EOF
Verify:
ldapsearch -xLLL -D "cn=directory manager" -W -b dc=example,dc=com '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
Enter LDAP Password:
dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 4f6e5d8e000000040000
nsds50ruv: {replica 4 ldap://ipaserver1.example.com:389} 4f9ee89f000000040000
4fcf496f000000040000
nsds50ruv: {replica 5 ldap://ipaclient1.example.com:389} 4f6e6ba1001d00050000
4f6e8256000000050000
dc: example
nsruvReplicaLastModified: {replica 4 ldap://ipaserver1.example.com:389} 4fcf49
6c
nsruvReplicaLastModified: {replica 5 ldap://ipaclient1.example.com:389} 000000
00
[root@ipaserver1 ~]#
NOTE: This operation does not replicate. You will have to perform this operation on all of your servers (masters, hubs, consumers).
If needed on the master:
ipa-replica-manage del <removed-replica> --force
ipa host-del <removed-replica>
And try again to install/configure the IPA replica, ipaserver2.example.com in this test, it did work this time...:
[root@ipaserver2 ~]# ipa-replica-install -d --skip-conncheck /root/replica-info-ipaserver2.example.com.gpg
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments