How to remove duplicate packages from a failed yum transaction

Solution Verified - Updated -

Red Hat Lightspeed can detect this issue

Proactively detect and remediate issues impacting your systems.
View matching systems and remediation

Environment

  • Red Hat Enterprise Linux 10
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5

Issue

  • How to remove the older versions of a package which has multiple versions installed on Red Hat Enterprise Linux?
  • After a failed yum transaction, yum-complete-transaction fails to complete and the duplicates are still present.
  • yum fails with <package_name-version1> is a duplicate with <package_name-version2>
  • package-cleanup --cleandupes fails with Error: Depsolving loop limit reached

Resolution

  1. Try to complete the transaction
  2. Manually remove duplicates
  3. Identify remaining rpm corruption

Try to complete the transaction

RHEL 5,6,7

  • Before manually removing the duplicates, confirm if yum-complete-transaction will resolve the issue.

    # yum-complete-transaction

  • If this completes successfully, so long as there are no further duplicates, you do not need to run the remaining steps.

  • If there are no duplicates, we can clear this message with the --cleanup-only option

    # yum-complete-transaction --cleanup-only

RHEL 8,9,10

  • yum history redo can reattempt the transaction. We can use last or the transaction-id as the target

    # yum history redo last

  • RHEL 8, 9 & 10 comes with ability to remove the lower version and reinstall the higher version, the same as the manual steps do.

    # man yum
       dnf [options] remove --duplicates
              Removes older versions of duplicate packages. To ensure  the  integrity  of
              the system it reinstalls the newest package. In some cases the command can‐
              not resolve conflicts. In such cases the  dnf  shell  command  with  remove
              --duplicates and upgrade dnf-shell sub-commands could help.

# yum remove --duplicates

  • If this completes successfully, so long as there are no further duplicates, problems, or concerns, you do not need to run the remaining steps.

Manually remove duplicates

The first step is to backup the rpm database. The remaining steps create a list of one of the duplicate package and removes them from the database only. With the --justdb flag we ensure no necessary files are removed until after we update. If there is any concern that the files are incorrect we can verify with rpm -Va.

RHEL 5

  • RHEL 5 does not have yum check so we must use an alternate method.

    # tar cjf /tmp/rpm_db.tar.bz2 /var/lib/{rpm,yum}
# package-cleanup --dupes &> /tmp/duplicate     

Note: Check the /tmp/duplicate file and remove the line(s) which is not a package name.

# cat /tmp/duplicate | sort -r| awk 'NR%2==0' &> /tmp/remove
# cat /tmp/remove | egrep -v "\:" > /tmp/remove1
# cat /tmp/remove |  egrep ":" | awk -F':' '{ print $NF }' >> /tmp/remove1 
# for i in `cat /tmp/remove1`; do rpm -e --justdb --nodeps $i; done
# yum update

RHEL 6,7

  • yum check can take quite a while but the output is necessary for these steps

    # tar cjf /tmp/rpm_db.tar.bz2 /var/lib/{rpm,yum}
# yum check &> /tmp/yumcheck
# grep "duplicate" /tmp/yumcheck | awk '{ print $NF }' | egrep -v "\:" > /tmp/duplicaterpms
# grep "duplicate" /tmp/yumcheck | awk '{ print $NF }' | egrep ":" | awk -F':' '{ print $NF }' >> /tmp/duplicaterpms
# for i in $(cat /tmp/duplicaterpms); do rpm -e --justdb --nodeps $i; done
# yum update

RHEL 8,9

  • Backup rpmdb and removing older version duplicate packages

    # tar cjf /tmp/rpm_db.tar.bz2 /var/lib/{rpm,dnf} /etc/{dnf*,os-release} /usr/lib/os-release
# dnf repoquery --duplicates --latest-limit=-1 -q &> /tmp/duplicateolder
# for i in $(cat /tmp/duplicateolder); do rpm -ev --justdb --nodeps $i; done
# yum update

RHEL 10

  • Backup rpmdb and removing older version duplicate packages

    # tar cjf /tmp/rpm_db.tar.bz2 /var/lib/{rpm,dnf} /etc/{dnf*,os-release} /usr/lib/os-release /usr/lib/sysimage/rpm
# dnf repoquery --duplicates --latest-limit=-1 -q &> /tmp/duplicateolder
# for i in $(cat /tmp/duplicateolder); do rpm -ev --justdb --nodeps $i; done
# dnf update

After removing the Duplicates

  • If a new kernel was part of the transaction that failed, you may have to reinstall that kernel as well to generate the correct grub entries and initrd/initramfs

    For RHEL 6 & 7
# yum remove kernel-<newversion>-<release>.<arch>
# yum install kernel-<newversion>-<release>.<arch>

For RHEL 8, 9 & 10
# yum remove kernel-core-<newversion>-<release>.<arch> kernel-<newversion>-<release>.<arch>
# yum install kernel-core-<newversion>-<release>.<arch> kernel-<newversion>-<release>.<arch>

Identify remaining rpm corruption

  • To identify any remaining corruption we can use rpm -Va which verifies all files on the system that are provided by rpms, against the rpms you have installed.

    • If there are duplicates on the system, an rpm -Va will be extremely misleading suggesting that most of the binaries are wrong when in fact it's comparing 1 binary against 2 packages and telling you that it doesn't match one of the 2 it checked. By removing the duplicates first, we can then use rpm -Va to identify any remaining rpm vs file mismatch.

  • If you have prelinking enabled, we want to clear that cache first before running rpm -Va or it obscures the output.

    # prelink -ua; rpm -Va &> /tmp/rpm_Va.out

  • Now ruling out configuration files which we expect to show up in this output, we can identify a list of packages to reinstall.

    # awk '$2 !~/^[cg]$/ {print $NF}' /tmp/rpm_Va.out | xargs rpm -qf 2>/dev/null | sort -u |& tee /tmp/pkgs.out

  • Reinstall the packages to replace any files that don't match the rpms installed.

    # yum reinstall $(cat /tmp/pkgs.out)

Root Cause

When yum is interrupted during an update, it can leave the old versions installed in addition to the new packages it was updating to leaving duplicates. The next time yum is used, the duplicate packages cause confusion in yum's logic and it can fail with "protected multilib" errors or misleading dependency errors.

Having more than one version of the same package can introduce inconsistencies or vulnerabilities in the system. It breaks the linking of binaries or libraries. The system may still have files from older version which are being used in the name of new version.

Note: There is a tool called package-cleanup which older versions will remove dependencies of duplicates while newer versions simply won't remove a duplicate that something depends on. Since, after we remove the duplicates, we update and confirm no further issues are present, we can safely remove the packages with the steps provided here.

This issue most commonly occurs when yum is interrupted. A few common reasons are below:

  • If you run yum update over ssh and connectivity is interrupted during the update
  • If you send SIGINT, kill -9, or issue aCTRL+c to the yum process while it's running
  • If the system reboots during the update

There are many possible reasons this could occur. If this occurs repetitively, it could be caused by automation software interrupting the updates when it decides it's taking too long or the command has timed out when often it has not.

How to debug recurring rpm database corruption

Diagnostic Steps

  • Identify any duplicates on the system

    # package-cleanup --dupes               <--- Needs yum-utils package installed
# dnf repoquery --duplicates            <--- Available default in RHEL 8 and higher versions.
# yum check

  • With RHEL 6 and newer, you can use yum history to look for recent failed yum transactions marked by a *

    # yum history

  • For reviewing rpm -Va output, you can refer to rpm's man page.

           The format of the output is a string of 9 characters, a  possible  attribute
       marker:

       c %config configuration file.
       d %doc documentation file.
       g %ghost file (i.e. the file contents are not included in the package payload).
       l %license license file.
       r %readme readme file.

       from  the  package header, followed by the file name.  Each of the 9 charac‐
       ters denotes the result of a comparison of attribute(s) of the file  to  the
       value of those attribute(s) recorded in the database.  A single "." (period)
       means the test passed, while a single "?" (question mark) indicates the test
       could  not  be performed (e.g. file permissions prevent reading). Otherwise,
       the (mnemonically emBoldened) character denotes failure of the corresponding
       --verify test:

       S file Size differs
       M Mode differs (includes permissions and file type)
       5 digest (formerly MD5 sum) differs
       D Device major/minor number mismatch
       L readLink(2) path mismatch
       U User ownership differs
       G Group ownership differs
       T mTime differs
       P caPabilities differ

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Was this helpful?

Yes No

Get notified when this content is updated

Comments