How do I resolve this "Unable to install RHEL 9 userspace packages" error?

Latest response

I've been trying to run a leapp upgrade from RHEL8.9 to 9.3 for a week and I keep getting the same error over and over again despite seemingly doing everything outlined in the instructions found at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/upgrading_from_rhel_8_to_rhel_9/index

Since 8.9 is a relatively recent minor update, all the instructions I found to attempt to resolve this issue are typically for upgrades up to v.8.8. Whether this matters or not, I am uncertain but none of the solutions I found have made any difference to my problem.

Essentially when I run leapp preupgrade, it always fails at the same point:

====> * target_userspace_creator
        Initializes a directory to be populated as a minimal environment to run binaries from the target system.
Updating Subscription Management repositories.
Unable to read consumer identity
subscription-manager is operating in container mode.

This system is not registered with an entitlement server. You can use subscription-manager to register.

rhel-8-for-x86_64-baseos-rpms                   7.9 kB/s | 153  B     00:00

============================================================
                           ERRORS
============================================================

2023-12-11 09:45:24.446264 [ERROR] Actor: target_userspace_creator
Message: Unable to install RHEL 9 userspace packages.
Summary:
    Details: Command ['systemd-nspawn', '--register=no', '--quiet', '--keep-unit', '--capability=all', '-D', '/var/lib/leapp/scratch/mounts/root_/system_overlay', '--setenv=LEAPP_HOSTNAME=server1.domain.com', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_UNSUPPORTED=0', '--setenv=LEAPP_NO_RHSM=0', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.3', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_IPU_IN_PROGRESS=8to9', '--setenv=LEAPP_EXECUTION_ID=546be438-d7d3-4fd7-830c-303f7ac4a9f6', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el8toel9/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files', 'dnf', 'install', '-y', '--setopt=module_platform_id=platform:el9', '--setopt=keepcache=1', '--releasever', '9.3', '--installroot', '/el9target', '--disablerepo', '*', '--enablerepo', 'AppStream', '--enablerepo', 'codeready-builder-for-rhel-9-x86_64-rpms', '--enablerepo', 'rhel-9-for-x86_64-baseos-rpms', '--enablerepo', 'rhel-9-for-x86_64-appstream-rpms', '--enablerepo', 'BaseOS', '--enablerepo', 'rhel-9-for-x86_64-supplementary-rpms', 'dnf', 'dnf-command(config-manager)'] failed with exit code 1.
    Stderr: Host and machine ids are equal (01a3f6cd41774ef1b6619c0a9a7ab169): refusing to link journals
            Errors during downloading metadata for repository 'BaseOS':
              - Status code: 404 for http://127.0.0.1/BaseOS/repodata/repomd.xml (IP: 127.0.0.1)
            Error: Failed to download metadata for repo 'BaseOS': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

============================================================
                       END OF ERRORS
============================================================

Debug output written to /var/log/leapp/leapp-preupgrade.log

============================================================
                      REPORT OVERVIEW
============================================================

Following errors occurred and the upgrade cannot continue:
    1. Actor: target_userspace_creator
       Message: Unable to install RHEL 9 userspace packages.

HIGH and MEDIUM severity reports:
    1. Packages not signed by Red Hat found on the system
    2. GRUB2 core will be automatically updated during the upgrade

Reports summary:
    Errors:                      1
    Inhibitors:                  0
    HIGH severity reports:       2
    MEDIUM severity reports:     0
    LOW severity reports:        2
    INFO severity reports:       3

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                   END OF REPORT OVERVIEW
============================================================

Answerfile has been generated at /var/log/leapp/answerfile

Looking at this failure report there are a few points I need to address:

  • The system is a virtual machine running on vmware
  • It is managed with RedHat Satellite Server v 6.13, running on a separate, dedicated virtual machine
  • It runs in Simple Content Access Mode
  • I have previously ran a successful upgrade to a very similar machine, albeit from 8.8 to 9.2 at the time
  • It is properly entitled and has a valid subscription.

Aspects of the error messages I don't understand:

  • subscription-manager is operating in container mode.: I assume this is part of the leapp upgrade process as I don't run RHEL in a containerised environment of any kind
  • This system is not registered with an entitlement server: This feels like a red herring or a result of running in SCA mode but it is definitely registered.
  • Status code: 404 for http://127.0.0.1/BaseOS/repodata/repomd.xml (IP: 127.0.0.1): This is the most puzzling of them all as it suggests it's looking for a repository located on localhost, which is wrong as the system is managed in Satellite, but I couldn't find any configuration file that forces it to look for a repo on localhost.

Things I tried:

  • subscription-manager release --set 8.9, making sure all 8.9 RHEL repos are activated, as well as all 9.3 repos and for the VM, enabling both 8 and 9 repos.
  • dnf versionlock clear, this basically did nothing.
  • force reregister system with Satellite: subscription-manager register --force, again this made no difference
  • force enable RHEL8 repos: subscription-manager repos --enable rhel-8-for-x86_64-baseos-rpms --enable rhel-8-for-x86_64-appstream-rpms. Those were already enabled so this made no difference.
  • made sure all packages are up to date.
  • creating a content view in Satellite with all required repos but since those are already enabled in the default content view default organizational view, it again made no difference.
  • disabling UFW Firewall

At this point, I don't know what else to do to diagnose the problem or why it fails at looking for the BaseOS repo on localhost and not on Satellite.

SI

Responses