Chapter 9. Troubleshooting

You can refer to the following tips to troubleshoot upgrading from RHEL 8 to RHEL 9.

9.1. Troubleshooting resources

You can refer to the following troubleshooting resources.

Console output

By default, only error and critical log level messages are printed to the console output by the Leapp utility. To change the log level, use the --verbose or --debug options with the leapp upgrade command.

  • In verbose mode, Leapp prints info, warning, error, and critical messages.
  • In debug mode, Leapp prints debug, info, warning, error, and critical messages.

Logs

  • The /var/log/leapp/leapp-upgrade.log file lists issues found during the initramfs phase.
  • The /var/log/leapp/dnf-debugdata/ directory contains transaction debug data. This directory is present only if the leapp upgrade command is executed with the --debug option.
  • The /var/log/leapp/answerfile contains questions required to be answered by Leapp.
  • The journalctl utility provides complete logs.

Reports

9.2. Troubleshooting tips

You can refer to the following troubleshooting tips.

Pre-upgrade phase

  • Verify that your system meets all conditions listed in Planning an upgrade.
  • Make sure you have followed all steps described in Preparing for the upgrade for example, your system does not use more than one Network Interface Card (NIC) with a name based on the prefix used by the kernel (eth).

  • Make sure you have answered all questions required by Leapp in the /var/log/leapp/answerfile file. If any answers are missing, Leapp inhibits the upgrade. For example:

    • Are there no VDO devices on the system?
  • Make sure you have resolved all problems identified in the pre-upgrade report, located at /var/log/leapp/leapp-report.txt. To achieve this, you can also use the web console, as described in Assessing upgradability and applying automated remediations through the web console.

Example 9.1. Leapp answerfile

The following is an example of an unedited /var/log/leapp/answerfile file that has one unanswered question: 

[check_vdo]
# Title:              None
# Reason:             Confirmation
# ============================= check_vdo.confirm =============================
# Label:              Are all VDO devices, if any, successfully converted to LVM management?
# Description:        Enter True if no VDO devices are present on the system or all VDO devices on the system have been successfully converted to LVM management. Entering True will circumvent check of failures and undetermined devices. Recognized VDO devices that have not been converted to LVM management can still block the upgrade despite the answer.All VDO devices must be converted to LVM management before upgrading.
# Reason:             To maximize safety all block devices on a system that meet the criteria as possible VDO devices are checked to verify that, if VDOs, they have been converted to LVM management. If the devices are not converted and the upgrade proceeds the data on unconverted VDO devices will be inaccessible. In order to perform checking the 'vdo' package must be installed. If the 'vdo' package is not installed and there are any doubts the 'vdo' package should be installed and the upgrade process re-run to check for unconverted VDO devices. If the check of any device fails for any reason an upgrade inhibiting report is generated. This may be problematic if devices are dynamically removed from the system subsequent to having been identified during device discovery. If it is certain that all VDO devices have been successfully converted to LVM management this dialog may be answered in the affirmative which will circumvent block device checking.
# Type:               bool
# Default:            None
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
# confirm =

The Label field specifies the question that requires an answer. In this example, the question is Are all VDO devices, if any, successfully converted to LVM management?

To answer the question, uncomment the last line and enter an answer of True or False. In this example, the selected answer is True: 

[check_vdo]
...
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
confirm = True

Download phase

  • If a problem occurs during downloading RPM packages, examine transaction debug data located in the /var/log/leapp/dnf-debugdata/ directory.

    Note

    The /var/log/leapp/dnf-debugdata/ directory is empty or does not exist if no transaction debug data was produced. This might occur when the required repositories are not available.

Initramfs phase

  • During this phase, potential failures redirect you to the Dracut shell. Check the Journal log: 

    # journalctl

    Alternatively, restart the system from the Dracut shell using the reboot command and check the /var/log/leapp/leapp-upgrade.log file.

Post-upgrade phase

  • If your system seems to be successfully upgraded but booted with the old RHEL 8 kernel, restart the system and check the kernel version of the default entry in GRUB.
  • Make sure you have followed the recommended steps in Verifying the post-upgrade state.

  • If your application or a service stops working or behaves incorrectly after you have switched SELinux to enforcing mode, search for denials using the ausearch, journalctl, or dmesg utilities: 

    # ausearch -m AVC,USER_AVC -ts boot
# journalctl -t setroubleshoot
# dmesg | grep -i -e selinux -e type=1400

    The most common problems are caused by incorrect labeling. See Troubleshooting problems related to SELinux for more details.

9.3. Known issues for the RHEL 8.9 to RHEL 9.3 upgrade

The following are Known Issues you may encounter when upgrading from RHEL 8.9 to RHEL 9.3.

  • Network teaming currently does not work when the in-place upgrade is performed while Network Manager is disabled or not installed.
  • If you use an HTTP proxy, Red Hat Subscription Manager must be configured to use such a proxy, or the subscription-manager command must be executed with the --proxy <hostname> option. Otherwise, an execution of the subscription-manager command fails. If you use the --proxy option instead of the configuration change, the upgrade process fails because Leapp is unable to detect the proxy. To prevent this problem from occurring, manually edit the rhsm.conf file as described in How to configure HTTP Proxy for Red Hat Subscription Management. (BZ#1689294)
  • If your RHEL 8 system uses a device driver that is provided by Red Hat but is not available in RHEL 9, Leapp inhibits the upgrade. However, if the RHEL 8 system uses a third-party device driver that Leapp does not have data for in the /etc/leapp/files/device_driver_deprecation_data.json file, Leapp does not detect such a driver and proceeds with the upgrade. Consequently, the system might fail to boot after the upgrade.

  • If the name of a third-party package (not signed by Red Hat) installed on your system is the same as the name of a package provided by Red Hat, the in-place upgrade fails. To work around this problem, choose one of the following options prior to upgrading:

    1. Remove the third-party package
    2. Replace the third-party package with the package provided by Red Hat
  • In RHEL 8, you can manage Virtual Data Optimizer (VDO) volumes by using either the VDO manager or the Logical Volume Manager (LVM). In RHEL 9, it is only possible to manage VDO volumes by using LVM. To continue using VDO-managed volumes on RHEL 9, import those volumes to LVM-managed VDO volumes before the upgrade. For more information, see Importing existing VDO volumes to LVM.
  • The in-place upgrade might fail on systems with Software Redundant Array of Independent Disks (RAID). (BZ#1957192)

  • During the in-place upgrade, the Leapp utility usually preserves the network interface controller (NIC) names between RHEL 8 and RHEL 9. However, on some systems, such as systems with network bonding, the NIC names might need to be updated between RHEL 8 and RHEL 9. On those systems, perform the following steps:

    1. Set the LEAPP_NO_NETWORK_RENAMING=1 environment variable to prevent the Leapp utility from incorrectly preserving the original RHEL 8 NIC names.
    2. Perform the in-place upgrade.

    3. Verify that your network is working correctly. If needed, manually update the network configuration.

      (BZ#1919382)

  • If your system boots by using BIOS, the in-place upgrade fails when upgrading the GRUB2 bootloader if the boot disk’s embedding area does not contain enough space for the core image installation. This results in a broken system, and can occur when the disk has been partitioned manually, for example using the RHEL 6 fdisk utility. To verify whether this issue affects you, perform the following steps:

    1. Determine which sector starts the first partition on the disk with the installed bootloader: 

      # fdisk -l

      The standard partitioning, which ensures enough space for the core image, starts on sector 2048.

    2. Determine whether the starting sector provides enough space. The RHEL 9.0 core image requires at least 36 KiB. For example, if the sector size is the standard 512 bytes, then starting on sector 73 or lower would not provide enough space.

      Note

      The RHEL 9 core image might be larger than 36 KiB and require a higher starting sector. Always verify how much space the current RHEL 9 core requires.

    3. If the embedding area does not contain enough storage space, perform a fresh installation of the RHEL 9 system instead of performing an in-place upgrade.

      (BZ#2181380)

  • After the in-place upgrade, SSH keys are no longer auto-generated if the system meets the following conditions:

  • VMWare virtual machines that were created at Hardware Level 13 and are booting with UEFI might experience issues during the upgrade because the NVRAM file is too small. For more information about this issue and how to resolve it, see VMWare: Getting "No space left on device" when executing efibootmgr or mokutil command to add entries. (RHEL-3362)
  • The upgrade might fail if you are upgrading by using RHUI with an ISO image. You can work around this issue by not using the --iso option with the upgrade or by following the instructions in Offline Leapp upgrade using ISO fails with "Failed to synchronize cache for repo 'rhul-microsoft-azure-rhel8', ignoring this repo. (RHEL-3296)

  • If too many file systems are mounted, the pre-upgrade process might fail with the following error message: 

    OperationalError: unable to open database file

    If this issue occurs, complete the following steps:

    1. Unmount any file systems that are not related to the system partitions and are not needed during the upgrade process.
    2. Comment out the unmounted file systems’ entries in the /etc/fstab file to prevent them from being mounted during the upgrade process.

    3. Restore the original file system configuration after the upgrade.

      (RHEL-3320)

  • If any of the mounted file systems that are defined in the /etc/fstab file do not have the shared propagation flag set, the upgrade might fail. To prevent this issue, remount these file systems to set them as shared: 

    # mount -o remount --make-shared <mountpoint>

    Replace mountpoint with the mountpoint of each file system.

    For more information, see Leapp "Can not load RPM file" during the DNF transaction check. (RHEL-23449)

9.4. Known issues for the RHEL 8.6 to RHEL 9.0 and RHEL 8.8 to RHEL 9.2 upgrade

The following are Known Issues you may encounter when upgrading from RHEL 8.6 to RHEL 9.0 or RHEL 8.8 to RHEL 9.2.

  • Network teaming currently does not work when the in-place upgrade is performed while Network Manager is disabled or not installed.
  • If you use an HTTP proxy, Red Hat Subscription Manager must be configured to use such a proxy, or the subscription-manager command must be executed with the --proxy <hostname> option. Otherwise, an execution of the subscription-manager command fails. If you use the --proxy option instead of the configuration change, the upgrade process fails because Leapp is unable to detect the proxy. To prevent this problem from occurring, manually edit the rhsm.conf file as described in How to configure HTTP Proxy for Red Hat Subscription Management. (BZ#1689294)
  • If your RHEL 8 system uses a device driver that is provided by Red Hat but is not available in RHEL 9, Leapp inhibits the upgrade. However, if the RHEL 8 system uses a third-party device driver that Leapp does not have data for in the /etc/leapp/files/device_driver_deprecation_data.json file, Leapp does not detect such a driver and proceeds with the upgrade. Consequently, the system might fail to boot after the upgrade.

  • If the name of a third-party package (not signed by Red Hat) installed on your system is the same as the name of a package provided by Red Hat, the in-place upgrade fails. To work around this problem, choose one of the following options prior to upgrading:

    1. Remove the third-party package
    2. Replace the third-party package with the package provided by Red Hat
  • In RHEL 8, you can manage Virtual Data Optimizer (VDO) volumes by using either the VDO manager or the Logical Volume Manager (LVM). In RHEL 9, it is only possible to manage VDO volumes by using LVM. To continue using VDO-managed volumes on RHEL 9, import those volumes to LVM-managed VDO volumes before the upgrade. For more information, see Importing existing VDO volumes to LVM.
  • The in-place upgrade might fail on systems with Software Redundant Array of Independent Disks (RAID). (BZ#1957192)

  • During the in-place upgrade, the Leapp utility usually preserves the network interface controller (NIC) names between RHEL 8 and RHEL 9. However, on some systems, such as systems with network bonding, the NIC names might need to be updated between RHEL 8 and RHEL 9. On those systems, perform the following steps:

    1. Set the LEAPP_NO_NETWORK_RENAMING=1 environment variable to prevent the Leapp utility from incorrectly preserving the original RHEL 8 NIC names.
    2. Perform the in-place upgrade.

    3. Verify that your network is working correctly. If needed, manually update the network configuration.

      (BZ#1919382)

  • If your system boots by using BIOS, the in-place upgrade fails when upgrading the GRUB2 bootloader if the boot disk’s embedding area does not contain enough space for the core image installation. This results in a broken system, and can occur when the disk has been partitioned manually, for example using the RHEL 6 fdisk utility. To verify whether this issue affects you, perform the following steps:

    1. Determine which sector starts the first partition on the disk with the installed bootloader: 

      # fdisk -l

      The standard partitioning, which ensures enough space for the core image, starts on sector 2048.

    2. Determine whether the starting sector provides enough space. The RHEL 9.0 core image requires at least 36 KiB. For example, if the sector size is the standard 512 bytes, then starting on sector 73 or lower would not provide enough space.

      Note

      The RHEL 9 core image might be larger than 36 KiB and require a higher starting sector. Always verify how much space the current RHEL 9 core requires.

    3. If the embedding area does not contain enough storage space, perform a fresh installation of the RHEL 9 system instead of performing an in-place upgrade.

      (BZ#2181380)

  • After the in-place upgrade, SSH keys are no longer auto-generated if the system meets the following conditions:

  • If too many file systems are mounted, the pre-upgrade process might fail with the following error message: 

    OperationalError: unable to open database file

    If this issue occurs, complete the following steps:

    1. Unmount any file systems that are not related to the system partitions and are not needed during the upgrade process.
    2. Comment out the unmounted file systems’ entries in the /etc/fstab file to prevent them from being mounted during the upgrade process.

    3. Restore the original file system configuration after the upgrade.

      (RHEL-3320)

9.5. Obtaining support

To open a support case, select RHEL 8 as the product, and provide a sosreport from your system.

  • To generate a sosreport on your system, run: 
# sosreport

Note that you can leave the case ID empty.

For details on generating a sosreport, see the solution What is an sosreport and how to create one in Red Hat Enterprise Linux?.

For more information about opening and managing a support case on the Customer Portal, see the article How do I open and manage a support case on the Customer Portal?.