Menu Close

Chapter 9. Troubleshooting

You can refer to the following tips to troubleshoot upgrading from RHEL 8 to RHEL 9.

9.1. Troubleshooting resources

You can refer to the following troubleshooting resources.

Console output

By default, only error and critical log level messages are printed to the console output by the Leapp utility. To change the log level, use the --verbose or --debug options with the leapp upgrade command.

  • In verbose mode, Leapp prints info, warning, error, and critical messages.
  • In debug mode, Leapp prints debug, info, warning, error, and critical messages.

Logs

  • The /var/log/leapp/leapp-upgrade.log file lists issues found during the initramfs phase.
  • The /var/log/leapp/dnf-debugdata/ directory contains transaction debug data. This directory is present only if the leapp upgrade command is executed with the --debug option.
  • The /var/log/leapp/answerfile contains questions required to be answered by Leapp.
  • The journalctl utility provides complete logs.

Reports

9.2. Troubleshooting tips

You can refer to the following troubleshooting tips.

Pre-upgrade phase

  • Verify that your system meets all conditions listed in Planning an upgrade.
  • Make sure you have followed all steps described in Preparing for the upgrade for example, your system does not use more than one Network Interface Card (NIC) with a name based on the prefix used by the kernel (eth).
  • Make sure you have answered all questions required by Leapp in the /var/log/leapp/answerfile file. If any answers are missing, Leapp inhibits the upgrade. For example:

    • Are there no VDO devices on the system?
  • Make sure you have resolved all problems identified in the pre-upgrade report, located at /var/log/leapp/leapp-report.txt. To achieve this, you can also use the web console, as described in Assessing upgradability and applying automated remediations through the web console.

Example 9.1. Leapp answerfile

The following is an example of an unedited /var/log/leapp/answerfile file that has one unanswered question:

[check_vdo]
# Title:              None
# Reason:             Confirmation
# ========================== check_vdo.no_vdo_devices =========================
# Label:              Are there no VDO devices on the system?
# Description:        Enter True if there are no VDO devices on the system and False continue the upgrade. If the system has no VDO devices, then it is safe to continue the upgrade. If there are VDO devices they must all be converted to LVM management before the upgrade can proceed.
# Reason:             Based on installed packages it is possible that VDO devices exist on the system.  All VDO devices must be converted to being managed by LVM before the upgrade occurs. Because the 'vdo' package is not installed, Leapp cannot determine whether any VDO devices exist that have not yet been converted.  If the devices are not converted and the upgrade proceeds the data on unconverted VDO devices will be inaccessible. If you have any doubts you should choose to install the 'vdo' package and re-run the upgrade process to check for unconverted VDO devices. If you are certain that the system has no VDO devices or that all VDO devices have been converted to LVM management you may opt to allow the upgrade to proceed.
# Type:               bool
# Default:            None
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
# no_vdo_devices =

The Label field specifies the question that requires an answer. In this example, the question is Are there no VDO devices on the system?

To answer the question, uncomment the last line and enter an answer of True or False. In this example, the selected answer is True:

[check_vdo]
...
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
no_vdo_devices = True

Download phase

  • If a problem occurs during downloading RPM packages, examine transaction debug data located in the /var/log/leapp/dnf-debugdata/ directory.

Initramfs phase

  • During this phase, potential failures redirect you to the Dracut shell. Check the Journal log:

    # journalctl

    Alternatively, restart the system from the Dracut shell using the reboot command and check the /var/log/leapp/leapp-upgrade.log file.

Post-upgrade phase

  • If your system seems to be successfully upgraded but booted with the old RHEL 8 kernel, restart the system and check the kernel version of the default entry in GRUB.
  • Make sure you have followed the recommended steps in Verifying the post-upgrade state of the RHEL 9 system.
  • If your application or a service stops working or behaves incorrectly after you have switched SELinux to enforcing mode, search for denials using the ausearch, journalctl, or dmesg utilities:

    # ausearch -m AVC,USER_AVC -ts boot
    # journalctl -t setroubleshoot
    # dmesg | grep -i -e selinux -e type=1400

    The most common problems are caused by incorrect labeling. See Troubleshooting problems related to SELinux for more details.

9.3. Known issues

The following are Known Issues you may encounter when upgrading from RHEL 8 to RHEL 9.

  • Network teaming currently does not work when the in-place upgrade is performed while Network Manager is disabled or not installed.
  • If you use an HTTP proxy, Red Hat Subscription Manager must be configured to use such a proxy, or the subscription-manager command must be executed with the --proxy <hostname> option. Otherwise, an execution of the subscription-manager command fails. If you use the --proxy option instead of the configuration change, the upgrade process fails because Leapp is unable to detect the proxy. To prevent this problem from occurring, manually edit the rhsm.conf file as described in How to configure HTTP Proxy for Red Hat Subscription Management. (BZ#1689294)
  • If your RHEL 8 system uses a device driver that is provided by Red Hat but is not available in RHEL 9, Leapp inhibits the upgrade. However, if the RHEL 8 system uses a third-party device driver that Leapp does not have data for in the /etc/leapp/files/device_driver_deprecation_data.json file, Leapp does not detect such a driver and proceeds with the upgrade. Consequently, the system might fail to boot after the upgrade.
  • If the name of a third-party package (not signed by Red Hat) installed on your system is the same as the name of a package provided by Red Hat, the in-place upgrade fails. To work around this problem, choose one of the following options prior to upgrading:

    1. Remove the third-party package
    2. Replace the third-party package with the package provided by Red Hat
  • In RHEL 8, it is possible to manage Virtual Data Optimizer (VDO) volumes using either the VDO manager or the Logical Volume Manager (LVM). In RHEL 9, it is only possible to manage VDO volumes using LVM. To continue using VDO-managed volumes after the upgrade, import those volumes to LVM-managed VDO volumes:

    1. Verify that the latest versions of VDO and LVM are installed on your RHEL 8 system.
    2. Import the VDO-managed VDO volumes to LVM-managed VDO volumes before beginning the in-place upgrade:

      # lvm_import_vdo --name <volume_group_name>/<lvm_name> /dev/mapper/<vdo_name>

      Replace volume_group_name with the new volume group name, lvm_name with the new name of the LVM-managed VDO volume, and vdo_name with the name of the VDO-managed volume that you are importing.

      Important

      You cannot import an LVM-managed VDO volume to a VDO-managed VDO volume. As a result, it is not possible to reverse the import procedure if you intend to access these VDO volumes through the VDO manager in the future. For more information on LVM-managed VDO volumes, see the Deduplicating and compressing logical volumes on RHEL guide.

  • The in-place upgrade fails on systems with Software Redundant Array of Independent Disks (RAID). (BZ#1957192)
  • During the in-place upgrade, the Leapp utility usually preserves the network interface controller (NIC) names between RHEL 8 and RHEL 9. However, on some systems, for example systems with network bonding, the NIC names might need to be updated between RHEL 8 and RHEL 9. On those systems, set the LEAPP_NO_NETWORK_RENAMING=1 environment variable, perform the in-place upgrade, and then verify that your network is working as expected. If needed, manually update the network configuration. (BZ#1919382)
  • On systems with the NSFD service running on NFS servers, a non-existent NFS partition might be incorrectly detected during the in-place upgrade, inhibiting the upgrade. To prevent this issue, stop the NFSD service before running the in-place upgrade:

    # systemctl stop /proc/fs/nfsd

    (BZ#2036069)

  • The in-place upgrade might be stopped before the upgrade is performed because the Leapp utility incorrectly detects that there is not enough free disk space. If your system contains partitions formatted with the XFS filesystem without ftype attributes, you can work around this issue by changing the default size in the LEAPP_OVL_SIZE environment variable to account for, at minimum, the specified missing disk space inside the container. It is recommended to increase the default size to greater than the specified missing disk space to prevent repeated error messages. For example, if the Leapp utility detects that an additional 400 MB is needed, increase the default size from 2048 MB to at least 2500 MB.

    Note

    This workaround can require a large amount of free space in the /var partition.

    If this workaround does not resolve the issue, or if your system does not contain these partitions without ftype attributes, contact Red Hat support. (BZ#1832730)

  • On systems with 64-bit ARM architecture, the kernel page size has changed from 64k in RHEL 8 to 4k in RHEL 9. As a result, the swap partition is not mounted automatically after the in-place upgrade. To work around this issue, manually mount the swap partition and reinitialize the swap after performing the in-place upgrade:

    # swapon -a --fixpgsz

    (BZ#2040470)

  • The in-place upgrade breaks networking on IBM Z with RoCE Express adapters. RHEL 9.0 uses Predictable Interface Names for RoCE Express adapters. These are different from the names available in the RHEL 8.6 distribution. Therefore, existing networking configurations of RoCE Express adapters will break with the current RHEL 9 minor release if you perform an in-place upgrade from RHEL 8 to RHEL 9.

9.4. Obtaining support

You can open a support case, select RHEL 7 as the product, and provide a sosreport from your system.

  • To generate a sosreport on your system, run:
# sosreport

Note that you can leave the case ID empty.

For details on generating a sosreport, see the solution What is an sosreport and how to create one in Red Hat Enterprise Linux?.

For more information on opening and managing a support case on the Customer Portal, see the article How do I open and manage a support case on the Customer Portal?.