Chapter 6. Applying patches with kernel live patching

The Red Hat Enterprise Linux kernel live patching solution allows you to patch a running kernel without rebooting or restarting any processes. The solution enables system administrators to apply critical security patches to the kernel immediately, without having to wait for long-running tasks to complete, for users to log off, or for scheduled downtime. It gives more control over uptime without sacrificing security or stability.

Note that not every critical or important CVE will be resolved using the kernel live patching solution. Our goal is to reduce the required reboots for security-related patches, not to eliminate them entirely.

Warning

Some incompatibilities exist between kernel live patching and other kernel subcomponents. Read the Section 6.9, “Limitations of kpatch” section carefully before using kernel live patching.

6.1. Access to kernel live patches

Kernel live patching capability is implemented as a kernel module (.ko file) that is delivered as an RPM package.

All customers have access to kernel live patches, which are delivered through the usual channels. However, customers who do not subscribe to an extended support offering will lose access to new patches for the current minor release once the next minor release becomes available. For example, customers with standard subscriptions will only be able to live patch the RHEL 8.2 kernels until RHEL 8.3 is released.

6.2. Components of kernel live patching

The components of kernel live patching are as follows:

Kernel patch module

  • The delivery mechanism for kernel live patches.
  • This is a kernel module which is built specifically for the kernel being patched.
  • The patch module contains the code of the desired fixes for the kernel.
  • The patch modules register with the livepatch kernel subsystem and provide information about original functions to be replaced, with corresponding pointers to the replacement functions. Kernel patch modules are delivered in kpatch-patch-<kernel-version> RPMs.
The kpatch utility
A command-line tool which allows you to manage patch modules.
The kpatch service
A systemd service required by multiuser.target. This target loads the kernel patch module at boot time.

6.3. How kernel live patching works

The kpatch kernel patching solution uses the livepatch kernel subsystem to redirect old functions to new ones. When a live kernel patch is applied to a system, the following things happen:

  1. The kernel patch module is copied to the /var/lib/kpatch/ directory and registered for re-application to the kernel via systemd on next boot.
  2. The kpatch module is loaded into the running kernel and the patched functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.
  3. When the kernel accesses the patched function, it is redirected by the ftrace mechanism which bypasses the original functions and redirects the kernel to patched version of the function.

Figure 6.1. How kernel live patching works

rhel kpatch overview

6.4. Enabling kernel live patching

A kernel patch module is delivered in an RPM package, specific to the version of the kernel being patched. Each RPM package will be cumulatively updated over time.

The following subsections describe how to ensure you receive all future cumulative live patching updates for a given kernel.

Warning

Red Hat strongly discourages from using any patch modules which were not provided for your specific usage by Red Hat itself.

6.4.1. Subscribing to the live patching stream

This procedure describes installing a particular live patching package. By doing so you subscribe to the live patching stream for a given kernel and ensure that you receive all future cumulative live patching updates for that kernel.

Prerequisites
Procedure
  1. Optionally, check your kernel version:

    $ uname -r
    3.10.0-1062.el7.x86_64
  2. Search for a live patching package that corresponds to the version of your kernel:

    # yum search $(uname -r)

    The live patching package contains a patch module, if the package’s version is 1-1 or higher. In that case the kernel will be automatically patched during the installation of the live patching package.

  3. Install the live patching package:

    # yum install "kpatch-patch = $(uname -r)"

    The example command above installs and applies the latest cumulative live patches for that specific kernel only. Later, if you install a new kernel and reboot into it, you will have to re-enable the live patching for the new kernel by running yum install kpatch-patch-<new_kernel_version>.

    The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during the future reboots.

    Note

    If there are not yet any live patches available for the given kernel, an empty live patching package will be installed. An empty live patching package will have a Version-Release of 0-0, for example kpatch-patch-3_10_0-1062-0-0.el7.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

  4. Optionally, verify the kernel is patched:

    $ kpatch list
    Loaded patch modules:
    kpatch_3_10_0_1062_1_1 [enabled]
    
    Installed patch modules:
    kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64)
    …​

    The output shows that the kernel patch module has been loaded into the kernel, which is now patched with the latest fixes from the kpatch-patch-3_10_0-1062-1-1.el7.x86_64.rpm package.

Additional resources
  • For more information about the kpatch command-line utility, refer to the kpatch(1) manual page.

6.5. Updating kernel patch modules

Since kernel patch modules are delivered and applied through RPM packages, updating a cumulative kernel patch module is like updating any other RPM package.

Prerequisites

Procedure

  • Run yum update to update to a new cumulative version.

    The command above automatically installs and applies any future released cumulative live patches to the running kernel.

    Note

    When the system reboots into the same kernel, the kernel is automatically live patched again by the kpatch.service systemd unit file.

6.6. Disabling kernel live patching

In case system administrators encountered some unanticipated negative effects connected with the Red Hat Enterprise Linux kernel live patching solution and they no longer wish to immediately apply critical security patches to their running kernel, they can disable the mechanism in three distinct ways:

6.6.1. Removing the live patching package

The following procedure describes how to disable the Red Hat Enterprise Linux kernel live patching solution by removing the live patching package that you installed in your system.

Prerequisites
  • Root permissions
  • You are familiar with basic package administration in Red Hat Enterprise Linux 7.
  • The live patching package is installed.
Procedure
  1. Select the live patching package:

    $ yum list installed | grep kpatch-patch
    kpatch-patch-3_10_0-1062.x86_64        1-1.el7        @@commandline
    …​

    The example output above lists live patching packages that you installed.

  2. Remove the live patching package:

    # yum remove kpatch-patch-3_10_0-1062.x86_64
    Updating Subscription Management repositories.
    Dependencies resolved.
    =========================================================================
     Package                   Arch     Version     Repository       Size
    =========================================================================
    Removing:
     kpatch-patch-3_10_0-1062    x86_64   1-1.el7     @@commandline     14 k
    
    Transaction Summary
    =========================================================================
    Remove  1 Package
    
    Freed space: 14 k
    Is this ok [y/N]: y
    Running transaction check
    Transaction check succeeded.
    Running transaction test
    Transaction test succeeded.
    Running transaction
      Preparing        :                                                  1/1
      Erasing          : kpatch-patch-3_10_0-1062-1-1.el7.x86_64          1/1
      Running scriptlet: kpatch-patch-3_10_0-1062-1-1.el7.x86_64          1/1
    uninstalling kpatch-3_10_0-1062-1-1.ko (3.10.0-1062.el7.x86_64)
    
      Verifying        : kpatch-patch-3_10_0-1062-1-1.el7.x86_64          1/1
    Installed products updated.
    
    Removed:
      kpatch-patch-3_10_0-1062-1-1.el7.x86_64
    
    Complete!

    When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. After the next reboot, the corresponding kernel will no longer be patched.

  3. Reboot your system.
  4. Verify that the live patching package has been removed:

    $ yum list installed | grep kpatch-patch
    <NO_RESULT>
  5. Verify that the kernel live patching solution is disabled:

    $ kpatch list
    Loaded patch modules:
    <NO_RESULT>

    The example output above shows that the kernel is not patched and the kernel live patching solution is not active.

Additional resources
  • For more information about the kpatch command-line utility, refer to the kpatch(1) manual page.

6.6.2. Uninstalling the kernel patch module

The following procedure describes how to prevent the Red Hat Enterprise Linux kernel live patching solution from applying a kernel patch module on subsequent boots.

Prerequisites
  • Root permissions
  • A live patching package is installed.
  • A kernel patch module is installed and loaded.
Procedure
  1. Select a kernel patch module:

    $ kpatch list
    Loaded patch modules:
    kpatch_3_10_0_1062_1_1 [enabled]
    
    Installed patch modules:
    kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64)
    …​
  2. Uninstall the selected kernel patch module:

    # kpatch uninstall kpatch_3_10_0_1062_1_1
    uninstalling kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64)
    • Note that the uninstalled kernel patch module is still loaded:

      $ kpatch list
      Loaded patch modules:
      kpatch_3_10_0_1062_1_1 [enabled]
      
      Installed patch modules:
      <NO_RESULT>

      When the selected module is uninstalled, kernel remains patched until the next reboot, but the kernel patch module is removed from disk.

  3. Reboot your system.
  4. Verify that the kernel patch module has been uninstalled:

    $ kpatch list
    Loaded patch modules:
    <NO_RESULT>

    The example output above shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.

Additional resources
  • For more information about the kpatch command-line utility, refer to the kpatch(1) manual page.

6.6.3. Disabling the kpatch service unit file

The following procedure describes how to prevent the Red Hat Enterprise Linux kernel live patching solution from applying all kernel patch modules globally on subsequent boots.

Prerequisites
  • Root permissions
  • A live patching package is installed.
  • A kernel patch module is installed and loaded.
Procedure
  1. Verify kpatch.service is enabled and running:

    # systemctl status kpatch.service
    ● kpatch.service - "Apply kpatch kernel patches"
       Loaded: loaded (/usr/lib/systemd/system/kpatch.service; enabled; vendor preset: disabled)
       Active: active (exited) since Sat 2019-07-27 13:46:07 CEST; 34min ago
      Process: 831 ExecStart=/usr/sbin/kpatch load --all (code=exited, status=0/SUCCESS)
     Main PID: 831 (code=exited, status=0/SUCCESS)
    
    Jul 27 13:46:07 redhat systemd[1]: Starting "Apply kpatch kernel patches"…​
    Jul 27 13:46:07 redhat kpatch[831]: loading patch module: /var/lib/kpatch/3.10.0-1062.el7.x86_64/kpatch-3_10_0-1062-1-1.ko
    Jul 27 13:46:07 redhat systemd[1]: Started "Apply kpatch kernel patches".
  2. Disable kpatch.service:

    # systemctl disable kpatch.service
    Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.
    • Note that the applied kernel patch module is still loaded:

      $ kpatch list
      Loaded patch modules:
      kpatch_3_10_0_1062_1_1 [enabled]
      
      Installed patch modules:
      kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64)
  3. Reboot your system.
  4. Optionally, verify the status of kpatch.service:

    # systemctl status kpatch.service
    ● kpatch.service - "Apply kpatch kernel patches"
       Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled)
       Active: inactive (dead)

    The example output testifies that kpatch.service has been disabled and is not running. Thereby the kernel live patching solution is not active.

  5. Verify that the kernel patch module has been unloaded:

    # kpatch list
    Loaded patch modules:
    <NO_RESULT>
    
    Installed patch modules:
    kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64)

    The example output above shows that a kernel patch module is still installed but the kernel is not patched.

Additional resources
  • For more information about the kpatch command-line utility, refer to the kpatch(1) manual page.
  • For more information about the systemd system and service manager, unit configuration files, their locations, as well as a complete list of systemd unit types, see the relevant sections in System Administrator’s Guide.

6.7. kpatch support

  • Live kernel patches are available for all RHEL subscriptions and on AMD64 and Intel 64 and IBM Power Systems architectures.
  • Not all CVE remediation patches that are kernel related may be able to be live patched
  • Red Hat will generate, maintain and support live kernel patches for kernels that are up to 1 year old.
  • Live kernel patches will be produced and supported for RHEL Releases throughout their EUS coverage period.
  • Live patches for RHEL kernels will be produced and supported for RHEL for SAP Solutions Releases throughout their Update Services (E4S) coverage period.
  • Live kernel patches will be for the following RHEL minor releases with current or planned EUS streams: RHEL 7.6, 7.7 RHEL 8.1, 8.2 ,8.4.

6.8. Support for third-party live patching

The kpatch utility is the only kernel live patching utility supported by Red Hat with the RPM modules supplied through your Red Hat support contract. Red Hat will not support any live patches which were not provided by Red Hat itself.

If you require support for an issue that arises with a third-party live patch, Red Hat recommends that you open a case with the live patching vendor at the outset of any investigation in which a root cause determination is necessary. This allows the source code to be supplied if the vendor allows, and for their support organization to provide assistance in root cause determination prior to escalating the investigation to Red Hat Support.

For any system running with third-party live patches, Red Hat reserves the right to ask for reproduction with Red Hat shipped and supported software. In the event that this is not possible, we require a similar system and workload be deployed on your test environment without live patches applied, to confirm if the same behavior is observed.

For more information about third-party software support policies, see How does Red Hat Global Support Services handle third-party software, drivers, and/or uncertified hardware/hypervisors or guest operating systems?

6.9. Limitations of kpatch

  • kpatch is not a general-purpose kernel upgrade mechanism. It is used for applying simple security and bug fix updates when rebooting the system is not immediately possible.
  • Do not use the SystemTap or kprobe tools during or after loading a patch. The patch could fail to take effect until after such probes have been removed.