Contents

1. What is kpatch?
2. What Is the Scope of Support for kpatch?
3. Access and Delivery of Live Kernel Patching
4. Limitations
5. How Do I Enable and Use kpatch?
   • Installing kpatch Tools
   • Installing a kpatch Hot Fix
   • Listing Installed kpatch Hot Fixes
   • Updating a kpatch Hot Fix
   • Removing a kpatch Hot Fix
6. How Does kpatch Work?
7. Are Any Third Party Live Patching Solutions Supported?

What is kpatch?

kpatch is a live kernel patching solution that allows you to patch a running kernel without rebooting or restarting any processes. It enables system administrators to apply critical security patches to the kernel immediately, without having to wait for long-running tasks to complete, for users to log off, or for scheduled reboot windows. It gives more control over uptime without sacrificing security or stability.

WARNING: Some incompatibilities exist between kpatch and other kernel subcomponents. Read the Limitations section carefully before using kpatch.

What Is the Scope of Support for kpatch?

• Live kernel patching with kpatch is supported from Red Hat Enterprise Linux 7.2 onwards.
• Live kernel patching is supported for customers with Premium SLA subscriptions.
• Live kernel patching is only supported on the active Red Hat Enterprise Linux 7 maintenance stream that is within the current async errata phase. Refer to the Red Hat Enterprise Linux Life Cycle for information about current support phases.
• Live kernel patching is available for Extended Update Support (EUS) for Premium SAP subscribers only, starting with RHEL 7.4.
• kpatch is currently not supported on the Red Hat Enterprise Linux Realtime (RT) kernel.
• kpatch is not supported on Red Hat Enterprise Linux 5 and Red Hat Enterprise Linux 6.
• Only one live kernel patch may be installed on the kernel at any given time.
• Not all issues may be covered under live kernel patching, including hardware enablement.

Access and Delivery of Live Kernel Patching

• Live kernel patching capability is implemented via a kernel module (kmod) that is delivered as an RPM package. The kpatch utility is used to install and remove the kernel modules for live kernel patching.
• Customers with Premium subscriptions are eligible to request a live kernel patch as part of an accelerated fix solution from Red Hat Support.
• Customers with Premium subscriptions who typically used 'hotfix' kernels which required a reboot can now request a kpatch kmod that requires no down time. The kpatch patch will be supported 30 days after the errata that contains the fix is released, in the same manner as 'hotfix' kernels.
• Customers should open a support case directly in the Red Hat Customer Portal and discuss appropriate accelerated fix options.

Limitations

• kpatch is not a general purpose kernel upgrade mechanism. It is used for applying simple security and bugfix updates when rebooting the system is not immediately possible.
• Do not use the SystemTap or kprobe tools during or after loading a patch. The patch could fail to take effect until after the probe has been removed.
• Do not directly access the ftrace output file, for example by running cat /sys/kernel/debug/tracing/trace. Using the trace-cmd utility is supported instead.
• Do not suspend or hibernate the system when using kpatch. This can result in a patch being temporarily disabled for a small amount of time.

NOTE: Red Hat is actively working to remove many of the limitations of kpatch for future releases.

How Do I Enable and Use kpatch?

The components of kpatch are as follows:

1. systemd Integration Point
   • A systemd service called kpatch.service that is required by multiuser.target which loads the kpatch modules at boot time.
2. Patch Module
   • The delivery mechanism for new kernel code.
   • This is another kernel module that is named to match the kpatch being applied.
   • The patch module contains the compiled code from the latest hot fixes introduced to the kernel.
   • The patch modules register with the core module, kpatch.ko, and provide information about which original functions will be replaced, with corresponding pointers to the replacement functions.
3. kpatch utility
   • A command-line tool which allows you to manage patch modules.

Installing kpatch Tools

Before you can install a kpatch module, you must install the kpatch tools package. To do so, type the following at a shell prompt as as root:

 # yum install kpatch

Installing a kpatch Hot Fix

To install a kpatch hot fix, install the supplied kpatch-patch RPM package with yum. For example, to install kpatch-patch-7.0-1.el7.x86_64.rpm, issue the following command as root:

 # yum install kpatch-patch-7.0-1.el7.x86_64.rpm

Listing Installed kpatch Hot Fixes

To verify a patch is loaded and installed, run the kpatch list command as root:

 # kpatch list
Loaded patch modules:
kpatch_7_0_1_el7
Installed patch modules:
kpatch-7-0-1-el7.ko (3.10.0-121.el7.x86_64)
kpatch-7-0-1-el7.ko (3.10.0-123.el7.x86_64)

The example output above shows that the module has been loaded into the kernel, meaning the kernel is now patched with the latest hot fixes contained in the kpatch-patch-7.0-1.el7.x86_64.rpm package. It also shows that it has been saved to /var/lib/kpatch to be loaded by systemd during future reboots for kernel versions 3.10.0-121 and 3.10.0-123.

Updating a kpatch Hot Fix

If a new version of the kpatch-patch RPM package is later released, upgrade the applied patch with yum. For example, to upgrade to kpatch-patch-7.0-2.el7.x86_64.rpm, run as root:

 # yum update kpatch-patch-7.0-2.el7.x86_64.rpm

Upgrading the RPM package atomically replaces the patch module in the running kernel and updates the /var/lib/kpatch structures used by systemd on reboot.

Note The patch modules in the kpatch-patch RPM packages are cumulative. Consequently, you could skip installing kpatch-patch-7.0-1 and instead start with installing kpatch-patch-7.0-2 if it were available.

Loading a patch module sets the TAINT_LIVEPATCH kernel taint flag (which corresponds to bit 15), TAINT_OOT_MODULE (which corresponds to bit 12), and TAINT_TECH_PREVIEW (which corresponds to bit 29). To determine whether the kernel has been patched, use the cat command to display the contents of /proc/sys/kernel/tainted and check the value in the file. Unless you have other taint flags set, the value should be 536907776 (0x20009000) when the kernel is patched.

Removing a kpatch Hot Fix

To unload a kpatch module from the running kernel, use the kpatch unload command, specifying the name of the patch module. For example, to unload kpatch_7_0_2_el7, type the following at a shell prompt as root:

 # kpatch unload kpatch_7_0_2_el7

The patch module must also be uninstalled from /var/lib/kpatch with the kpatch uninstall command as follows:

 # kpatch uninstall kpatch_7_0_2_el7

The default behavior of this command is to uninstall the kpatch from the kernel corresponding to the current kernel version, but you can specify a different kernel version by using the --kernel-version option:

 # kpatch uninstall --kernel-version 3.10.0-121.el7.x86_64 kpatch_7_0_2_el7

Alternatively, you can uninstall the kpatch-patch RPM package, which also removes the patch module from /var/lib/kpatch.

Note: Uninstalling the RPM package does not unload the kpatch module from the kernel. An explicit call to kpatch unload as described above is required.

How Does kpatch Work?

The kpatch utilities leverage the existing ftrace capabilities to enable arbitrary remapping of pointers to kernel functions. When a live kernel patch is applied to a system, the following things happen:

1. The new compiled code in the module is copied to /var/lib/kpatch and registered for re-application to the kernel via systemd on next boot.
2. The kpatch module is loaded into the running kernel and the new functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.
3. When the kernel accesses the patched function, it is redirected to the ftrace mechanism which bypasses the original functions and redirects the kernel to patched version of the function.

Are Any Third Party Live Patching Solutions Supported?

Although several third party and proprietary tools that provide live kernel patching are available, Red Hat only supports kpatch and the RPM modules supplied through your Red Hat support contract. Red Hat cannot support third party live-patches, however requests for Engineering and an official Red Hat kpatch can be opened at any time.

For any Red Hat review of a third party live-patch, the source code would need to be supplied to determine if they meet the following criteria:

1. Impact the same subsystems and codepaths as the kernel encountered during a failure.
2. Applying the same patches using supported means, within the applicable streams, result in no failure being encountered.

Red Hat recommends that you open a case with the live kernel patching vendor at the outset of any investigation in which a root cause determination is necessary. This will allow the source code to be supplied if the vendor allows, and for their support organization to provide assistance in root cause determination prior to escalating the investigation to Red Hat Support.

For any system running with third party live kernel patches, Red Hat reserves the right to ask for reproduction with Red Hat shipped and supported software. In the event that this is not possible, we will require a similar system and workload be deployed on your test environment without live patches applied, to confirm if the same behavior is observed.

For more information about third party software support policies, see How does Red Hat Global Support Services handle third-party software, drivers, and/or uncertified hardware/hypervisors or guest operating systems?.