Chapter 6. Working with kpatch
6.1. What is kpatch?
kpatch is a live kernel patching solution that allows you to patch a running kernel without rebooting or restarting any processes. It enables system administrators to apply critical security patches to the kernel immediately, without having to wait for long-running tasks to complete, for users to log off, or for scheduled downtime. It gives more control over uptime without sacrificing security or stability.
Some incompatibilities exist between
kpatch and other kernel subcomponents. Read the Section 6.4, “Limitations” section carefully before using
6.2. What is the scope of support for kpatch?
Live kernel patching with
kpatchis supported from Red Hat Enterprise Linux 7.2 onwards.
- Live kernel patching is supported for customers with Premium SLA subscriptions.
- Live kernel patching is only supported on the active Red Hat Enterprise Linux 7 maintenance stream that is within the current async errata phase. See Red Hat Enterprise Linux Life Cycle for information about current support phases.
- Live kernel patching is not available on Extended Update Support releases.
kpatchis not supported on the Red Hat Enterprise Linux Realtime (RT) kernel.
kpatchis not supported on Red Hat Enterprise Linux 5 or Red Hat Enterprise Linux 6.
- Only one live kernel patch can be installed on the kernel at any given time.
- It is not certain that all issues are covered under live kernel patching, including hardware enablement.
6.3. Access and delivery
Live kernel patching capability is implemented via a kernel module (kmod) that is delivered as an RPM package. The
kpatch utility is used to install and remove the kernel modules for live kernel patching.
Customers with Premium subscriptions are eligible to request a live kernel patch as part of an accelerated fix solution from Red Hat Support.
Customers with Premium subscriptions who typically used
hotfix kernels which required a reboot can now request a
kpatch kmod that requires no down time. The
kpatch patch is supported for 30 days after the errata that contains the fix is released, in the same manner as
Customers have the option to open a support case directly in the Red Hat Customer Portal and discuss appropriate accelerated fix options.
kpatch is not a general-purpose kernel upgrade mechanism. It is used for applying simple security and bug fix updates when rebooting the system is not immediately possible.
Do not use the
kprobe tools during or after loading a patch. The patch could fail to take effect until after the probe has been removed.
Do not directly access the
ftrace output file, for example by running
cat /sys/kernel/debug/tracing/trace. The
trace-cmd utility is supported instead.
Do not suspend or hibernate the system when using
kpatch. This can result in a patch being temporarily disabled for a small amount of time.
Red Hat is actively working to remove many of the limitations of
kpatch for future releases.
6.5. How do I enable and use kpatch?
The components of
kpatch are as follows:
Components of kpatch
- A systemd Integration Point
kpatch.servicethat is required by
multiuser.targetwhich loads the kpatch modules at boot time.
- A Patch Module
- The delivery mechanism for new kernel code.
This is another kernel module that is named to match the
- The patch module contains the compiled code from the latest hot fixes introduced to the kernel.
The patch modules register with the core module,
kpatch.koand provide information about original functions to be replaced, with corresponding pointers to the replacement functions.
- A command-line tool which allows you to manage patch modules.
6.5.1. Installing the kpatch tools
Before you can install a
kpatch module, you must install the
kpatch tools package. To do so, type the following at a shell prompt as
# yum install kpatch
6.5.2. Installing a kpatch hot fix
To install a
kpatch hot fix, install the supplied
kpatch-patch RPM package with
yum. For example, to install
kpatch-patch-7.0-1.el7.x86_64.rpm, issue the following command as
# yum install kpatch-patch-7.0-1.el7.x86_64.rpm
6.5.3. Listing installed kpatch hot fixes
To verify a patch is loaded and installed, run the
kpatch list command as
# kpatch list Loaded patch modules: kpatch_7_0_1_el7 Installed patch modules: kpatch-7-0-1-el7.ko (3.10.0-121.el7.x86_64) kpatch-7-0-1-el7.ko (3.10.0-123.el7.x86_64)
The example output above shows that the module has been loaded into the kernel, meaning the kernel is now patched with the latest hot fixes contained in the
kpatch-patch-7.0-1.el7.x86_64.rpm package. It also shows that it has been saved to
/var/lib/kpatch to be loaded by
systemd during future reboots for kernel versions 3.10.0-121 and 3.10.0-123.
6.5.4. Updating a kpatch hot fix
If a new version of the
kpatch-patch RPM package is later released, upgrade the applied patch with
yum For example, to upgrade to
kpatch-patch-7.0-2.el7.x86_64.rpm run as
# yum update kpatch-patch-7.0-2.el7.x86_64.rpm
Upgrading the RPM package automatically replaces the patch module in the running kernel and updates the
/var/lib/kpatch structures used by
systemd on reboot.
The patch modules in the
kpatch-patch RPM packages are cumulative. Consequently, you could skip installing kpatch-patch-7.0-1 and instead start with installing kpatch-patch-7.0-2 if it were available.
Loading a patch module sets the
TAINT_LIVEPATCH kernel taint flag (which corresponds to bit 15) and
TAINT_OOT_MODULE (which corresponds to bit 12). To determine whether the kernel has been patched, use the
cat command to display the contents of
/proc/sys/kernel/tainted and check the value in the file. Unless you have other taint flags set, the value is
36864 when the kernel is patched.
6.5.5. Removing a kpatch Hot Fix
To unload a
kpatch module from the running kernel, use the
kpatch unload command, specifying the name of the patch module. For example, to unload
kpatch_7_0_2_el7 type the following at a shell prompt as
# kpatch unload kpatch_7_0_2_el7
The patch module must also be uninstalled from
/var/lib/kpatch with the
kpatch uninstall command as follows:
# kpatch uninstall kpatch_7_0_2_el7
The default behavior of this command is to uninstall the
kpatch from the kernel corresponding to the current kernel version, but you can specify a different kernel version by using the
# kpatch uninstall --kernel-version 3.10.0-121.el7.x86_64 kpatch_7_0_2_el7
Alternatively, you can uninstall the
kpatch-patch RPM package, which also removes the patch module from
Uninstalling the RPM package does not unload the
kpatch module from the kernel. An explicit call to
kpatch unload as described above is required.
6.6. How does kpatch work?
kpatch utilities use
ftrace for arbitrary remapping of pointers to kernel functions. When a live kernel patch is applied to a system, the following things happen:
The new compiled code in the module is copied to
/var/lib/kpatchand registered for re-application to the kernel via
systemdon next boot.
kpatchmodule is loaded into the running kernel and the new functions are registered to the
ftracemechanism with a pointer to the location in memory of the new code.
When the kernel accesses the patched function, it is redirected to the
ftracemechanism which bypasses the original functions and redirects the kernel to patched version of the function.
Figure 6.1. How kpatch Works
6.7. Are any third-party live patching solutions supported?
Although several third-party and proprietary tools that provide live kernel patching are available, Red Hat only supports
kpatch and the RPM modules supplied through your Red Hat support contract. Red Hat cannot support third-party live-patches, however requests for Engineering and an official Red Hat
kpatch can be opened at any time.
For any Red Hat review of a third-party live-patch, the source code would need to be supplied to determine if they meet the following criteria:
- Impact the same subsystems and codepaths as the kernel encountered during a failure.
- Applying the same patches using supported means, within the applicable streams, result in no failure being encountered.
Red Hat recommends that you open a case with the live kernel patching vendor at the outset of any investigation in which a root cause determination is necessary. This allows the source code to be supplied if the vendor allows, and for their support organization to provide assistance in root cause determination prior to escalating the investigation to Red Hat Support.
For any system running with third-party live kernel patches, Red Hat reserves the right to ask for reproduction with Red Hat shipped and supported software. In the event that this is not possible, we require a similar system and workload be deployed on your test environment without live patches applied, to confirm if the same behavior is observed.
For more information about third-party software support policies, see How does Red Hat Global Support Services handle third-party software, drivers, and/or uncertified hardware/hypervisors or guest operating systems? in the knowledgebase at https://access.redhat.com/articles/1067.