- What is kpatch?
- What Is the Scope of Support for kpatch?
- Access and Delivery of Live Kernel Patching
- How Do I Enable and Use kpatch?
- How Does kpatch Work?
- Are Any Third Party Live Patching Solutions Supported?
kpatch is a live kernel patching solution that allows you to patch a running kernel without rebooting or restarting any processes. It enables system administrators to apply critical security patches to the kernel immediately, without having to wait for long-running tasks to complete, for users to log off, or for scheduled reboot windows. It gives more control over uptime without sacrificing security or stability.
WARNING: Some incompatibilities exist between kpatch and other kernel subcomponents. Read the Limitations section carefully before using kpatch.
- Live kernel patching with kpatch is supported from Red Hat Enterprise Linux 7.2 onwards.
- Live kernel patching is supported for customers with Premium SLA subscriptions.
- Live kernel patching is only supported on the active Red Hat Enterprise Linux 7 maintenance stream that is within the current async errata phase. Refer to the Red Hat Enterprise Linux Life Cycle for information about current support phases.
- Live kernel patching is available for Extended Update Support (EUS) for Premium SAP subscribers only, starting with RHEL 7.4.
- kpatch is currently not supported on the Red Hat Enterprise Linux Realtime (RT) kernel.
- kpatch is not supported on Red Hat Enterprise Linux 5 and Red Hat Enterprise Linux 6.
- Only one live kernel patch may be installed on the kernel at any given time.
- Not all issues may be covered under live kernel patching, including hardware enablement.
- Live kernel patching capability is implemented via a kernel module (kmod) that is delivered as an RPM package. The kpatch utility is used to install and remove the kernel modules for live kernel patching.
- Customers with Premium subscriptions are eligible to request a live kernel patch as part of an accelerated fix solution from Red Hat Support.
- Customers with Premium subscriptions who typically used 'hotfix' kernels which required a reboot can now request a kpatch kmod that requires no down time. The kpatch patch will be supported 30 days after the errata that contains the fix is released, in the same manner as 'hotfix' kernels.
- Customers should open a support case directly in the Red Hat Customer Portal and discuss appropriate accelerated fix options.
- kpatch is not a general purpose kernel upgrade mechanism. It is used for applying simple security and bugfix updates when rebooting the system is not immediately possible.
- Do not use the SystemTap or kprobe tools during or after loading a patch. The patch could fail to take effect until after the probe has been removed.
- Do not directly access the ftrace output file, for example by running cat
/sys/kernel/debug/tracing/trace. Using the
trace-cmdutility is supported instead.
- Do not suspend or hibernate the system when using kpatch. This can result in a patch being temporarily disabled for a small amount of time.
NOTE: Red Hat is actively working to remove many of the limitations of kpatch for future releases.
How Do I Enable and Use kpatch?
The components of kpatch are as follows:
- systemd Integration Point
kpatch.servicethat is required by
multiuser.targetwhich loads the kpatch modules at boot time.
- Patch Module
- The delivery mechanism for new kernel code.
- This is another kernel module that is named to match the kpatch being applied.
- The patch module contains the compiled code from the latest hot fixes introduced to the kernel.
- The patch modules register with the core module,
kpatch.ko, and provide information about which original functions will be replaced, with corresponding pointers to the replacement functions.
- kpatch utility
- A command-line tool which allows you to manage patch modules.
Installing kpatch Tools
Before you can install a kpatch module, you must install the
kpatch tools package. To do so, type the following at a shell prompt as as
# yum install kpatch
Installing a kpatch Hot Fix
To install a kpatch hot fix, install the supplied
kpatch-patch RPM package with
yum. For example, to install
kpatch-patch-7.0-1.el7.x86_64.rpm, issue the following command as
# yum install kpatch-patch-7.0-1.el7.x86_64.rpm
Listing Installed kpatch Hot Fixes
To verify a patch is loaded and installed, run the
kpatch list command as
# kpatch list Loaded patch modules: kpatch_7_0_1_el7 Installed patch modules: kpatch-7-0-1-el7.ko (3.10.0-121.el7.x86_64) kpatch-7-0-1-el7.ko (3.10.0-123.el7.x86_64)
The example output above shows that the module has been loaded into the kernel, meaning the kernel is now patched with the latest hot fixes contained in the
kpatch-patch-7.0-1.el7.x86_64.rpm package. It also shows that it has been saved to
/var/lib/kpatch to be loaded by
systemd during future reboots for kernel versions 3.10.0-121 and 3.10.0-123.
Updating a kpatch Hot Fix
If a new version of the
kpatch-patch RPM package is later released, upgrade the applied patch with
yum. For example, to upgrade to
kpatch-patch-7.0-2.el7.x86_64.rpm, run as
# yum update kpatch-patch-7.0-2.el7.x86_64.rpm
Upgrading the RPM package atomically replaces the patch module in the running kernel and updates the
/var/lib/kpatch structures used by
systemd on reboot.
Note The patch modules in the
kpatch-patch RPM packages are cumulative. Consequently, you could skip installing kpatch-patch-7.0-1 and instead start with installing kpatch-patch-7.0-2 if it were available.
Loading a patch module sets the
TAINT_LIVEPATCH kernel taint flag (which corresponds to bit 15),
TAINT_OOT_MODULE (which corresponds to bit 12), and
TAINT_TECH_PREVIEW (which corresponds to bit 29). To determine whether the kernel has been patched, use the
cat command to display the contents of
/proc/sys/kernel/tainted and check the value in the file. Unless you have other taint flags set, the value should be
536907776 (0x20009000) when the kernel is patched.
Removing a kpatch Hot Fix
To unload a kpatch module from the running kernel, use the
kpatch unload command, specifying the name of the patch module. For example, to unload
kpatch_7_0_2_el7, type the following at a shell prompt as
# kpatch unload kpatch_7_0_2_el7
The patch module must also be uninstalled from
/var/lib/kpatch with the
kpatch uninstall command as follows:
# kpatch uninstall kpatch_7_0_2_el7
The default behavior of this command is to uninstall the kpatch from the kernel corresponding to the current kernel version, but you can specify a different kernel version by using the
# kpatch uninstall --kernel-version 3.10.0-121.el7.x86_64 kpatch_7_0_2_el7
Alternatively, you can uninstall the
kpatch-patch RPM package, which also removes the patch module from
Note: Uninstalling the RPM package does not unload the kpatch module from the kernel. An explicit call to
kpatch unload as described above is required.
How Does kpatch Work?
kpatch utilities leverage the existing
ftrace capabilities to enable arbitrary remapping of pointers to kernel functions. When a live kernel patch is applied to a system, the following things happen:
- The new compiled code in the module is copied to
/var/lib/kpatchand registered for re-application to the kernel via
systemdon next boot.
- The kpatch module is loaded into the running kernel and the new functions are registered to the
ftracemechanism with a pointer to the location in memory of the new code.
- When the kernel accesses the patched function, it is redirected to the
ftracemechanism which bypasses the original functions and redirects the kernel to patched version of the function.
Are Any Third Party Live Patching Solutions Supported?
Although several third party and proprietary tools that provide live kernel patching are available, Red Hat only supports
kpatch and the RPM modules supplied through your Red Hat support contract. Red Hat cannot support third party live-patches, however requests for Engineering and an official Red Hat kpatch can be opened at any time.
For any Red Hat review of a third party live-patch, the source code would need to be supplied to determine if they meet the following criteria:
- Impact the same subsystems and codepaths as the kernel encountered during a failure.
- Applying the same patches using supported means, within the applicable streams, result in no failure being encountered.
Red Hat recommends that you open a case with the live kernel patching vendor at the outset of any investigation in which a root cause determination is necessary. This will allow the source code to be supplied if the vendor allows, and for their support organization to provide assistance in root cause determination prior to escalating the investigation to Red Hat Support.
For any system running with third party live kernel patches, Red Hat reserves the right to ask for reproduction with Red Hat shipped and supported software. In the event that this is not possible, we will require a similar system and workload be deployed on your test environment without live patches applied, to confirm if the same behavior is observed.
For more information about third party software support policies, see How does Red Hat Global Support Services handle third-party software, drivers, and/or uncertified hardware/hypervisors or guest operating systems?.