Is live kernel patch (kpatch) supported in Red Hat Enterprise Linux ?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7.7
  • Red Hat Enterprise Linux 7.6
  • kpatch
  • AMD64, Intel 64 and ppc64le architectures

Issue

  • Does Red Hat offer a live kernel patching mechanism?
  • What is kpatch, and when will it be available?

Resolution

Live kernel patches avoids the need for a reboot when patching the kernel for select important and critical CVEs. Live kernel patch is supported for customers who have an active subscription. Live kernel patches are cumulative, so when you get a new live kernel patch for a kernel, it will have all the fixes of the previous live kernel patch, along with new fixes. You can safely upgrade the loaded live kernel patch to a newer live patch.

Current scope and limitations of kpatch

  • Starting with RHEL 8.1, RHEL 7.7; RHEL-7.6, starting with kernel-3.10.0-957.35.1.el7 -- live kernel patches will be available on the Red Hat Content Delivery Network(CDN) and can be installed via the yum command.

  • Live kernel patches will be made available for selected Important and Critical CVEs.

  • Live Patches for CVEs that occur between minor kernel releases are available with standard subscriptions. Customers who purchase Extended Update Support will be able to use live patching for a full year after a particular kernel release.

  • No support for Red Hat Enterprise Linux 5 and Red Hat Enterprise Linux 6.

  • Live patches will only release for minor releases where Extended Update Support (EUS) is planned.

  • Unloading a kpatch from the kernel is not supported. The workaround is to uninstall the kpatch, and to reboot.

Access and Delivery of live kernel patches:

  • The live kernel patch capability is implemented as a kernel module (kmod) that is delivered as an RPM. The kpatch utility, currently available on RHEL 7 and RHEL 8, is used to install and remove the kernel modules for live kernel patch.

  • Customers with active subscriptions are eligible to receive live kernel patches via the Red Hat CDN.

For directions to enable live patching, see:
Enabling kernel live patching

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

26 Comments

So I need to contact support every time I want a kpatch for a kernel? Are there any plans to just make kpatches available for any kernel updates?

Erinn, There are no plans to provide KPatch kmods for every z or EUS kernel Red Hat releases. However, if KPatch adoption picks up in the future is could be a possibility. This Kbase solution will be updated if changes like that are made.

1) Any plans to make kpatch patches available for any kernal updates ? Or will it always be based on customer request ? 2) Are there any solutions available or similar plans in future to dynamically patch packages like glibc, which also requires reboot as recommended by Redhat?

Staish, There are no plans to provide KPatch kmods for every z or EUS kernel Red Hat releases. However, if KPatch adoption picks up in the future is could be a possibility. As for your second question, there are currently no plans to provide dynamic patching for packages like glibc.

I see the current comments regarding kpatch module availability, but they are several months old. Is it still the case the kpatch modules will not be provided with each kernel release, but instead having to request them specifically? What is the rationale behind that? Why "support" a function/binary without actually supporting it? Patching without a reboot is something sysadmins have been screaming for for years. IBM AIX will be supporting live patching in their upcoming TL for AIX 7.2 - Red Hat should have already been doing this as a fully supported capability rather than a "by request only" solution.

Hello Kevin, You are correct. Kpatch kmods are not released for every kernel but customers can request a kpatch kmod be created for a particular issue they are experiencing. The rationale behind that is it allows customers to schedule downtime at their convenience rather than having to reboot the system right away to boot the kernel that contains the fix. The kpatch kmod is fully supported until 30 days after the errata that contains the fix has been released.

The use case you describe is a possibility in the future. It's difficult to be precise as to when Red Hat will support kpatch in that manner but it is something we are working towards. Thank you for your comments and I will definitely pass them on to Product Management.

how is that different from Oracle ksplice? from what I read ksplice have no days limit to reboot, and is already supported in Fedora and Ubuntu, this will increase their test base. are there any good informative non-marketing article comparing ksplice to kpatch, and do we know what will be out in 7.3?

So, this is available only for Premium subscription; what about standard, shoul we wait for kernel update release? .........Talking'bout https://access.redhat.com/security/vulnerabilities/2706661

Hello, Red Hat Product Security has granted an one time exception allowing customer's who do not have premium support to request kpatch packages related to CVE-2016-5195. Please open a support case and reference URL: https://access.redhat.com/security/vulnerabilities/2706661 for assistance.

We have premium support, and have requested the kpatch through a support case but we were told to go to a page and that page only says to request the the kpatch through a support case :-/

We should have you all sorted Adam. Thanks for the feedback!

Clearly there's great interest in kpatch. Has this KB moved at all? One of the big selling points with EUS is stability; kpatch is a great tool to that end. So I'd like to know what timelines there are and/or future plans for kpatch.

Hello where can I download the packages kpatch-patch RPM ?

Hello Edson Luna Navarrete,

The only way to obtain a kpatch-patch RPM is to open a case and request a kpatch-patch. Make sure to include which CVEs or BZs that you would fixed, and which specific kernel version(s) as returned by uname -r.

If you already have a case open for the specific issue, you can add a comment to the case to request the kpatch-patch.

This isn't valid:

yum update “kpatch-patch = (uname -r)”

I'm guessing you mean:

yum update “kpatch-patch-$(uname -r)”

Though the packages don't seem to exist yet.

Erinn,

Thanks for pointing out the missing $. I updated the article with the correction.

The yum install "kpatch-patch = $(uname -r)" command will only work if you are running a kernel for which there will be kpatches released. Currently there are no kpatches that fix an issue available for kernel-3.10.0-1062.el7. After another RHEL-7.7 kernel is released that fixes an important or critical CVE, check for the available of a kpatch.

Is this something available with satellite ?

Dhaval,

kpatch-patch is distributed as an rpm. Satellite can be used to install kpatches just like it can install any other rpm.

Is this still a process where RH Support has to be contacted to receive kernel patches kpatch loads, or once kpatch is installed the kernel found in yum checks will be any updates RH has made to the current kpatch kernel available?

Hi Ryan,

Starting with the RHEL 7.7 kernel, there is no need to open a case to obtain a kpatch. Subscribe to kpatch for the current kernel:

# yum install "kpatch-patch = $(uname -r)"

After that, you will get the latest kpatches for the subscribed kernel with:

# yum update

For other kernels that can receive kpatches, it is still required to open a case in order to request a kpatch.

Regards,

Marc

Marc,

Are there plans to make this available to CentOS users, now that this is generally available for RHEL 7.7?

This would be huge!

-Matt

"uname -r" is still showing the old version after live patch. This is not very convenient to check whether the OS is patched or not. When I use ksplice, it shows a new kernel version.

$ uname -r
3.10.0-1062.el7.x86_64

$ sudo yum install -y "kpatch-patch = $(uname -r)"
Package kpatch-patch-3_10_0-1062-0-0.el7.x86_64 already installed and latest version
Nothing to do
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager

$ sudo yum update -y
$ sudo kpatch list
Loaded patch modules:
kpatch_3_10_0_1062_1_1 [enabled]

Installed patch modules:
kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64)
### Note: uname -r is still showing the old release
$ uname -r
3.10.0-1062.el7.x86_64

$ rpm -q kernel
kernel-3.10.0-1062.el7.x86_64
kernel-3.10.0-1062.1.2.el7.x86_64


$ ls -l /var/lib/kpatch/3.10.0-1062.el7.x86_64/kpatch-3_10_0-1062-1-1.ko
-rw-r--r--. 1 root root 30633 Sep 23 09:37 /var/lib/kpatch/3.10.0-1062.el7.x86_64/kpatch-3_10_0-1062-1-1.ko

Hello Ken,

kpatch does not affect the kernel version as returned by uname -r. A kpatch-patch fixes important and critical security issues. A kernel update usually fixes more bugs than a kpatch-patch.

You can see which bugs have been fixed in the current kpatch, but it is not completely straighforward:

$ rpm -qa | grep kpatch-patch
kpatch-patch-3_10_0-1062-1-1.el7.x86_64

$ rpm -q --changelog kpatch-patch-3_10_0-1062-1-1.el7.x86_64
* Mon Sep 16 2019 Artem Savkov <asavkov@redhat.com> [1-1.el7]
- vhost-net: guest to host kernel escape during migration [1750892] {CVE-2019-14835}

* Fri Jul 19 2019 Artem Savkov <asavkov@redhat.com> [0-0.el7]
- An empty patch to subscribe to kpatch stream for kernel-3.10.0-1062.el7 [1730784]

I will make a suggestion to Engineering to make an easy way to see which kernel version were used for the latest fixes in the kpatch-patch.

Hi, I think we should add "Red Hat Enterprise Linux 8" in to "Environment" section.

Regards, Sam

Xinhua,

kpatch-patch hasn't shipped via the CDN for RHEL-8 yet. When that happens, RHEL 8 will be added to the Environment section.

Regards, Marc

"Kernel Live patching for the kernel, kpatch, is now available, which enables you to consume Critical and Important CVEs fixes without the need to reboot your system. See Section 4.7, “Kernel” for more information."

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/8.1_release_notes/overview