Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715 and CVE-2017-5753 using Red Hat Enterprise Linux Tunables

Updated -

Overview

Red Hat Customer Portal Labs provides a Spectre And Meltdown Detector to help you detect if your systems are vulnerable to these flaws.

The recent speculative execution CVEs address three potential attack vectors across a wide variety of processor architectures and platforms. Each platform requiring slightly different fixes. In many cases these fixes also require matching microcode updates provided by hardware vendors.

The security vulnerabilities described in these three CVEs may be found in modern microprocessors and operating systems on major hardware platforms including x86 (Intel and AMD chipsets), System Z, Power and ARM.

Red Hat has made updated kernels available to address these security vulnerabilities. These patches are enabled by default because Red Hat prioritizes out of the box security. Speculative execution is a performance optimization technique which these updates change (both kernel and microcode) and may result in workload-specific performance degradation.

Some customers who feel confident that their systems are well protected may wish to disable some or all of protection mechanisms. If the system administrator wishes elects to enable the protection mechanisms in the interest of security, this article provides a method to conduct performance characterizations with and without the fixes enabled.

Retpoline Kernels

For Red Hat Enterprise Linux versions up through RHEL-7.6, Red Hat uses “retpoline” code sequences for indirect branches in the kernel to isolate those branches from speculative execution. In those OS releases, for Intel processors prior to Skylake, retpolines are used instead of the ibrs feature for mitigation against Spectre variant 2. For Skylake, ibrs is used instead of retpolines.

With the upcoming Red Hat Enterprise Linux 7.7 release, it is planned that all new installations on all Intel processors up through and
including Skylake will default to use retpolines. However updates from versions from RHEL 7.6, and before, to RHEL-7.7, and later, will preserve the existing Spectre variant 2 mitigation methods that were in place before the upgrade.

Any updated system that is using ibrs can be switched from using ibrs to retpolines at any time simply by adding the "spectre_v2=retpoline" flag to the kernel boot command line.

A patched GCC compiler with Retpoline support is required for compiling the Retpoline patched kernel and third party modules. Any third party kernel module supplied prior to the update will require recompiling from source. SystemTap is one example that uses kernel modules to run code in kernel space, so it also needs the patched compiler. Please contact your software provider of the kernel module and request an update if the kernel module fails to work correctly.

Disabling the protection mechanisms:

For Red Hat Enterprise Linux kernels on x86, three debugfs tunables control the behaviour of the various patches in the updated kernel. These patches require updated microcode, which can be obtained from the hardware platform providers.

These debugfs tunables can be enabled or disabled on the kernel command line at boot or at runtime via debugfs controls. The tunables control Page Table Isolation (pti), Indirect Branch Restricted Speculation (ibrs), and retpolines (retp). Depending on the CPU type, Red Hat enables each of these features by default as needed to protect the architecture detected at boot.

For those wanting to disable the security mitigation for these protection mechanisms to recover lost performance, the changes can be set at run time or persistently across reboots.

Persistently disable - Effective across a reboot

The first option is to disable them via the kernel command line by adding these flags, then reboot the kernel to have them take effect: There are several flags available to do this, noted below.

     spectre_v2=off nopti

Note: you can individually disable each parameter, for performance characterization it is not required that all be simultaneously disabled.

Red Hat recommends using 'grubby' to manage these changes. See the Systems Adminstrators guide section on grubby for how to use this tool effectively.

Runtime disable - Does not persist through a reboot

The second option is to disable them at runtime with the following three commands. The change is immediately active and does not require a reboot.

    # echo 0 > /sys/kernel/debug/x86/pti_enabled
    # echo 0 > /sys/kernel/debug/x86/retp_enabled
    # echo 0 > /sys/kernel/debug/x86/ibrs_enabled

Note this requires that the debugfs filesystem be mounted. In addition, the above "retp_enabled" alteration is only available at runtime for RHEL7 systems. This tunable is read-only for RHEL 6 systems. In RHEL 7 the debugfs is mounted by default. In RHEL 6 you can mount it manually with the command:

  mount -t debugfs nodev /sys/kernel/debug

Verifying changes

To verify the fixes for these CVEs are correctly disabled, check the contents of the three files to verify their values are all set to 0.

    # cat /sys/kernel/debug/x86/pti_enabled
    # cat /sys/kernel/debug/x86/retp_enabled
    # cat /sys/kernel/debug/x86/ibrs_enabled

Some applications may still see a small performance loss even with the above CVE flags disabled.

Details:

The rest of this article describes more specifics about each CVE variant.

  • CVE-2017-5753 (variant #1/Spectre) is a Bounds-checking exploit during branching. This issue is fixed with a kernel patch. Variant #1 protection is always enabled; it is not possible to disable the patches. Red Hat’s performance testing for variant #1 did not show any measurable impact.

  • CVE-2017-5715 (variant #2/Spectre) is an indirect branching poisoning attack that can lead to data leakage. This attack allows for a virtualized guest to read memory from the host system. This issue is corrected with microcode, along with kernel and virtualization updates to both guest and host virtualization software. This vulnerability requires both updated microcode and kernel patches. Variant #2 behavior is controlled by the ibrs tunable which work in conjunction with the microcode, or the retp tunable. The ibpb tunable is still visible, but now read-only and is set by the kernel.

  • CVE-2017-5754 (variant #3/Meltdown) is an exploit that uses speculative cache loading to allow a local attacker to be able to read the contents of memory. This issue is corrected with kernel patches. Variant #3 behavior is controlled by the pti tunable (nopti/pti_enabled).

As noted, installing the microcode update for your hardware, if provided by the hardware vendor, is necessary to protect against variant 2. Please contact your hardware vendor for microcode updates.

Page Table Isolation (pti)

"nopti"/pti_enabled controls the Kernel Page Table Isolation feature, which isolates kernel pagetables when running in userland. This feature addresses CVE-2017-5754, also called variant #3 (Meltdown)

Customers and vendors can disable the PTI feature by passing "nopti" to the kernel command line at boot, or dynamically with the runtime debugfs control below:

    # echo 0 > /sys/kernel/debug/x86/pti_enabled

Indirect Branch Restricted Speculation (ibrs)

The "noibrs"/ibrs_enabled controls the IBRS feature in the SPEC_CTRL model-specific register (MSR) when SPEC_CTRL is present in cpuid (post microcode update). When ibrs_enabled is set to 1 (spectre_v2=ibrs) the kernel runs with indirect branch restricted speculation, which protects the kernel space from attacks (even from hyperthreading/simultaneous multi-threading attacks). When IBRS is set to 2 (spectre_v2=ibrs_always), both userland and kernel runs with indirect branch restricted speculation. This protects userspace from hyperthreading/simultaneous multi-threading attacks as well, and is also the default on certain old AMD processors (family 10h, 12h and 16h). This feature addresses CVE-2017-5715, variant #2.

When ibrs_enabled is set to 3, only userland runs with indirect branch restricted speculation. This can be used in combination with retpoline (spectre_v2=retpoline,ibrs_user) to provide similar security to ibrs_always with less performance overhead.

The ibrs implementation can be disabled in microcode by passing "noibrs" to the kernel command line at boot, or dynamically with the debugfs control below:

    # echo 0 > /sys/kernel/debug/x86/ibrs_enabled

Indirect Branch Prediction Barriers (ibpb)

Note: The ibpb tuning knob is now read-only and will be set by the kernel if either ibrs or retp is set. As with ibrs, ibpb needs updated microcode in order to work (and be set) correctly.

Ibpb controls the IBPB feature in the PRED_CMD model-specific register (MSR) if either IBPB_SUPPORT or SPEC_CTRL is present in cpuid (post microcode update). When ibpb_enabled is set to 1, an IBPB barrier that flushes the contents of the indirect branch prediction is run across user mode or guest mode context switches to prevent user and guest mode from attacking other applications or virtual machines on the same host. In order to protect virtual machines from other virtual machines, ibpb_enabled=1 is needed even if ibrs_enabled is set to 2. This feature addresses CVE-2017-5715, variant #2.

Architectural Defaults

By default, the appropriate tunables that apply to an architecture will be enabled automatically at boot time, based upon the architecture detected. Per Intel's guidance, (https://software.intel.com/security-software-guidance/api-app/sites/default/files/Retpoline-A-Branch-Target-Injection-Mitigation.pdf), both retpolines and IBRS are mitigations for Spectre variant 2. Retpolines will likely have a reduced performance impact.

Red Hat Enterprise Linux defaults on Intel CPUs:

Listed below are configurations that the kernel would default to assuming no kernel command line parameters are provided.

For all pre-Skylake CPUs, and for Skylake with new Red Hat Enterprise Linux 7.7 installations and beyond:

pti=1 ibrs=0 retp=1 ibpb=1-> fix variant#1 #2 #3

For Skylake CPUs for RHEL installations prior to RHEL-7.7:

pti=1 ibrs=1 retp=0 ibpb=1-> fix variant#1 #2 #3

For for older Intel systems with no microcode update available:

pti=1 retp=1 ibrs=0 ibpb=0 -> fix variant#1 #3

Red Hat Enterprise Linux defaults on AMD CPUs:
Due to the differences in underlying hardware implementation, AMD X86 systems are not vulnerable to variant #3. The correct default values will be set on AMD hardware based on dynamic checks during the boot sequence.

pti=0 ibrs=0 ibpb=1 retp=1 -> fix variant #1 #2 if the microcode update is applied
pti=0 ibrs=2 ibpb=1 retp=1 -> fix variant #1 #2 on older processors that can disable indirect branch prediction without microcode updates

Note: A microcode patch provided by the vendor must be applied in order for the tunables to be visible.

s390x Defaults
s390x is affected by Spectre (Variants 1 and 2), but not by Meltdown (Variant 3).

The mitigation for variant 1 (ppa15) is always active if the feature is available in microcode. It cannot be switched on/off dynamically, you can only disable instruction patching completely via the kernel parameter noaltinstr.
For variant 2, mitigation is done via the bpb feature. As for ppa15, it is always active if the feature is available in microcode. It cannot be switched on/off dynamically either, but there is a kernel parameter nobp enabled by default which can be disabled with the nobp=off kernel parameter. (This patch uses a 'big hammer' approach, which has measurable performance impact).

IBM POWER Defaults
The mitigation for Spectre (Variants 1 and 2) are provided by the system firmware provided by the vendor. Currently there is no tunable available on Linux to disable these mitigations. The mitigation for variant 3 is provided by the Linux kernel, without depending on system firmware (although an optimized implementation is used in case system firmware provides support for it). It is enabled by default, and can be disabled on boot time, with the kernel command line parameters no_rfi_flush or nopti), or at run time, with either of the following tunables:

On initial zstream/errate updates:
/sys/devices/system/cpu/rfi_flush

The above tunable will change with upcoming zstream/errata updates to the following:
/sys/kernel/debug/powerpc/rfi_flush

Tuned for automation

Customers may control these settings by adding the above mentioned tuning commands to a customized tuned-adm profile via this method:

How to create a customized tuned profile

Note that these security fixes for variants #1 #2 #3 are enabled by default. Therefore creating a custom tuned profile is only required if the user intends to disable the security fixes.

33 Comments

It appears that tunables control steps described above are only for RHEL 7.

For RHEL 6, /sys/kernel/debug/x86/pti_enabled and /sys/kernel/debug/x86/ibpb_enabled and /sys/kernel/debug/x86/ibrs_enabled files are not there by default.

Please advise on how to perform runtime disable on RHEL 6

If created, will they work in RHEL6?

I don't know.

Hi Daniel,

The debugfs is not mounted by default in RHEL 6. Run the below to mount then follow the directions above. # mount -t debugfs nodev /sys/kernel/debug

Best, Nick

To make this mount persistent across reboots add the following to /etc/fstab: debugfs /sys/kernel/debug debugfs defaults 0 0

Two year old Dell PowerEdge R720 server with RHEL 7.4. Have run "yum clean all; yum update; reboot". Noticed that a bunch of *firmware-packages were updated by yum, as well as microcode_ctl and the kernel. Given this article, I would now expect the following commands to yield 1: cat /sys/kernel/debug/x86/ibrs_enabled cat /sys/kernel/debug/x86/ibpb_enabled However, they return 0. What am I missing?

i see the similar results as well with Dell R610. I guess those are not enabled by default based on the architecture detected, The article says "By default, each of the 3 tunables that apply to an architecture will be enabled automatically at boot time, based upon the architecture detected."

I have now patched an HPE ProLiant BL460c Gen9 with CPU E5-2667 v4 (3.2 GHz / 8-core) CPUs. On that server, both /sys/kernel/debug/x86/ibrs_enabled and /sys/kernel/debug/x86/ibpb_enabled have the value of 1 after boot, to that's better. I wonder how microcode gets into the right firmware RPM packages: Does it originate from Intel, or from server vendors? In other words: Who am I waiting for for microcode_ctl to be able to do its magic?

The microcode comes from Intel. So far they've only released updates for a handful of CPUs. More updates are supposedly coming later this week.

Will be patching RHEL 6.x test system later today, but at present the options ibpb & ibrs are not installed.

[root@ ~]# ls -l /sys/kernel/debug/x86/* -r-------- 1 root root 0 Aug 31 13:04 /sys/kernel/debug/x86/pat_memtype_list [root@cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.7 (Santiago) [root@uname -r 2.6.32-573.12.1.el6.x86_64

For RHEL5 does not have the /sys/kernel/debug/x86 even after mount -t debugfs nodev /sys/kernel/debug. Is this normal? Is the RHEL5 fix will have the process to disable the fix?

As shown in the 'resolve' tab of https://access.redhat.com/security/vulnerabilities/speculativeexecution , at this point no rhel5 errata including the fixes for these CVE has been released.

Hi,

Here is a little background:

Our team is trying to remediate meltdown/spectre. We ran this and we're getting IBRS failures (https://raw.githubusercontent.com/speed47/spectre-meltdown-checker/master/spectre-meltdown-checker.sh). I reviewed the code there and it looks solid.

I'm running RHEL 6.9 in AWS EC2 running kernel 2.6.32-696.18.7.el6.x86_64.

[root@ip-10-81-30-15 ~]# mount -t debugfs nodev /sys/kernel/debug [root@ip-10-81-30-15 ~]# cat /sys/kernel/debug/x86/ibrs_enabled 0 [root@ip-10-81-30-15 ~]# echo 2 > /sys/kernel/debug/x86/ibrs_enabled -bash: echo: write error: No such device

Here is the question:

Any thoughts on why I can't update /sys/kernel/debug/x86/ibrs_enabled to a value of 2? (FYI: It doesn't work if I try to update it to a 1 either.)

For anyone having the same problem, our team identified the root cause.

If we run on AWS Nitro Hypervisors (C5/M5 EC2 instance types) we can remediate (echo 2 works). If we run on AWS Xen Hypervisors we can't remediate (the issue I outlined above).

So it looks like AWS Nitro is patched but not AWS Xen.

Hope this removes some frustration for anyone having problems remediating their hosts in AWS.

Are there any plans to back-port the PCID support from Linux kernel 4.14 ?

Is there any statement from Red Hat on whether RHEL7.4 on Intel x86 processors makes use of the processor context ID (PCID) feature to reduce the performance impact of "Meltdown"/KPTI patches?

The last part of the page indicates that this can be controlled via tuned. If there specific variables which can be defined directly or is it done via using the [script] section and literally running the echo commands from a separate script file?

+1 to this question. I don't see it yet as a sysctl parameter either.

If someone can post a tuned.conf for this, it would be greatly appreciated.

You can do it with a [bootloader] section, just add 1 line in that section with "cmdline = nopti noibrs noibpb". Of course this will require rebooting to take effect, after changing profile.

Just finished patching about 30 running Red Hat Enterprise Linux Server release 7.4 (Maipo); kernel 3.10.0-693.11.6.el7.x86_64, and only one flag seems enabled by default cat /sys/kernel/debug/x86/pti_enabled; cat /sys/kernel/debug/x86/ibpb_enabled; cat /sys/kernel/debug/x86/ibrs_enabled 1 0 0

Trying to enable the other 2 manually, but I'm getting: echo 1 > /sys/kernel/debug/x86/ibrs_enabled -bash: echo: write error: No such device

Are you running microcode with the required fixes for you CPU type? The updated microcode could technically come via an updated BIOS, or a microcode_ctl package from RHEL.

Hi Christian, In my case it came from a package: yum info microcode_ctl-2.1-22.2.el7.x86_64 Installed Packages Name : microcode_ctl Arch : x86_64 Epoch : 2 Version : 2.1 Release : 22.2.el7 dmesg| grep microcode [ 9.994181] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x710 [ 9.994187] microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x710 [ 9.994247] microcode: Microcode Update Driver: v2.01 tigran@aivazian.fsnet.co.uk, Peter Oruba

ll /sys/devices/system/cpu/microcode/reload --w------- 1 root root 4096 Jan 16 18:00 /sys/devices/system/cpu/microcode/reload

Hi, we had released a microcode_ctl errata with updated microcode, yet just for certain cpu types, and it did lead to issues on certain hardware. Please verify with your hardware vendor which version of the firmware/BIOS they provide includes the microcode fixing the issue for your cpu. Details: https://access.redhat.com/solutions/3315431

Hi. Thank you for the link and your reply. I have just been made aware that our vendor's firmware applied a few days ago was faulty, so we have to wait a new one. That's most probably why I couldn't set those flags on/off.

Seems that microcode_ctl-2.1-22.2.el7.x86_64 RPM is only for Intel processor, model 79

Refer - /usr/lib/systemd/system/microcode.service ExecStart=/usr/bin/bash -c 'grep -l GenuineIntel /proc/cpuinfo | xargs grep -l -E "model[[:space:]]*: 79$" > /dev/null || echo 1 > /sys/devices/system/cpu/microcode/reload'

My RHEL6 AWS EC2 instances are still showing as "Variant #2 (Spectre): Vulnerable". I've applied the patches. Does this mean that AWS still needs to apply firmware patching on their end?

+1 to this questions, also looking for an answer

Let's say you're on a VMware VM. You installed the patched kernel. Now you want to try these tunables and you see the result below while logged in as root.

[root@vm.example.com ~]# echo 0 > /sys/kernel/debug/x86/pti_enabled bash: /sys/kernel/debug/x86/pti_enabled: Permission denied

Tried touching the file first and that also returns Permission denied

touch /sys/kernel/debug/x86/pti_enabled

I have a hunch this happens because the underlying VMware infrastructure hasn't been patched yet, and so the kernel in the guest VM is still running on old hardware types. Am I on solid ground?

I have one question: There is a new microcode_ctl - Upgrade because of stability issues with Spectre Variant#2. When I have disabled Spectre with Kernel-Parameters (ibpb_enabled = 0 and ibrs_enabled=0) is the system still affected or not, when I do not apply the updates?

Thanks

Are KVM based virtual machines vulnerable, if the kernel is not patched in them or patched but tuned following the above instructions and even if the host machine is completely patched (kernel,libvirt and qemu-kvm)?

And if we do patching for kernel in Virtual machine too, will it add more performance impact?

Simply putting, is it necessary to apply patch on Virtual machines if the host machines are completely patched?

The 32 bit kernels version 2.6.32-696.18.7.el6.i686 and above still does not address CVE-2017-5753, CVE-2017-5715 , CVE-2017-5754 do they ?

# uname -r
2.6.32-696.18.7.el6.i686

Running spectre-meltdown-checker.sh from speed47's spectre-meltdown-checker in github

# ./spectre-meltdown-checker.sh --no-color
Spectre and Meltdown mitigation detection tool v0.35

Checking for vulnerabilities on current system
Kernel is Linux 2.6.32-696.18.7.el6.i686 #1 SMP Thu Dec 28 20:16:03 EST 2017 i686
CPU is Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates IBRS capability: NO
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available: NO
    * CPU indicates IBPB capability: NO
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates STIBP capability: NO
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability: NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO): NO
  * CPU microcode is known to cause stability problems: NO (model 44 stepping 2 ucode 12)
* CPU vulnerability to the three speculative execution attacks variants
  * Vulnerable to Variant 1: YES
  * Vulnerable to Variant 2: YES
  * Vulnerable to Variant 3: YES

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Kernel has array_index_mask_nospec: NO
* Kernel has the Red Hat/Ubuntu patch: NO
* Checking count of LFENCE instructions following a jump in kernel... NO (only 0 jump-then-lfence instructions found, should be >= 30 (heuristic))
> STATUS: VULNERABLE (Kernel source needs to be patched to mitigate the vulnerability)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support: NO
  * Currently enabled features
    * IBRS enabled for Kernel space: NO
    * IBRS enabled for User space: NO
    * IBPB enabled: NO
* Mitigation 2
  * Kernel compiled with retpoline option: NO
  * Kernel compiled with a retpoline-aware compiler: NO
> STATUS: VULNERABLE (IBRS hardware + kernel support OR kernel with retpoline are needed to mitigate the vulnerability)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Kernel supports Page Table Isolation (PTI): NO
* PTI enabled and active: NO
* Running as a Xen PV DomU: NO
> STATUS: VULNERABLE (PTI is needed to mitigate the vulnerability)

A false sense of security is worse than no security at all, see --disclaimer

Is the sysctl parameter for a sysctl.ktune tune profile missing for these? I need to set ibrs = "2" to enable user space security also.

echo 2 > /sys/kernel/debug/x86/ibrs_enabled

If so will that be added soon?

Hi, When using the latest retpoline kernel (e.g. 3.10.0-693.21.1.el7.x86_64), can we disable and/or enable the retpoline feature dynamically without system reboot or something else ?

## disable

# echo 0 > /sys/kernel/debug/x86/retp_enabled

## enable

# echo 1 > /sys/kernel/debug/x86/retp_enabled